Definition of "forgetting"

[Susumu_Katayama]

Not-forgetting is a must for both prizes now, but I think its definition is not exactly given.

The third item of Evaluation Criteria in p. 6 of "General AI Challenge: Specifications of the First (Warm-Up) Round: Gradual Learning - Learning Like a Human" states
"For example, after solving tasks 1, 2, 3 and then 4, agent must still be able to solve tasks 1, 2, 3 (with the same or better performance)".
Literally, this only means "The agent must be able to re-learn 1, 2, and 3 after learning 4, and the speed must not deteriorate." I think this is a very weak, unimportant restriction, not exactly corresponding to the meaning of "forgetting", but at first I thought this was what the organizers have in mind, because the agent MUST forget last task and last task instance in order to solve micro-tasks A.2.1-A.2.5.18.

However, in micro-tasks A.2.6-A.2.20, each input is prefixed with a "command" word which is redundant when solving each task independently, and it is quite natural and meaningful to require that the agent can solve, say, task A.2.6, A.2.7, and A.2.8 WITHOUT LEARNING after learning A.2.6, A.2.7, A.2.8, and A.2.9 in this order. In order to do this, the agent must "understand commands", i.e. learn the map from commands in strings to functions resulting from interpreting the commands.

So, which do you mean by forgetting?

[Kir]

It would be nice to provide the code that will test the gradual learning and non-forgetting properties.

[Susumu_Katayama]

I think if the organizers are expecting us to induce the rule from one example, that is OK, if the actual rule is the simplest generalization. My concern is that the term "forgetting" can be interpreted in a more natural way, so I just like to know which is the actual rule. Or, if the organizers are expecting us to write an AI that can cope with both cases, that can be another option.

[martin.poliak]

The idea was indeed that the agent should not forget stuff we've taught it unless we're explicitly forcing it to replace the previous knowledge with a new knowledge.
This way, we expect the agent to retain knowledge from tasks 2.6 onwards for further use. For instance, the last tasks in the curriculum expect the agent to understand the meaning of keywords and switch to a respective solution based on which keywords occur.
At the same time, we don't expect a fully trained agent to solve the first task without any re-training. Knowledge gained in the first task was meant to be replaced by knowledge gained in the second task and so on until task 2.5.3.
Evaluation of the non-forgetting ability will go along these lines.

[Susumu_Katayama]

I see. Thank you for the answer.

One thing I would like to suggest is to intermingle old questions with the current one, e.g. intermingle questions from A.2.6 while doing A.2.7, in order to make sure that the learner does not forget old ones, as human teachers usually do.

[martin.poliak]

This is a good idea, we'll consider this for the evaluation, at least for some of the tasks.