I have a task that I am not able to identify if it is of transfer or meta learning. I want to know this, in order to ask help in solving it, because there are some parts that I have not understood.
The task is the following:
We want that our neural network learns to sum weights.
1)Do the training on MNIST, and on CIFAR10 (as support dataset). We want that performance (accuracy) on MNIST, before and after the sum, to be as close as possible. We use CIFAR10 only to try to learn the operation of sum of models. We are interested in the weights of MNIST, we can theta_mnist, and we hope that they have learnt the operation of sum.
The way to do this is the following: have 3 NNs, with the same architecture, but with different weights, the first one for MNIST, second for CIFAR10, and the third one that have weights (for each layer) of
theta_mnist+theta_cifar10. The loss function for this network must be
CrossEntropy(MNIST) + CrossEntropy(CIFAR10) + CrossEntropy([MNIST+CIFAR10]-MNIST) (this last term of loss should guarantee that the network on MNIST have similar performances before and after the operation of sum.
2)Do the same thing of 1)replacing MNIST with SVHN and again CIFAR10 as support dataset.
3)Building a NN with weights:
theta_mnist+theta_svhn, what are the performances?
Now, step by step, I have to start from point 1. I highlighted the parts that I have not clear. In particular, i have not understood, if I have to define a loss and do the training, how is it possible that the network has the
theta_mnist+theta_cifar10? Do you think It means to create a network with those parameters and then start doing learning (but i think it is not useful), or doing like a simultaneously training? But I have no clear how is it possible to define weights and at the same time doing training.