Validation sets when training from code Options

[Paul Heideman]

Hi,

I'm just getting going with training a neural net from code. I've got to the point of getting it training, but was wondering:

1) How can I change the interval for when the validation set is used?
2) How can I get the MSE of the validation set?

Thanks,
Paul

[luka]

There are two possible ways. One is to set the appropriate channels in the data unit and to define the number of training and validation epochs. There is however a quicker way and that is to simply set the NoUpdate flag to true before using the StepEpoch function just like you would in training. To get the MSE, just pick it off the Delta Terminator block.
So you do something like this:

double MSE = deployedSystem.Topology.DeltaTerminator1.MSE:

[Paul Heideman]

Hi,

Let me check if I've got this right... Here is basically what I'm doing.

CODE

            solution.Data.CSV.CSV_Input.SetSignal(validationMatrix, Peltarion.Data.SetType.Validation);
            solution.Data.CSV.CSV_Input.SetSignal(trainingMatrix, Peltarion.Data.SetType.Training);
            while (!halt)
            {
                // step
                solution.StepEpoch();
                double MSE = solution.Topology.DeltaTerminator1.MSE;
                logger.Debug("Type: " + solution.ControlSystem.CurrentSet + " \tMSE: " + MSE);                    
            }

Question: Is solution.ControlSystem.CurrentSet equal to the set that will be used by solution.StepEpoch() next or is it the set that was previously used by solution.StepEpoch()?

The values I get for MSE seem to always be from the same set, regardless of the value in solution.ControlSystem.CurrentSet. Am I doing something wrong?

Thanks,
Paul

[luka]

Hi,

Let me check if I've got this right... Here is basically what I'm doing.

CODE

            solution.Data.CSV.CSV_Input.SetSignal(validationMatrix, Peltarion.Data.SetType.Validation);
            solution.Data.CSV.CSV_Input.SetSignal(trainingMatrix, Peltarion.Data.SetType.Training);
            while (!halt)
            {
                // step
                solution.StepEpoch();
                double MSE = solution.Topology.DeltaTerminator1.MSE;
                logger.Debug("Type: " + solution.ControlSystem.CurrentSet + " \tMSE: " + MSE);                    
            }

Question: Is solution.ControlSystem.CurrentSet equal to the set that will be used by solution.StepEpoch() next or is it the set that was previously used by solution.StepEpoch()?

The values I get for MSE seem to always be from the same set, regardless of the value in solution.ControlSystem.CurrentSet. Am I doing something wrong?

Thanks,
Paul

1) CurrentSet is the one is being used or that has just been used. NextSet shows you the next one that will be used.

The things that if you want to use the full approach with using the channels of the data units then the matrix input format (default at deployment) won't cut it. It supports only one channel. So in your example you are only getting the training channel as you set it last. When the control system requests the validation channel, you'll get the training data as the validation channel will be empty.

There is however a much easier way to do it (in very verbose form):

CODE

int validationFrequency = 10;
int epoch =0;

while(!halt)
{    

     if(epoch%validationFrequency!=0)
     {
          solution.NoUopdate = false;
          solution.Input_CSV = trainingMatrix;
          solution.StepEpoch();
        
          double MSE = solution.Topology.DeltaTerminator1.MSE;
          logger.Debug("Type: Training \tMSE: " + MSE);              
     }
    else
     {
           solution.NoUpdate = true;
           solution.Input_CSV = validationMatrix;
           solution.StepEpoch();

          double MSE = solution.Topology.DeltaTerminator1.MSE;
          logger.Debug("Type: Validation\tMSE: " + MSE);    
      }

      epoch++;

}

Or in more compact form:

CODE

int validationFrequency = 10;
int epoch =0;

while(!halt)
{    

     bool v = epoch%validationFrequency==0;

     solution.NoUpdate = v;
     solution.Input_CSV = v ? validationMatrix : trainingMatrix;
     solution.StepEpoch();

     double MSE = solution.Topology.DeltaTerminator1.MSE;
     logger.Debug("Type: " + (v?"Validation":"Training") + "\tMSE: " + MSE);  

     epoch++;
    
}

This way you don't have to concern yourself with data units and formats.

[Paul Heideman]

That makes sense. Thank you!