Script filter

From Piki

Currently0.00/5

The script filter is a powerful meta-filter that allows the use of C# code to manipulate the data in the data unit.

Script

Name	Script
Deployable	Yes
Static	Yes/No
Sample modifier	Yes/No
Feature modifier	Yes/No

1 Usage
- 1.1 Code editor
- 1.2 Debugging
2 Settings
3 Important to know before starting
4 Anatomy of a filter
5 Code examples
- 5.1 Simple mode examples
- 5.2 Advanced examples
  - 5.2.1 Example 1
  - 5.2.2 Example 2
6 General advice
7 See also

Usage

The script filter allows the fast creation of custom filter functionality. This filter can be used when data requires special preprocessing that isn't covered by any of the standard filters.

To use the script filter open the Code editor window (from the settings browser).

Code editor

Main article: Synapse:Code editor

The code editor

The code editor is an advanced editor that features C# syntax highlighting, intellisense and other advanced features for code editing. It is similar to the editor found in Microsoft Visual Studio.

For detailed instructions on using the code editor see the core editor manual.

Filter related highlights:

The editor features two modes: simple and advanced. The former hides all unnecessary code and presents just the core methods to be implemented. The latter shows all available code. Advanced mode also allows you to link in external assemblies. The code editor is in simple mode by default.

To load sample filter code select "Load Sample.." which is found under the "File" menu.

To compile the script filter, click on the "Build" button in the toolbar. To compile and apply the result directly, click the "Build and Apply" button.

To switch from simple mode to advanced mode, click on the "Switch to advanced mode" button.

Debugging

The code editor does not have a debugger, but there are two tools that you have to your aid. The first one is that when you click on "Build and Apply" the filter will be updated and so will all visualizers so you can immediately see the effects of your code.

The second one is the Synapse debug console that can be started from the main Synapse window. It is found under the "Tools->Debug Console" menu. To write to the console you use the Context.Console object:

Context.Console.WriteLine("Hello World");

Settings

The settings can be modified using the settings browser.

Script filter settings
(Info) Mode: Shows the code editor mode. Latest Compiled: States if the latest saved code was compiled. (Edit) Code: Opens the code editor

Important to know before starting

A filter is a very powerful tool. You get the full data unit at your disposal which means that you can do anything to it, including breaking it. When you change a data unit, make sure that:

1. You have read and understood the theory behind data units and filters.

2. Only modify the OutputBuffer and never the InputBuffer.

3. The channels (training,validation and testing) must have an equal number of columns. They can be null. The test channel is not used.

4. All changes to features (count, names, order etc) must be reflected in the meta-data. Failing to do so will most likely cause the filter stack and visualizers to malfunction.

Anatomy of a filter

If you work in simple mode, most of this code will be hidden.

There are four includes used:

using System;
using Peltarion.Core;
using Peltarion.Data;
using Peltarion.Maths;

A class has to be created that extends the CoreUnitFilter class:

public class Script : CoreUnitFilter
{
}

The class needs two methods that need to be overriden:

protected override FilterInfo BuildOutgoingInfo(FilterInfo incomingInfo);
public override void Apply(DataUnit du)

If no modifications are made to the features (count, name, order etc) the BuildOutgoingInfo function reduces to:

override protected FilterInfo BuildOutgoingInfo(FilterInfo incomingInfo)
{
     return incomingInfo;
}

Apply is the main method and looks like this:

public override void Apply(DataUnit du)
{
 
}

Code examples

You can copy-paste these examples into the code window of the script filter to test them.

Simple mode examples

Example 1

Divide all data in the data unit by 2:

/// <summary>
/// The main filtering method
/// </summary>
/// <param name="du">Data unit</param>
public override void Apply(DataUnit du) 
{
    //For each data channel (Training, Validation, Testing)
    for (int i = 0; i < du.OutputBuffer.Length; i++)
        //If the data channel exists
        if (du.OutputBuffer[i] != null)
        {
            Matrix m = du.OutputBuffer[i].Data;
            //divide all values by two
            m = m / 2;
            //Square all values
            m = Matrix.Pow(m, 2);
            //set a
            du.OutputBuffer[i].Data = m;
        }
}

Example 2

Subtract the mean of the the second column from the first column. Note that if there is only one column in the data unit, the filter will malfunction.

public override void Apply(DataUnit du) 
{
    //For each data channel (Training, Validation, Testing)
    for (int i = 0; i < du.OutputBuffer.Length; i++)
        if (du.OutputBuffer[i] != null)
        {
            Matrix m = du.OutputBuffer[i].Data;
            //first column
            Matrix col1 = m[false, 0];
            //second column
            Matrix col2 = m[false, 1];
            //subtract the col2 mean from col1
            m[false, 0] = col1 - Stats.Mean(col2)[0];
            //set the data to the data unit
            du.OutputBuffer[i].Data = m;
        }
}

Example 3

Create a new column called "Random" and fill it with random values.

//Handle the necessary column metadata
protected override FilterInfo BuildOutgoingInfo(FilterInfo incomingInfo)
{
    //Make a copy of the filter info that describes the columns
    FilterInfo info = incomingInfo.GClone();
    //Add a new column called "Random"
    info.Add(new ColumnInfo("Random"));
 
    return info;
}
 
 
// The main filtering method
public override void Apply(DataUnit du) 
{
    //For each data channel (Training, Validation, Testing)
    for (int i = 0; i < du.OutputBuffer.Length; i++)
        //If the data channel exists
        if (du.OutputBuffer[i] != null)
        {
            //Get the data matrix        
            Matrix m = du.OutputBuffer[i].Data;
            //Create a new column filled with random numbers
            Matrix random = new Matrix(m.Rows,1);
            random.Randomize();
            //Concat the two matrices to produce the final result
            du.OutputBuffer[i].Data = Matrix.Concat(true,m,random);
        }
}

Advanced examples

Example 1

Replace column 4 with a copy of column 2 and rename the copy. Finally, the fifth column is removed. Note that this filter has no real error checking implemented so for instance if the data unit doesn't have five or more columns, it will fail.

using System;
using Peltarion.Core;
using Peltarion.Data;
using Peltarion.Maths;
 
namespace Peltarion.Plugins.OnTheFly
{
    public class Script : CoreUnitFilter
    {
        /// <summary>
        /// Manage the FilterInfo metadata.
        /// This is necessary when changing features.
        /// </summary>
        /// <param name="incomingInfo">Metadata from the previous filter</param>
        /// <returns>Metadata sent to the next filter</returns>
        protected override FilterInfo BuildOutgoingInfo(FilterInfo incomingInfo)
        {
            //clone the metadata from the previous filter
            FilterInfo info = incomingInfo.GClone();
            //Replace column4 with a copy of column2
            info[3] = info[1];
            //Rename the new column4
            info.Rename(3, info[1].Name + " 2");
            //Remove column5
            info.RemoveAt(4); 
            return info;
        }
 
        /// <summary>
        /// The main filtering method
        /// </summary>
        /// <param name="du">Data unit</param>
        public override void Apply(DataUnit du)	
        {
            //For each data channel (Training, Validation, Testing)
            for (int i = 0; i < du.OutputBuffer.Length; i++)
                if (du.OutputBuffer[i] != null)
                {
 
                    Matrix m = du.OutputBuffer[i].Data;
                    //Replace column4 with a copy of column2
                    m[0, -1, 3, 1] = m[0, -1, 2, 1];
                    // Remove column5
                    m = Matrix.DropColumns(m, 4);
                    //Set the data on the output buffer
                    du.OutputBuffer[i].Data = m;
                }
        }
    }
}

Example 2

Create two new columns, one containing the average of the current row and one containing random numbers.

using System;
using Peltarion.Core;
using Peltarion.Data;
using Peltarion.Maths;
 
namespace Peltarion.Plugins.OnTheFly
{
    public class Script : CoreUnitFilter
    {
        //Handle the metadata.
        protected override FilterInfo BuildOutgoingInfo(FilterInfo incomingInfo)
        {
            //clone the metadata from the previous filter
            FilterInfo info = incomingInfo.GClone();
            //define a new column called "Average"
            info.Add(new ColumnInfo("Average"));
            //define a new column called "Random"
            info.Add(new ColumnInfo("Random"));
 
            return info;
        }
 
        public override void Apply(DataUnit du)
        {
            for (int i = 0; i < du.OutputBuffer.Length; i++)
                if (du.OutputBuffer[i] != null)
                {
                    //original data
                    Matrix original = du.OutputBuffer[i].Data;
                    //matrix to be filled with random numbers
                    Matrix random = new Matrix(original.Rows, 1);
                    //random bounds (-1 to 1)
                    random.NoiseLevel = 1;
                    //randomize the matrix
                    random.Randomize();
                    //calculate an average
                    Matrix average = Matrix.Sum(false, original) / (double)original.Columns;
                    //concat the original matrix with the two new columns
                    du.OutputBuffer[i].Data = Matrix.Concat(true, original, average, random);
                }
        }
    }
}

General advice

For many custom filtering operations the expression filter will suffice and is easier to use.
As you have complete access to the data unit incorrect filter implementations may cause system instability and can cause other Synapse components to fail.

Personal tools

Views

Script filter

From Piki

Contents

Usage

Code editor

Debugging

Settings

Important to know before starting

Anatomy of a filter

Code examples

Simple mode examples

Example 1

Example 2

Example 3

Advanced examples

Example 1

Example 2

General advice

See also

Navigation

Search

Toolbox