Script format

From Piki

The script format is a powerful meta-input format that allows the use of C# code to import or generate data.

Script format
Deployable	Yes
Type	Script

Usage

The script format allows the fast creation input formats. This filter can be used to either import data from a custom external data source or simply by generating the data.

You can create a script format like any other format - create a new data unit using the Data unit manager and select "Script format" as the input format.

In the format GUI, click on the "Open Code Editor" button to open the code editor. Ifd the "Preview" check box is checked then "Build and Apply" from the code editor will fill the grid below with the data for preview.

Script format GUI

Code editor

Main article: Synapse:Code editor

The code editor

The code editor is an advanced editor that features C# syntax highlighting, intellisense and other advanced features for code editing. It is similar to the editor found in Microsoft Visual Studio.

For detailed instructions on using the code editor see the core editor manual.

Format related highlights:

• The editor features two modes: simple and advanced. The former hides all unnecessary code and presents just the core methods to be implemented. The latter shows all available code. Advanced mode also allows you to link in external assemblies. The code editor is in simple mode by default.

• To load sample format code select "Load Sample.." which is found under the "File" menu.

• To compile the script format , click on the "Build" button in the toolbar. To compile and apply the result directly, click the "Build and Apply" button.

• To switch from simple mode to advanced mode, click on the "Switch to advanced mode" button.

Debugging

The code editor does not have a debugger, but there are two tools that you have to your aid. The first one is that when you click on "Build and Apply" the filter will be updated and so will all visualizers so you can immediately see the effects of your code.

The second one is the Synapse debug console that can be started from the main Synapse window. It is found under the "Tools->Debug Console" menu in the main Synapse window. To write to the console you use the Context.Console object:

Context.Console.WriteLine("Hello World");

Important to know before starting

While the script format provides more insulation than the script filter, it is still possible to make the system unstable by misuse of the API.

• When returning the data in the NamedSignal format make sure that the column names have been defined. If they haven't or if the number of features in the data and the number of columns don't match visualizers and filters may malfunction.
• Data loading is not guaranteed to be thread-safe. Use multiple threads on your own risk.
• Use the provided function and do not try to access and edit the data unit directly. In order to work with all types of data update models the formats function depends on specific event chains. Should you short-circuit them the data system will at best be desync and at worst crash.

Anatomy of a format

If you work in simple mode most of this code will be hidden.

As input formats can deal with a variety of data many includes are used by default. The bare necessity are the following:

using System;
using Peltarion.Core;
using Peltarion.Data;

The base class that is extended is a simplified filter form that hides the inner workings:

class ScriptFormatDynamic : SimpleInputFormat
{
}

There is one method that is overridden:

public override NamedSignal GetData();

The named signal takes a matrix object and a string array as input.

Code examples

You can copy-paste these examples into the code window of the script filter to test them. They can also be loaded as sample code directly in the editor. All examples use simple mode.

Example 1

Create two features and fill them with random data. Name the features "Random1" and "Random2":

public override NamedSignal GetData()
{
  //Create a matrix with 1000 rows and 2 columns
  Matrix data = new Matrix(1000,2);
  data.Randomize(); // randomize the data;
  //Define the feature names
  string[] columns = new string[] { "Random1", "Random2" };
 
  //Return the data and column names capsulated in a NamedSignal object
   return new NamedSignal(data, columns);
}

Example 2

Create 1000 samples of the time series

where and is random noise.

public override NamedSignal GetData()
{
    Matrix m = new Matrix(1000,1); //create data matrix
  Matrix noise = new Matrix(1000,1); //create noise matrix
  noise.NoiseLevel = 0.5; //assign nose level
  noise.Randomize(); //randomize noise matrix
 
  double fx = Math.PI/100; //define period
 
  for(int i = 0; i < m.Rows;i++)
  {
    //calcuate value as f(t) = sqrt(l*t)*sin(l*t) + N
    m[i,0] = Math.Sqrt(fx*i)*Math.Sin(fx*i) + noise[i,0];
 
  }
  return new NamedSignal(m,new string[] { "NoisySqrtSine"});
}

Example 3

Load and parse XML data based on the CIA world factbook from the mondial project at the University of Göttingen: http://www.dbis.informatik.uni-goettingen.de/Mondial/

public override NamedSignal GetData()
{
  XmlDocument xdoc = new XmlDocument();
  xdoc.Load("http://peltarion.com/data/mondial-europe-flat.xml");
  XmlNodeList list = xdoc.SelectNodes("/Mondial/country");
 
 
  string[] columnNames = new string[] { "Area", "Population", "Population Growth", 
  "Infant mortality", "GDP_pcapita", "GDP_Agricultural", "GDP_Industrial", 
  "GDP_Service", "Inflation" };
 
  List<Matrix> samples = new List<Matrix>();
 
  foreach (XmlNode n in list)
  {
    if (n.Attributes["total_area"] != null && n.Attributes["population"] != null 
    && n.Attributes["population_growth"] != null && n.Attributes["infant_mortality"] != null 
    && n.Attributes["gdp_total"] != null && n.Attributes["gdp_agri"] != null 
    && n.Attributes["gdp_ind"] != null && n.Attributes["gdp_serv"] != null 
    && n.Attributes["inflation"] != null)
    {
      Matrix sample = new Matrix(1, 9);
      sample[0] = double.Parse(n.Attributes["total_area"].Value);
      sample[1] = double.Parse(n.Attributes["population"].Value);
      sample[2] = double.Parse(n.Attributes["population_growth"].Value);
      sample[3] = double.Parse(n.Attributes["infant_mortality"].Value);
      sample[4] = double.Parse(n.Attributes["gdp_total"].Value) / sample[1];
      sample[5] = double.Parse(n.Attributes["gdp_agri"].Value);
      sample[6] = double.Parse(n.Attributes["gdp_ind"].Value);
      sample[7] = double.Parse(n.Attributes["gdp_serv"].Value);
      sample[8] = double.Parse(n.Attributes["inflation"].Value);
 
      samples.Add(sample);
    }
 
  }
 
  Matrix data = Matrix.Concat(false, samples.ToArray());
 
  return new NamedSignal(data,columnNames);
 
}

See also

• Input format - Article covering input formats in general.
• Data unit - Article covering the data unit fundamentals.
• List of Input format components - List of all available input format components.