Outlier filter

From Piki

The outlier remover filter eliminates outliers from data.

Oultier remover
Name	Oultier remover
Deployable	No
Static	Yes
Sample modifier	Yes
Feature modifier	No

Contents 1 Usage 2 Operation 3 Settings 4 General advice 5 See also

Usage

The outlier remover is used to purge the data of samples that are outside the range of normal values. This filter can only be used with static systems and should not be used with time series data.

Operation

The outlier remover has two basic operational modes: statistical and histogram modes. The statistical (or sigma) mode removes samples that fall outside of a normal distribution. This works very well when the data roughly follows a bell shaped distribution.

The histogram mode removes samples that occur less than a specified frequency in a histogram of the features of the data unit are removed. This will work with any distribution but is sensitive to the histogram settings.

The two modes can be combined.

Settings

The settings can be modified using the settings browser.

Outlier remover settings

(Mode) Mode: Operation mode: Sigma - Uses the statistical mode for removal. Histogram - Uses the histogram mode for removal. Full - Uses both modes for removal (Operation) Training: Specifies if the filter is applied on the training channel. Validation: Specifies if the filter is applied on the validation channel. (Operator) Invert: Flips the selection status of all features. Clear: Clears the selection status of all features (sets them to false) (Settings) (If Sigma or Full) xSigma: Sigma multiplier for normal distributions. Beyond the sigma value samples are removed. Sigma = 2 equals about 96% of a normal distribution. (If Histogram or Full) Bins: Number of bins in histogram. Higher number of bins results in higher resolution, but also fewer samples per bin. (If Histogram or Full) Cutoff: Cutoff level. Bins that have fewer samples than the cutoff level are removed. (Selected) Show Count: The number of features to show in the list. Useful if there is a very large number of features. Show From: The feature number to start the list from. Only used if the number of features > show count. <Feature name>: The selection status of a feature. If this is set to true the feature will be affected by the filter.

General advice

• Be careful not to remove too much with the outlier remover. It is advisable to keep the validation set intact.

See also

• Filter - Article covering general filter principles.
• List of Filter components - List of all available filters.
• Data unit - Article explaining data units in detail.
• Data - Article explaining essential data concepts.