Sensitivity analyzer postprocessor
From Piki
The sensitivity analyzer postprocessor is a testing tool that gives a qualitative and quantitative idea of which inputs affect the outputs.
Contents |
Usage
Sensitivity analysis adds a small dither value to each input feature and sees how it affects a selected output feature. Features with high sensitivity affect the system more than features with small sensitivity. This can be seen as a non-linear version of a correlation visualizer.
The sensitivity analyzer can only be used with systems that use a terminator (that marks the end of the signal flow in a system), such as the delta terminator block and use a source that supports dithering, such as the data source block or the function source block.
| Sensitivity analyzer GUI |
|---|
|
- Dither (slider): The percentage by which each input feature should be perturbed. The text boxes on the sides of the slider allow you to enter the maximum and minimum values for the slider.
- Auto: Analysis is performed directly for each movement of the slider. Works well for smaller data sets. If you turn it off you need to press "Update" to perform the analysis.
- Correlation compensation: Makes the system compensate for the relative correlations. If variable A is perturbed by 20% and variable B has a 0.5 correlation, it will be perturbed 10%. Correlation compensation gives a more realistic combination of test variables however it prevents traditional sensitivity analysis where the question asked is how much the change of a single variable affects the overall error.
- Source: The input source that performs the dithering and that contains the features needed for analysis. Selecting an non-input source (for instance a source connected to the terminator) will produce nonsensical values.
- Terminator: The terminator component that ends the signal flow in the system.
- Refresh: Updates the list of data sources and terminators.
- Selected Output Feature: The affected output feature to be analyzed.
Important considerations
What sensitivity analysis does is to go through the input features, one by one and dither them a defined percentage while holding the rest fixed. It runs the system for each feature and records how much the output changed due to the dithering of the input feature. This information is then assembled and you can see how sensitive the system is to changes in the features.
In plain English this means that you can see how much the output changes if you change an input feature a certain percentage. This is a measure of how important a feature is at a certain level.
It is tempting to say that sensitivity equals importance. In a way, it does, but you have to take the level into account. A variable may be very important around a certain value and completely irrelevant around another.
One other thing that is important to understand is that sensitivity is per model as long as input variables are correlated (which they nearly always are in real-world problems) . A neural net can choose which information it wishes to extract from what feature. How that choice is made depends on many factors, the initial random weights in the network being a major one. This directly impacts the sensitivity and two models trained on the same data are unlikely to have equal sensitivity.
If you are careful with the interpretation, sensitivity analysis can give a lot of information about the model. The non-linearity is a great thing, if you use it correctly.
Y-axis (bar height)
Sensitivity analysis works by changing the input features one by one by a specified amount (original data multiplied by the dither level). So for each feature there will be an original value and a dithered value and hence an associated original error and dithered error. The error uses a linear absolute average metric which means that it takes the average of the absolute value of the difference between the desired output and the actual output.
The height of the bars in the sensitivity analyzer is 
meaning the absolute value of the difference between the original error and the error when the feature is dithered.
Given that a system can use a variety of normalization and scaling methods and that different error metrics can be used, the values on the Y-axis are only relevant for comparison within one system and that uses the same data.
For a practical example: Suppose that with a dither level of 0.1, the highest bar has a value of 0.005. When you set a dither level of 1, suppose you get a value of 0.15. That means that for that particular feature a change of 10% to 100% dither will increase the error by 0.15/0.005 = 30 = 3000%. (Of course a 100% dither is very much over the top, but the number was just used as an illustration.)
The conclusion is that the absolute value of the Y-axis has no simple explicit meaning as it depends on the system, on the data, on the normalization and on the error metric – to mention a few of the parameters that affect it. It is however useful as a metric relative different dither levels for the same system.
See also
- Tutorial 4 - Tutorial 4 ("Cows") has an in-depth practical example of using the sensitivity analyzer.
- Postprocessor - Article covering general postprocessor.
- List of postprocessor components - List of all available postprocessors.