Data filter won't go away! Options

[RyanA1084]

I have encountered a very frustrating problem. I loaded one data file and applied a filter that removed the first 14 columns. I then started over, deleting all data files and filters, but now when I load a new file, it automatically chops off the first 14 columns! I thought I had deleted the filter that was doing this, since it is not displayed on the list, but it still gets applied when I load new data. I have tried restarting Synapse and rebooting the computer. Oddly, when I tried loading the Abalone sample data, the problem did not occur.

Possibly related, I encountered an error message several times in the process, I assume because the data files that I am loading are large (~500 samples and ~6000 features; they are spectra). I clicked the "details" button and got the following:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.Split(Char[] separator, Int32 count, StringSplitOptions options)
at Peltarion.Plugins.Data.CVSFileGUI.ReadText(String inputString, Int32 maxrows)
at Peltarion.Plugins.Data.CVSFileControl.Fname_TextChanged(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnTextChanged(EventArgs e)
at System.Windows.Forms.TextBoxBase.OnTextChanged(EventArgs e)
at System.Windows.Forms.Control.set_Text(String value)
at System.Windows.Forms.TextBoxBase.set_Text(String value)
at System.Windows.Forms.TextBox.set_Text(String value)
at Peltarion.Plugins.Data.CVSFileControl.OpenCSVFile()
at Peltarion.Plugins.Data.CVSFileControl.BrowseButton_Click(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

I'm attaching the full details. Any help would be greatly appreciated!

Attached File(s)

 error.txt ( 36.68K ) Number of downloads: 1

 

[luka]

The Out Of Memory exception is that you've run out of system memory. My guess would be based on your previous posts that you are importing a text file that has sample names. The default settings for the CSV file format is to expand strings i.e. convert each string to a column of binary values (see string expansion). This means that your 500 samples become 500 additional features.

Generally speaking, I would not recommend pushing Synapse beyond a couple of hundred of features if you want a pleasant usage experience. Having more features than samples is fairly pointless. The systems you build in Synapse are statistical in nature and for each feature that you add, you should have sufficient data coverage. 500 samples would have been a questionable data size if you only had 50 features, let alone 6000! My advice would be to see if you really need all the 6000 features. If you are using just a raw FFT dump, you may want to change the window resolution or do a simple subsampling. A more advanced approach would involve using ICA or PCA to extract independent or principal components and thus reduce the number of features significantly.

Anyway, to answer your specific question - the system runs out of memory and once that happens all bets are off. Out of memory problems cause all forms of strange and unpredictable problems. Normally my advice would be to run Synapse on a machine with more memory, but I think the set up of your problem is the primary issue and that more RAM won't help.