Greetings.
I am trying to open a raw sleep EEG ascii file in the brainstorm database but it is stuck on a message saying reading file header. My data is a single ascii file (2.5 GB) of 6 hours continuous EEG recording and I dont have the channel location files at the moment.I just want to do some processing on the time series. How can I load this heavy ascii data in the brainstorm data base ?
Hi Ali,
Just to check a couple of things, did you select the correct filetype when reading it in?
Sometimes when a loading screen seems stuck there is output in the matlab command window that there is an error why it cannot proceed, is that the case? As a last question, were you already able to read in similar files like this before?
Kind regards,
Steven
Dear Steven,
Thank you for your quick response.What I really ment from stuck is that progress bar keeps moving (the green marker going from left to right for hours). I have selected the ascii file type and that is the only file type in which my desired file becomes visible for selection in the browse window.
Regarding your last question I have not personally opened this kind of file befor but my associates are working on the same kind of files (different subjects) using brainstorm and they can open them just fine.
Best regards,
Ali
Hi Ali,
That's good to know! Note that even if the file becomes visible it is not a guarantee that it's the correct filetype in every case, since the same extension can be used for a different file build up.
Did you get any error message in the Matlab main screen? (this can still happen with the moving progress bar)
Kind regards,
Steven
Dear Steven,
I dont have any error messages in the Matlab screen.the last visible line is "Loading current protocol...".
And the progress bar is saying "Reading file header".
Could this be because of entering the orientation (time × channels) or number of header files incorrectly?
Also if the problem is with the file format what do you think could be done? The type of the file is "File" when I view it in the properties section after right clicking in it and as I said there weren't any alternative file formats in the browser for it to become visible .
Hi Ali,
No I don't think the problem is within the file format directly, sometimes the wrong option is used in files of format, and it then gives an error. To be honest, I'm not sure then what the problem could be and I hope the more advanced users can chime in to this. Usually it helps to share your EEG file in a way, maybe shortened via a wetransfer or something like that, so the developers can have a look at it.
Again my apologies for not being able to help you further at the moment.
Kind regards,
Steven
Dear Steven,
Not at all. Thank you so much for your help and advice.
Best regards,
Ali
1 Like
Dear Steven , it seems like that problem is fixed.but now it is saying that array exceeds maximum array size preference. How can we load long eeg files into brainstorm?
Dear Ali,
Good to hear that it has been fixed. How did you fix it if I may ask? The error that you get is not related to Brainstorm directly but more to Matlab, which limits . I would try to do this kind of analysis on a PC with a lot of RAM memory or a server maybe.
You might want to look into this page if really needed on your machine: http://jmdvinodjmd.blogspot.com/2017/04/matlab-error-requested-array-exceeds.html
Kind regards,
Steven
Dear steven,
I was using the "Review raw data" option in the interface as usual but when I used "load EEG/MEG data" things started moving again. But at the end the oyher problem occurred.
You're right its better that I do this on another pc or a server.
Thanks a alot for your help
Hi Ali,
I hope you get that sorted out and can work as soon as possible on your data 
Good luck with your research.
Kind regards,
Steven
1 Like
Thank you very much for your support Steven.
Good luck to you too.
Best regards.
Ali
Dear Steven its me again!
Do you have any suggestions on how I can load very large data by turning them into smaller pieces ?
Hi Ali,
I don't have it directly, with ascii files it might be possible by scripting something.
Furthermore you could try to use the techniques mentioned in the site i mentioned before.
How big is the array that will be produced?
Kind regards,
Steven
It is a six-hour long High-density EEG recording. the array is approximately 64x30000000 and about 15 Gb.I did check out the site you sent. But as you see there is a relatively large memory required to process the data all at once.
Best regards.
Ali
Hi Ali,
The Brainstorm reading functions for text-based are not designed to handle gigantic files like yours. The reading is not optimized to read only the part of the file that is needed for review or processing: the entire file has to be at once. Also, text files is a terribly inefficient way of storing signals (only 10 values are used for every byte stored, which could store 256 values - resulting in files that are 10-20x larger and slower to read).
In the first place, ASCII file formats should never be used for storing continuous recordings. My first recommendation would be that you import directly the native binary recordings file in Brainstorm (I doubt that your acquisition software saved text files in the first place - if this is the case for real, change its configuration to save binary files), and if the native format is not supported by Brainstorm go back to your acquisition workstation and export the file in EDF+ or any other generic binary format.
The list of supported file formats in Brainstorm:
https://neuroimage.usc.edu/brainstorm/Introduction#Supported_file_formats
If you definitely can't access anything else, your best option might be to use Brainstorm to convert your recordings to a binary file: Use the menu "Import MEG/EEG" to load your file, then right-click on the imported file > Review as raw.
This might cause "out of memory" errors (all the data can't be held in memory at once, which would not surprise me for a 15Gb text file). In that case, your only chance is to find a text editor that would let you open and chop manually the file in smaller blocks. Notepad++ probably would not work with such a big file, so you would need to do some online searches... Maybe an hexadecimal editor that does not try to read the file as formatted ASCII text. Good luck!
Dear Francois,
Thank you very much for your explanation. You're absolutely right. I think the only option I have now is to convert the format as you said. I am doing this on a virtual machine with a huge memory. I am going to bring back the data in a more convenient format.
Sincerely,
Ali
Dear Francois,
Just to be clear, the actual ASCII file is only 2.3 GB but when I try to open it in Matlab I get the error " Requested (17GB) array exceeds maximum array size preference". Is it safe to say that the problem is essentially with the file and not the way I am opening it?
Best regards,
Ali
Please post the full error message (including the stack trace).
Copy-paste it from the Matlab command window.
Dear Francois,
I have pasted the message here. This is what happens: I try to load the data as an ASCII .txt file then in a panel I specify some features (Number of header lines (which I am not sure of !) , channel X time ...). Then there are a series of progress bars : txt2mat : reading file -> replacing strings -> updating break positions ->
ERROR in txt2mat :
Line 1438 Out of memory.
Please unload some datasets (or restart Matlab) an try again.
Here is the command window (As I said the file itself is about 2.3 GB) :
BST> Emptying temporary directory...
- C:\Users\SONY\Desktop\Data\s_1
- read mode: line
- 61 data lines analysed
- 2 header line(s)
- 310 data column(s)
- 1 string replacement(s) »,/:;!"#$%&'()*<=>?@ABCDGHJKLMOPQRSTUVWXYZ[]^_`bcdghjklmopqrstuvwxyz{|}~ ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ «
** Error: Line 2656: NaN
** Requested 7599641x236 (13.4GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.
**
** Call stack:
** >txt2mat.m>InitializeMatrix at 2656
** >txt2mat.m at 1417
** >in_ascii.m at 52
** >in_data_ascii.m at 43
** >in_fopen.m at 163
** >in_data.m at 487
** >import_data.m at 200
** >bst_call.m at 28
** >tree_callbacks.m>@(h,ev)bst_call(@import_data,[],[],[],iStudy,iSubject) at 637
**
Thank you for your time and consideration,
Sincerely,
Ali