Folder share for exporting patient zip files

[mattiher]

Dear Francois!

I have a problem when exporting patients as a zip file. I'll send you some large files if you would have time to explore what is wrong in our system.

So, our situation is as follows. I have some old sleep studies (Neuroscan, 64ch) that are in two or more parts, because the recording software has crashed down during the recordings. I have been joining those recording parts together. I imported the data into database and then I concatenated the datasets. After concatenating I selected review as raw. After that I epoched the data to 1min epochs. I computed different delta powers of these epochs and then I extracted data to get the time curve of delta power during the whole night, because our professor wants to search sleep spindels from the continuous data.
I copied the patient in Brainstorm and removed all unnecessary data from that patient. Then I exported patient and I got zip which is approx 8.8GB. We were not able to import it as a subject, but we managed to import it as a new protocol.
With the patient number 2 something went wrong during the patient copy and the software was not able to proceed with copying. I also noticed that in the database folder there appeared two (temporary?) folders: copy_anat_tmp and copy_data_tmp. I would not like to do this, but would it be possible to remove those _tmp folders manually if they are temporary folders?

The size of the zip of the second patient is now gigantic 33GB, because I didn't remove those extra files. We were not able to import this patient as a subject or as a new protocol.
What is common for these both zip-files is that they cannot be opened in Windows file explorer. Should it be so?
I did a third test patient where I reviewed some raw test file (NicoletOne EEG file) and did some test imports to database. At least this test patient, as exported to zip file (approx 400MB), it was possible to open zip in Windows file explorer.
I checked from this test patient what folders was zipped from brainstorm_db folder and I tried to zip folders manually with 7-zip software by selecting compression type to zip. The zipped patient was this 33GB size patient. However, the brainstorm rejected this immediately when I was trying to import this zip as a new patient.
Is it possible to do this copying/zipping phase manually? Of course I would like to use the Brainstorm export patient functionality.
I have only four patients. I would like to join the data within each individual patients and send these whole night recordings to the professor. She would mark the sleep spindels and then she would export the patient again to our junior team member for EEG cleaning and for further processing. I though that this would be the right way of sharing data. Am I right?
My version of the Brainstorm is quite old, but I cannot update yet, since I have one research project going on. My version is precompiled 20-May-2019. The version of the professor is from this spring, but I cannot remember exact version. I can check that if you need it. Also when the professor is exporting the data, she is not able to explore the zip-file in Windows and the junior member was not able to import it. The junior member has also a recent precompiled versions of Brainstorm from this spring. What do you think, could this export problem be a consequence of Brainstorm version miss match between professor and I?

If you (or some other member in your team) has time to help me, which folders/screen captures you would like to have? The folder I shared to you in my Google Drive is still empty, because I don't know where to begin.

I'll send this message without link also to forum if you think that there is some useful information to other members.

Kind Regards,
Herkko

[Francois]

(I made your topic public and removed the link - which was empty btw)

[Francois]

@mattiher

I have been joining those recording parts together. I imported the data into database and then I concatenated the datasets. After concatenating I selected review as raw. After that I epoched the data to 1min epochs.

Why not simply epoch the original separated files as 1s epochs?
And why do you need epoching in the first place? Reviewing data would be easier directly from the continuous files, using custom shortcuts:
https://neuroimage.usc.edu/brainstorm/Tutorials/EventMarkers#Custom_shortcuts

I would not like to do this, but would it be possible to remove those _tmp folders manually if they are temporary folders?

Yes, you can delete them and then reload your database.
I hope you made a full backup of your database before starting to play with these copying and exports. If you don't have a decent backup system set up, it's time to invest a few minutes to make your data safe.

What is common for these both zip-files is that they cannot be opened in Windows file explorer. Should it be so?

I'm not sure. I've had tons of problems with large zip files with the Windows tools. Try with something more robust instead, like 7zip.

She would mark the sleep spindels and then she would export the patient again to our junior team member for EEG cleaning and for further processing. I though that this would be the right way of sharing data. Am I right?

These export to zip / import from zip features were not really designed to handle such gigantic files.
I would recommend you find a way to skip these import/export parts in Brainstorm and directly exchange the actual files (zipping them with 7zip if it's more convenient for you to send only one file).

• You can copy directly the protocol folder and attach it the brainstorm database on another computer with the menu File > Load protocol > Load from folder
• If you want to manipulate the different subjects completely independently: create one protocol per subject. If at some point you need to have all the patients in the same protocol, you can simply copy-paste the subject folders in the same data and anat folders and reload the protocol
• If you want to manually create a stand-alone protocol (zipped or not) from a single subject, you need to include all the extra folders described in the tutorials: anat/@default_subject, data/@default_study, data/@inter (https://neuroimage.usc.edu/brainstorm/Tutorials/ExploreAnatomy#Interaction_with_the_file_system)

The junior member has also a recent precompiled versions of Brainstorm from this spring. What do you think, could this export problem be a consequence of Brainstorm version miss match between professor and I?

It's probably just because the files are too big. Don't use the Export/Import features, copy the files manually from your file explorer.

[mattiher]

Hi Francois!

@Francois

Why not simply epoch the original separated files as 1s epochs?
And why do you need epoching in the first place? Reviewing data would be easier directly from the continuous files, using custom shortcuts:
https://neuroimage.usc.edu/brainstorm/Tutorials/EventMarkers#Custom_shortcuts

Sorry, my description was too unclear. Yes, we are using the continuous file for reviewing data. However, because the recording had crashed in the middle of the night, I had to reconstruct one whole night raw data recording from two parts A and B. So now the senior scientist can review one continuous raw data (A+B).
The reason I epoched this to 1 minute epochs, was that I wanted to do time-frequency analysis only to delta band for whole night recording. This was something like what you have advised in the following topic:

How can I do time-frequency analysis on continuous data?

Greetings, I am trying to acquire a spectrogram for a long steady-state study. Naturally methods such as Morlet wavelet convolution are not applicable to lengthy and continuous data. How can I have a Time-frequency plot for my lengthy steady-state Data (Hours long) using brainstorm?

We wanted to have a rough estimation of structure of the sleep and in what time range the spindles might be occuring. I didn't create the real spectrogram, but only one dimensional curve of the delta power during the night in few channels. Example below:

@Francois

I hope you made a full backup of your database before starting to play with these copying and exports.

Yes, the backups should be ok. I have both automatic backups and manual snapshots of brainstorm_db folder.

@Francois

I'm not sure. I've had tons of problems with large zip files with the Windows tools. Try with something more robust instead, like 7zip.

@Francois

These export to zip / import from zip features were not really designed to handle such gigantic files.

The 7-zip was not able to open them either, so as you said it is probably the big size of the files.

@Francois

I would recommend you find a way to skip these import/export parts in Brainstorm and directly exchange the actual files (zipping them with 7zip if it's more convenient for you to send only one file).

I believe this solves my problem. I let you know if I still have problems with this. Actually, I was considering this direct change of files, but I saw a line in Tutorial 1 in Database chapter which says:
"All the files in the database have to be imported through the GUI. Do not try to copy files by yourself in the brainstorm_db folder, it won't work." I misunderstood this and assumed that also file export by copying database folders is a bad idea.

Thank you Francois!

[Francois]

@mattiher

I believe this solves my problem. I let you know if I still have problems with this. Actually, I was considering this direct change of files, but I saw a line in Tutorial 1 in Database chapter which says:
"All the files in the database have to be imported through the GUI. Do not try to copy files by yourself in the brainstorm_db folder, it won't work." I misunderstood this and assumed that also file export by copying database folders is a bad idea.

No, you understood correctly, we're definitely discouraging the average user from trying to manipulate the files in the brainstorm_db, as it was the #1 cause of broken databases.
But for more advanced users or specific use cases, it's totally fine to move around the Brainstorm files. This is one of the reasons for which we're using a file base instead of a proper database system: it gives a lot of freedom. You can move/edit files and then simply reload the protocol so that it gets displayed correctly in the database explorer.
Just make sure everything you move around is properly backed-up, because it's easy to destroy everything