Sorting files in database

DNAcombo · July 3, 2019, 2:40pm

Hi,
Is there a way to sort files in the database ?
I'm pushing all my subjects to the default brain, and they appear in groupanalysis/intra-subject in the order they were pushed. With many subject, it gets complicated to find out which ones are in and which ones not. This is especially relevant if I reprocess one subject for some reason.
If they were sorted, it would be really easy...
Thanks!
Max

MartinC · July 3, 2019, 2:55pm

Hi Max,

We have various processes to select the appropriate files for your pipelines. Have a look at our list of processes under File -> Select Files. You can also filter the process box. Refer to this tutorial page for more information: https://neuroimage.usc.edu/brainstorm/SelectFiles

Martin

Francois · July 3, 2019, 4:04pm

No, there is no easy way to sort the files within one folder... If you reload the folder, they will appear by alphabetical order of the files, not by the order of creation, this is the only thing you can change.

Two solutions for easier sorting can be used, but need to think about it in advance.

Add tags to the files after they are created, so you can easily get them with the search box or the selection processes:
https://neuroimage.usc.edu/brainstorm/Tutorials/PipelineEditor#Control_the_output_file_names
https://neuroimage.usc.edu/brainstorm/Tutorials/Scripting#Running_the_script_again
As you go, move the files in other folders than the intra-subject folder.

The group analysis tutorial uses both solutions, you have the GUI version illustrated in the page and the scripted version accessible at the bottom of the page:
https://neuroimage.usc.edu/brainstorm/Tutorials/VisualGroup#Sources
https://neuroimage.usc.edu/brainstorm/Tutorials/VisualGroup#Group_analysis:_Sources

pdkieffaber · September 9, 2020, 2:57pm

I have a question related to this topic. I would like to create figures similar to those produced by "Display as image", but I would like to use a parameter to sort files/subjects (equivalent to EEGlab's "EEG image"). Rather than adding tags to filenames, I would like to add fields to the data structure. For example, a "Behavioral" field with subfields like "RT" and "ACC" etc. I realize this approach wouldn't work with the file grouping processes or the GUI filter, but I'm wondering if Brainstorm will delete unrecognized fields added to each files underlying Matlab structure variable.

Francois · September 12, 2020, 8:48am

I'm wondering if Brainstorm will delete unrecognized fields added to each files underlying Matlab structure variable

No. If you save additional fields in the .mat files, they will remain in place. You can store any additional data you want in them. However:

If you transform the files (eg. filtering), they might not be carried to the new file, but I guess you want to add fields to the imported epochs, which are already fully pre-processed, therefore this should not be a limitation.
The extra fields are not loaded in memory, and therefore not available from the GUI. If you want to create new displays, you would need to read the extra data from the files instead of the data structures in memory. This is a very minor problem, but I just wanted to point out the fact that the data structures loaded in memory when viewing the files are not exactly the files that are on the hard drive.

What you want to do is probably achievable with a custom version of view_erpimag.m, in which you would add an extra block that sorts the input files based on the extra fields (maybe around line 62).

github.com

brainstorm-tools/brainstorm3/blob/master/toolbox/gui/view_erpimage.m#L62


      
              DataFiles = {DataFiles};
              DisplayMode = 'trialimage';
          end
          % Progress bar
          bst_progress('start', 'ERP image', 'Loading data...');
          % Get file type
          FileType = file_gettype(DataFiles{1});
          % Are we going to create a new figure?
          isNewFig = isempty(hFig) || isequal(hFig, 'NewFigure');
          
          
          % ===== LOAD DATA =====
          switch (FileType)
              % ===== RECORDINGS =====
              case {'data','pdata'}
                  % Load data file
                  iDS = bst_memory('LoadDataFile', DataFiles{1});
                  if isempty(iDS)
                      return;
                  end
                  bst_memory('LoadRecordingsMatrix', iDS);

pdkieffaber · June 9, 2021, 8:09pm

Francois,
I'm revisiting this topic as I am now ready to try implementing it with some new data. I noticed that the "Events" structure associated with each file has a "reactTime" field, which I am assuming would hold the kind of event metadata I'm talking about. Do you know if I add other metadata fields to my event structure in the continuous file, whether or not those will carry forward into segmented data? What I would like to do is add a "metadata" field to the "Events" structure and then write a process that would include a drop-down menu allowing the user to select any of the sub-fields of "metadata" for sorting trials. Does this sound like it would work?

Thanks!
Paul

Francois · June 10, 2021, 12:10pm

This field reactTime is defined only in Neuroscan recordings. It's carried through all the different processes and always kept in the data. You can set to anything you'd like, as long as it has exactly the same dimension as the times field.

Do you know if I add other metadata fields to my event structure in the continuous file, whether or not those will carry forward into segmented data?

No, it is not safe to add any field to the documented events structure.
https://neuroimage.usc.edu/brainstorm/Tutorials/EventMarkers#On_the_hard_drive

In the next few years we will add metadata to document the trials properly.
Unfortunately, I don't have any easy solution for you before we start working on this.
You can add fields to the imported epochs, and you can write scripts to process the files in the way you want, but this won't be easy to integrate with the Brainstorm interface.

pdkieffaber · June 10, 2021, 5:53pm

Francois,
Thanks for the quick replies. I managed to write a process that does the trick. I store metadata in a .csv file where the first row indicates the name of the metadata (e.g., 'RT', 'Accuracy', etc.). The process asks the user to select the metadata file and designate the column name. Provided the number of sorting values in the specified column is the same as the number of trials, the trials will be sorted (ascending or descending) and submitted to view_erpimage.m. I'm sure you will cringe at the code I added to include the trend line, but it serves its purpose. I'm posting the code here in case any other users are interested.
Thanks again!
Paul

P.S. How can I enable the "Smooth display" option that is available when visualizing a spectogram?

process_ERPimage.m (6.2 KB)

pdkieffaber · June 10, 2021, 7:17pm

I just noticed that the "smooth display" option for this type of image is on the "What's next" list.

Francois · June 11, 2021, 10:13am

Indeed, this is not available yet...

I'm posting the code here in case any other users are interested.

Thanks.
If you are interested in sharing this more formally, you could open a PR on this repo (include a README with instructions for other users to understand how to use it, see examples in other processes):