Importing MEG/EEG: Events question

mcp0228 · April 19, 2023, 6:10am

Hi BrainStorm community,

I have 27 participants in my AEP study and each participant went through 8 different conditions and each condition has 6000 trials/epochs. I am currently using the below pipeline for my analysis and realised that the "Import MEG/EEG" process can only be applied to a single person. Therefore, I cannot select 27 participants all at once and run them through the same pipeline otherwise ALL the epochs will go to the FIRST person. In the below example, if I apply the same pipeline to all the participants, all the epochs from all participants will be imported to the participant displayed here (i.e. "UD16_R").

How can I amend this pipeline so that I can run it for the 27 participants all at once?

Raymundo.Cassani · April 19, 2023, 1:33pm

Hi @mcp0228,

Unfortunately, loop over subjects is not supported in the Pipeline editor

To run your pipeline for each of the subjects you would need to make use of scripting.

See this scripting in Brainstorm tutorial:
https://neuroimage.usc.edu/brainstorm/Tutorials/Scripting

The looping over subjects is described in the Loop over subjects section of that tutorial
https://neuroimage.usc.edu/brainstorm/Tutorials/Scripting#Loop_over_subjects

If you have any question after reading the scripting tutorial, do not hesitate in posting it in the Forum.

Best,
Raymundo

mcp0228 · April 20, 2023, 5:15am

Hello @Raymundo.Cassani
Thank you for your answer, the link provided was helpful to get things started.

Each subject (n = 27) went through 8 acquisition runs therefore the same analysis pipeline needs to be looped over the 8 runs over the 27 subjects. Hence, Loop over acquisition runs is better suited for what I am trying to do.

What do you think?

Here is a simplified snapshot of what I am trying to explain. In this case, I am only showing 2 subjects each with 2 acquisition runs. What needs to happen is that the same analysis pipeline needs to loop over the 2 runs AND over the 2 subjects.

The process that is causing difficulty from the pipeline is the "Import MEG/EEG: Events" process. Therefore, to go through some trial and error for this specific process, I made the following script using the 2 subjects that I showed above. The script was made using the step-by-step instructions from the Loop over acquisition runs tutorial section. The script is as follows. There are two nested "for loops" to loop over both RUNS and SUBJECTS.

% Script generated by Brainstorm (17-Jul-2021)

% Input files
% Added the other subject names and all the runs for all the subjects
% as per the tutorial instructions
SubjectNames = {...
    'UD14',...
    'UD15'};
sFiles = {...
    {'/.../@rawUD14_1kQ60/data_0raw_UD14_1kQ60.mat', ...
    '/.../@rawUD14_1kQ45/data_0raw_UD14_1kQ45.mat'}, ...
    {'/.../@rawUD15_1kQ60/data_0raw_UD15_1kQ60.mat', ...
    '/.../@rawUD15_1kQ45/data_0raw_UD15_1kQ45.mat'}};

% Loop on subjects
% Added as per the tutorial instructions
for iSubject = 1:length(SubjectNames)
    % Loop on runs for each subject 
    % Added as per the tutorial instructions
    for iRun = 1:length(sFiles{iSubject})
      % Start a new report
      bst_report('Start', sFiles);

      % Process: Import MEG/EEG: Events
      sFiles = bst_process('CallProcess', 'process_import_data_event', sFiles, [], ...
        'subjectname', SubjectNames{iSubject}, ...
        'condition',   '', ...
        'datafile',    {sFiles{iSubject}{iRun},'MAT'}, ... 
        'eventname',   '1', ...
        'timewindow',  [], ...
        'epochtime',   [-0.2002, 0.5996], ...
        'createcond',  0, ...
        'ignoreshort', 1, ...
        'usectfcomp',  1, ...
        'usessp',      1, ...
        'freq',        [], ...
        'baseline',    []);

    % Save and display report
    ReportFile = bst_report('Save', sFiles);
    bst_report('Open', ReportFile);
    % bst_report('Export', ReportFile, ExportDir);
    end
end

Unfortunately, I am facing some issues with this script with numerous attempts not going anywhere other than receiving error messages like the one below.

>> script_new
Operator '==' is not supported for operands of type 'cell'.

Error in file_win2unix (line 31)
if (FileName(1) == '/') && ~file_exist(FileName)

Error in bst_process>GetInputStruct (line 1464)
    FileNames = cellfun(@file_win2unix, FileNames, 'UniformOutput', 0);

Error in bst_process>CallProcess (line 2185)
        sInputs = GetInputStruct(sInputs);

Error in bst_process (line 37)
eval(macro_method);

Error in script_new (line 25)
      sFiles = bst_process('CallProcess', 'process_import_data_event', sFiles, [], ...

Hence, my conclusion is that just following the Loop over acquisition runs tutorial section is not enough to make this work (unless I misunderstood the tutorial material).

What do you think I should try now?

Raymundo.Cassani · April 20, 2023, 5:13pm

Indeed, that's what you are aiming to do.

In the tutorial script, the process process_import_data_raw is executed to link the raw recordings to Brainstorm database. In your case the raw recordings are already in your database. Thus the code will not work as it is to do what you want.

There are few errors in your code:

sFiles are the relative path, with respect to the data folder. So instead of:
'/.../@rawUD14_1kQ60/data_0raw_UD14_1kQ60.mat', ...
there should be something like:
'/.../UD14/@rawUD14_1kQ60/data_0raw_UD14_1kQ60.mat', ...
The error you get is in this line:
sFiles = bst_process('CallProcess', 'process_import_data_event', sFiles, [], ...

When calling bst_process your code uses sFiles as input, but it is [1 x nSubjects] cell array, but it must be as [1 x nRuns] cell array.

Moreover, sFiles is overwritten when running the process, this will just break your code. I'd use a variable something like sAllFiles for all the files, and then sFiles = sAllFiles{iSubject} in the first loop.

Best,
Raymundo

mcp0228 · April 20, 2023, 8:11pm

Thanks Raymundo, but what do you mean by the above?

Raymundo.Cassani · April 20, 2023, 9:01pm

I meant the shape and content of the variable sFiles.
Run the code snippets below to make it clearer.

The sFiles is an (cell) array with 2 elements (one element for each Subject)

sFiles = {...
    {'/.../@rawUD14_1kQ60/data_0raw_UD14_1kQ60.mat', ...
    '/.../@rawUD14_1kQ45/data_0raw_UD14_1kQ45.mat'}, ...
    {'/.../@rawUD15_1kQ60/data_0raw_UD15_1kQ60.mat', ...
    '/.../@rawUD15_1kQ45/data_0raw_UD15_1kQ45.mat'}};
% Shape of sFiles
disp(size(sFiles))    % [1,2]
% Type of elements in sFiles
disp(class(sFiles{1}))    % cell

Each elements in sFiles is a itself a (cell) array that contains 2 filenames

firstElementFromsFiles = sFiles{1};
% Shape of sFiles
disp(size(firstElementFromsFiles))         % [1,2]
% Type of elements in sFiles
disp(class(firstElementFromsFiles{1}))   % char

In this case the input for bst_process is must be "a cell array of file names (full path, or relative path from the protocol folder)"
https://neuroimage.usc.edu/brainstorm/Tutorials/Scripting#Line_by_line:_Body

mcp0228 · April 27, 2023, 3:45am

Thanks for the answer Raymundo,

If I already have imported files like the below and would like to use a custom-made process (see attached) for looping over the acquisition runs, what modifications would you suggest that I need to do?

process_wavg_rnoise_nepoch.m (10.0 KB)

That is, I need the script to apply the custom-made process over all the folders one by one (acquisition run) and over the subjects.

To do this I tried two different methods.

The conventional way.
I first drag-dropped the four folders into the "Process 1" Box, generated a Matlab script with the custom-process, and then modified it to get something like the one below based on what I made previously in the previous questions. However I could not get this to work - what do you think I should do?
As I could not get the below to work, I am now trying a different way.

% Input files
sFiles = {...
    'UD14/UD14_1kQ45/data_1_trial001.mat', ...
...% all the File names go here
    'UD15/UD15_1kQ60/data_1_trial005.mat'};

for iSubject = 1:length(SubjectNames)
    sFiles = sAllFiles{iSubject};

    for iRun = 1:length(sFiles{iSubject})
        
% Start a new report
bst_report('Start', sFiles);

% Process: WAvg RNoise NEpoch
sFiles = bst_process('CallProcess', 'process_wavg_rnoise_nepoch', sFiles, [], ...
    'info', []);

    end
end

The manual way.
This way involves manually putting in the relevant custom-process at the appropriate places in the X number of files which is shown below. This way is indeed longer but it certainly works so I might try this way. Unless you had any better ideas.
One thing I did notice with this method is that if the number of files becomes large (e.g. 6000 trials), this method becomes tedious to use. This is because at every 6000th trial you have to insert the same code for custom-processing.
Hence, is there a way to call a FOLDER rather than the FILES of the folder?

Sorry for the long texts!

% Script generated by Brainstorm (17-Jul-2021)

% Input files
sFiles = {...
    'UD14/UD14_1kQ45/data_1_trial001.mat', ...
    'UD14/UD14_1kQ45/data_1_trial002.mat', ...
    'UD14/UD14_1kQ45/data_1_trial003.mat', ...
    'UD14/UD14_1kQ45/data_1_trial004.mat', ...
    'UD14/UD14_1kQ45/data_1_trial005.mat'};

% Process: WAvg RNoise NEpoch
sFiles = bst_process('CallProcess', 'process_wavg_rnoise_nepoch', sFiles, [], ...
    'info', []);

sFiles = {...
    'UD14/UD14_1kQ60/data_1_trial001.mat', ...
    'UD14/UD14_1kQ60/data_1_trial002.mat', ...
    'UD14/UD14_1kQ60/data_1_trial003.mat', ...
    'UD14/UD14_1kQ60/data_1_trial004.mat', ...
    'UD14/UD14_1kQ60/data_1_trial005.mat'};

% Process: WAvg RNoise NEpoch
sFiles = bst_process('CallProcess', 'process_wavg_rnoise_nepoch', sFiles, [], ...
    'info', []);

sFiles = {...
    'UD15/UD15_1kQ45/data_1_trial001.mat', ...
    'UD15/UD15_1kQ45/data_1_trial002.mat', ...
    'UD15/UD15_1kQ45/data_1_trial003.mat', ...
    'UD15/UD15_1kQ45/data_1_trial004.mat', ...
    'UD15/UD15_1kQ45/data_1_trial005.mat'};

% Process: WAvg RNoise NEpoch
sFiles = bst_process('CallProcess', 'process_wavg_rnoise_nepoch', sFiles, [], ...
    'info', []);

sFiles = {...
    'UD15/UD15_1kQ60/data_1_trial001.mat', ...
    'UD15/UD15_1kQ60/data_1_trial002.mat', ...
    'UD15/UD15_1kQ60/data_1_trial003.mat', ...
    'UD15/UD15_1kQ60/data_1_trial004.mat', ...
    'UD15/UD15_1kQ60/data_1_trial005.mat'};

% Process: WAvg RNoise NEpoch
sFiles = bst_process('CallProcess', 'process_wavg_rnoise_nepoch', sFiles, [], ...
    'info', []);

Raymundo.Cassani · April 27, 2023, 2:55pm

Method 1 is not working, as the nested for loops do not select specific files from the inputs

Method 2 seems more appropriated, and in fact it can be automated. Any repetitive task is a task for code rather for people

The approach would be to find all the recoding files for a given Subject and a given Condition (folder). The automated Method 2 will do these tasks:

For every Subject in Subjects
- For every Condition in Conditions
  - Find / Select Trials in Condition
  - Run custom process on Trials in Condition

The code would look like:

% Subject names
SubjectNames = {'UD14', 'UD15'};
% Condition name after "_" . It assumes same condition name endings for all Subjects
Conditions   = {'1kQ45', '1kQ60'}; 

for iSubject = 1 : length(SubjectNames)
    subjectName = SubjectNames{iSubject};
    for iCondition = 1 : length(Conditions)
        conditionName = [subjectName, '_', Conditions{iCondition}];
        % Process: Select data files
        sFiles = bst_process('CallProcess', 'process_select_files_data', [], [], ...
            'subjectname',   subjectName, ...
            'condition',     conditionName, ... 
            'tag',           '', ...
            'includebad',    0, ...
            'includeintra',  0, ...
            'includecommon', 0);
        
        % Check selected files
        if isempty(sFiles)
            warning('No files were selected for Subject: "%s", Condition: "%s"\n', subjectName, conditionName);
        else
            fprintf('Selected files for Subject: "%s", Condition: "%s"\n', subjectName, conditionName);
            fprintf('%s\n', sFiles.FileName)
        end
        
        % Process: WAvg RNoise NEpoch
        sFiles = bst_process('CallProcess', 'process_wavg_rnoise_nepoch', sFiles, [], 'info', []);        
    end
end

mcp0228 · May 9, 2023, 5:48am

Thanks a lot Raymundo - all working fine with a little modification from my end.
Cheers!

mcp0228 · August 1, 2023, 1:12am

Hello Raymundo,
Sorry to bring this up again but I would appreciate your thoughts on the matter below:

The current script written by you is to select a subject then select a condition from that subject and run a specified process. I.e. Select subject > select condition > run process.

How should I modify this script if there were different events for each condition?

For example, in the data-tree displayed below, under the subject there are three conditions and the last condition (AP_C60) has two different event names.

How should I modify this script if I want to run the process separately for each event name?
I.e. Select subject > select condition > select event > run process?

Raymundo.Cassani · August 1, 2023, 4:35pm

One of the options for process_select_files_data is tag, you can set it to 1_02

In that it will find all (the 10) files that have 1_02, for a given Condition (e.g. AP_C60) for a given Subject (e.g. CodingPractice1)

Alternatively, you can have a separate (Condition) folder for each trial group (1 and 1_02) if at Import in database, the option of Create a separate folder for each event type is checked.