Scripting to create multiple subjects

Discussions
1

Hi

I'm trying the scripting function for the first time to automate most of my pre-processing. I've no prior experience with coding in Matlab/Brainstorm. I have 60 subjects, one EEG acquisition one per subject. I tried entering the below script which I found in another thread on this forum to create multiple subjects. Under a new protocol, I selected file > command window and copied pasted the below script in MATLAB command window (I put in a number for 'n' in first line, also tried for 'no' in first and second line). I got the below answer in the cmd.exe window. Could you advise what the right steps are to run this script?

function create_nSubjects(start, noSubject)
range = start:noSubject;
for i = range
if i < 10
subjectname = [‘Subject’, num2str(0), num2str(i)];
else
subjectname = [‘Subject’, num2str(i)];
end
[sSubject, iSubject] = db_add_subject(subjectname);
end

BST> Error executing command:
Error using panel_command>CreatePanel/ButtonRun_Callback (line 66)
Error: Function definitions are not permitted in this context.

2

Hi David,

The compiled version of Brainstorm has limited capabilities in terms of scripting, it seems functions are not allowed at all. If you want to be able to run more advanced scripts, I recommend you install the full version of Matlab going forward.

I adapted your script to make it work in this context. You can try the following script, which creates subjects 1 to 15. You are free to change the value of the first two variables to your needs.

firstSubject = 1;
lastSubject = 15;

for i = firstSubject:lastSubject
    if i < 10
        subjectname = ['Subject0' num2str(i)];
    else
        subjectname = ['Subject', num2str(i)];
    end
    [sSubject, iSubject] = db_add_subject(subjectname);
end
db_reload_database('current');

Does that work for you?

Cheers,
Martin

1 Like
3

Hi Martin

Thank you. That worked perfectly. I ran it in MatLab no problem.

Could you advise how I link a raw file to each subject (one acquisition run for each subject)? I copied below script from the tutorials. The raw files (BDF) are all labelled with the same word followed by a random ID number (MEMORYTASK_XXX). I assume I add label for each subject in SubjectNames. I have all my files in an external hard drive. Is there a way to extract one raw file and link to each subject heading? Like - {'/...D:\Brainstorm dB\MEMORYTASK. The location of the files is D:\Brainstorm db\MEMORYTASK

% Input files
SubjectNames = {'sub01', 'sub02'};
RawFiles = {...
{'/.../sample_group/ds117/sub001/MEG/run_01_sss.fif', ...
{'/.../sample_group/ds117/sub002/MEG/run_01_sss.fif', ...
% Process: Create link to raw file
sFileRaw = bst_process('CallProcess', 'process_import_data_raw', [], [], ...
'subjectname', SubjectNames{1}, ...
'datafile', {RawFiles{1}, 'FIF'}, ...
'channelreplace', 0, ...
'channelalign', 0, ...
'evtmode', 'value');

4

Hi David,

You are on the right path. It really depends how your data is organized in your MEMORYTASK folder. It needs to be in a standardized way so that you can write a script that works for every subject. Assuming your MEMORYTASK folder is organized in the following structure:

MEMORYTASK/sub001/MEG/run_01_sss.fif
MEMORYTASK/sub002/MEG/run_01_sss.fif
etc...

You could run the following script:

firstSubject = 1;
lastSubject = 15;
RawFile = 'D:\Brainstorm dB\MEMORYTASK\sub0%s\MEG\run_01_sss.fif';

for i = firstSubject:lastSubject
    if i < 10
        SubjectId = ['0' num2str(i)];
    else
        SubjectId = num2str(i);
    end

bst_process('CallProcess', 'process_import_data_raw', [], [], ...
    'subjectname',    ['Subject' SubjectId], ...
    'datafile',       {sprintf(RawFile, SubjectId), 'FIF'}, ...
    'channelreplace', 0, ...
    'channelalign',   0, ...
    'evtmode',        'value');
end

You can always prototype your scripts using our Pipeline Editor, and you can refer to our Scripting tutorial page for more thorough documentation.

Best,
Martin

1 Like
5

Hi Martin

I tried to create the link a raw file to each subject alone so I could make the correct code. I have all my EEG files in the one folder (D:\Brainstorm db\MEMORYTASK). They are all in this format MEMORYTASK_XXX.bdf (XXX is a three digit random subject ID number). I had a go with the below coding but no luck. I got the output further below. Also tried variations on this. Is this part (MEMORYTASK_sss.bdf) supposed to be the actual name of the file?

RawFile = 'D:\Brainstorm dB\MEMORYTASK\sub0%s\EEG\MEMORYTASK_sss.bdf';

bst_process('CallProcess', 'process_import_data_raw', [], [], ...
'subjectname', ['Subject' SubjectId], ...
'datafile', {sprintf(RawFile, SubjectId), 'BDF'}, ...
'channelreplace', 0, ...
'channelalign', 0, ...
'evtmode', 'value');
end

** Error: Line 184: panel_command>ExecuteScript (line 184)
** Error while executing script:
** panel_command>ExecuteScript (line 168)
** Error: The input character is not valid in MATLAB statements or expressions.
**
** Call stack:
** >panel_command.m>ExecuteScript at 184
** >panel_command.m at 26
** >bst_call.m at 28
** >gui_brainstorm.m>@(h,ev)bst_call(@panel_command,'ExecuteScript') at 142

6

Hi David,

Once again I must stress that you install the full version of Matlab if you want to do scripting in Brainstorm. The error message that the compiled version gives us is not very helpful. Perhaps there is someone working with you with more programming experience that could help?

The above script won't work as it ends with "end" when there was no block open. I have adapted it and fixed minor issues based on what you describe above. I can't help much without a full understanding of your directory structure. From what you describe, your MEMORYTASK folder looks like the following:

D:\Brainstorm db\MEMORYTASK\MEMORYTASK_001.bdf
D:\Brainstorm db\MEMORYTASK\MEMORYTASK_002.bdf
etc.

If that's not correct the following script won't work but you could adapt the first variable to your custom directory structure, so long as it's consistent between subjects and runs. You can use the star character (*) to match multiple strings, e.g. "\sub0*" would match any directory that starts with "sub0".

RawFiles = 'D:\Brainstorm dB\MEMORYTASK\MEMORYTASK_*.bdf';
allFiles = dir(RawFiles);
for iFile = 1:length(allFiles)
    % Get path to raw file
    RawFile = fullfile(allFiles(iFile).folder, allFiles(iFile).name);
    % Extract subject name and create it
    [path, name, ext] = fileparts(allFiles(iFile).name);    
    iUnderscore = strfind(name, '_');
    SubjectName = ['Subject' name(iUnderscore+1:end)];
    db_add_subject(SubjectName);
    
    % Import raw file to subject
    bst_process('CallProcess', 'process_import_data_raw', [], [], ...
        'subjectname', SubjectName, ...
        'datafile', {RawFile, 'EEG-BDF'}, ...
        'channelreplace', 0, ...
        'channelalign', 0, ...
        'evtmode', 'value');
end
db_reload_database('current');

Cheers,
Martin

7

Hi Martin,

Thank you for sending that. That worked. My directory structure is in that format you described. I got some help from someone versed in Matlab. I have the full version of Matlab. When I tried the file > execute script in brainstorm only I couldn't run it. When I ran it in Matlab it worked great once we added brainstorm to line 3 of the code between Lines starting "allFiles" and "for iFile". This re-opens Brainstorm and all the raw files are loaded under subject folders.

Thanks
David

1 Like
8

Hi Martin,

Could I get some help with the following script? I generated it on a single file in Brainstorm. I'm trying to re-code triggers and add EEG positions for all my data files. I replaced 005 with * for multiple subjects.

When I ran the script I got the following error message in Matlab - Error in script_combine_responses_and_add_EEG_positions (line 17) 'channelfile', {RawFiles{1}, RawFiles{1}}, ...

When I ran it on one file with single subject number instead of an asterisk it successfully re-coded the triggers but the EEG positions were not added.

I'm assuming regards the 'add EEG positions' part that this follows the approach Francois advised so I don't share a channel file across subjects - "I'd recommend the other approach Martin mentioned: select all the raw files in the protocol (you can select directly a subject folder in the Process1 list, or even the entire protocol) and run the process "Add EEG positions"."

% Script generated by Brainstorm (29-Jan-2020)

% Input files
sFiles = {...
'Subject*/@rawmemorytask_/data_0raw_MEMORYTASK_.mat'};

% Start a new report
bst_report('Start', sFiles);

% Process: Combine stim/response
sFiles = bst_process('CallProcess', 'process_evt_combine', sFiles, [], ...
'combine', ['11_wrong, 14_wrong, 11, 14' 10 '12_wrong, 14_wrong, 12, 14' 10 '21_wrong, 24_wrong, 21, 24' 10 '22_wrong, 24_wrong, 22, 24' 10 '11_right, 13_right, 11, 13' 10 '12_right, 13_right, 12, 13' 10 '21_right, 23_right, 21, 23' 10 '22_right, 23_right, 22, 23' 10 ''], ...
'dt', 15);

% Process: Add EEG positions
sFiles = bst_process('CallProcess', 'process_channel_addloc', sFiles, [], ...
'channelfile', {RawFiles{1}, RawFiles{1}}, ...
'usedefault', 8, ... % Colin27: BioSemi 128 A1
'fixunits', 1, ...
'vox2ras', 1);

% Save and display report
ReportFile = bst_report('Save', sFiles);
bst_report('Open', ReportFile);
% bst_report('Export', ReportFile, ExportDir);

9

Hi David,

The script you posted won't work because variable "RawFiles" is not defined. It should be defined as a cell array of the path to your EEG position file and its format. I recommend you generate a script from the pipeline first to see how it looks like. For example:

'channelfile', {'C:\MATLAB\tutorial_electrodes.elc', 'XENSOR'}, ...

Also, the star (*) in front of Subject won't work in this context. If you want to apply this process to all data files, you can run the process File -> Select files: recordings.

sFiles = bst_process('CallProcess', 'process_select_files_data', sFiles, , ...
'subjectname', 'All', ...
'condition', '', ...
'tag', '', ...
'includebad', 0, ...
'includeintra', 0, ...
'includecommon', 0);

Good luck!
Martin

10

Hi Martin,

I tried this script for combine stim/response and it worked perfectly but no luck for adding EEG positions. I added the file location of the default EEG positions template ICBM152 Biosemi 128 channels to the code. When I run the process a box opens to select the template which I did and the process seems to run on all 59 files )see image) but when I check the channel files under each subject folder there's no EEG positions. Could I advise if I have appropriately added the default template?

% Script generated by Brainstorm (29-Jan-2020)

% Input files
sFiles = ;

% Start a new report
bst_report('Start', sFiles);

% Process: Select data files in: /
sFiles = bst_process('CallProcess', 'process_select_files_data', sFiles, , ...
'subjectname', 'All', ...
'condition', '', ...
'tag', '', ...
'includebad', 0, ...
'includeintra', 0, ...
'includecommon', 0);

% Process: Add EEG positions
sFiles = bst_process('CallProcess', 'process_channel_addloc', sFiles, , ...
'channelfile', {'C:\Users\me\Desktop\brainstorm3\defaults\eeg\ICBM152', 'Matlab Data'}, ...
'usedefault', 49, ... % ICBM152: BioSemi 128 A1
'fixunits', 1, ...
'vox2ras', 1);

% Save and display report
ReportFile = bst_report('Save', sFiles);
bst_report('Open', ReportFile);
% bst_report('Export', ReportFile, ExportDir);

BST> Warning: No channel information was read from the file.
Warning: WARNING: When importing sensor positions for multiple subjects: the SCS
transformation from the first subject is used for all of them.
Please consider importing your subjects seprately.

In import_channel (line 353)
In channel_add_loc (line 57)
In process_channel_addloc>Run (line 149)
In process_channel_addloc (line 24)
In bst_process>Run (line 229)
In bst_process>CallProcess (line 2205)
In bst_process (line 36)
In script_add_EEG_positions (line 21)

image

11

Hi David,

To add EEG positions from a built-in template, you must not select a channel file and instead populate the UseDefault parameter. The correct process call would look like the following:

% Process: Add EEG positions
sFiles = bst_process('CallProcess', 'process_channel_addloc', sFiles, [], ...
    'channelfile', [], ...
    'usedefault', 49, ... % ICBM152: BioSemi 128 A1
    'fixunits', 1, ...
    'vox2ras', 1);

Best,
Martin

12

Hi Martin,

That worked. Thank you.

Best
David

13

Hi Martin,

Could I get your help again? I generated the below script from Brainstorm to cover some pre-processing steps for all my files. I trialed it on one raw file. The output looked fine but I got the below warning under 'process_resample' (see screenshot). The same message was listed numerous times. Could you advise if the code worked appropriately?

Also, I had originally tried to include ICA after the PSD step but the PSD and ICA steps don't seem to run together. I assume placing 'resample(256Hz)' after 'tag' under '% Process: Select data files in: /' will allow an ICA script to identify the filter file?

% Process: Band-pass:0.5Hz-30Hz
sFiles = bst_process('CallProcess', 'process_bandpass', sFiles, , ...
'sensortypes', 'EEG', ...
'highpass', 0.5, ...
'lowpass', 30, ...
'tranband', 0, ...
'attenuation', 'strict', ... % 60dB
'ver', '2019', ... % 2019
'mirror', 0, ...
'read_all', 1);

% Process: Resample: 256Hz
sFiles = bst_process('CallProcess', 'process_resample', sFiles, , ...
'freq', 256, ...
'read_all', 0);

% Process: Re-reference EEG
sFiles = bst_process('CallProcess', 'process_eegref', sFiles, , ...
'eegref', 'AVERAGE', ...
'sensortypes', 'EEG');

% Process: Power spectrum density (Welch)
sFiles = bst_process('CallProcess', 'process_psd', sFiles, , ...
'timewindow', [0, 2398.999756], ...
'win_length', 4, ...
'win_overlap', 50, ...
'sensortypes', 'EEG', ...
'win_std', 0, ...
'edit', struct(...
'Comment', 'Power', ...
'TimeBands', , ...
'Freqs', , ...
'ClusterFuncTime', 'none', ...
'Measure', 'power', ...
'Output', 'all', ...
'SaveKernel', 0));

% Save and display report
ReportFile = bst_report('Save', sFiles);
bst_report('Open', ReportFile);
% bst_report('Export', ReportFile, ExportDir);

image

14

Hi David,

The code should have run properly, even though you got a warning. You will get a warning for each file processed so it's normal you had multiple ones.

The PSD process returns a power spectrum object, while the ICA process needs to be executed on a raw file, hence why it is incompatible to run the ICA process on the output of the PSD process. Make sure you don't overwrite the sFiles variable when running the PSD process so that you run the ICA process on the actual raw data.

Hence, in your code above, replace

sFiles = bst_process('CallProcess', 'process_psd', sFiles, , ...

By

bst_process('CallProcess', 'process_psd', sFiles, , ...

Or run the ICA process before the PSD.

Best,
Martin

15

Hi Martin,

That's great, thank you. If I decide to run the analysis in steps, or any analysis after processing, could you tell me how I can run a script on the processed file (after down-sampling/band-pass filter) rather than the original raw file?

Best
David

17

Hi David,

If it's only a few files, you can manually drag and drop the files you want to process in the process box, and either generate a new script from there or right click on the process box and copy the list of files to clipboard. You can then replace the inputs of your script to these new files.

Another more efficient way would be to add tags to your processed files in order to easily identify and select them. You can use the File -> Add tag process after a specific step of your processing/analysis, and then the File -> Select files: By tag to start another processing step with these same files.

Finally, this is a bit more advanced but you could write a search query that exactly identifies your files, if it's possible to do so from metadata only. For example, a query that selects all non-raw data files: [type EQUALS "Data"]. You can build this query from the search database interface and then use it in a script using the File -> Select files: Search query process.

All of these methods are documented in our Our Select files tutorial page, I recommend you give it a read! https://neuroimage.usc.edu/brainstorm/Tutorials/PipelineEditor

Best,
Martin

18

Thanks Martin!

19

Hi Martin,

I tried to import the epochs from my files that have been filter (First bandpass and then resample). I used the Search Database (name contains "resample" AND file type equals raw data) to generate the script to identify the files to import data from (I kept the pre-filtered files in my database. This identified the correct files in the database when i ran the search. I got the below message in Matlab. I tried removing the "'datafile', {RawFiles{1}, RawFiles{1}}, ..." line so the script matches that in the EEG/epilepsy tutorial but no difference. I also tried to identify by tag (again using resample). Could you advise how I might fix it?

% Script generated by Brainstorm (29-Jan-2020)

% Input files
sFiles = [];

% Start a new report
bst_report('Start', sFiles);

% Process: Select files using search query
sFiles = bst_process('CallProcess', 'process_select_search', sFiles, [], ...
'search', '(([name CONTAINS "resample"] AND [type EQUALS "RawData"]))');

% Process: Import MEG/EEG: Events
sFiles = bst_process('CallProcess', 'process_import_data_event', sFiles, [], ...
'subjectname', SubjectNames{1}, ...
'condition', '', ...
'datafile', {RawFiles{1}, RawFiles{1}}, ...
'eventname', '11_right, 12_right, 21_right, 22_right', ...
'timewindow', [-500, 500], ...
'epochtime', [-0.1992, 1], ...
'createcond', 1, ...
'ignoreshort', 1, ...
'channelalign', 1, ...
'usectfcomp', 1, ...
'usessp', 1, ...
'freq', [], ...
'baseline', []);

% Save and display report
ReportFile = bst_report('Save', sFiles);
bst_report('Open', ReportFile);
% bst_report('Export', ReportFile, ExportDir);

Undefined variable RawFiles.

Error in script_import_wo_dc (line 17)
'datafile', {RawFiles{1}, RawFiles{1}}, ...

20

Hi David,

Yes, you need to remove the line you mentioned, as you are not importing epochs from new raw data but rather existing raw data.

'datafile', {RawFiles{1}, RawFiles{1}}, ...

The rest looks fine at quick glance. If it is not working, can you please post the full error message?

Best,
Martin

21

I re-ran it and got this;

Undefined variable SubjectNames.

Error in script_import_wo_dc (line 15)
'subjectname', SubjectNames{1}, ...

I changed the subject name line to;

'subjectname', 'All', ...

I didn't get any error message in Matlab just this message box (see screenshot).

image