Scripting to create multiple subjects

Discussions
14

Hi David,

The code should have run properly, even though you got a warning. You will get a warning for each file processed so it's normal you had multiple ones.

The PSD process returns a power spectrum object, while the ICA process needs to be executed on a raw file, hence why it is incompatible to run the ICA process on the output of the PSD process. Make sure you don't overwrite the sFiles variable when running the PSD process so that you run the ICA process on the actual raw data.

Hence, in your code above, replace

sFiles = bst_process('CallProcess', 'process_psd', sFiles, , ...

By

bst_process('CallProcess', 'process_psd', sFiles, , ...

Or run the ICA process before the PSD.

Best,
Martin

15

Hi Martin,

That's great, thank you. If I decide to run the analysis in steps, or any analysis after processing, could you tell me how I can run a script on the processed file (after down-sampling/band-pass filter) rather than the original raw file?

Best
David

17

Hi David,

If it's only a few files, you can manually drag and drop the files you want to process in the process box, and either generate a new script from there or right click on the process box and copy the list of files to clipboard. You can then replace the inputs of your script to these new files.

Another more efficient way would be to add tags to your processed files in order to easily identify and select them. You can use the File -> Add tag process after a specific step of your processing/analysis, and then the File -> Select files: By tag to start another processing step with these same files.

Finally, this is a bit more advanced but you could write a search query that exactly identifies your files, if it's possible to do so from metadata only. For example, a query that selects all non-raw data files: [type EQUALS "Data"]. You can build this query from the search database interface and then use it in a script using the File -> Select files: Search query process.

All of these methods are documented in our Our Select files tutorial page, I recommend you give it a read! https://neuroimage.usc.edu/brainstorm/Tutorials/PipelineEditor

Best,
Martin

18

Thanks Martin!

19

Hi Martin,

I tried to import the epochs from my files that have been filter (First bandpass and then resample). I used the Search Database (name contains "resample" AND file type equals raw data) to generate the script to identify the files to import data from (I kept the pre-filtered files in my database. This identified the correct files in the database when i ran the search. I got the below message in Matlab. I tried removing the "'datafile', {RawFiles{1}, RawFiles{1}}, ..." line so the script matches that in the EEG/epilepsy tutorial but no difference. I also tried to identify by tag (again using resample). Could you advise how I might fix it?

% Script generated by Brainstorm (29-Jan-2020)

% Input files
sFiles = [];

% Start a new report
bst_report('Start', sFiles);

% Process: Select files using search query
sFiles = bst_process('CallProcess', 'process_select_search', sFiles, [], ...
'search', '(([name CONTAINS "resample"] AND [type EQUALS "RawData"]))');

% Process: Import MEG/EEG: Events
sFiles = bst_process('CallProcess', 'process_import_data_event', sFiles, [], ...
'subjectname', SubjectNames{1}, ...
'condition', '', ...
'datafile', {RawFiles{1}, RawFiles{1}}, ...
'eventname', '11_right, 12_right, 21_right, 22_right', ...
'timewindow', [-500, 500], ...
'epochtime', [-0.1992, 1], ...
'createcond', 1, ...
'ignoreshort', 1, ...
'channelalign', 1, ...
'usectfcomp', 1, ...
'usessp', 1, ...
'freq', [], ...
'baseline', []);

% Save and display report
ReportFile = bst_report('Save', sFiles);
bst_report('Open', ReportFile);
% bst_report('Export', ReportFile, ExportDir);

Undefined variable RawFiles.

Error in script_import_wo_dc (line 17)
'datafile', {RawFiles{1}, RawFiles{1}}, ...

20

Hi David,

Yes, you need to remove the line you mentioned, as you are not importing epochs from new raw data but rather existing raw data.

'datafile', {RawFiles{1}, RawFiles{1}}, ...

The rest looks fine at quick glance. If it is not working, can you please post the full error message?

Best,
Martin

21

I re-ran it and got this;

Undefined variable SubjectNames.

Error in script_import_wo_dc (line 15)
'subjectname', SubjectNames{1}, ...

I changed the subject name line to;

'subjectname', 'All', ...

I didn't get any error message in Matlab just this message box (see screenshot).

image

22

Hi Martin,

I also tried the below script using the tag process but this didn't work either. I can run the process accurately if I copy the entire protocol into the run process box and generate a script listing all the database files and then use the search function within this script. It will be useful to know if there's another way to do it though rather than have every file in the database listed in the script.

% Script generated by Brainstorm (29-Jan-2020)

% Input files
sFiles = [];

% Start a new report
bst_report('Start', sFiles);

% Process: Select file names with tag: resample
sFiles = bst_process('CallProcess', 'process_select_tag', sFiles, [], ...
'tag', 'resample', ...
'search', 2, ... % Search the file names
'select', 1); % Select only the files with the tag

% Process: Import MEG/EEG: Events
sFiles = bst_process('CallProcess', 'process_import_data_event', sFiles, [], ...
'subjectname', '', ...
'condition', '', ...
'eventname', '11_right, 12_right, 21_right, 22_right', ...
'timewindow', [], ...
'epochtime', [-0.1992, 1], ...
'createcond', 1, ...
'ignoreshort', 1, ...
'usectfcomp', 1, ...
'usessp', 1, ...
'freq', [], ...
'baseline', []);

% Save and display report
ReportFile = bst_report('Save', sFiles);
bst_report('Open', ReportFile);
% bst_report('Export', ReportFile, ExportDir);

23

Hi David,

Yes, it's fine to leave the subjectname parameter empty since the search query process before populates this field for you.

Are you sure your search query is appropriate? Raw files are typically all named "Link to raw file" yet you're trying to find one with the keyword "resample", so there are probably simply no matches. You can test your query from the GUI by clicking the magnifying glass icon at the top right of the Brainstorm database and pasting/typing your query. If you are looking for a raw file whose parent folder contains the word "resample", the following query would be more appropriate:

(([type EQUALS "RawData"] AND [parent CONTAINS "resample"]))

Best,
Martin

24

Hi Martin

When I type in resample under search/magnifying glass (name contains resample) the correct files are filtered. I see the subject folder, then just the resampled file folder and then the raw file. I tried this code but no luck.

% Script generated by Brainstorm (29-Jan-2020)

% Input files
sFiles = [];

% Start a new report
bst_report('Start', sFiles);

% Process: Select files using search query
sFiles = bst_process('CallProcess', 'process_select_search', sFiles, [], ...
'search', '(([type EQUALS "RawData"] AND [parent CONTAINS "resample"]))');

% Process: Import MEG/EEG: Events
sFiles = bst_process('CallProcess', 'process_import_data_event', sFiles, [], ...
'subjectname', '', ...
'condition', '', ...
'eventname', '11_right, 12_right, 21_right, 22_right', ...
'timewindow', [-500, 500], ...
'epochtime', [-0.1992, 1], ...
'createcond', 1, ...
'ignoreshort', 1, ...
'channelalign', 1, ...
'usectfcomp', 1, ...
'usessp', 1, ...
'freq', [], ...
'baseline', []);

% Save and display report
ReportFile = bst_report('Save', sFiles);
bst_report('Open', ReportFile);
% bst_report('Export', ReportFile, ExportDir);

25

Original files are named "Link to raw file".
Processed files (resampled, filtered, etc) are tagged by the processes, eg "Raw | notch(60Hz 120Hz 180Hz)" or "Raw | resample(256Hz)".

26

HI Francois,

Thank you. I tried both "resample" and "Raw | band(0.5-30Hz) | resample(256Hz)". They work in the search function (magnifying glass in database) but not in the script. I have used both Select files using search query and by tag.

27

Hi

This relates to the issue I described above. I have extracted 4 epochs from my data (11_right, 12_right, 21_right and 22_right). Now I'm trying to remove all channels from each trial with +/-100uv. When I used the search function (at magnifying glass) the right files were filtered through. I generated a code from this and combined with the script for detect bad channels-peak to peak. When I run the script I get this message (screenshot). I can filter the files suing the search query and generate a script listing all the files I want to run this process on but it leads to a much longer script.

% Script generated by Brainstorm (29-Jan-2020)

% Input files
sFiles = ;

% Start a new report
bst_report('Start', sFiles);

% Process: Select files using search query
sFiles = bst_process('CallProcess', 'process_select_search', sFiles, , ...
'search', '([name CONTAINS "11_right"] OR [name CONTAINS "12_right"] OR [name CONTAINS "21_right"] OR [name CONTAINS "22_right"])');

% Process: Detect bad channels: Peak-to-peak EEG(0-100)
sFiles = bst_process('CallProcess', 'process_detectbad', sFiles, , ...
'timewindow', [-0.19921875, 1], ...
'meggrad', [0, 0], ...
'megmag', [0, 0], ...
'eeg', [0, 100], ...
'ieeg', [0, 0], ...
'eog', [0, 0], ...
'ecg', [0, 0], ...
'rejectmode', 1); % Reject only the bad channels

% Save and display report
ReportFile = bst_report('Save', sFiles);
bst_report('Open', ReportFile);
% bst_report('Export', ReportFile, ExportDir);

image
image1920×1080 241 KB

28

Hi David,

I think the issue stems from the fact that your version of Brainstorm is outdated. It is a relatively new feature that the search query process (process_select_search) works without inputs, which is the error you are getting (No input). Please try updating Brainstorm to the latest version.

Let us know,
Martin

29

Hi Martin,

You were right, that seems to have fixed it. Matlab had been reading the Brainstorm folder as read only.
Thank you.

David

31

Hi

I'm averaging EEG responses across subjects and then two groups. I'm not sure how but a couple of my subjects have been given extra channels which led to an error in the process (in first screenshot, 8 warnings) (I think possibly when I ran the EEG acquisition I selected 256 channels to be saved instead of 128 for these two). I originally imported the Biosemi 128 A1 template (using a script to import for all subjects). This gave the other subjects 144 channels (the extra 16 channels were not used) but for two subjects there are 272 channels. I manually deleted the additional 128 channels for both subjects and re-ran the averaging process. I then got the second error message below (4 warnings). Could you advise how I can fix this?

image

image

32

Use the process "Standardize > Uniform list of channels".
Make a full backup of your database first as this maybe damage it if anything goes wrong.

For future questions unrelated with your initial scripting questions, please create new topics.

33

Hi Francois,

Thank you, I will do that. I currently have three protocols saved in my database.

When I run this process via Matlab will it only run on the protocol open in Brainstorm like other processes or on all three protocols? I assume if I copy the entire protocol into the process one box it will standardise both those tied to subject folders and those tied to epochs in the protocol?

Also will the excess non-used channels in the two subjects folders have affected the ICA/SSP screening for artifacts?

Best
David

34

When I run this process via Matlab will it only run on the protocol open in Brainstorm like other processes or on all three protocols? I assume if I copy the entire protocol into the process one box it will standardise both those tied to subject folders and those tied to epochs in the protocol?

It would be applied to the data files you selected in Process1.

Also will the excess non-used channels in the two subjects folders have affected the ICA/SSP screening for artifacts?

Not if they were marked as bad or as non-EEG.
If you are seeing them when you double-click on the recordings: yes.
If you don't see them: no.

35

Thank you Francois