Thanks a lot Dimitrios for the profiling results, it's quite telling actually!
@Francois: The bottleneck is not in the processes but after, due to this line: https://github.com/brainstorm-tools/brainstorm3/blob/master/toolbox/process/bst_process.m#L271
The process outputs a list of files (string), but bst_process creates an input structure for all of them, which is extremely slow when you have thousands of files.
From what I see, the created structure is only used twice:
- To find which studies to redraw in the database. This could be skipped here since this is a read-only process, the database was not modified.
- To update the report, but I don't think the structure is required for that, the list of files would be enough.
How about we add a new "read-only" field to certain processes and when it is set, we skip this step in bst_process? I am fairly confident this would cut most of the overhead here.
Or there might be an even better solution here, let me know.
Martin