Files getting replaced

Bugs
1

Hi @Raymundo.Cassani!

I'm experiencing an issue with Brainstorm when using a loop to create multiple files under the same subject and condition within a minute. For instance, when I subtract a specific time series from all trials using the difference function, I typically use a loop like the one below, as batch processing from the process tab isn't an option:

        for file = 1:length(action_files_sub)
            % Process: Difference: A-B
            sFiles = bst_process('CallProcess', 'process_diff_ab', action_files_sub{file}, tone_files_sub);
            
            % Process: Add tag: base_f_tone_2_45
            sFiles = bst_process('CallProcess', 'process_add_tag', sFiles, [], ...
                'tag',            'tone_subtracted', ...
                'output',         'name_path');  % Add to file name and file path

            %pause(60);
        end

Brainstorm typically names each file based on the clock. Despite changing the name by adding a tag, files get replaced if they are processed within the same minute. To avoid this, I had to use pause(60), which significantly increases the script's runtime.

To address this, I modified the following naming scheme in the GetNewFilename function of the bst_process.m file to include seconds in the timestamp, allowing me to reduce the pause to just 1 second:

Before:

    if isTimestamp
        c = clock;
        strTime = sprintf('_%02.0f%02.0f%02.0f_%02.0f%02.0f', c(1)-2000, c(2:5));
    else
        strTime = '';
    end

After:

    if isTimestamp
        c = ceil(clock);
        strTime = sprintf('_%02.0f%02.0f%02.0f_%02.0f%02.0f%02.0f', c(1)-2000, c(2:6));
    else
        strTime = '';
    end

This change has significantly improved the runtime. While I haven't encountered any issues so far, I'm uncertain if this modification might affect other Brainstorm functionalities. It would be great if you could review and potentially implement this change on your end, considering any necessary additional adjustments.

Thanks in advance,
Sainath Murali

2

Hello Sainath,

This is true, if and only if, the file does not exist. In your provided example, if you comment the Process: Add tag, you will notice that despite being created in the same minute, the files have different filenames.

What is happening with your code is a bit tricky. For example:

  1. For the first file, with the first process (difference) it will create a file: data_something_240527_1100.mat
  2. Then adding the tag to the Comment and Filepath will rename the file to data_something_240527_1100_tag.mat
  3. The second file, with the first process (difference) it will create a file: data_something_240527_1100.mat as this name is not used in the database, it was changed in Step 2.
  4. As such, adding the tag to the second file will rename the file to data_something_240527_1100_tag.mat again and replace the file from Step2

While this has the potential to fix your current issue, it will only move the post, as this error will come back if files are created in the same second. The bug lies in the add process tag, as it should not overwrite files, now solved at: e6a158e

In the shared script, it is possible to complete avoid the FOR loop by calling the process_diff_a_bwith arguments action_files_sub and repmat({tone_files_sub}, 1, length(action_files_sub)), to create N unique files, then add the tag to all of them.

3

Yeah! This still happened for me. So I used a wait time of 1 second which temporarily fixed my concern. Thank you for fixing the issue.

Oh! I didn't think of this. This will reduce my codes even further. Thank you!