Linking tsss processed fif files

Discussions
1

Hello all,
I am trying to import a series of fif files that are tsss processed into brainstorm.
When trying to import the raw fif files, Brainstorm links them
For example,
SPONT_3_xx_raw.fif
SPONT_3_xx_raw-1.fif
SPONT_3_xx_raw-2.fif
.
.
SPONT_3_xx_raw-7.fif
But when I try to import the same files that are tsss processed, sometimes Brainstorm links them, sometimes it does not.
SPONT_3_xx_raw_tsss_script.fif
SPONT_3_xx_raw-1_tsss_script.fif
SPONT_3_xx_raw-2_tsss_script.fif
.
.
SPONT_3_xx_raw-7_tsss_script.fif

Can you advise why it might be the case? What does Brainstorm look for when linking them.
I have a series of 7 files, and its laborious to individually import them. Is there a roundabout or alternate ?

Thanks,
Raghavan

2

The FIF files are chained using the tag FIFF_REF_FILE_NAME, within the FIF files.
The next file name is either explicitly mentioned in the FIF file, or it is guessed after parsing the list of other .fif files in the same folder.

If your files are initially linked correctly when using the menu Review raw file in Brainstorm, but then not linked after preprocessing, it is most likely because your preprocessing does not maintain the link tags correctly and/or alters the file names in a way that is not compatible with the FIF or BIDS standards.

See the details directly in the code:

3

Hi Francois,

I am having the same issue as the original poster: inconsistent FIF-linking performance. Like the OP, I have 3 or more files per subject, always named sid_tsss_mc, sid-1_tsss_mc, sid-2_tsss_mc, etc. As far as I can tell, in_fopen_fif parses these filenames/numbers correctly if I select sid_tsss_mc at the ‘review raw file’ stage. When the procedure “fails” for a subject, sid_tsss_mc and sid-1_tsss_mc are correctly linked, but the recursive call exits prematurely (with raw.next_num = [] at line 228) before linking sid-2_tsss_mc. When the procedure “succeeds” for a different subject, raw.next_num = 2 and sid-2_tsss_mc is found and linked. I do not receive any warnings about incompatibility (e.g., lines 290, 292).

This FIF-linking is a great feature and I hope there’s a simple fix on your or my end. Thank you!

4

If in file sid-1_tsss_mc you have raw.next_num=[] AND raw.next_fname=[], then the files are NOT linked correctly.
What makes you say they are "correctly linked"? Can you read them as linked in another program?

Please share an example sequence of files (3 first files only: sid_tsss_mc, sid-1_tsss_mc, sid-2_tsss_mc), where the first two are correctly linked but not the 3rd one. Zip the files, upload them somewhere and post the download link here.
Include a full description of how these files where processed before you try to import them in Brainstorm.

5

Thanks so much for looking into this. You can find the set of files that link the first and the second but not the third here: fif.zip. The only processing done was maxfilter. However, I noticed that the second (-1) file is much smaller than the other two, which is abnormal.

I say "correctly linked" to mean that sid and sid-1 become sid_tsss_mc (2) in the Brainstorm GUI. But for the set of files I have provided, I should get sid_tsss_mc (3).

6

I can't download the file you shared, but I'm not sure if this is because of the way it is shared or because of my broken Microsoft account.
Could you try either making this link fully open (accessible to everybody without logging in) or uploading the file somewhere else?
Thanks

7

This link should work without a login: fif.zip. The password is brain.

8

Thank you for the example file.
I checked the contents of the FIF files, and indeed the files -1 and -2 are not linked correctly.

Put a breakpoint at line 228 in in_fopen_fif and then run the execution, then try to link the file to the database. The first time the debugger stops, type raw in the Matlab command window:

raw = 

  struct with fields:

    first_samp: 411000
     last_samp: 2028999
        rawdir: [1×1618 struct]
    next_fname: []
      next_num: 1

This indicates that aphsis36_tsss_mc.fif is followed by aphsis36-1_tsss_mc.fif.
Resume the execution, now it reads recursively the header of file aphsis36-1_tsss_mc.fif. The debugger stops again, but not we see:

raw = 

  struct with fields:

    first_samp: 2029000
     last_samp: 2326999
        rawdir: [1×298 struct]
    next_fname: []
      next_num: []
       next_id: []

Which indicates that the file aphsis36-1_tsss_mc.fif is not followed by anything, it is the last in the chain.

Double-check your pre-processing pipeline. If some subjects work correctly and others don't, it probably means that some have been pre-processed in a different way, you should fix that and try again.

@Alexandre Would you have any other suggestion?
Could anybody check if this sequence of 3 files is correctly read in MNE-Python?
Maybe there is something wrong in the MNE-matlab function, or in the Matlab reader?

9

Hi Francois, thanks for checking. I get the same output with the breakpoints. I think I may have misunderstood what Brainstorm is doing here: it is only reading in files which are already marked/chained/linked as continuous (i.e., the 2GB limit is reached) during acquisition. I guess there was a pause in recording between my files -1 and -2. I don't suppose you could concatenate them by force at this stage of the pipeline? (Mostly joking.) The only goal is to simplify artifact rejection and event exporting/importing in my pipeline.

Now for the very niche use case: now I'd like to import -2 and -3. I'm pretty sure these are continuous (time stamps and event triggers match, plus -2 is 2GB), but if I select -2 in my file browser, I get an error at line 307 because the value of raw.next_num is 1. It would work well for me if sFile were parsed and next_num were set to the current file number +1. But I understand this is likely not a top priority in Brainstorm functionality!

Thanks again for looking into this.

10

Indeed. If the files are not registered as being linked, they would not be linked automatically in Brainstorm.

I don't suppose you could concatenate them by force at this stage of the pipeline?

If the files are sharing exactly the same sensor positions and SSPs (recorded together or coregistered with MaxFilter), you can concatenate them in Brainstorm. Select all the files in the correct order in Process1, then run process Standardize > Concatenate time.

now I'd like to import -2 and -3. I'm pretty sure these are continuous (time stamps and event triggers match, plus -2 is 2GB), but if I select -2 in my file browser, I get an error at line 307 because the value of raw.next_num is 1.

These are files you didn't share, I can't check what you mention. But I don't think I need them to address your question.
It looks like you added these -2 and -3 tags to the filenames yourself. I would not recommend doing this, especially if you label the files with indices that are not what the files contain. Apparently, the file in which you added the tag -2 is the first file in a sequence (next index is 1 => the file index is 0). This means that this file should not contain any index tag, or if it contains one, it should be 0.

I recommend you reorganize the way you name your files, in order to make your data readable by all the programs that support the FIF format.

11

Hi Francois, you're probably right that those files were inappropriately renamed, but they were collected years ago, so who knows. In any case, concatenating in time works great for this subject -- thank you for drawing my attention to that -- as we need all the experimental events to be on the same "clock" for further processing. Thanks for your help!