Segmentation fault during OpenMEEG head model calculation

Hello. I've got T1 images, BESA EEG sensor coordinates, and EEG data from several individual subjects. I have segmented all the MRIs with FreeSurfer and imported them into Brainstorm. I have also imported all the individual BESA coordinates into Brainstorm. I am now starting to construct head models for the individual subjects. My source model consists of ~15,000 vertices of neocortex, as well as a volume composed of some subcortical structures.

On Ubuntu: After starting "Compute head model", it runs for about 24 minutes, gets to the point where the dialog says "Assembling EEG leadfield", then the process crashes with the attached segmentation fault error. I tried it with several different subjects, and the same thing happens every time.

On macOS: The head models finish successfully to completion in about 32 minutes, identical data.

I'm using the mac laptop for now, but my Ubuntu machine is more powerful so I'd prefer to use that, so I am wondering if there is anything that can be done about this error, on my end or yours. I will probably need to provide more information besides just the error message for this to be addressed, and will be ready and willing to do so. Thanks very much!

Karl Lerud

Thanks for sharing.

Can you please provide some data, and instructions to see if I can replicate the crash.

mean while can you try this for me:

  1. check if openmeeg_hminv.mat, openmeeg_dsm.mat, openmeeg_h2em.mat, opemeeg_gain_eeg.mat flies in /home/pythagoras/.brainstorm/tmp/ they still exist.

  2. cd /home/pythagoras/.brainstorm/openmeeg/linux64 and execute the following line:

LD_LIBRARY_PATH=$PWD ./om_gain --help
  1. if everything worked, and the files exist run:
LD_LIBRARY_PATH=$PWD ./om_gain  -EEG /home/pythagoras/.brainstorm/tmp/openmeeg_hminv.mat /home/pythagoras/.brainstorm/tmp/openmeeg_dsm.mat /home/pythagoras/.brainstorm/tmp/openmeeg_h2em.mat /home/pythagoras/.brainstorm/tmp/openmeeg_gain_eeg.mat

All those files exist except for openmeeg_gain_eeg.mat, but I tried it anyway. Another segmentation fault:

pythagoras@pythagoras:~/.brainstorm/openmeeg/linux64$ LD_LIBRARY_PATH=$PWD ./om_gain  -EEG /home/pythagoras/.brainstorm/tmp/openmeeg_hminv.mat /home/pythagoras/.brainstorm/tmp/openmeeg_dsm.mat /home/pythagoras/.brainstorm/tmp/openmeeg_h2em.mat /home/pythagoras/.brainstorm/tmp/openmeeg_gain_eeg.mat
./om_gain version 2.4.0 compiled at Jul 13 2018 15:57:20 using OpenMP
 Executing using 12 threads.


| ------ ./om_gain
| -EEG
| /home/pythagoras/.brainstorm/tmp/openmeeg_hminv.mat
| /home/pythagoras/.brainstorm/tmp/openmeeg_dsm.mat
| /home/pythagoras/.brainstorm/tmp/openmeeg_h2em.mat
| /home/pythagoras/.brainstorm/tmp/openmeeg_gain_eeg.mat
| -----------------------

Using variable with name : matrix
Segmentation fault (core dumped)
pythagoras@pythagoras:~/.brainstorm/openmeeg/linux64$

I’d be glad to share whatever data is needed to replicate the error, but I’m not super familiar with FreeSurfer and Brainstorm yet. What would be the minimum things needed to replicate this particular part of the process? My guess is the mat files that Brainstorm has for this protocol so far in its database, for this particular subject. Here are those files, organized into anat and data directories like Brainstorm does. I am attempting to make a head model out of the “mixed” cortex+subcortical. Let me know if this is sufficient, thanks.

https://www.dropbox.com/s/2kjki8hqeojigt8/test.zip?dl=1

Karl

I’m not familiar with brainstorm either, if you can provide me with the openmeeg_xxxxx.mat files that would be great.

meanwhile two things:

  1. in your side ( I don’t think it would have any effect but just in case) change the number of threads from 12 to 1. like this:
LD_LIBRARY_PATH=$PWD OMP_NUM_THREADS=1 ./om_gain  -EEG /home/pythagoras/.brainstorm/tmp/openmeeg_hminv.mat /home/pythagoras/.brainstorm/tmp/openmeeg_dsm.mat /home/pythagoras/.brainstorm/tmp/openmeeg_h2em.mat /home/pythagoras/.brainstorm/tmp/openmeeg_gain_eeg.mat
  1. On my side I’ll try to generate those intermediate files

@francoise could you point me to some tutorial doing something similar to what @lerud is trying to do ? thx.

sorry, I put an extra ‘e’ at the name. @francois can you provide such link?

The EEG/Epilepsy tutorial you’ve already used?
https://neuroimage.usc.edu/brainstorm/Tutorials/Epilepsy

Wouldn’t it be better to restore the 2.2 version as the default in Brainstorm while you guys fix all these issues with the new release?..

@francois, I don’t think that reverting to 2.2 is the solution. So far we have mainly encountered openmeeg-brainstorm integration issues (missing libraries, paths, lib incompatibility, etc…) This is the first one with something internal to openmeeg.

What it would be great is to find a manner to replicate brainstorm pipelines so that I can just run script like the epilepsy tutorial and generate the data. I’m sure there’s a way to do so. Its just that I’m not familiar with it.

@lerud could you please try this for me:

  1. set a break point in line 249 of brainstorm3/toolbox/script/tutorial_epilepsy.m
  2. follow the data download steps here https://neuroimage.usc.edu/brainstorm/Tutorials/Epilepsy#Download_and_installation
  3. open brainstorm
  4. execute tutorial_epilepsy('/path_to/where_you_download_the_zip', '/tmp/report_dir/')

Once the execution has stop at the break point:

  1. cd ~/.brainstorm/openmeeg/linux64
  2. execute LD_LIBRARY_PATH=$PWD OMP_NUM_THREADS=1 ./om_gain -EEG ~/.brainstorm/tmp/openmeeg_hminv.mat ~/.brainstorm/tmp/openmeeg_dsm.mat ~/.brainstorm/tmp/openmeeg_h2em.mat ~/.brainstorm/tmp/openmeeg_gain_eeg.mat

and see if it still segfaults

You mean: something else that the script tutorial_epilepsy.m ?

That one is completely fine. Something like that one but that breaks

Here is a zip of everything in my tmp directory after the segmentation fault on my data. And as I mentioned, openmeeg_gain_eeg.mat does not appear in my case:

https://www.dropbox.com/s/pc7iwx8mt062ajk/tmp.zip?dl=1

I tried it with a single thread and the same thing happened.

I did what you suggested with the epilepsy tutorial and it looks like the same commands finish to completion in this case:

pythagoras@pythagoras:~/.brainstorm/openmeeg/linux64$ LD_LIBRARY_PATH=$PWD OMP_NUM_THREADS=1 ./om_gain -EEG ~/.brainstorm/tmp/openmeeg_hminv.mat ~/.brainstorm/tmp/openmeeg_dsm.mat ~/.brainstorm/tmp/openmeeg_h2em.mat ~/.brainstorm/tmp/openmeeg_gain_eeg.mat
./om_gain version 2.4.0 compiled at Jul 13 2018 15:57:20 using OpenMP
 Executing using 1 threads.


| ------ ./om_gain
| -EEG
| /home/pythagoras/.brainstorm/tmp/openmeeg_hminv.mat
| /home/pythagoras/.brainstorm/tmp/openmeeg_dsm.mat
| /home/pythagoras/.brainstorm/tmp/openmeeg_h2em.mat
| /home/pythagoras/.brainstorm/tmp/openmeeg_gain_eeg.mat
| -----------------------
Using variable with name : matrix
-------------------------------------------
| Elapsed Time: 2.00185 s.
-------------------------------------------
pythagoras@pythagoras:~/.brainstorm/openmeeg/linux64$ 

I’m not sure what you expect, and from who. Writing a script that reproduces the crash means that we already know how to reproduce it, hence how to fix it…

Thx, @lerud. I can reproduce.

./om_gain -EEG /tmp/yy/openmeeg_hminv.mat /tmp/yy/openmeeg_dsm.mat /tmp/yy/openmeeg_h2em.mat /tmp/yy/gain_eeg.mat

./om_gain version 2.4.0 compiled at Jun 22 2018 18:01:35 using OpenMP
 Executing using 4 threads.


| ------ ./om_gain
| -EEG
| /tmp/yy/openmeeg_hminv.mat
| /tmp/yy/openmeeg_dsm.mat
| /tmp/yy/openmeeg_h2em.mat
| /tmp/yy/gain_eeg.mat
| -----------------------
Using variable with name : matrix
zsh: segmentation fault (core dumped)  ./om_gain -EEG /tmp/yy/openmeeg_hminv.mat /tmp/yy/openmeeg_dsm.mat  

I'll try to do some forensics on it.
here is the ticket:

I see that changes to both OpenMEEG and Matio have been merged. I now get the new error message from OpenMEEG. But how can I utilize the fix that was merged in Matio? Would I have to build and install Matio manually, or will OpenMEEG incorporate the fix somehow?

Karl

Sorry for the delay. As you can se here https://github.com/openmeeg/openmeeg/issues/334#issuecomment-413913953 we have working binaries.

If you want to speed up and baypass the system I think that you can download the OpenMEEG universal linux binaries from https://github.com/openmeeg/openmeeg_packages untar them and swap the old backed for the new one. Like so:

just to make sure that its broken

~/.brainstorm/openmeeg/linux64
❯ LD_LIBRARY_PATH=$PWD OMP_NUM_THREADS=1 ./om_gain -EEG ~/.brainstorm/tmp/openmeeg_hminv.mat ~/.brainstorm/tmp/openmeeg_dsm.mat ~/.brainstorm/tmp/openmeeg_h2em.mat ~/.brainstorm/tmp/openmeeg_gain_eeg.mat
./om_gain version 2.4.0 compiled at Jul 13 2018 15:57:20 using OpenMP
 Executing using 1 threads.


| ------ ./om_gain
| -EEG
| /home/sik/.brainstorm/tmp/openmeeg_hminv.mat
| /home/sik/.brainstorm/tmp/openmeeg_dsm.mat
| /home/sik/.brainstorm/tmp/openmeeg_h2em.mat
| /home/sik/.brainstorm/tmp/openmeeg_gain_eeg.mat
| -----------------------
Using variable with name : matrix
zsh: segmentation fault (core dumped)  LD_LIBRARY_PATH=$PWD OMP_NUM_THREADS=1 ./om_gain -EEG    

Keep a copy of the current OpenMEEG backend. (see that there would be 4 broken links)

~/.brainstorm/openmeeg/linux64
❯ mv libOpenMEEG.so.1.1.0 _libOpenMEEG.so.1.1.0

~/.brainstorm/openmeeg/linux64
❯ mv libOpenMEEGMaths.so.1.1.0 _libOpenMEEGMaths.so.1.1.0

~/.brainstorm/openmeeg/linux64
❯ ls -l    
total 152640
lrwxrwxrwx 1 sik sik       21 Jul 13 17:58 libOpenMEEGMaths.so -> libOpenMEEGMaths.so.1
lrwxrwxrwx 1 sik sik       25 Jul 13 17:58 libOpenMEEGMaths.so.1 -> libOpenMEEGMaths.so.1.1.0
-rw-r--r-- 1 sik sik 20966161 Jul 13 17:56 _libOpenMEEGMaths.so.1.1.0
lrwxrwxrwx 1 sik sik       16 Jul 13 17:58 libOpenMEEG.so -> libOpenMEEG.so.1
lrwxrwxrwx 1 sik sik       20 Jul 13 17:58 libOpenMEEG.so.1 -> libOpenMEEG.so.1.1.0
-rw-r--r-- 1 sik sik  1925873 Jul 13 17:57 _libOpenMEEG.so.1.1.0
-rwxr-xr-x 1 sik sik    91362 Jul 13 17:57 om_assemble
-rwxr-xr-x 1 sik sik 14744873 Jul 13 17:57 om_check_geom
-rwxr-xr-x 1 sik sik    37065 Jul 13 17:57 om_forward
-rwxr-xr-x 1 sik sik    83188 Jul 13 17:57 om_gain
-rwxr-xr-x 1 sik sik 14744875 Jul 13 17:57 om_geometry_info
-rwxr-xr-x 1 sik sik 14767946 Jul 13 17:57 om_make_nerve
-rwxr-xr-x 1 sik sik    47597 Jul 13 17:57 om_matrix_convert
-rwxr-xr-x 1 sik sik    31855 Jul 13 17:57 om_matrix_info
-rwxr-xr-x 1 sik sik 14744839 Jul 13 17:57 om_mesh_concat
-rwxr-xr-x 1 sik sik 14768785 Jul 13 17:57 om_mesh_convert
-rwxr-xr-x 1 sik sik 14744813 Jul 13 17:57 om_mesh_info
-rwxr-xr-x 1 sik sik 14745097 Jul 13 17:57 om_mesh_smooth
-rwxr-xr-x 1 sik sik    36507 Jul 13 17:57 om_mesh_to_dip
-rwxr-xr-x 1 sik sik    31471 Jul 13 17:57 om_minverser
-rwxr-xr-x 1 sik sik 14756335 Jul 13 17:57 om_project_sensors
-rwxr-xr-x 1 sik sik 14786966 Jul 13 17:57 om_register_squids
-rwxr-xr-x 1 sik sik    31529 Jul 13 17:57 om_squids2vtk
-rw-r--r-- 1 sik sik       68 Jul 25 17:57 url

Overwrite with the new backend. (see that now the links are proper again)

~/.brainstorm/openmeeg/linux64
❯ cp $PATH_TO_UNZIP/OpenMEEG-2.4.0-Linux/lib/libOpenMEEG.so.1.1.0 ./

~/.brainstorm/openmeeg/linux64
❯ cp $PATH_TO_UNZIP/OpenMEEG-2.4.0-Linux/lib/libOpenMEEGMaths.so.1.1.0 ./ 

~/.brainstorm/openmeeg/linux64 19s
❯ ls -l
total 175168
lrwxrwxrwx 1 sik sik       21 Jul 13 17:58 libOpenMEEGMaths.so -> libOpenMEEGMaths.so.1
lrwxrwxrwx 1 sik sik       25 Jul 13 17:58 libOpenMEEGMaths.so.1 -> libOpenMEEGMaths.so.1.1.0
-rw-r--r-- 1 sik sik 20966161 Jul 13 17:56 _libOpenMEEGMaths.so.1.1.0
-rw-r--r-- 1 sik sik 21121792 Aug 17 18:20 libOpenMEEGMaths.so.1.1.0
lrwxrwxrwx 1 sik sik       16 Jul 13 17:58 libOpenMEEG.so -> libOpenMEEG.so.1
lrwxrwxrwx 1 sik sik       20 Jul 13 17:58 libOpenMEEG.so.1 -> libOpenMEEG.so.1.1.0
-rw-r--r-- 1 sik sik  1925873 Jul 13 17:57 _libOpenMEEG.so.1.1.0
-rw-r--r-- 1 sik sik  1925344 Aug 17 18:20 libOpenMEEG.so.1.1.0
-rwxr-xr-x 1 sik sik    91362 Jul 13 17:57 om_assemble
-rwxr-xr-x 1 sik sik 14744873 Jul 13 17:57 om_check_geom
-rwxr-xr-x 1 sik sik    37065 Jul 13 17:57 om_forward
-rwxr-xr-x 1 sik sik    83188 Jul 13 17:57 om_gain
-rwxr-xr-x 1 sik sik 14744875 Jul 13 17:57 om_geometry_info
-rwxr-xr-x 1 sik sik 14767946 Jul 13 17:57 om_make_nerve
-rwxr-xr-x 1 sik sik    47597 Jul 13 17:57 om_matrix_convert
-rwxr-xr-x 1 sik sik    31855 Jul 13 17:57 om_matrix_info
-rwxr-xr-x 1 sik sik 14744839 Jul 13 17:57 om_mesh_concat
-rwxr-xr-x 1 sik sik 14768785 Jul 13 17:57 om_mesh_convert
-rwxr-xr-x 1 sik sik 14744813 Jul 13 17:57 om_mesh_info
-rwxr-xr-x 1 sik sik 14745097 Jul 13 17:57 om_mesh_smooth
-rwxr-xr-x 1 sik sik    36507 Jul 13 17:57 om_mesh_to_dip
-rwxr-xr-x 1 sik sik    31471 Jul 13 17:57 om_minverser
-rwxr-xr-x 1 sik sik 14756335 Jul 13 17:57 om_project_sensors
-rwxr-xr-x 1 sik sik 14786966 Jul 13 17:57 om_register_squids
-rwxr-xr-x 1 sik sik    31529 Jul 13 17:57 om_squids2vtk
-rw-r--r-- 1 sik sik       68 Jul 25 17:57 url

Try it out (now you should be able to finally see this .brainstorm/tmp/openmeeg_gain_eeg.mat file, that was missing all the time)

~/.brainstorm/openmeeg/linux64
❯ LD_LIBRARY_PATH=$PWD OMP_NUM_THREADS=1 ./om_gain -EEG ~/.brainstorm/tmp/openmeeg_hminv.mat ~/.brainstorm/tmp/openmeeg_dsm.mat ~/.brainstorm/tmp/openmeeg_h2em.mat ~/.brainstorm/tmp/openmeeg_gain_eeg.mat
./om_gain version 2.4.0 compiled at Jul 13 2018 15:57:20 using OpenMP
 Executing using 1 threads.


| ------ ./om_gain
| -EEG
| /home/sik/.brainstorm/tmp/openmeeg_hminv.mat
| /home/sik/.brainstorm/tmp/openmeeg_dsm.mat
| /home/sik/.brainstorm/tmp/openmeeg_h2em.mat
| /home/sik/.brainstorm/tmp/openmeeg_gain_eeg.mat
| -----------------------
Using variable with name: matrix
-------------------------------------------
| Elapsed Time: 18.7218 s.
-------------------------------------------

Ok, I will try this. But meanwhile, there is apparently another bug in OpenMEEG which is causing my head models being calculated in macOS to be incorrect, even though they were finishing. We are talking about it here:

I have been asked to report this. Would this be easy for you to report and/or diagnose, or should I go to the Github page and make an issue?

Thanks,
Karl

Sure. Be my gest, the more issues we get (and fix) the more reliable the software would be.

you can also find us here https://gitter.im/openmeeg/openmeeg

just to close this thread.

@lerud you should now be able to update brainstorm and get an updated version (2.4.1) of OpenMEEG that should work out of the box.

Alex

Thank you. OpenMEEG is now producing a gain matrix with no zeros on both Linux and macOS.