Automatic Resection identification

Bugs
1

Hello,

I just tried the new module for automatirc resection identification. but it seems to be failing:

BST> Emptying temporary directory...
RESEC_ID> pre-op MRI:  sub-pat01/subjectimage_sub-pat01_ses-preop_res-cmp_T1w.mat
RESEC_ID> post-op MRI: sub-pat01/subjectimage_sub-pat01_ses-postop_space-individual_desc-brain_T1w_spm.mat
BST> Adding plugin resection-identification to path: /NAS/home/edelaire/.brainstorm/plugins/resection-identification/bin
RESEC_ID > System call: "/NAS/home/edelaire/.brainstorm/plugins/resection-identification/bin/resection_identification" "/home/edelaire/.brainstorm/tmp/resection_identification_260427_143812/preop.nii" "/home/edelaire/.brainstorm/tmp/resection_identification_260427_143812/postop.nii" "/home/edelaire/.brainstorm/tmp/resection_identification_260427_143812"
Using existing temp directory: /home/edelaire/.brainstorm/tmp/resection_identification_260427_143812
Write permission confirmed for: /home/edelaire/.brainstorm/tmp/resection_identification_260427_143812
PyInstaller will extract to: /home/edelaire/.brainstorm/tmp/resection_identification_260427_143812/_MEIxxxxxx
Starting resection identification analysis...
Traceback (most recent call last):
  File "numpy/_core/__init__.py", line 23, in <module>
  File "pyimod02_importers.py", line 457, in exec_module
  File "numpy/_core/multiarray.py", line 10, in <module>
  File "pyimod02_importers.py", line 457, in exec_module
  File "numpy/_core/overrides.py", line 7, in <module>
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/edelaire/.brainstorm/tmp/resection_identification_260427_143812/_MEI0vQxDs/libstdc++.so.6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "numpy/__init__.py", line 114, in <module>
  File "pyimod02_importers.py", line 457, in exec_module
  File "numpy/__config__.py", line 4, in <module>
  File "pyimod02_importers.py", line 457, in exec_module
  File "numpy/_core/__init__.py", line 49, in <module>
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.10 from "/NAS/home/edelaire/.brainstorm/plugins/resection-identification/bin/resection_identification_core"
  * The NumPy version is: "2.2.6"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/edelaire/.brainstorm/tmp/resection_identification_260427_143812/_MEI0vQxDs/libstdc++.so.6)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "pyi_rth_matplotlib_headless.py", line 3, in <module>
  File "pyimod02_importers.py", line 457, in exec_module
  File "matplotlib/__init__.py", line 161, in <module>
  File "pyimod02_importers.py", line 457, in exec_module
  File "matplotlib/cbook.py", line 24, in <module>
  File "pyimod02_importers.py", line 457, in exec_module
  File "numpy/__init__.py", line 119, in <module>
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.
[PYI-3964483:ERROR] Failed to execute script 'pyi_rth_matplotlib_headless' due to unhandled exception!
2

FYI @chinmay.chinara
@edelaire, which OS are you using?

3

i am using Ubuntu 20.04.6 LTS (Focal Fossa) with matlab 2024b

4

@Raymundo.Cassani I do not have access to a Linux system. Can you test it at your end?

5

@chinmay.chinara, when the binary was built, what was the OS (and versions) that you use to build the binary?

6

@edelaire check this command in your termina, and paste the result:
ldd --version

7

It was built on Windows Subsystem for Linux Ubuntu 22.04.5 LTS (GNU/Linux 6.6.87.2-microsoft-standard-WSL2 x86_64)

8

ldd (Ubuntu GLIBC 2.31-0ubuntu9.18) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

9

i just tried on ubuntu 24.04.4 (LTS), but get the following error:

BST> Adding plugin resection-identification to path: /NAS/home/edelaire/.brainstorm/plugins/resection-identification/bin
RESEC_ID > System call: "/NAS/home/edelaire/.brainstorm/plugins/resection-identification/bin/resection_identification" "/home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/preop.nii" "/home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/postop.nii" "/home/edelaire/.brainstorm/tmp/resection_identification_260427_155234"
Using existing temp directory: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234
Write permission confirmed for: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234
PyInstaller will extract to: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxxxxxx
Starting resection identification analysis...
PyInstaller extracted to: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q
Current temp directory: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234
Using device: cuda
Temporary files will be stored in /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection
Running as PyInstaller bundle, using base path: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q
BrainSuite binaries found at: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q/BrainSuite/bin/linux
1. cost(3,26.875,0.576,1)=54.6137
2. cost(3,26.875,0.576,1)=54.6137
3. cost(3,27.5,0.512,1)=53.6907
4. cost(3,27.5,0.512,1)=53.6907
1. cost(3,26.25,0.608,2)=15.6487
2. cost(3,27.5,0.608,2)=15.6423
3. cost(3,26.875,0.608,2)=15.6407
4. cost(3,26.875,0.608,2)=15.6407
lowest cost(3,26.875,0.608,2)=15.6407
Wrote mask file /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/preop_1mm.mask.nii.gz
Wrote skull-stripped MRI volume /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/preop_1mm.bse.nii.gz
Running as PyInstaller bundle, using base path: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q
BrainSuite binaries found at: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q/BrainSuite/bin/linux
Wrote corrected image /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/preop_1mm.bfc.nii.gz
Running as PyInstaller bundle, using base path: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q
BrainSuite binaries found at: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q/BrainSuite/bin/linux
Wrote tissue label file /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/preop_1mm.pvc.label.nii.gz
Wrote tissue fraction file /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/preop_1mm.pvc.frac.nii.gz
Running as PyInstaller bundle, using base path: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q
BrainSuite binaries found at: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q/BrainSuite/bin/linux
1. cost(3,26.875,0.576,1)=14.3304
2. cost(3,29.6875,0.544,1)=14.1231
3. cost(3,32.3438,0.4,1)=13.9206
4. cost(3,32.3438,0.4,1)=13.9206
1. cost(3,26.875,0.576,2)=15.8367
2. cost(3,29.6875,0.544,2)=14.3745
3. cost(3,32.3438,0.4,2)=13.8729
4. cost(3,32.3438,0.4,2)=13.8729
lowest cost(3,32.3438,0.4,2)=13.8729
Wrote mask file /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/postop_1mm.mask.nii.gz
Wrote skull-stripped MRI volume /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/postop_1mm.bse.nii.gz
Running as PyInstaller bundle, using base path: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q
BrainSuite binaries found at: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q/BrainSuite/bin/linux
Wrote corrected image /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/postop_1mm.bfc.nii.gz
Running as PyInstaller bundle, using base path: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q
BrainSuite binaries found at: /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/_MEIxtGe8Q/BrainSuite/bin/linux
Wrote tissue label file /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/postop_1mm.pvc.label.nii.gz
Wrote tissue fraction file /home/edelaire/.brainstorm/tmp/resection_identification_260427_155234/temp_dir_resection/postop_1mm.pvc.frac.nii.gz
Using device for affine registration: cuda
torch/cuda/__init__.py:235: UserWarning: 
NVIDIA GeForce RTX 5070 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA GeForce RTX 5070 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
Traceback (most recent call last):
  File "standalone_wrapper.py", line 131, in <module>
  File "standalone_wrapper.py", line 121, in main
  File "auto_resection_mask/auto_resection_mask.py", line 11, in auto_resection_mask
  File "autoresec.py", line 821, in delineate_resection_post
  File "aligner.py", line 141, in affine_reg
  File "aligner.py", line 75, in performAffine
  File "monai/transforms/intensity/array.py", line 1417, in __call__
  File "monai/transforms/intensity/array.py", line 1390, in _normalize
  File "monai/transforms/utils_pytorch_numpy_unification.py", line 135, in percentile
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[PYI-3817206:ERROR] Failed to execute script 'standalone_wrapper' due to unhandled exception!
>>

it seems we need to use newer version of pytorch for that specific GPU: NVIDIA GeForce RTX 5070 Ti with CUDA capability sm_120 - PyTorch Forums

10

Ok, one issue solved.

The problems of state of the art HW.

@chinmay.chinara, what do you suggest?

  • Update the binary? Or
  • Run from source code? If so, do we have the instructions?
11

Oh boy. New hardware new issues. I had an RTX 4090 when the binary was built.

  1. I will look into the binary if I can also make it also agnostic to OS versions within Linux. Should be doable.
  2. But the issue with cuda that @edelaire reported I can fix the source code to handle the newer versions but again with future hardwares new issues might come up.
  3. Currently the instructions for running from source code lies in the repo it is pulled from: GitHub - ajoshiusc/auto_resection_mask at brainstorm-plugin · GitHub . While I look into the binaries a quick thing for now would be add a troubleshooting section in the Brainstorm page linking this discussion and add source code instructions by redirecting users to the above repo.
12

Maybe a good time to turn it into a container

jar

:sweat_smile:

2 Likes
13

Indeed. This too is an option.

14

Sure, support is working fine.
There is need of extra documentation

@chinmay.chinara, please check this

1 Like
15

so, on the computer with linux 20.04.6, i rebuild tool using the build_linux script; and it was able to run until i had a out-of-memory error:

Using device for affine registration: cuda
 size of largest connected component is: 220  
Traceback (most recent call last):
  File "standalone_wrapper.py", line 131, in <module>
    main()
  File "standalone_wrapper.py", line 121, in main
    result = auto_resection_mask(
  File "auto_resection_mask/auto_resection_mask.py", line 11, in auto_resection_mask
  File "autoresec.py", line 1062, in delineate_resection_post
    image_movedo = apply_warp(nonlin_reg.ddf[None,], moving[None,], target[None,])
  File "warp_utils.py", line 113, in apply_warp
    grid = ref_grid[None,].to(disp_field) + disp_field
  File "monai/data/meta_tensor.py", line 283, in __torch_function__
  File "torch/_tensor.py", line 1512, in __torch_function__
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 1.95 GiB of which 50.00 MiB is free. Including non-PyTorch memory, this process has 1.60 GiB memory in use. Of the allocated memory 1.45 GiB is allocated by PyTorch, and 40.02 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[PYI-153533:ERROR] Failed to execute script 'standalone_wrapper' due to unhandled exception!

GPU is NVIDIA GeForce GTX 1050

I tried to do the same thing, on the newest hardward (linux 24.4) but was not able to update PyTorch (for some reason it keeps taking the old version)