Error when running Kilosort

Xihuitljh · May 12, 2023, 6:20pm

Hi! I´m trying to use Kilosort for Unsupervised Spike Sorting, however qhen I run it I am getting the following error:

***************************************************************************
** Error: [process_spikesorting_kilosort]  Electrophysiology > Unsupervised Spike Sorting > KiloSort
** Line 26:  gpuDevice (line 26)
** Failed to load graphics driver. Unable to load library 'nvcuda.dll'. The error was:
** No se puede encontrar el m�dulo especificado.

** Update or reinstall your graphics driver. For more information on GPU support, see GPU Support by Release.
** 
** Call stack:
** >gpuDevice.m at 26
** >process_spikesorting_kilosort.m>Run at 271
** >process_spikesorting_kilosort.m at 33
** >bst_process.m>Run at 203
** >bst_process.m at 38
** >panel_process1.m>RunProcess at 124
** >panel_process1.m at 26
** >gui_brainstorm.m>CreateWindow/ProcessRun_Callback at 777
** >bst_call.m at 28
** >gui_brainstorm.m>@(h,ev)bst_call(@ProcessRun_Callback) at 298

It asks me for a library that belongs to the NVIDIA graphic driver, but that graphic driver isn´t compatible with my computer graphic card: ADM Radeon(TM) RX Vega 10 Graphics. My question is, is there an option where I tell Kilosort to use the graphics driver of my graphic card? Instead of Kilosort trying to use NVIDIA, wich it doesn´t exists in my computer?

Info of my hardware and software:
HP Pavilion Laptop 15-cw1xxx
AMD Ryzen 7 3700U with Radeon Vega Mobile Gfx 2.30 GHz
RAM 16GB
Operative System Windows 11 Home Single Language x64

Thanks in advance for your support.

Raymundo.Cassani · May 12, 2023, 7:06pm

Hi @Xihuitljh,

The issue here is that the option: GPU processing was selected when running the spike sorting.
Uncheck that option and run the process.

The GPU processing option is used to accelerate the processing by using the GPU for computations. Graphics will be handle properly with your GPU

Unfortunately, GPU computing in Matlab is only available for NVIDIA GPUs

Best,
Raymundo

Xihuitljh · May 12, 2023, 7:43pm

Hi Raymundo, thanks for your quick response.
I uncheck the option that you told me and ran the process again, but now I have this error:

***************************************************************************
** Error: [process_spikesorting_kilosort]  Electrophysiology > Unsupervised Spike Sorting > KiloSort
** Error with Kilosort while training on the CPU
** File: E220919/@rawE220919R19/data_0raw_E220919R19.mat

I hope you could help me with this one.

Raymundo.Cassani · May 12, 2023, 8:26pm

This is error is produced by the kilosort software itself, more specifically with their function fitTemplates() at line 276 in process_spikesorting_kilosort.

Were you able to run unsupervised spike sorting with kilosort on the tutorial data?
https://neuroimage.usc.edu/brainstorm/e-phys/Introduction

Xihuitljh · May 15, 2023, 8:12pm

Yes, I ran the same process with the tutorial data and it worked fine. This the command window of Matlab for the unsupervised spike sorting with kilosort:

BST> Processing dependencies: kilosort requires: kilosort-wrapper phy npy-matlab 
BST> Plugin kilosort already loaded: C:\Users\xihui\.brainstorm\plugins\kilosort
Time   0s. Loading raw data... 
Time   8s. Channel-whitening filters computed. 
Time   8s. Loading raw data and applying filters... 
Time 15.30. Whitened data written to disk... 
Time 15.30. Preprocessing complete!
Time  15s. Optimizing templates ...
Time 787.82, batch 2581/2592, mu 10.12, neg-err 38.628447, NTOT 65215, n100 0, n200 0, n300 0, n400 0
Time 791s. Running the final template matching pass...
Time 946.86, batch 401/432,  NTOT 71492

Loading data for spike group 1. 
Elapsed time is 0.026020 seconds.

Saving .clu files to disk (cluster indexes)
Saving .clu file for group 1. 
Elapsed time is 0.108902 seconds.

Saving .res files to disk (spike times)
Saving .res file for group 1. 
Elapsed time is 0.179717 seconds.

Extracting waveforms
Warning: Binary file does not exist:
C:\Users\xihui\.brainstorm\tmp\Unsupervised_Spike_Sorting\Tutorial_e-Phys\Floyd\@rawytu288c-01\data_0raw_ytu288c-01.mat\raw_data_no_header_ytu288c-01.dat 
> In Kilosort2Neurosuite/Kilosort_ExtractWaveforms (line 154)
In Kilosort2Neurosuite (line 80)
In process_spikesorting_kilosort>Run (line 304)
In process_spikesorting_kilosort (line 33)
In bst_process>Run (line 203)
In bst_process (line 38)
In panel_process1>RunProcess (line 124)
In panel_process1 (line 26)
In gui_brainstorm>CreateWindow/ProcessRun_Callback (line 777)
In bst_call (line 28)
In gui_brainstorm>@(h,ev)bst_call(@ProcessRun_Callback) (line 298) 
Loading xml for probe layout from root folder
Extraction of waveforms begun 
100 percent complete
Elapsed time is 16.853412 seconds.

Saving .spk files to disk (waveforms)
Saving .spk for group 1. 
Elapsed time is 17.934933 seconds.

Computing PCAs
Computing PCAs for group 1. Saving .fet files for group 1. 
Elapsed time is 26.538545 seconds.

Complete!BST> Saving protocol "Tutorial_e-Phys"...

Do you think the problem might be with my file? My file format is: EEG: Matlab matrix (*.mat).

Xihuitljh · May 17, 2023, 3:30am

Hi, I keep trying to use kilosort, now before it shows me the "Error with Kilosort while training on the CPU" it does this:

BST> File converted to binary .bst format: D:\Estancia\BrainStorm y Kilo Sort\brainstorm_db\Experimento_Electrofisiologia\data\E220701\@rawE220701R15_ColVolt\E220701R15_ColVolt.bst
BST> Processing dependencies: kilosort requires: kilosort-wrapper phy npy-matlab 
BST> Plugin kilosort is outdated and will be updated.
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\CUDA
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\Docs
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\configFiles
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\driftCorrection
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\eMouse
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\finalPass
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\initialize
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\mainLoop
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\mergesplits
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\preProcess
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\tests
BST> Removing plugin kilosort from path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\utils
BST> Deleting plugin kilosort: C:\Users\xihui\.brainstorm\plugins\kilosort
BST> Downloading URL : https://github.com/cortex-lab/KiloSort/archive/refs/heads/master.zip
BST> Saving to file  : C:\Users\xihui\.brainstorm\plugins\kilosort\plugin.zip
BST> Plugin kilosort-wrapper already loaded: C:\Users\xihui\.brainstorm\plugins\kilosort-wrapper
BST> Plugin phy already loaded: C:\Users\xihui\.brainstorm\plugins\phy
BST> Plugin npy-matlab already loaded: C:\Users\xihui\.brainstorm\plugins\npy-matlab
BST> Adding plugin kilosort to path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master
BST> Adding plugin kilosort to path: C:\Users\xihui\.brainstorm\plugins\kilosort\KiloSort-master\*
BST> Executing callback InstalledFcn: process_spikesorting_kilosort('copyKilosortConfig', bst_fullfile(bst_get('UserPluginsDir'), 'kilosort', 'KiloSort-master', 'configFiles', 'StandardConfig_MOVEME.m'), bst_fullfile(bst_get('UserPluginsDir'), 'kilosort', 'KiloSort-master', 'KilosortStandardConfig.m'));
Time   0s. Loading raw data... 
Time   1s. Channel-whitening filters computed. 
Time   1s. Loading raw data and applying filters... 
Time 0.76. Whitened data written to disk... 
Time 0.76. Preprocessing complete!

***************************************************************************
** Error: [process_spikesorting_kilosort]  Electrophysiology > Unsupervised Spike Sorting > KiloSort
** Error with Kilosort while training on the CPU
** File: E220701/@rawE220701R15_ColVolt/data_0raw_E220701R15_ColVolt.mat
** 
***************************************************************************

That doesn´t happen when I ran it with the tutorial data.

Raymundo.Cassani · May 17, 2023, 2:31pm

Hey, the first part of the message only just informs that the kilosort plugin was automatically updated in Brainstorm.

The error is produced in the fitTemplates.m from KiloSort plugin, which is called in line 276 in process_spikesorting_kilosort.m

To have a more insightful error message:

Could you modify this file:
brainstorm3/toolbox/process/functions/ process_spikesorting_kilosort.m
around line 275
(One line is Updated and 2 lines are Inserted)

From:

    try
        rez = fitTemplates(rez, DATA, uproj);  % fit templates iteratively
    catch
        if ops.GPU
            % ~\.brainstorm\plugins\kilosort\KiloSort-master\CUD?\mexGPUall.m
            % needs to be called and compile the .cu files.
            % Suggested environment: Matlab 2018a, CUDA 9.0, VS 13.
            bst_report('Error', sProcess, sInput, 'Error trying to spike-sort on the GPU. Have you set up CUDA correctly? Check https://github.com/cortex-lab/KiloSort for installation instructions');
            return;
        else
            bst_report('Error', sProcess, sInput, 'Error with Kilosort while training on the CPU');
            return;
        end
    end

To:

    try
        rez = fitTemplates(rez, DATA, uproj);  % fit templates iteratively
    catch ME  % UPDATED LINE
        if ops.GPU
            % ~\.brainstorm\plugins\kilosort\KiloSort-master\CUD?\mexGPUall.m
            % needs to be called and compile the .cu files.
            % Suggested environment: Matlab 2018a, CUDA 9.0, VS 13.
            bst_report('Error', sProcess, sInput, 'Error trying to spike-sort on the GPU. Have you set up CUDA correctly? Check https://github.com/cortex-lab/KiloSort for installation instructions');
            disp(['ME.message: ', ME.message]) % INSERTED LINE
            return;
        else
            bst_report('Error', sProcess, sInput, 'Error with Kilosort while training on the CPU');
            disp(['ME.message: ', ME.message]) % INSERTED LINE
            return;
        end
    end

And run again with your data
Let us know the messages you get

P.S. In the reply, you can add ``` (3 backticks) in the line before in and the line after the error message to make it easier to read. For example:

***************************************************************************
** Error: [process_spikesorting_kilosort]  Electrophysiology > Unsupervised Spike Sorting > KiloSort
...

Xihuitljh · May 17, 2023, 3:57pm

Hi, I modified the file and it appeared the same error:

BST> Processing dependencies: kilosort requires: kilosort-wrapper phy npy-matlab 
BST> Plugin kilosort already loaded: C:\Users\xihui\.brainstorm\plugins\kilosort
BST> File already converted to KiloSort input.
Time   0s. Loading raw data... 
Time   0s. Channel-whitening filters computed. 
Time   0s. Loading raw data and applying filters... 
Time 0.29. Whitened data written to disk... 
Time 0.29. Preprocessing complete!
ME.message: Incorrect dimensions for matrix multiplication. Check that the number of columns in the first matrix matches the number of rows in the second matrix. To perform elementwise multiplication, use '.*'.

***************************************************************************
** Error: [process_spikesorting_kilosort]  Electrophysiology > Unsupervised Spike Sorting > KiloSort
** Error with Kilosort while training on the CPU
** File: E220919/@rawE220919R19Col2/data_0raw_E220919R19Col2.mat
** 
***************************************************************************

Raymundo.Cassani · May 17, 2023, 4:22pm

The message was not enough, my bad.
Please update once more the file to, so it rethrows the error:

    try
        rez = fitTemplates(rez, DATA, uproj);  % fit templates iteratively
    catch ME  % UPDATED LINE
        if ops.GPU
            % ~\.brainstorm\plugins\kilosort\KiloSort-master\CUD?\mexGPUall.m
            % needs to be called and compile the .cu files.
            % Suggested environment: Matlab 2018a, CUDA 9.0, VS 13.
            bst_report('Error', sProcess, sInput, 'Error trying to spike-sort on the GPU. Have you set up CUDA correctly? Check https://github.com/cortex-lab/KiloSort for installation instructions');
            rethrow(ME); % INSERTED LINE
            return;
        else
            bst_report('Error', sProcess, sInput, 'Error with Kilosort while training on the CPU');
            rethrow(ME); % INSERTED LINE
            return;
        end
    end

Also, does this error appear for all your files, or for a specific file?

Xihuitljh · May 17, 2023, 4:49pm

This are the messages I got:

And yes, this error happens with all my files.

mian · August 30, 2023, 8:35am

Hi
I get the same error message when running KiloSort. Was this error solved?

Kind regards

Raymundo.Cassani · September 5, 2023, 11:58pm

@Xihuitljh and @mian, the error happens inside a KiloSort function. It may be caused by issues with the data.

Which pre-processing was performed on it?
How many channels are in the data?
Are there repeated channels?
Are there flat channels?

Lastly, could you share a file that is giving the error to replicate on my side?
Thanks,