How the Corr matix is saved?

thismoment · November 23, 2014, 10:33pm

I calculate the correlation matrix by the way NxN.
when I check the saved file content, I did not find a N*N matrix. instead of it, I find a variable TF , stored M * 1 matrix.

My question,

how you experts stored the correlation matrix? In which way?
where the pvalue associated with Correlation matrix is stored?
How the cohr and other connectivity metris store the results.
I tried to do further analysis , so I ask such question.
In fact, your software is user friendly and almost perfect.

Thank you again.

Best wishes,
Junpeng

Francois · November 24, 2014, 7:43am

Hi Junpeng,

The connectivity matrices are compressed before they are saved. The coherence, correlation and phase-locking value are symmetric measures (AxB=BxA), so we don’t have to save twice all the values.

To get the connectivity matrix correctly formatted, you can do the following:

Read the file timefreq_connect…mat into the variable TfMat, or right-click > File > Export to Matlab > “TfMat”
R = bst_memory(‘GetConnectMatrix’, TfMat);
The matrix R is the connectivity matrix [signalsA x signalsB x Frequencies]
For PLV and coherence: he frequency bins are described in the TfMat.Freqs: R(:,:,1) is frequency TfMat.Freqs(1)

Alternatively, you can use the function export_timefreq.m: right-click on a connectivity file > File > Export to file > Save in an ASCII format (simple text file)

The p-values associated with the correlation measure are not saved. The threshold is asked as an input parameter and the output matrices are already thresholded (connectivity value set to zero).

Cheers,
Francois

thismoment · November 24, 2014, 8:00am

Thank you. I got it!
Another question,
Assuming the imaging kernel (K) is sensors x the number of sources x 3, then the lead field matrix of the source 1 is K(:,1, for three orientation, or others?
What I understand is right?

Francois · November 24, 2014, 8:25am

Please refer to this thread:
http://neuroimage.usc.edu/forums/showthread.php?918

thismoment · November 24, 2014, 7:15pm

Hi Great coder,
I used
R = bst_memory(‘GetConnectMatrix’, TfMat);
successfully.
However, after I change R value, how to save back the R.
using R = bst_memory(‘SaveConnectMatrix’, TfMat)?

Thank you very much for your answer.

Cheers,
Junpeng

Francois · November 25, 2014, 5:40am

You can compress the matrix again using:
TfMat.TF = process_compress_sym(‘Compress’, R(:));

Or if you want to keep the full matrix:
TfMat.TF = R(:);
TfMat.Options.isSymmetric = 0;

Francois

thismoment · June 13, 2016, 7:25pm

Hi experts,
I tried to use the command
R = bst_memory(‘GetConnectMatrix’, tf_plv);

The following error appeared,

** Error: Line 1613: Undefined variable “TimefreqMat” or class “TimefreqMat.RowNames”.
**
** Call stack:
** >bst_memory.m>GetConnectMatrix at 1613
** >bst_call.m at 26
** >macro_methodcall.m at 37
** >bst_memory.m at 71
**

Would you like to solve it or tell us how to read the tfmat in other ways?
The version of brainstorm is 7 Jun, 2016.
Thank you very much.

Best wishes,
Junpeng

Francois · June 14, 2016, 8:43am

Hi Junpeng,

I added this bug last week when updating some other part of the code…
Thanks for reporting it!
It’s fixed, please update Brainstorm and try again.

Francois

thismoment · November 1, 2016, 6:29pm

Hi Expert,

The correlation NxN matrix means the matrix calculated by matlab function cov ? or means correlation coefficient matrix?

How to explain the detailed info about the correlation NxN matrix in our paper?

Best wishes,
Junpeng

Francois · November 2, 2016, 7:19am

Hi Junpeng,

In the processes “Correlation 1xN”, “Correlation NxN” and “Correlation AxB”, the measure that is used is the same.
In the NxN case, it just repeats it for all the pairs of signals, there is no covariance of the other signals involved.
The results are the same as the non-diagonal values returned by the corrcoef function with only two signals in input.

The full code of the function and some additional explanations are available here:

github.com

brainstorm-tools/brainstorm3/blob/master/toolbox/connectivity/bst_corrn.m

function [R, pValues] = bst_corrn(X, Y, RemoveMean)
% BST_CORRN: Calculates the same correlation coefficients as Matlab function corrcoef (+/- rounding errors), but in a vectorized way
%            Equivalent to bst_correlation with nDelay=1 and maxDelay=0 
%
% INPUTS:
%    - X: [Nx,Nt], Nx signals varying in time
%    - Y: [Ny,Nt], Ny signals varying in time
%    - RemoveMean: If 1, removes the average of the signal before calculating the correlation
%                  If 0, computes a scalar product instead of a correlation
%
% NOTE: The rounding errors
%    Corrcoef computes the correlation coefficients based on the variance values computed with cov(),
%    instead of a direct sum of the squared values (sum(Xc.^2,2)).
%    Hence it uses a corrected algorithm for the computation of the variance, that is not sensible to
%    the rounding errors for large number of time samples. We do not divide the values by the number 
%    of samples here, so if the two signals are the same range of dynamics, those rounding errors
%    should not be a problem, even for a very large number of time samples.

% @=============================================================================
% This function is part of the Brainstorm software:

This file has been truncated. show original

Cheers,
Francois

thismoment · November 2, 2016, 7:01pm

Hi Francois,

I got it, thank you.
In fact, the function calculate the correlation coefficient between each pairs of channels.

Best wishes,
Junpeng

[QUOTE=Francois;11245]Hi Junpeng,

In the processes “Correlation 1xN”, “Correlation NxN” and “Correlation AxB”, the measure that is used is the same.
In the NxN case, it just repeats it for all the pairs of signals, there is no covariance of the other signals involved.
The results are the same as the non-diagonal values returned by the corrcoef function with only two signals in input.

The full code of the function and some additional explanations are available here:

github.com

brainstorm-tools/brainstorm3/blob/master/toolbox/connectivity/bst_corrn.m

function [R, pValues] = bst_corrn(X, Y, RemoveMean)
% BST_CORRN: Calculates the same correlation coefficients as Matlab function corrcoef (+/- rounding errors), but in a vectorized way
%            Equivalent to bst_correlation with nDelay=1 and maxDelay=0 
%
% INPUTS:
%    - X: [Nx,Nt], Nx signals varying in time
%    - Y: [Ny,Nt], Ny signals varying in time
%    - RemoveMean: If 1, removes the average of the signal before calculating the correlation
%                  If 0, computes a scalar product instead of a correlation
%
% NOTE: The rounding errors
%    Corrcoef computes the correlation coefficients based on the variance values computed with cov(),
%    instead of a direct sum of the squared values (sum(Xc.^2,2)).
%    Hence it uses a corrected algorithm for the computation of the variance, that is not sensible to
%    the rounding errors for large number of time samples. We do not divide the values by the number 
%    of samples here, so if the two signals are the same range of dynamics, those rounding errors
%    should not be a problem, even for a very large number of time samples.

% @=============================================================================
% This function is part of the Brainstorm software:

This file has been truncated. show original

Cheers,
Francois[/QUOTE]

hasanparvez · February 22, 2017, 4:27am

When I export correlation array file from brainstorm to matlab it is exported as structure and it is in structure.F = 2701x1
i want to convert it to 73x73 in matlab, how to do it?

Francois · February 24, 2017, 11:28am

This is illustrated in previous posts in the same thread:
http://neuroimage.usc.edu/forums/showthread.php?1796-How-the-Corr-matix-is-saved&p=7616#post7616

Other related discussions:
http://neuroimage.usc.edu/forums/showthread.php?1681-PLV-NxN-Read-matrix
http://neuroimage.usc.edu/forums/showthread.php?1556

jfboasen · October 12, 2019, 3:30am

Hi Francois,

I was exporting the maps, modifying them, and reimporting as per your instructions above. Everything was working fine. Then when I went to do it again this evening, something changed and it won't work.

I am now getting an error:
Call stack:
** >bst_memory.m>ReshapeConnectMatrix at 1772
** >bst_memory.m at 72
** >view_connect.m at 124
** >tree_callbacks.m at 380
** >bst_call.m at 28
** >panel_protocols.m>CreatePanel/protocolTreeClicked_Callback at 109

I noticed one peculiar thing that changed when I did: TfMat.TF=R(:);,
That was, the resulting matrix size for TF was larger than it had been when it was working.

I cannot figure out why this is happening. And I think it is difficult for you to figure out. But one workaround maybe is if you can explain how to get the R matrix back into the same shape as the original TF matrix. The extracted R is a 4-D matrix. But the TF matrix is not. In my case the TF matrix was a 1953x1x10 double. Now, after importing the R using the command TfMat.TF=R(:);, the TF matrix is 38440x1 double.

Again, this double matrix shape had worked before, but the size was smaller. I don't know why it has increased.

Any ideas?

Thank you so much.

Jared

Francois · October 13, 2019, 11:54am

Depending on the connectivity measure that is used, the resulting connectivity matrix can be symmetric or not. Symmetric matrix can be compressed, with only half of the values stores (+ the diagonal).

Check the corresponding code if you need to understand what you are not doing correctly:

github.com

brainstorm-tools/brainstorm3/blob/master/toolbox/core/bst_memory.m#L1787


        if isempty(iSel)

            return

        end

        % Select only the required row

        R = R(iSel, :, :, :);

    end

end





%% ===== GET CONNECT MATRIX =====

function R = GetConnectMatrix(Timefreq) %#ok<DEFNU>

    % Expand symmetric matrix

    %if isfield(Timefreq.Options, 'isSymmetric') && Timefreq.Options.isSymmetric

    if (length(Timefreq.RowNames) == length(Timefreq.RefRowNames)) && (size(Timefreq.TF,1) < length(Timefreq.RowNames)^2)

        Timefreq.TF = process_compress_sym('Expand', Timefreq.TF, length(Timefreq.RowNames));

    end

    % Reshape TF matrix: [Nrow x Ncol x Ntime x nFreq]

    nTime = size(Timefreq.TF, 2);

    nFreq = size(Timefreq.TF, 3);

    R = reshape(Timefreq.TF, [length(Timefreq.RefRowNames), length(Timefreq.RowNames), nTime, nFreq]);

end

github.com

brainstorm-tools/brainstorm3/blob/master/toolbox/process/functions/process_compress_sym.m#L67


%% ===== FORMAT COMMENT =====
function Comment = FormatComment(sProcess) %#ok<DEFNU>
    switch(sProcess.options.method.Value)
        case 1,  Comment = 'Expand symmetric matrix';
        case 2,  Comment = 'Compress symmetric matrix';
    end
end




%% ===== RUN =====
function OutputFile = Run(sProcess, sInput) %#ok<DEFNU>
    OutputFile = sInput.FileName;
    % Load TF file
    TimefreqMat = in_bst_timefreq(sInput.FileName, 0);
    % Get options
    switch(sProcess.options.method.Value)
        % EXPAND
        case 1
            % Check file status
            if ~TimefreqMat.Options.isSymmetric
                bst_report('Warning', sProcess, sInput, 'File is already expanded.');

jfboasen · October 14, 2019, 12:49am

Hi Francois,

Awesome! Thank you! I will study this.

In the meantime, based on the other information in this thread, I was able to understand how the extracted 2-D TF matrix is arranged in the original single column, multidimensional TF matrix. I was then able to build a dummy file for a single dimension in Excel and enter values such that only the values I want to show will show, and I can display them in different colors by adjusting the color bar according to the directionality of the statistical comparison that was made. By avoiding even using the code, "get connectivity matrix" I avoid changing its shape, and can simply replace the TF matrix with the dummy one that I have made.

It's a sketchy workaround, and was a little tedious, but got the job done without me having troubleshoot code (which is not one of my strong points...yet).

Anyway, thank you so much again!

Jared

Francois · January 22, 2022, 11:53am

The storage of connectivity matrices is now documented here: https://neuroimage.usc.edu/brainstorm/Tutorials/Connectivity#On_the_hard_drive