What is the algorithm of permutation test?

Discussions
1

Dear all,

Out team has been performing source reconstruction statistics with the help of cluster-based non-parametric permutation test. The part of the script is the following:

bst_report('Start', sFiles_proj);

        sFiles = bst_process('CallProcess', 'process_ft_sourcestatistics', sFiles_proj, sFiles2_proj, ...
        'timewindow',     PostStim, ...
        'scoutsel',       {}, ...
        'scoutfunc',      2, ...  % PCA
        'isabs',          0, ...
        'avgtime',        0, ...
        'randomizations', 2000, ...
        'statistictype',  2, ...  % Paired t-test
        'tail',           'two', ...  % Two-tailed
        'correctiontype', 2, ...  % cluster
        'minnbchan',      0, ...
        'clusteralpha',   0.05);

However, every time we run it (either 1000 or 2000 permutations), we get different significant clusters. The p-value is far below 0.05. How is it possible that the get different results after the very same test on the same data? What is the exact algorythm of this test?

2

This uses a random set of permutations. Each execution will lead to different results.
In Brainstorm, this random selection of permutations is done in bst_permtest.m. I'm not sure where this happens in the FieldTrip functions you are using.

If you want to get the exact same values in output, you need to initialize the random generator in Matlab with a fixed value (e.g. see the rng function).

Anything that is not shown as significant across multiple executions should be questioned.

1 Like