6605
Comment:
|
11467
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
= Tutorial 18: Statistics = | = Tutorial 25: Statistics = '''[TUTORIAL UNDER DEVELOPMENT: NOT READY FOR PUBLIC USE] ''' |
Line 3: | Line 5: |
Until now we have been computing measures of the brain activity in time or time-frequency domain. We were able to observe clear effects or slight tendencies, but what we were observing was always dependent on an arbitrary amplitude threshold or on the configuration of the colormap. With appropriate statistical tests, we can go beyond these empirical observations and assess what are the significant effects in a more formal way. We are typically interested in comparing different groups of samples. We want to know what is significantly different in the brain responses of two experimental conditions or two groups of subjects. So we will be essentially estimating differences and testing if these differences are significantly different from zero. |
|
Line 6: | Line 12: |
= From CTF = This tutorial introduces the ''Process2 ''tab, which is mainly used to test two different sets of data. This tab works the same way as the ''Process1 ''tab, except that you have to define two sets of files instead of one. This tool will be presented very briefly now, and the results will not have any scientific meaning, because the protocol '' TutorialCTF'' do not contain enough data to perform any interesting statistical test. |
== Difference deviant-standard == In this auditory oddball experiment, we can test for the significant differences between the brain response to the deviant beeps and the standard beeps, time sample by time sample. |
Line 9: | Line 15: |
== Binary processes == The first type of processes you can find in the Process2 panel are binary file-by-file operations. You define a list of files A and a list of files B, with the same number of files in both lists, and you run a process that is going to create one output file for each couple of files (Ai, Bi). The only two process of that type that are available now are: a difference and an average. Let's try the difference, just for illustrating the interface. |
Before running complicated statistical tests that will take weeks of computation, you can start by checking what the difference of the average responses looks like. If in this difference you observe obvious effects that are clearly not what you are expecting, it's not worth moving forward with finer analysis: either the data is not clean enough or your initial hypothesis is wrong. |
Line 12: | Line 17: |
==== Files selection ==== Drag and drop the ''Right ''condition in "Files A" list, and the ''Left ''condition in "Files B" list. Select the "recordings" button. Click on Run. |
We are going to use the Process2 tab, at the bottom of the Brainstorm figure. It works exactly like the Process1 tab but with two lists of input files, referred to as FilesA (left) and FilesB (right). |
Line 15: | Line 19: |
. {{http://neuroimage.usc.edu/brainstorm/Tutorials/TutStat?action=AttachFile&do=get&target=files_sel.gif|files_sel.gif|class="attachment"}} | * In Process2, drag and drop the '''non-normalized deviant average''' on the left (FilesA) and the non-normalized '''standard average''' on the right (FilesB). |
Line 17: | Line 21: |
==== Process selection ==== Select process "Other > Difference A - B" to compute a "file by file" difference between files A and files B. Click on Ok to run the process. |
* Run the process "'''Difference > Difference A-B'''".<<BR>>Select the option "'''Use absolute values'''", which will convert the unconstrained source maps (three dipoles at each cortex location) into a flat cortical map by taking the norm of the three dipole orientations before computing the difference.<<BR>><<BR>> {{attachment:diff_process.gif||height="311",width="478"}} |
Line 20: | Line 23: |
. {{http://neuroimage.usc.edu/brainstorm/Tutorials/TutStat?action=AttachFile&do=get&target=diff.gif|diff.gif|class="attachment"}} | * Rename the new file in "Deviant-Standard", double-click on it and explore it in time. <<BR>><<BR>> {{attachment:diff_contact.gif||height="307",width="638"}} * We are looking at the difference (Deviant-Standard) so positive/red regions indicate higher activity levels for the deviant beeps, and negative/blue regions higher activity for the standard beeps. * '''Before 50ms''': The motor activity in the deviant is probably due to the previous stims. * '''P50''': Maybe a stronger response in the primary auditory cortex for the standard condition. * '''MMN '''(125ms): Stronger response for the deviant (left temporal, inferior frontal). * '''150ms''': Stronger response in the auditory system for the standard condition. * '''175ms''': Stronger response in the motor regions for the standard condition (motor inihibition). * '''After 200ms''': Stronger response in the deviant condition. |
Line 22: | Line 32: |
==== Results ==== There where 2 x 2 files in input, there are 2 files in output, in a newly created condition "Right - Left". |
== Difference of means == Another process can compute the average and the difference at the same time. We are going to compute the difference of all the trials from both runs at the sensor level. This is usually not recommended because the subject might have moved between the runs. Averaging the recordings across runs is not accurate but can give a good first approximation, in order to make sure we are on the right tracks. |
Line 25: | Line 35: |
. {{http://neuroimage.usc.edu/brainstorm/Tutorials/TutStat?action=AttachFile&do=get&target=diff_result.gif|diff_result.gif|class="attachment"}} | * In Process2, select all the '''deviant trials''' (Files A) and all the '''standard trials''' (Files B). * Run the process "'''Test > Difference of means'''", select the option "'''Arithmetic average'''". <<BR>><<BR>> {{attachment:diff_mean_process.gif||height="341",width="419"}} * Rename the new file in "Deviant-Standard", double-click on it to display it. The difference deviant-standard does not show anymore the early responses (P50, P100) but emphasizes the difference in the later process (MMN/P200 and P300). <<BR>><<BR>> {{attachment:diff_mean_ts.gif||height="164",width="608"}} |
Line 27: | Line 39: |
Now delete this condition before going any further. | == Parametric Student's t-test == Using a t-test instead of the difference of the two averages, we can reproduce similar results but with a significance level attached to each value. |
Line 29: | Line 42: |
== Warning: Mixing recordings from different runs == We should not have done this subtraction because it combines recordings that were acquired in two different runs, with different head positions. The runs should have been registered together to have the same head position. |
* In the Process2 tab, select the following files: * Files A: All the deviant trials, with the '''[Process sources]''' button selected. * Files B: All the standard trials, with the '''[Process sources]''' button selected. * Run the process "'''Test > Student's t-test'''", Equal variance, '''Absolute value of average'''.<<BR>>We have to select this last option if we want to do the test on the norm of the three dipole orientations for unconstrained sources. The second option (absolute value of average) is better indicated for constrained source maps, but here it would return one value per orientation, which is then difficult to interpet. <<BR>><<BR>> {{attachment:ttest_process.gif||height="516",width="477"}} * Double-click on it for displaying it. With the new tab "Stat" you can control the p-value threshold and the correction you want to apply for multiple comparisons. <<BR>><<BR>> {{attachment:ttest_file.gif||height="243",width="492"}} * Set the options in the Stat tab: p-value threshold: '''0.05''', Multiple comparisons: '''Uncorrected'''. <<BR>>What we see in this figure are the t-values corresponding to p-values under the threshold. We can make similar observations than with the difference of means, but without the arbitrary amplitude threshold (this slider is now disabled in the Surface tab). If at a given time point a vertex is red in this view, the mean of the deviant condition is significantly higher than the standard conditions (p<0.05).<<BR>><<BR>> {{attachment:ttest_contact.gif||height="303",width="512"}} * This approach considers each time sample and each surface vertex separately. This means that we have done Nvertices*Ntime = 15000*361 = 5415000 t-tests. The threshold at p<0.05 controls correctly for false positives at one point but not for the entire cortex. We need to correct the p-values for '''multiple comparisons'''. The logic of two types of corrections available in the Stat tab (FDR and Bonferroni) is explained in [[http://scan.oxfordjournals.org/content/4/4/417.full|Bennett et al (2009)]]. * Select the correction for multiple comparison "'''False discovery rate (FDR)'''". You will see that a lot less elementrs survive this new threshold. In the Matlab command window, you can see the average corrected p-value, that replace for each vertex the original p-threshold (0.05):<<BR>>BST> Average corrected p-threshold: 0.000315138 (FDR, Ntests=5415722) * From the Scout tab, you can also plot the scouts time series and get in this way a summary of what is happening in your regions of interest. Positive peaks indicate the latencies when '''at least one vertex''' of the scout has a value that is significantly higher in the deviant condition. The values that are shown are the averaged t-values in the scout. <<BR>><<BR>> {{attachment:ttest_scouts.gif||height="169",width="620"}} |
Line 32: | Line 52: |
To try minimizing the localization errors due to the different head positions, Brainstorm created a channel file in the new condition by averaging the positions from the channel files of the two initial condition. To observe that: select the three channel files in the tree, right-click on one, select the ''Display sensors'' menu. Red=Left, Green=Right, Blue=Right-Left (intermediate positions). | == FieldTrip: Non-parametric cluster-based statistic [TODO] == We have the possibility to call some of the FieldTrip functions from the Brainstorm environment. For this, you need first to [[http://www.fieldtriptoolbox.org/download|install the FieldTrip toolbox]] on your computer and [[http://www.fieldtriptoolbox.org/faq/should_i_add_fieldtrip_with_all_subdirectories_to_my_matlab_path|add it to your Matlab path]]. |
Line 34: | Line 55: |
To avoid the problems related with the position of the head, two options: | For an introduction to non-parametric cluster-based statistics in FieldTrip, watch this video: |
Line 36: | Line 57: |
* interpolate the recordings of both conditions on the same head position right after you import them using the process '''Standardize > Co-register MEG runs''', * or work in '''source space only'''. |
<<HTML(<iframe width="680" height="382" src="https://www.youtube.com/embed/vOSfabsDUNg" frameborder="0" allowfullscreen></iframe>)>> |
Line 39: | Line 59: |
. {{http://neuroimage.usc.edu/brainstorm/Tutorials/TutStat?action=AttachFile&do=get&target=sel_channels.gif|sel_channels.gif|class="attachment"}} {{http://neuroimage.usc.edu/brainstorm/Tutorials/TutStat?action=AttachFile&do=get&target=view_channels.gif|view_channels.gif|class="attachment"}} | Permuation-based non-parametric statistics are more flexible and do not require to do any assumption on the distribution of the data, but on the other hand they are a lot more complicated to process. Calling FieldTrip's function ft_sourcestatistics requires a lot more memory because all the data has to be loaded at once, and a lot more computation time because the same test is repeated many times. |
Line 41: | Line 61: |
== Tests == We do not have enough data to illustrate those tests, it will be done soon in another tutorial, based on single trials instead of average recordings. Its works exactly the same way with many files or many subjects: put all the files from condition A in the first list, and all the files from condition B in second list, no matter if they belong to the same subject or not. Then select your process. |
Running this function in the same way as the parametric t-test previously (full cortex, all the trials and all the time points) would require 45000*461*361*8/1024^3 = '''58 Gb of memory''' just to load the data. This is impossible on most computers, we have to give up at least one dimension (running the test only for one time sample, or one region of interest). |
Line 44: | Line 63: |
In the multiple subjects case, the exploration mode "''Functional data (by condition)''" is usually useful to select a given condition for all the subjects. To switch to this mode, click on the third button on top of the database explorer in the main window. | We will run this FieldTrip function first on the scouts time series and then on a short time window. |
Line 46: | Line 65: |
=== Difference of mean === Calculates separately the average of files A and the average of files B, and the do the subtraction average(A) - average(B). |
* Keep the same selection in Process2: all the deviant trials in FilesA, all the standard trials in FilesB. * Run process: '''Test > FieldTrip: ft_sourcestatistics''', select the options as illustrated below.<<BR>><<BR>>{{attachment:ft_process_scouts.gif}} * Double-click on the new file to display it: <<BR>><<BR>> |
Line 49: | Line 69: |
* It gives cortical maps that are similar to the t-tests (following processes), but without the measure of significance of the results. * If there is only one file in each set, it's going to perform the same operation as "Other > Difference A-B" * In the case of sources, there is an extra option to define what type of "Average" function you want to use. |
Description of the process options: |
Line 53: | Line 71: |
. {{http://neuroimage.usc.edu/brainstorm/Tutorials/TutStat?action=AttachFile&do=get&target=diff_mean.gif|diff_mean.gif|class="attachment"}} === Student's t-test === Get the significant differences between two conditions. * '''Within one subject''': Test a set of individual trials of condition A vs. a set of trials of condition B. * '''Across subjects''': Test a set of subjects for condition A vs. a set of subjects for condition B. Each file (Subject #i, Condition X) is an average response computed across many trials.''' ''' * Same thing as for the difference of means: in the case of source files, there is an extra option to define how to compute the average across the trials. . {{http://neuroimage.usc.edu/brainstorm/Tutorials/TutStat?action=AttachFile&do=get&target=ttest.gif|ttest.gif|class="attachment"}} === Student's t-test (paired) === For the two previous processes, the number of files in sets A and B do not have to be the same. This one requires to have the same number of samples on both sides. It creates dynamically a new set of files C, where file Ci = (Ai - Bi), and then tests if this set C is equal to zero. . {{http://neuroimage.usc.edu/brainstorm/Tutorials/TutStat?action=AttachFile&do=get&target=ttest_paired.gif|ttest_paired.gif|class="attachment"}} |
* |
Line 70: | Line 74: |
Brainstorm does not offer a very large variety of statistical tests. However, it is easy to export your source results and process them with a specialized program. Some examples are illustrated in the advanced tutorials: | An alternative to running the statical tests in Brainstorm is to export all the data and compute the tests with an external program (R, Matlab, SPM, etc). Multiple menus exist to export files to external file formats (right-click on a file > File > Export to file). Two tutorials explain to export data specifically to SPM: |
Line 75: | Line 81: |
== On the hard drive == Right click one of the first TF file we computed > File > '''View file contents'''. == References == * Maris E, Oostendveld R, [[http://www.sciencedirect.com/science/article/pii/S0165027007001707|Nonparametric statistical testing of EEG- and MEG-data]] <<BR>>J Neurosci Methods (2007), 164(1):177-90. * Pantazis D, Nichols TE, Baillet S, Leahy RM. [[http://www.sciencedirect.com/science/article/pii/S1053811904005671|A comparison of random field theory and permutation methods for the statistical analysis of MEG data]], Neuroimage (2005), 25(2):383-94. * Bennett CM, Wolford GL, Miller MB, [[http://scan.oxfordjournals.org/content/4/4/417.full|The principled control of false positives in neuroimaging]] <<BR>> Soc Cogn Affect Neurosci (2009), 4(4):417-422. * FieldTrip video: Non-parametric cluster-based statistical testing of MEG/EEG data: <<BR>> https://www.youtube.com/watch?v=vOSfabsDUNg |
|
Line 76: | Line 92: |
Multiple comparisons: http://neuroimage.usc.edu/forums/showthread.php?1297 | * Forum: Multiple comparisons: http://neuroimage.usc.edu/forums/showthread.php?1297 * Forum: Cluster neighborhoods: [[http://neuroimage.usc.edu/forums/showthread.php?2132-Fieldtrip-statistics|http://neuroimage.usc.edu/forums/showthread.php?2132]] == Delete all your experiments == Before moving to the next tutorial, '''delete '''all the statistic results you computed in this tutorial. It will make it the database structure less confusing for the following tutorials. <<HTML(<!-- END-PAGE -->)>> |
Tutorial 25: Statistics
[TUTORIAL UNDER DEVELOPMENT: NOT READY FOR PUBLIC USE]
Authors: Francois Tadel, Dimitrios Pantazis, Elizabeth Bock, Sylvain Baillet
Until now we have been computing measures of the brain activity in time or time-frequency domain. We were able to observe clear effects or slight tendencies, but what we were observing was always dependent on an arbitrary amplitude threshold or on the configuration of the colormap. With appropriate statistical tests, we can go beyond these empirical observations and assess what are the significant effects in a more formal way.
We are typically interested in comparing different groups of samples. We want to know what is significantly different in the brain responses of two experimental conditions or two groups of subjects. So we will be essentially estimating differences and testing if these differences are significantly different from zero.
Contents
Difference deviant-standard
In this auditory oddball experiment, we can test for the significant differences between the brain response to the deviant beeps and the standard beeps, time sample by time sample.
Before running complicated statistical tests that will take weeks of computation, you can start by checking what the difference of the average responses looks like. If in this difference you observe obvious effects that are clearly not what you are expecting, it's not worth moving forward with finer analysis: either the data is not clean enough or your initial hypothesis is wrong.
We are going to use the Process2 tab, at the bottom of the Brainstorm figure. It works exactly like the Process1 tab but with two lists of input files, referred to as FilesA (left) and FilesB (right).
In Process2, drag and drop the non-normalized deviant average on the left (FilesA) and the non-normalized standard average on the right (FilesB).
Run the process "Difference > Difference A-B".
Select the option "Use absolute values", which will convert the unconstrained source maps (three dipoles at each cortex location) into a flat cortical map by taking the norm of the three dipole orientations before computing the difference.
Rename the new file in "Deviant-Standard", double-click on it and explore it in time.
- We are looking at the difference (Deviant-Standard) so positive/red regions indicate higher activity levels for the deviant beeps, and negative/blue regions higher activity for the standard beeps.
Before 50ms: The motor activity in the deviant is probably due to the previous stims.
P50: Maybe a stronger response in the primary auditory cortex for the standard condition.
MMN (125ms): Stronger response for the deviant (left temporal, inferior frontal).
150ms: Stronger response in the auditory system for the standard condition.
175ms: Stronger response in the motor regions for the standard condition (motor inihibition).
After 200ms: Stronger response in the deviant condition.
Difference of means
Another process can compute the average and the difference at the same time. We are going to compute the difference of all the trials from both runs at the sensor level. This is usually not recommended because the subject might have moved between the runs. Averaging the recordings across runs is not accurate but can give a good first approximation, in order to make sure we are on the right tracks.
In Process2, select all the deviant trials (Files A) and all the standard trials (Files B).
Run the process "Test > Difference of means", select the option "Arithmetic average".
Rename the new file in "Deviant-Standard", double-click on it to display it. The difference deviant-standard does not show anymore the early responses (P50, P100) but emphasizes the difference in the later process (MMN/P200 and P300).
Parametric Student's t-test
Using a t-test instead of the difference of the two averages, we can reproduce similar results but with a significance level attached to each value.
- In the Process2 tab, select the following files:
Files A: All the deviant trials, with the [Process sources] button selected.
Files B: All the standard trials, with the [Process sources] button selected.
Run the process "Test > Student's t-test", Equal variance, Absolute value of average.
We have to select this last option if we want to do the test on the norm of the three dipole orientations for unconstrained sources. The second option (absolute value of average) is better indicated for constrained source maps, but here it would return one value per orientation, which is then difficult to interpet.
Double-click on it for displaying it. With the new tab "Stat" you can control the p-value threshold and the correction you want to apply for multiple comparisons.
Set the options in the Stat tab: p-value threshold: 0.05, Multiple comparisons: Uncorrected.
What we see in this figure are the t-values corresponding to p-values under the threshold. We can make similar observations than with the difference of means, but without the arbitrary amplitude threshold (this slider is now disabled in the Surface tab). If at a given time point a vertex is red in this view, the mean of the deviant condition is significantly higher than the standard conditions (p<0.05).
This approach considers each time sample and each surface vertex separately. This means that we have done Nvertices*Ntime = 15000*361 = 5415000 t-tests. The threshold at p<0.05 controls correctly for false positives at one point but not for the entire cortex. We need to correct the p-values for multiple comparisons. The logic of two types of corrections available in the Stat tab (FDR and Bonferroni) is explained in Bennett et al (2009).
Select the correction for multiple comparison "False discovery rate (FDR)". You will see that a lot less elementrs survive this new threshold. In the Matlab command window, you can see the average corrected p-value, that replace for each vertex the original p-threshold (0.05):
BST> Average corrected p-threshold: 0.000315138 (FDR, Ntests=5415722)From the Scout tab, you can also plot the scouts time series and get in this way a summary of what is happening in your regions of interest. Positive peaks indicate the latencies when at least one vertex of the scout has a value that is significantly higher in the deviant condition. The values that are shown are the averaged t-values in the scout.
FieldTrip: Non-parametric cluster-based statistic [TODO]
We have the possibility to call some of the FieldTrip functions from the Brainstorm environment. For this, you need first to install the FieldTrip toolbox on your computer and add it to your Matlab path.
For an introduction to non-parametric cluster-based statistics in FieldTrip, watch this video:
Permuation-based non-parametric statistics are more flexible and do not require to do any assumption on the distribution of the data, but on the other hand they are a lot more complicated to process. Calling FieldTrip's function ft_sourcestatistics requires a lot more memory because all the data has to be loaded at once, and a lot more computation time because the same test is repeated many times.
Running this function in the same way as the parametric t-test previously (full cortex, all the trials and all the time points) would require 45000*461*361*8/1024^3 = 58 Gb of memory just to load the data. This is impossible on most computers, we have to give up at least one dimension (running the test only for one time sample, or one region of interest).
We will run this FieldTrip function first on the scouts time series and then on a short time window.
- Keep the same selection in Process2: all the deviant trials in FilesA, all the standard trials in FilesB.
Run process: Test > FieldTrip: ft_sourcestatistics, select the options as illustrated below.
Double-click on the new file to display it:
Description of the process options:
Export to SPM
An alternative to running the statical tests in Brainstorm is to export all the data and compute the tests with an external program (R, Matlab, SPM, etc). Multiple menus exist to export files to external file formats (right-click on a file > File > Export to file).
Two tutorials explain to export data specifically to SPM:
Export source maps to SPM8 (volume)
Export source maps to SPM12 (surface)
On the hard drive
Right click one of the first TF file we computed > File > View file contents.
References
Maris E, Oostendveld R, Nonparametric statistical testing of EEG- and MEG-data
J Neurosci Methods (2007), 164(1):177-90.Pantazis D, Nichols TE, Baillet S, Leahy RM. A comparison of random field theory and permutation methods for the statistical analysis of MEG data, Neuroimage (2005), 25(2):383-94.
Bennett CM, Wolford GL, Miller MB, The principled control of false positives in neuroimaging
Soc Cogn Affect Neurosci (2009), 4(4):417-422.FieldTrip video: Non-parametric cluster-based statistical testing of MEG/EEG data:
https://www.youtube.com/watch?v=vOSfabsDUNg
Additional discussions on the forum
Forum: Multiple comparisons: http://neuroimage.usc.edu/forums/showthread.php?1297
Forum: Cluster neighborhoods: http://neuroimage.usc.edu/forums/showthread.php?2132
Delete all your experiments
Before moving to the next tutorial, delete all the statistic results you computed in this tutorial. It will make it the database structure less confusing for the following tutorials.