Is eye tracker data support interesting to Brainstorm?

MartinFR · October 2, 2015, 6:57am

Hi Francois,

we are using the EyeLink (SR Research, Canada) to record eye movements and detect saccades and blinks in our EEG experiments. For us, it would certainly be nice to be able to include such data into the brainstorm environment (we are thinking about moving the whole analysis pipeline to brainstorm because it’s so nice, and also easy to include our own functions).

Now I’m not really sure if it is in your interest to include such data, as it is not really directly related to the main purpose of the toolbox; it could, however, be used nicely to control for saccades, microsaccades and blink artifacts.

In case you are interested, the following could help in implementing it:

There is some documentation from fieldtrip, how they use this kind of data: http://www.fieldtriptoolbox.org/getting_started/eyelink
A program that converts the eyelink data format (they called it .EDF, but it’s not the European Data Format, but the Eyelink Data Format…) into ascii, as mentioned in the fieldtrip tutorial, is attached here.
An example of an eyelink .edf: https://www.dropbox.com/s/8j5qof2c5kow4lz/eyelinkEDFexample.edf?dl=1
A program that converts the .edf into .mat and is able to extract events (this is what we use so far): http://kobi.nat.uni-magdeburg.de/edfImport

We are always sending some event triggers to the eyelink and to the EEG amplifier at the same time, so that we can synchronize them later in matlab.

However, I would of course understand if you don’t want to include eye tracking data into the toolbox.

Best,
Martin

Francois · October 2, 2015, 11:19am

Hi Martin,

It would be interesting to implement a way to get access to the Eyelink files in Brainstorm. It’s a widely used system and having its output in Brainstorm could be useful for various purposes.
However I have very limited resources at the moment to deal with this, so I would need help from you or your collaborators to implement this.
There are a few issues to deal with first, starting from legal considerations. To include anything in Brainstorm, we need it to be compatible with a GPL license.

This is not an open file format, and the EDF API doesn’t look like open software.
We probably don’t have the right to distribute this (neither as part of the Brainstorm forum, nor as a .zip file posted on this forum).
Could you contact the company and ask them what the actual license is for this library? What are we allowed to do with it, are we allowed to redistribute it, and if not what is the correct distribution path for it?
The license for the edfImport library is not explicit either, so we need to ask the author the same questions: license and redistribution rights.
It looks like the code comes from a German university, so there is a higher chance to have this solved easily.
We can note that the author didn’t include the “edfapi.dll” in the package so maybe he doesn’t have the right to.
Once the integration and distribution questions are solved, we have to deal with the technical ones.
Using the call "edfImport(‘file.edf’, [1 1 1]) returns tons of data. How would you like this to be available in Brainstorm?

Would you like all the values to be accessible as continuous signals?
If this is the case, there might be sampling issues: is the time sampled regularly? Brainstorm cannot handle irregular time sampling or recordings with gaps.
What is the difference between “Events” and “Samples” (sorry, I’ve never been using an eye tracking device myself, I’ll a lot of guidance)?
What is the meaning of all these fields?
In the, what is the final goal of having this data available in Brainstorm? Can you provide examples?

The last problem will be the synchronization with MEG/EEG data. If you want the eye tracker data displayed simultaneously with other signals coming from other acquisition devices, we need to solve two issues:

the sampling rate (Brainstorm can display signals simultaneously only if they have exactly the same time definition)
the synchronization itself (how do you match the start points after resampling)
=> This will require the development of a separate process, which might be a bit difficult to write properly (but doable with some resource).

I think all this is doable, and will happen a lot faster if you are willing to contribute to the development.

Cheers,
Francois

MartinFR · October 5, 2015, 5:34am

Hi Francois!

[QUOTE=Francois;9384]Hi Martin,

It would be interesting to implement a way to get access to the Eyelink files in Brainstorm. It’s a widely used system and having its output in Brainstorm could be useful for various purposes.
However I have very limited resources at the moment to deal with this, so I would need help from you or your collaborators to implement this.
There are a few issues to deal with first, starting from legal considerations. To include anything in Brainstorm, we need it to be compatible with a GPL license.
[/QUOTE]

Sounds good, we are of course happy to help in any way we can!

[QUOTE=Francois;9384]

This is not an open file format, and the EDF API doesn’t look like open software.
We probably don’t have the right to distribute this (neither as part of the Brainstorm forum, nor as a .zip file posted on this forum).
Could you contact the company and ask them what the actual license is for this library? What are we allowed to do with it, are we allowed to redistribute it, and if not what is the correct distribution path for it?
The license for the edfImport library is not explicit either, so we need to ask the author the same questions: license and redistribution rights.
It looks like the code comes from a German university, so there is a higher chance to have this solved easily.
We can note that the author didn’t include the “edfapi.dll” in the package so maybe he doesn’t have the right to.
[/QUOTE]

I will contact them and let you know of the answer.

[QUOTE=Francois;9384]
3) Once the integration and distribution questions are solved, we have to deal with the technical ones.
Using the call "edfImport(‘file.edf’, [1 1 1]) returns tons of data. How would you like this to be available in Brainstorm?

Would you like all the values to be accessible as continuous signals?
If this is the case, there might be sampling issues: is the time sampled regularly? Brainstorm cannot handle irregular time sampling or recordings with gaps.
What is the difference between “Events” and “Samples” (sorry, I’ve never been using an eye tracking device myself, I’ll a lot of guidance)?
What is the meaning of all these fields?
In the, what is the final goal of having this data available in Brainstorm? Can you provide examples?
[/QUOTE]

Running edfImport, you will get two variables: “Preamble”, which has a bit of text with infos about date and hardware, and “Trials”, which is an (1*X) struct, where X is the number of “trials” we recorded. The definition of such a trial is up to the user who records the eyetracker data. For example, we start the eyelink “trial” with each recording session, and stop it when we also stop the EEG recording; thus, an eyelink “trial” for us would be around 10-15 minutes of recording and thus not correspond with the trials of the actual EEG experiment.

The “Trials” struct includes 3 different fields:
[ul]
[li]“Header” with some general info and also start- and end-time
[/li][li]“Samples”. In there, all the raw, continuous data (sampled regularly) is saved. Including such data as gaze position, pupil size, and a lot of other infos (see data field description link below)
[/li][li] “Events”. Here, no continuous data is stored. Instead, the data is only stored at certain events, for example blinks, saccades, fixations or, most importantly, message triggers sent by the user. In Trials(X).Events.message, the event names which were sent from the user are saved. These are the same as event triggers and EEG and can be used to synchronize the data.
[/li][/ul]

You can run the function “edfExtractInterestingEvents” from the “edfImport”-library as following:
Trials = edfExtractInterestingEvents(Trials, ‘^TRIALID’)

Then, you will get additional fields for fixation, blinks and saccades events. Also a field for button presses, which in our case would be empty because we did not use this function. With “edfExtractMicrosaccades”, you can also extract microsaccades, if the data was recorded binocularely.

I think the Events and Blinks fields are the easiest to start with. After synchronizing it with the EEG, the user could very easily exclude trials with blinks or too many eye movements.

[B]Description of data fields: http://download.sr-support.com/dispdoc/page9.html [/B]
… I don’t know If we actually need all those fields.

At the moment, we have two main goals using the eytracker data:

[ul]
[li] Discarding trials with blinks or too much eye movement
[/li][li] Epoching EEG data based on saccade and microsaccade events and looking for possible neuronal sources
[/li][/ul]

One could also for example check if a subject acutally fixated on the spot (s)he should, and if not, exclude the trial. Or examine attention-based neuronal effects, by extracting from the eye movements what attracted the subject’s attention, for example if you present different stimuli at the same time.

Looking at continuous eye movements is of course also interesting, but I think the sorting out of ocular artifacts is the most important thing for the beginning.

[QUOTE=Francois;9384]
4) The last problem will be the synchronization with MEG/EEG data. If you want the eye tracker data displayed simultaneously with other signals coming from other acquisition devices, we need to solve two issues:

the sampling rate (Brainstorm can display signals simultaneously only if they have exactly the same time definition)
the synchronization itself (how do you match the start points after resampling)
=> This will require the development of a separate process, which might be a bit difficult to write properly (but doable with some resource).
[/QUOTE]

Yes, I also think that might need quite some programming to get this done properly.

The sampling rate is normally 500 Hz for binocular recording and 1000 for monocular recording.

For the synchronization itself, we are sending certain trigger Events in each trial at the same time to the eye tracker and to the EEG amplifier. These triggers are text strings, and we usually use numbers to code for different events. So this should be doable somehow, using the fields “Trials(X).Events.message” & “Trials(X).Events.sttime”.

[QUOTE=Francois;9384]
I think all this is doable, and will happen a lot faster if you are willing to contribute to the development.

Cheers,
Francois[/QUOTE]

We are definitely willing to contribute - let us know how we can help!

Best,
Martin

MartinFR · October 5, 2015, 7:18am

[QUOTE=MartinFR;9387]

I will contact them and let you know of the answer.
[/QUOTE]

SR Research support already answered:

[QUOTE=“W. Dan McEchron” <dan@sr-research.com>]
Hi Martin,

The people at Brainstorm would be welcome to include the EDFAccess API in their project however the licensing is maintained by SR Research and does not conform to the GPL. They could still include the functions, they would just need to add an exception to their GPL and state that the EDFAccess API is owned and maintained by SR Research.

Alternatively, they could build in the ability to import one of our ASC files or our MATLAB structures into their executable and allow users to independently convert the EDFs to one of the supported formats. This would allow them to maintain their GPL with no exceptions.

I hope that helps. Please let us know if you have any other questions.

Dan McEchron
Research Support Specialist
SR Research Limited
35 Beaufort Drive
Kanata, ON K2L2B9
Phone | 613-271-8686 Ex. 238
Toll Free | 1-866-821-0731 Ex. 238
Email | dan@sr-research.com

[/QUOTE]

Best,
Martin

MartinFR · October 9, 2015, 4:36am

Dr. Alexander Pastukhov, the author of the edfImport library, answered that he is happy that his code is used and that you are allowed to use and redistribute the code. He also wrote that he will most likely add a GPL license specification, so that it is clear to everybody!

Best,
Martin

Francois · October 9, 2015, 6:14am

This is great, I will add the code of both packages in the folder brainstorm3/external, which contains all code that was not written by our direct collaborators.
I’m sorry I didn’t get back to you yet, I didn’t have time to work on the import filters this week. Hopefully next week.

Francois

Francois · October 14, 2015, 8:06am

Hi Martin,

I added a basic support for reading the information from the .edf files from EyeLink systems. New files in the distribution:

brainstorm3/external/edfimport-1.0.4: Files from the SR EDF Access API and the edfImport library.
brainstorm3/toolbox/io/in_fopen_eyelink.m: Open a .edf file in Brainstorm
brainstorm3/toolbox/io/in_fread_eyelink.m: Read a block of .edf file

For now, I used the functions edfExtractInterestingEvents and edfExtractMicrosaccades as you suggested.
When linking the file to the database, it creates 8 event categories: Blink R/L, Saccade R/L, Fixation R/L, Button and Microsaccade.
The import is quite slow mainly because of the microsaccades detection, which requires the reading and processing of the entire file.

When you try to display the contents of the file after importing it, it shows all the fields of data read in the field Trials(i).Samples. I classified them randomly in two categories “EYE” and “STATUS”. If you double-click on the file, it shows by default the “EYE” group. Let me know what would be an appropriate grouping of channels (you can have as many as you want). You can display some channels separately by selecting them and right-click > Channels > View selected.
The trials are displayed separately because they are not necessarily contiguous. To switch between trials, use the box “Epoch” in the Record tab.
Scrolling in time is slow because all the file is read every time (there is no way with these libraries to read just a segment of the file).

It works for Windows computers only because I don’t have the shared library for MacOS or Linux systems (equivalent of edfapi.dll).

So now you will have access to the events in the Brainstorm environment. The second part of the problem is maybe more complicated: how to merge this information with your EEG recordings.
One way for you to do this is to write a process (=plugin) that takes two inputs (Process2): on one side the EEG recordings, on the other the EyeLink information.
The process would create new event categories in the EEG recordings, based on the information found in the eye tracker event lists. It would be responsible for the synchronization of the two sets of signals.

For information on how to write processes:
http://neuroimage.usc.edu/brainstorm/Tutorials/TutUserProcess
Description of the event structures:
http://neuroimage.usc.edu/brainstorm/Tutorials/EventMarkers#On_the_hard_drive

You can start working from two examples: one that takes two inputs (eg. process_diff_mean.m) and one that writes events in the input file (eg. process_evt_delete.m).

Let me know how it goes.
Cheers,
Francois

MartinFR · October 15, 2015, 8:36am

Hi Francois,

thank you! I will try it out and give you feedback.

One quick question for the start: I could so far not find out how to import the data into brainstorm; I did not find the option in the GUI, and if I run the “in_fopen_eyelink” function, I was able to import the “ChannelMat”, but not the “sFile” variable. Am I missing something?

Best,
Martin

Francois · October 15, 2015, 9:01am

Hi Martin,

Create a subject, right-click on it and import your file like any other file with the menus “Review raw file” or “Import MEG/EEG”.
At the bottom of the list of file formats in the dialog box you will an option “EyeLink eye tracker (*.edf)”.

If you are not familiar with these operations yet, I’d recommend you start by reading all the “Get started” tutorials:
http://neuroimage.usc.edu/brainstorm/Tutorials

Cheers,
Francois

MartinFR · October 16, 2015, 12:29am

Hi Francois,

this was were I expected it, but somehow, the EyeLink option is not available to me after the update, although the functions you mentioned are there. Maybe something went wrong with the update; but even a second update does not seem to help.

Best,
Martin

/edit: Now it works. The problem with the update was on our side, probably another user using the same brainstorm folder while I updated it.

MartinFR · October 23, 2015, 8:33am

Hi Francois,

I managed to synchronize the eyelink events with my EEG data; however, the functions are far from being ready for distribution.

The first thing I did was modifying your “in_fopen_eyelink”, because I needed to import events from the “Events.message” field, where the synchronization triggers are stored (modified function attached here). The modifications are in row 112 and in rows 156-181. I excluded the event message “TRIALID” from getting imported, because this would lead to too many event types (one for each trial…).

Then I wrote the function “process_transfer_events” (also attached here), a Process2-function which takes the events from datasets in A and stores them into the datasets within B after calculating the time offset between the two datasets. Of course one has to be careful to add the corresponding data sets, also in the correct order. This works for me now, however, only after importing the raw data into the brainstorm database.

One really important thing to add would be that the user could choose which event types of each data set to use for the synchronization of the files (must be event types which were sent at the same time to both eyelink and eeg amplifier). For now, I just hardcoded this within the function with one event type which works for me. I’m not sure how to add a drop-down menu with a list of available event types of both datasets into the gui.

Best,
Martin

Francois · October 30, 2015, 8:18am

Hi Martin,

Thanks for the update, and sorry for the late reply.
I added your modifications of the opening function to the standard distribution.

There is currently no way to select an event category from the options of a process, in other processes it is done by simply typing the event name in a text box.
Could you send me an example of synchronized EEG+Eyelink files, so I can test your process?

Would you be interested in writing a short online tutorial to illustrate how to add EyeLink data in an EEG file with your process?
A short document that shows what you need, the general logic and the expected results. It would be referenced in the “Recordings” section of the tutorials.
It could be nice to include an example dataset so that other people can replicate the results (as this MIT group did: http://neuroimage.usc.edu/brainstorm/Tutorials/Decoding), but it is not mandatory.

Cheers,
Francois

MartinFR · November 4, 2015, 4:35am

Hey Francois,

okay! Here is an example of an EEG recording session and the associated Eyelink data:
https://www.dropbox.com/s/u4l4jqpphfxim7k/SampleEyelinkSync.zip?dl=1

Please note that due to the importing functions, the trigger events in the EEG file are called e.g. 'E31', 'E32', etc, while in the Eyelink dataset they are named '31', '32', without the "E". Also, 'E1', 'E2', 'E3', 'E4' are associated with '01','02', '03', '04'.
Events that are there in both Eyelink and EEG are: 01, 02, 03, 04, 10, 20, 21, 30, 31, 32, 33, 40, 5, 50.

I can write a tutorial for that, no problem! Of course I will also need to make the synchronizing function more flexible, by adding for example a textbox input for the event types.

Best,
Martin

Francois · November 5, 2015, 1:30pm

Hi Martin,

In the package you sent me, I can see that you imported completely the EEG and EyeLink data.
This is not the way I would recommend you work with these tools. When working with the list of events, it is preferable to work at the level of continuous recordings (“Link to raw file”). I think you should edit your process so that it works with non-imported files. You can find many examples in the process folder (for instance process_evt_simple.m).
Do you need help with this?
When done, could you also send me the non-imported files?

There is also something weird with the Eyelink_File imported file: there are 334181 elements in the Time vector, but only 334180 samples in the .F matrix.
This causes an error when trying to load the file.

Cheers,
Francois

MartinFR · November 10, 2015, 7:19am

Hi Francois,

you are right, this would be a lot nicer to work with. I modified the “process_transfer_events” function so that you can use raw EEG files as destination for the event transfer (attached here).

However, I do have a problem using raw eyelink files as source. This is because, the way we use the eyelink, we get one .edf - file for the whole experiment, while the NeurOne EEG amplifier gives us one file per recording session. The synchronization will get a lot harder if I have 1 source file in dataset A and for example 10 destination files in dataset B. If i import the eyelink data, the data is again nicely split into epochs and thus more easily to manage for me.

Maybe you can help me with how to split the eyelink events into epochs without importing the data before? Right now, I am using the completely imported eyelink files as event source (dataset A) and the raw EEG recordings as destination (dataset B).

Here’s a small example recording so you can test it: https://www.dropbox.com/s/9b07dknsay8n2ur/TrigTest.zip?dl=1

It consists of 3 recording sessions with the NeurOne (however, no real EEG attached), with the associated eyelink edf-file (eyelink data is real).

I’m not sure why the last imported eyelink file had some sample number issues, but I didn’t really look into that for now (but I also had no error here). Please let me know if this problem is consistent in the new files; maybe there is a problem in how i imported the files.

Best,
Martin

Francois · November 11, 2015, 10:29am

Hi Martin,

Thanks for the example.
This process has the potential to be a very useful tool for many purposes and many file formats. However it would need to work completely on continuous files to be really efficient.

For the problem of the multiple sessions, I think the ideal option would be to allow multiple files on both sides.
The process would allow the pairing as long as the number of events is the same in the Files A and Files B.
This is indeed more complicated to program, but doable.

Alternatively, you can change the process type from “Custom” to “File” (process only one pair of file at a time) and allow a sub-selection of the event in A and/or B.
You put three times the same EyeLink file in A, and the three EEG sessions in B. Each pair is one execution of the process function, you need an extra parameter to indicate which ones of the events have to be used from file A at each execution.
Two options to select them:

Using a “time window” input, which selects the time segment in which we look for the events (not convenient, because we are doing this to avoid having to match the times manually between the two files)
Listing manually the indices that have to be used if there are more on one side (you can use an option similar to the frequency list in process_notch.m: Type=‘value’, Value={[], ‘list’, 0}). This is not a lot of work to add.

Does this sound reasonable?

I edited a bit the function, including the file name (attached).

Cheers,
Francois

MartinFR · November 20, 2015, 9:11am

Hi Francois,

sorry for the late response.

I edited the function extensively, so that now following things should be possible:

using raw files or data files (or even a mix of those) as source and destination.
using a different number of input files, as long as the number of sync events in both sets is identical; for example, 1 raw file in set A with 1000 trials and 10 files in set B with 100 trials each, or vice versa.

I tried to keep the function as general and flexible as possible; unfortunately, the code got a bit complicated because of that.

It would be nice to have the choice between overwriting the input files or creating new ones. The checkbox for that is already implemented, but I’m not quite sure how to create a new file and database connection for the “not overwrite” case.

Test file transfers [eyelink–>EEG] and [EEG–>eyelink] worked in my case.

Best,
Martin

Francois · November 23, 2015, 10:54am

Hi Martin,

Thanks for the update. I added a test to handle the case where the input event name is not found in the file.
It’s online, you can update Brainstorm to get the modified version.

There is not really any “overwrite” or “copy” in this process, because you don’t want to duplicate the input files. You just edit the metadata associated with the recordings. With the way the database is organized, we cannot have two different links to the same continuous file.
I understand that you would like to be able to revert easily the modifications made in the input files, but it cannot be done by duplicating the “Link to raw file”.

The easier solution would be to add a tag to the names of the events that are copied. For instance when transferring the event “microsaccade”, you could add a user defined tag “tag: microsaccade”.
It prevents the transfer to mess up with the events that are already existing, and if you want to revert the transfer, you just have to delete all the events with a name starting with “tag”.
This tag can be either something hardcoded, or an additional option for the process.

What do you think?
Francois

MartinFR · November 25, 2015, 9:12am

Hi Francois,

thanks!

I think the idea with the optional, user defined tag is really nice.

Regarding the tutorial: In what format could I send it to you (when I’ve written it, I haven’t started yet).

Best,
Martin

Francois · November 25, 2015, 10:47am

Hi Martin,

If you’d like, you could write directly a page on the Brainstorm wiki-based website, this would be the easier. I’ll send you the registration information in a separate email.
Something like: http://neuroimage.usc.edu/brainstorm/Tutorials/EyetrackSynchro

Thanks
Francois