= Tutorial 9: Select files and run processes =
''Authors: Francois Tadel, Elizabeth Bock, Sylvain Baillet''

The Brainstorm window includes a graphical batching    interface. With the two tabs Process1 and Process2 in the lower part of the window, you can select files from the database explorer and assemble a processing pipeline. Most of the operations available in the interface can also be executed this way, including everything we've been doing with Brainstorm so far.

On the other hand, some features are only available this way. It is the case for the frequency filters we will need for the pre-processing of our auditory recordings. This tutorial is a parenthesis to explain how to select files and run processes, we will resume with the cleaning of the recordings in the next tutorial.

<<TableOfContents(2,2)>>

<<Icon(image)>> /!\ --

 {*}
    {*}

== Selecting files to process /!\ ==
The tab '''Process1''' contains a empty box in which you can drag and drop any number of files or folders from the database explorer. The easier way to understand how it works is to try it.

 * Try to drag and drop in Process1 all the nodes you currently have in your database explorer.
 * You will see that it accepts all the folders and all the recordings, but not the channel files.
 * When you add a new node, the interface evaluates the number of files of the selected type that each of them contain. The number in the brackets next to each node represents the number of  data   files that where found in them.
 * On top of the list, a comment shows the total number of files that are currently selected. <<BR>><<BR>> {{attachment:process1_example.gif}}
 * The   buttons on the left side allow you to select what type of file you  want  to process: Recordings, sources, time-frequency, other. When you select another button, all the counts are updated to reflect the  number of files of the selected type that are found for each node.
 * Right now, if you select another file type, it would show only "0" everywhere because there are no sources or time-frequency decompositions available in the database yet. <<BR>><<BR>> {{attachment:process1_sources.gif||height="159",width="464"}}

 * To remove files from the Process1 list:
  * Select the nodes to remove (holding Shift or Ctrl key) and press the '''Delete''' key.
  * Right-click on the list > '''Clear list'''

== Filter by name or comment ==
When  you have lots of files in a folder, like multiple source  reconstructions or time-frequency files for each trial, it is difficult  to grab just the ones you are interested in. After selecting your folders in the Process1 box, you can refine the selection with the '''Filter search box''' at  the bottom-right corner of the window.

 * The example below shows how select the data file corresponding to the noise recordings: by typing "Noise" in the search box and selecting the option "Search file names". We cannot perform the search "by comment" because all the data files have the same comment "Link to raw file".<<BR>><<BR>> {{attachment:process1_search.gif||height="148",width="436"}}

 * Reminder: To see the file name corresponding to a node in the database, leave your mouse over it for a few seconds. You can do this both in the database explorer and the Process1 list.<<BR>><<BR>> {{attachment:process_filename.gif||height="148",width="490"}}

The options offered in the Filter menu are:

 * '''Search comments''':  Look for the string in the comments of the files, ie. what is displayed  in the database explorer to represent them (the .Comment field).
 * '''Search file names''': Look for the string in the full file names (including their relative path).
 * '''Select files''': Only the files that contain the string are selected.
 * '''Exclude files''': Only the files that DO NOT contain the string are selected.
 * '''Reset filters''': Removes the current file filters applied on Process1 and Process2.
 * '''Case insensitive''':  Note that the search is not sensitive to case.

== Selecting processes ==
 * Clear the file list and the search filters.
 * Select all the three datasets we have linked to our protocol. <<BR>>You can select the three "link to raw file" nodes, the three folders or the entire subject node.
 * Click on the '''[Run]''' button at the bottom-left corner of the Process1 tab.
 * The '''Pipeline editor''' window appears. You can use it to  create  an   analysis pipeline, ie. a list of processes that are applied on  the    selected files one after the other. The first button in the  toolbar    shows the list of processes that are currently available. If  you click    on a menu, it's added to the list.
 * Some    menus appear in grey. This    means that they are not designed to  be applied to the type of data that  you   have in input, or at the end  of the current pipeline.
 * In the current example, we have a file with the type "continuous raw recordings", so we have access mostly to menus to manipulate event markers, run cleaning procedures and import data blocks. You can recognize a few operations that we exectued in the previous tutorials: "Event > Read from channel" and "Event > Detect analog triggers". <<BR>><<BR>> {{attachment:process1_addprocess.gif||height="480",width="473"}}
 * When you select a process, a list of options specific to this process is shown in the window.
 * To delete a process: Select it and press the '''Delete''' key, or use the '''[X]''' button in the toolbar.
 * After selecting a first process, you can add another one. The output of the first process will be passed to the second process without giving back the control to the user. This is how you can build a full analysis pipeline with the interface.

 * After adding a few processes, you can move up/down a process in the pipeline with the [up arrow] and [down arrow] buttons in the toolbar. Click on a process in the pipeline to edit its options.

 * Select and delete a few processes to understand how this interface works. Just do not click on RUN.

== Plugin structure ==
All  the menus available in the pipeline editor are actually plugins for  Brainstorm. The processes are functions that are independent from each  other and detected automatically when starting Brainstorm.

Any   Matlab    script that is added to  the plugin  folder     (brainstorm3/toolbox/process/functions/) and has  the right  format    will   be automatically detected and made available  in the GUI.  This     mechanism  makes it easy for external contributors to develop their  own code an integrate it in the interface.

More information: [[Tutorials/TutUserProcess|How to write your own process]]

== Too much information ==
Everything below is advanced documentation. It might be confusing for beginners, you can skip it for now.

<<EmbedContent("http://neuroimage.usc.edu/bst/get_prevnext.php?skip=Tutorials/ArtifactsFilter")>>

== Saving a pipeline ==
After preparing your analysis pipeline by listing all the operations to run on your input files, you can either click on the [Run] button, or save/export your pipeline. The last button in the the toolbar offers a list of menus to save, load and export the pipelines.

 . {{attachment:pipeline_example.gif||height="340",width="334"}}

 * '''Load''': List of pipelines that are saved in the user preferences on this computer.
 * '''Load from .mat file''': Import a pipeline from a pipeline_...mat file.
 * '''Save''': Save the pipeline in the user preferences.
 * '''Save as .mat matrix''':    Exports the pipeline as Matlab structure in a .mat file. Allows    different users to exchange their analysis pipelines, or a single user    between different computers.
 * '''Generate .m script''': This option generates automatically a Matlab script.
 * '''Delete''': Remove a pipeline that is saved in the user preferences.
 * '''Reset options''':    Brainstorm saves automatically the options of all the    processes in the user preferences. This menu removes all the saved options and set them back  to   the default values.

== Automatic script generation ==
Here is the Matlab script that is generated automatically for this pipeline.

{{attachment:pipeline_script.gif||height="509",width="703"}}

Reading this script is easy: input files at the top, one block per process, one line per option. You can also modify them to add personal code, loops or tests. Many features are still missing in the pipeline editor, but the generated scripts are easy enough for users with basic Matlab knowledge to edit and improve them.

Running this script from Matlab or clicking on the [Run] button of the pipeline editor produce exactly the same results. In both cases you will not have any interaction with the script, it could be executed without any direct supervision. You just get a report in the end that describes everything that happened during the execution.

Those scripts cannot be reloaded in the pipeline editor window after being generated. If you work on a long analysis pipeline, save it in your user preferences before generating the corresponding Matlab script.

== Process: Select files with tag ==
Since we are discussing the file selection and the pipeline execution, we can explore a few more available options. We have seen how to filter the files in the Process1 box using the Filter search box. We can get  to the exact same result by using the process '''File > Select files: By tag''' before the process you want to execute, to keep only a subset  of the files that were placed in the Process1 list.

It is less  convenient in interactive mode because you don't see immediately the  effect of your file filter, but it can be very useful when writing  scripts. You can also combine search constraints by adding multiple times the same process in your pipeline, which is not possible with the search box.

 * Make sure you still have the three datasets selected in the Process1 list.
 * Select the process:  '''File > Select files: By tag'''
 * Select the options: Search: "Noise", Search the file names, Select only the files with the tag.
 * Click on [Run] to execute the process.  <<BR>><<BR>> {{attachment:process1_select.gif||height="333",width="598"}}
 * This process  is useless if not followed immediately by another process that does something with the selected files. It does nothing but selecting the file, but we can observe that the operation was actually executed with the report viewer.

== Report viewer ==
Everytime   the pipeline  editor is used to run a list of processes, a report is created and logs all the messages that are generated during the execution. These reports are saved in the user home folder: $HOME/.brainstorm/reports/.

The report viewer shows as an HTML page some   of the information saved  in this report structure: the date and  duration  of execution, the  list of processes, the input and output  files. It  reports all the  warnings and errors that occurred during the  execution.

The report is displayed at the end of the execution only if there were more than one process executed, or if an error or a warning was reported. In this case, nothing is displayed.

You can always open explicitly the report viewer to show the last reports: '''File > Report viewer'''.

{{attachment:reports.gif||height="379",width="711"}}

When   running processes manually from a script,  the calls to '''bst_report''' explicitly indicate when the  logging of the events should   start and stop.

You can add images to the reports for quality control using the process '''File > Save snapshot'''.

With the buttons in the toolbar, you can go back to the previous reports saved from the same protocol.

More information: [[Tutorials/Scripting|Complete tutorial script]]

== Error management ==
 * Select the same files and same process:  '''File > Select files: By tag'''
 * Note that the options you used during the previous call are now selected by default.
 * Instead of "Noise", now search for a string that doesn't exist in the file name, such as "XXXX". <<BR>><<BR>> {{attachment:search_xxxx.gif||height="303",width="321"}}
 * Click on [Run] to execute the process. You will get the following error. <<BR>><<BR>> {{attachment:search_error.gif||height="129",width="253"}}
 * If you open the report viewer, it should look like this. <<BR>><<BR>> {{attachment:search_report.gif||height="283",width="512"}}

== Control the output file names ==
If you are  running two processes with different parameters but that produce exactly  the same file names and file comments, you wouldn't be able to select  them with this process. But immediately after calling any process, you  can add the process '''File > Add tag '''to tag one specific set of  files, so that you can re-select them easily later.

Example: You run twice the time-frequency decomposition with different options  on the same files, tag the files after calculating them with different  tags.

{{http://neuroimage.usc.edu/brainstorm/SelectFiles?action=AttachFile&do=get&target=addTag.gif|addTag.gif|class="attachment"}}

== Additional documentation ==
Tutorial: [[Tutorials/Scripting|Complete tutorial script]]

Tutorial: [[Tutorials/TutUserProcess|How to write your own process]]

<<EmbedContent("http://neuroimage.usc.edu/bst/get_prevnext.php?prev=Tutorials/StimDelays&next=Tutorials/ArtifactsFilter")>>

<<EmbedContent(http://neuroimage.usc.edu/bst/get_feedback.php?Tutorials/PipelineEditor)>>