B

N3

Da

b

h

•

•

•

•

•

a

ARRAA

KE3VEEB

d

h0

Journal of Neuroscience Methods 235 (2014) 245–251

Contents lists available at ScienceDirect

Journal of Neuroscience Methods

jo ur nal ho me p age: www.elsev ier .com/ locate / jneumeth

asic Neuroscience

EDE: An open-source scripting suite for developing experiments inD virtual environments

avid C. Jangrawa, Ansh Johri a, Meron Gribetza, Paul Sajdaa,b,∗

Department of Biomedical Engineering, Columbia University, New York, NY, USADepartment of Radiology, Columbia University, New York, NY, USA

i g h l i g h t s

We present a design environment fordeveloping realistic 3D experiments.Subjects can explore custom virtualenvironments in search of random-ized stimuli.Leverages rendering power and com-munity support of popular 3D gameengine.Interfaces with an eye tracker in real-time and syncs with an EEG system.Simple runtime GUI controls taskprompts, eye tracker calibration, andnavigation.

g r a p h i c a l a b s t r a c t

r t i c l e i n f o

rticle history:eceived 23 January 2014eceived in revised form 13 June 2014ccepted 23 June 2014vailable online 23 July 2014

eywords:xperimental designD stimuliirtual environmentye trackingEGCI

a b s t r a c t

Background: As neuroscientists endeavor to understand the brain’s response to ecologically valid scenar-ios, many are leaving behind hyper-controlled paradigms in favor of more realistic ones. This movementhas made the use of 3D rendering software an increasingly compelling option. However, mastering suchsoftware and scripting rigorous experiments requires a daunting amount of time and effort.New method: To reduce these startup costs and make virtual environment studies more accessible toresearchers, we demonstrate a naturalistic experimental design environment (NEDE) that allows experi-menters to present realistic virtual stimuli while still providing tight control over the subject’s experience.NEDE is a suite of open-source scripts built on the widely used Unity3D game development software,giving experimenters access to powerful rendering tools while interfacing with eye tracking and EEG,randomizing stimuli, and providing custom task prompts.Results: Researchers using NEDE can present a dynamic 3D virtual environment in which randomizedstimulus objects can be placed, allowing subjects to explore in search of these objects. NEDE interfaceswith a research-grade eye tracker in real-time to maintain precise timing records and sync with EEG orother recording modalities.Comparison with existing methods: Python offers an alternative for experienced programmers who feel

comfortable mastering and integrating the various toolboxes available. NEDE combines many of thesecapabilities with an easy-to-use interface and, through Unity’s extensive user base, a much more sub-stantial body of assets and tutoConclusions: Our flexible, openroscientists interested in deve

Abbreviations: EEG, electroencephalography; BCI, brain–computer interface; 3D, 3-development kit; GUI, graphical user interface; UDK, unreal development kit.∗ Corresponding author at: 351 Engineering Terrace Building, Mail Code 8904, 1210 Am

E-mail addresses: djangraw@gmail.com (D.C. Jangraw), psajda@columbia.edu (P. Sajda

ttp://dx.doi.org/10.1016/j.jneumeth.2014.06.033165-0270/© 2014 Elsevier B.V. All rights reserved.

rials.-source experimental design system lowers the barrier to entry for neu-loping experiments in realistic virtual environments.

© 2014 Elsevier B.V. All rights reserved.

imensional; NEDE, naturalistic experimental design environment; SDK, software

sterdam Avenue, New York, NY 10027, USA. Tel.: +1 212 854 8997.).

2 roscience Methods 235 (2014) 245–251

1

tuwesgpr

eibst(iTa2ab2ineaars

rpiesdsuscNdr

2

tFEejcdtp

2

re

Fig. 1. Block diagram of the system used to display the virtual environment whilecollecting EEG, eye tracking, and display data simultaneously. Three computers(green) each control a peripheral device (red) and are connected on an intranet(dotted lines). A sync pulse is sent by the eye tracker computer every second andreceived by the EEG system. NEDE was tested with an EyeLink 1000 (tower mount)but should be compatible with any EyeLink model. (For interpretation of the refer-

46 D.C. Jangraw et al. / Journal of Neu

. Introduction

Studies of the physiological signals evoked in “naturalistic” vir-ual environments that approximate our real-life experience offernique advantages, allowing experimenters to present the subjectith dynamic stimuli, intuitive interactions, and a sense of “pres-

nce” in the task (Nash et al., 2000; Bohil et al., 2011). Until recently,uch studies have been limited by technological constraints: slug-ish graphics cards and complex rendering software have made itroblematically expensive and time-consuming to present natu-alistic stimuli at the desired level of spatial and temporal control.

As graphics processing capabilities improve and researchersxamine the neural signals evoked in increasingly realistic scenar-os, the benefits of introducing complex virtual environments haveegun to outweigh the costs. Rendering latencies have declined,ophisticated virtual reality tools have become increasingly easyo use, and stereoscopic hardware and software have proliferatedBohil et al., 2011). As a result, the use of virtual environmentsn neuroscience research has begun to accelerate in recent years.hey have been used to study visual decision-making, memory,ttention, social interactions, and spatial abilities (Jovancevic et al.,006; Piccione et al., 2008; Parsons et al., 2007; Bohil et al., 2011);nd they have been used as a proving ground for motor-imageryrain–computer interface (BCI) control of a wheelchair (Leeb et al.,007; Pfurtscheller et al., 2010). But due to the difficulty of adapt-

ng complex 3-dimensional (3D) rendering software to the specificeeds of neuroscience experiments, each study tends to use anxperimental design environment that is tailored to their paradigmnd not flexible enough to use for other tasks. The result is a rel-tively small, fractured set of studies and an inefficient use ofesources, as each experimenter builds a complex experiment fromcratch.

To make naturalistic paradigms more accessible to neuroscienceesearchers, we have developed a design environment for theresentation of realistic 3D experiments. This naturalistic exper-

mental design environment (NEDE, pronounced “Neddy”) willnable 3D studies featuring custom environments, randomizedtimuli, and integration with established neuroscience tools. Theesign environment is built on a popular 3D game developmentoftware platform, giving researchers access to extensive doc-mentation and support as they develop custom settings andcripting. NEDE scripts are open-source, allowing the neuroscienceommunity to refine and extend their capabilities over time. TheEDE system could serve as a useful testbed for real-world BCIevelopment and provide new momentum in the study of natu-alistic visual experience.

. Materials and methods

The NEDE platform is a collection of novel scripts written forhe Unity 3D game development platform (Unity Technologies, Sanrancisco, CA) to interface with a research-grade eye tracker andEG system, facilitate the use of randomized stimuli, and give thexperimenter precise control over important elements of the sub-ect’s experience. The resulting setup lets experimenters designustom experiments that combine Unity’s state-of-the-art 3D ren-ering with the eye tracker’s physiological signals and preciseiming. NEDE’s role is to facilitate this interface and simplify therocess of building naturalistic experiments.

.1. Data collection system

Eye movements will be an important part of many virtual envi-onment experiments, as they are crucial to our natural visualxperience. Central to NEDE, therefore, is a real-time interface with

ences to color in this figure legend, the reader is referred to the web version of thearticle.)

an EyeLink eye tracker (SR Research, Ontario, Canada). Using theEyeLink Software Development Kit (SDK), we constructed a customwrapper to send experimental parameters and updates to the ded-icated eye tracker computer during the experiment, so that theseare recorded using EyeLink’s precise clock and synchronized witheye position data.

Like eye position, EEG data are recorded on a separate com-puter to avoid overly taxing either processor. We used a BioSemiActiveTwo EEG amplifier and recording software for testing(BioSemi, Amsterdam, The Netherlands), but another system couldbe substituted without any change in scripting.

A block diagram of the system used for NEDE data collection isshown in Fig. 1. Three computers are connected on a network, anda TCP/IP connection is used to send data between the eye trackerand display computers during each session. Placing the EEG com-puter on the network expedites the transfer of EyeLink data to thiscomputer for analysis, which is currently performed offline.

2.2. Unity platform

The Unity platform was chosen because it is highly flexibleand customizable, easily integrated with third-party software andhardware, and has a large community of users that can provideextensions and crowd-sourced support. Unity also supports customanimation and multiplayer modes. Custom scripts can be writtenin Javascript, C#, or Boo (an object-oriented language with syntaxsimilar to Python); and compiled C, C++, and objective-C code can beincorporated in the form of plugins. Integration with common anal-ysis tools can be accomplished using TCP/IP (e.g., to interface withMATLAB). Currently available extensions include random environ-ment generators, SQL database integration, a Python interpreter,and a plugin for stereoscopic displays and other virtual realityequipment. A screenshot of the Unity application is shown in Fig. 2.

2.3. Scripting

NEDE scripts are written primarily in JavaScript, as this lan-

guage is used most widely in the Unity documentation. C# is usedfor reading and writing text files and for interfacing with the eyetracker. Two snippets of NEDE code are shown in Fig. 3. The Eye-Link wrapper was written in C and compiled into a dynamic-link

D.C. Jangraw et al. / Journal of Neuroscience Methods 235 (2014) 245–251 247

Fig. 2. Screenshot of the Unity 3D game development software running a preview of the sample experiment. Unity allows the experimenter to see the virtual environmentf ill be

w e filesa

lR

3

3

atteitocidpf

3

cAnalM

also designate locations or groups of locations where objects willbe randomly placed.

Both 2D images and 3D models can be used as objects in the vir-tual environment. 3D models are inserted directly, while 2D images

rom any angle (top left), from the subject’s point of view (bottom left), and as it windows to select or modify the objects in the environment (“Hierarchy” tab), th

ssociated with various objects in the scene (“Inspector” tab).

ibrary (DLL) using Microsoft Visual Studio (Microsoft Corporation,edmond, WA).

. Results

.1. Capabilities

Our scripts represent an extension of Unity’s capabilities intoreas of likely importance to neuroscience experiments. Withhese new capabilities, an experimenter can use the NEDE “Con-rol Center” graphical user interface (GUI) to choose the virtualnvironment in which the experiment takes place, the size, pos-tions, and categories of the randomized stimuli to be presented,he speed and agency of the subject, and the presence of a sec-ndary task. A video tutorial describing the practical usage of theseapabilities is presented in Supplementary Movie S1. Perhaps mostmportantly, NEDE scripts make these capabilities part of a largerata collection system (described in Section 2.1), allowing the dis-lay computer to communicate with an eye tracker and EEG systemor data collection and analysis.

.1.1. Environments and objectsThe standard Unity game engine allows the experimenter to

reate a custom environment for subjects to observe or explore. custom environment might be particularly helpful for studies of

avigation, occlusion, or stimulus context. Environments can bessembled from basic building blocks in the Unity editor or fromarger models constructed in 3D editors like Google Sketch-Up or

aya. The use of “seamless” textures can avoid artificial-looking

recorded in the data file (bottom-most bar, reading “Eye at...”). Unity also provides available to the experiment scripts (“Project” tab), and the variables and scripts

edges on walls and floors; these are available on many 3D model-ing or web design sites. NEDE scripting allows the experimenter to

Fig. 3. (a) This sample JavaScript code includes the Start and LateUpdate functions,which can be used to run code once on startup or on each frame update, respectively.(b) This sample C# code demonstrates the use of the System. IO package, which isused in NEDE for querying, loading, and writing files within a Unity project’s filestructure.

248 D.C. Jangraw et al. / Journal of Neuroscience Methods 235 (2014) 245–251

F the ot ce of ei on cale ure le

awiwoWsp

gsccat

3

“omev

snbd

ig. 4. NEDE Control Center GUI displayed to the experimenter at runtime. Amonghe desired stimulus categories and the type (i.e., target or distractor) and prevalent (green boxes), the response type and navigation type (blue boxes), and eye positixplanation of these options. (For interpretation of the references to color in this fig

re projected onto billboard objects. Thus, researchers can placeell-studied images in more natural contexts (say, on wall posters

n an office), or they can perform novel visual discrimination studiesith groups of 3D objects. Many researchers need not design their

wn objects – 3D models are available for download on Google 3Darehouse, TurboSquid, and other 3D modeling websites. NEDE

cripting can automatically normalize the sizes of objects beforelacing them in the environment.

The Control Center GUI provided to experimenters at runtimeives various options for experimental parameters. The current ver-ion of the GUI is shown in Fig. 4. From the GUI, the experimenteran select whether each stimulus category is used, whether it isonsidered a target or distractor (which will affect later analysisnd the automatically-adjusted prompts shown to the subject), andhe prevalence of the category in the environment.

.1.2. NavigationThe experimenter can select from two navigation modes:

active,” in which the subject moves his avatar with a keyboardr joystick, or “passive,” in which the subject is navigated auto-atically through a subset of waypoints specified in a text file. The

xperimenter can specify the elapsed time or number of objectsiewed before the session is over.

The experimenter may also add a secondary task in which the

ubject is asked to press a button when a car in front of him illumi-ates its brake lights and slows to half its normal speed. When theutton is pressed, this “leading car” speeds back up to return to itsefault distance in front of the subject. The time between braking

ptions the experimenter can select are the subject and session number (red box),ach (orange box), the environment to be used and the size of the objects placed in

ibration parameters (purple box). See Supplementary Movie S1 for a more detailedgend, the reader is referred to the web version of the article.)

events is randomly selected within ranges set by the experimenter.The secondary task serves to keep subjects focused and default theirgaze to the center of the screen.

3.1.3. Experimenter toolsIn addition to the parameters governing the experiment itself,

the Control Center GUI gives the experimenter access to other toolsthat can be used between sessions. The experimenter can select“Browse Objects” to show the subject examples of the objects fromeach category (Fig. 5(a)) or “Calibrate Eye” to see the subject’s eyeposition in real time and adjust the horizontal and vertical offsetand gain (Fig. 5(b)). When the desired parameters are selected,the experimenter can select “Start Session” to display promptsto the subject. These prompts are automatically adapted to theparameters set by the experimenter in the main Control CenterGUI (Fig. 5(c)). NEDE scripts record the experimental parameters,the camera’s position at each frame, and the bounds of all visiblestimuli on the screen for purposes of offline analysis. NEDE can alsoread this data back in using “replay” mode, which allows the exper-imenter to play back a previously recorded session and see exactlywhat the subject saw. Replays can be paused or shifted to slow orfast motion at any time using a GUI slider (Fig. 5(d)).

3.2. Synchronization testing

The EEG system and eye tracker computer each have clocks withsub-millisecond precision but are independent from each other.To synchronize the two clocks before analysis, NEDE scripting

D.C. Jangraw et al. / Journal of Neuroscience Methods 235 (2014) 245–251 249

Fig. 5. NEDE capabilities. In addition to the experiment itself and the Control Center GUI (Fig. 4), custom scripting gives the design environment other useful tools. (a)The Object Browser lets the experimenter display some of the objects that will appear during the experiment, allowing the subject to become accustomed to the virtualenvironment before the task begins. (b) The Eye Tracker Calibration screen allows the experimenter to check and manually adjust the horizontal and vertical offset and gainw sessioR (i.e., fc can b

clpo

d9iea(ndtstdc

3

e

object were analyzed. The secondary task described in Section 3.1.2was also employed.

−3 −2 −1 0 1 2 30

50

100

150

# ev

ents

ithout re-running a separate calibration script. (c) Prompts displayed before theeplay GUI allows the experimenter to play back a session at various “time scales”an step through the session frame by frame. The session time is also displayed and

ommands the Eye Tracker computer to periodically send paral-el port pulses to the EEG system. Samples recorded between theseulses can have their timing interpolated to translate the data fromne system into the clock of the other.

To test the synchronization between the eye tracker and EEGata, we recorded five sessions of data lasting approximately6 s each using the NEDE data collection system (as illustrated

n Fig. 1, but with no subject). Sync pulses were sent from theye tracker computer to the EEG system (a BioSemi ActiveTwomplifier recording at 2056 Hz) at intervals of approximately 1 sNEDE’s default setting). We compiled the time between sync sig-als reported by the eye tracker data file and the EEG system. Theisagreement between the two is a conservative measure of theemporal misalignment introduced by the system. As Fig. 6 demon-trates, the discrepancy between the two systems’ records of theime between pulses is typically very small. The root-mean-squareiscrepancy is 0.380 ms, and the maximum magnitude of the dis-repancy is 1.26 ms.

.3. Sample experiment

NEDE has been used to implement a visual decision-makingxperiment in a recent study (Jangraw et al., 2014). Images from

n begins automatically reflect the target category(ies) (in this case, laptop). (d) Aast or slow motion). To examine part of the session very closely, the experimentere compared to the last time a saccade was made.

four different categories were placed throughout a large grid envi-ronment, and subjects were navigated passively: their car weavedthrough the grid systematically. Subjects were asked to countstimuli of one category and ignore the others, and the neural andocular data time-locked to the first moment of fixation on each

Discrepancy (ms)

Fig. 6. Results of synchronization tests from NEDE. Discrepancy is defined as thetime between received sync signals reported by the EEG system minus the timebetween the corresponding sent sync signals reported by the eye tracker.

250 D.C. Jangraw et al. / Journal of Neuroscience Methods 235 (2014) 245–251

Table 1Comparison of several popular 3D game engines that could be used for naturalistic experiments.

UDK PANDA3D Unity + NEDE

Platforms (editor) Windows Windows, OSX, Linux Windows, OSXPlatforms (experiment) Windows, OSX, iOS Windows, OSX, Linux Windows, OSX, Linux, Android, iOS, WebScripting languagea UnrealScript Python JavaScript, C#, BooCostb Free Free $750 (Unity)Forum sizec ∼250, 000 ∼7, 300 ∼400, 000 (Unity)Video tutorialsd 116 16 170Open source

√ √(NEDE)

Eye tracker integration√

(PyGaze)√

Neuroscience-focused GUI√

a UnrealScript is an object-oriented language with syntax similar to Java. Boo is an object-oriented language with syntax similar to Python.b Cost as of June 10, 2014 for a single non-commercial Unity Pro license (required for EyeLink plugin) at an educational institution. Contact Unity Sales for student/faculty

d

2014.

ittptwpm

3

ebmtasiicsdtGa

4

4

olToEoiaeapao

pp

iscounts. A free 30-day trial of Unity Pro is also available. NEDE is free.c Members reported by each engine’s forum website on December 3, 2013.d Official tutorials posted on each engine’s website or YouTube page as of June 3,

Using this NEDE paradigm, the authors were able to character-ze the EEG signals, gaze patterns, and pupil dilation evoked byargets and distractors during a naturalistic search. They were fur-her able to fuse multimodal features from EEG, eye position, andupillometry data to create a hybrid BCI classifier, which they usedo improve navigation in a mobile visual search. These advancesere made possible by NEDE’s capabilities for randomized stimuli,recise timing, and multimodal synchronization in virtual environ-ents.

.4. Download, documentation, and support

NEDE is available for download at http://nede-neuro.org. Wexpect that learning to manipulate objects in Unity will take theulk of experimenters’ time as they develop a custom environ-ent for their experiment, but Unity’s extensive documentation,

utorials, and user forums should make this task manageable. Inddition, several NEDE-centric tutorials are posted on the NEDEite to get readers up and running quickly. Once an environments constructed, NEDE makes the introduction of new stimuli intot relatively simple. To keep NEDE usage up to date and expand itsapabilities as rapidly as possible, we have made the software open-ource and will allow users to edit and update all the scripts andocumentation on the site. A community forum is also available onhe site, and users can post to the forum by joining the nede-neurooogle group. Documentation is available in the form of a wiki thatny member of the community forum can edit or expand.

. Discussion

.1. Comparison to existing tools

The popularity of game development has led to a proliferationf 3D game engines; dozens are currently available for down-oad or purchase. Some of the most popular, apart from Unity, arehe Unreal Engine (available through the Unreal Development Kit,r UDK), PANDA3D, Torque 3D, OGRE3D, Irrlicht, Leadwerks, C4ngine, and ShiVa. Each has powerful capabilities and a communityf at least 1000 users. For the purposes of neuroscientists develop-ng their first 3D experiments, we prioritize two elements: supportnd flexibility. Unity and UDK have many more users than otherngines, leading to stronger forums and additional downloadablessets. Unity boasts OSX editing, more users, a greater body of sup-ort and tutorials, and additional flexibility in scripting: while Unityllows scripting in several languages (see Section 2.3), UDK allows

nly UnrealScript, a language similar to Java.

PANDA3D is notable because it is an open-source Pythonackage able to leverage the significant power of other Pythonackages. Several Python toolboxes have been developed for use in

neuroscience and psychophysics experiments, including PsychoPy(Peirce, 2007), Pyff (Venthur et al., 2010), and PyGaze (Dalmaijeret al., 2013). PANDA3D, used in combination with these packages,can be a powerful and transparent tool for experienced program-mers. Unity and NEDE offer a simpler solution for those looking toenter the domain of naturalistic experiments with lower startupcosts and a greater body of support and tutorials.

A side-by-side comparison of UDK, PANDA3D, and the Unity +NEDE combination is presented in Table 1.

4.2. Potential extensions

The flexibility of the Unity platform allows a multitude of oppor-tunities to expand the capabilities of NEDE. Unity projects ofteninterface with audio hardware, and NEDE could be extended topresent auditory stimuli. Unity’s large user base has developedcustom scripting that interfaces with a variety of input and dis-play hardwares, available through the Unity Asset Store and forum.They can interface with stereoscopic displays, including 3D TVsand head-mounted displays, or with Microsoft Kinect, introducingthe exciting possibility of incorporating and sensing movement innaturalistic experiments.

Unity’s broad and expanding capabilities offer further oppor-tunities to extend NEDE in the software domain. For example,Unity features increasingly realistic animation, which could allowresearchers to study human–autonomy interaction or immersiontreatments for phobias. To act as a testing environment for mobileBCI technology, NEDE could be adapted so that real-time analysiscould be accomplished by streaming EEG data in using an SDK and aTCP/IP connection, as in other BCI systems (Pohlmeyer et al., 2011).

5. Conclusion

Taken together, the capabilities of NEDE described in this paperrepresent an extensive framework for naturalistic visual neuro-science and psychophysics experiments. An experimenter with anEyeLink eye tracker and a recent installation of Unity can copy theNEDE scripts to a Unity project folder and begin using or extendingits capabilities with minimal setup time or changes to the code,enabling a wide variety of visual neuroscience experiments farmore realistic than the status quo.

Acknowledgments

Research was sponsored by the Army Research Laboratory

and was accomplished under Cooperative Agreement NumberW911NF-10-2-0022. The views and conclusions contained in thisdocument are those of the authors and should not be interpretedas representing the official policies, either expressed or implied,

roscie

oGfh

A

i2

R

B

D

J

cortically-coupled computer vision: a brain–computer interface for searchingimage databases. J Neural Eng 2011];8(3):036025.

D.C. Jangraw et al. / Journal of Neu

f the Army Research Laboratory of the U.S. Government. The U.S.overnment is authorized to reproduce and distribute reprints

or Government purposes notwithstanding any copyright notationerein.

ppendix A. Supplementary Data

Supplementary data associated with this article can be found,n the online version, at http://dx.doi.org/10.1016/j.jneumeth.014.06.033.

eferences

ohil CJ, Alicea B, Biocca FA. Virtual reality in neuroscience research and therapy.Nat Rev Neurosci 2011];12(12):752–62.

almaijer ES, Mathot S, Van der Stigchel S. Pygaze: an open-source, cross-platform

toolbox for minimal-effort programming of eyetracking experiments. Behav ResMethods 2013]:1–9.

angraw DC, Wang J, Lance BJ, Chang S-F, Sajda P. Neurally and ocularlyinformed graph-based models for searching 3D environments. J Neural Eng2014];11(4):046003.

nce Methods 235 (2014) 245–251 251

Jovancevic J, Sullivan B, Hayhoe M. Control of attention and gaze in complex envi-ronments. J Vis 2006];6(12).

Leeb R, Friedman D, Muller-Putz GR, Scherer R, Slater M, Pfurtscheller G. Self-paced(asynchronous) BCI control of a wheelchair in virtual environments: a case studywith a tetraplegic. Comput Intell Neurosci 2007]:79642.

Nash EB, Edwards GW, Thompson JA, Barfield W. A review of presence and per-formance in virtual environments. Int J Hum Comput Interact 2000];12(1):1–41.

Parsons TD, Rizzo AA, Bamattre J, Brennan J. Virtual reality cognitive performanceassessment test. Annu Rev CyberTherapy Telemed 2007];5:163–71.

Peirce JW. Psychopy – psychophysics software in python. J Neurosci Methods2007];162(12):8–13.

Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer R,et al. The hybrid BCI. Front Neurosci 2010];4:30.

Piccione F, Priftis K, Tonin P, Vidale D, Furlan R, Cavinato M, et al. Task and stimulationparadigm effects in a p300 brain computer interface exploitable in a virtualenvironment: a pilot study. PsychNol J 2008];6(1):99–108.

Pohlmeyer EA, Wang J, Jangraw DC, Lou B, Chang SF, Sajda P. Closing the loop in

Venthur B, Scholler S, Williamson J, Dahne S, Treder MS, Kramarek MT, et al. Pyff –a pythonic framework for feedback applications and stimulus presentation inneuroscience. Front Neuroinform 2010];4.