Multimodal acquisition and processing of psychophysiological

variables in complex decision-making

Marta Ferreiramarta.ramalho.ferreira@tecnico.ulisboa.pt

Instituto Superior Tecnico, Lisboa, Portugal

November 2016

Abstract

Humans are constantly making decisions. From the perspective of cognitive neuroscience, suchdecision-making is complex, since it involves the integration of several cognitive processes in a variety ofsituations and contexts. While simple experimental tasks have been used to study the basic mechanismsof decision-making in great detail, more complex paradigms, such as action planning, haven’t beenmodeled accurately. Additionally, the applicability of psychophysiology in complex decision-makingtasks is still unclear.

The purpose of this study was to develop a multimodal acquisition setup to record physiologicalsignals during a new decision-making task, and evaluate its reliability and potential in estimatingthe internal cognitive states in a dynamic environment. Measures of the cardiorespiratory system,perspiration, and eye pupil were simultaneously acquired with high quality and processed to extractrelevant features, such as heart rate (HR), skin conductance responses (SCR) and pupil size (PS).

Experiments conducted in 7 subjects revealed that the dynamics of these signals is related to thecharacteristics of the task. The results show that this task robustly induces global changes in arousalacross different levels of difficulty and after stimulus onset. These changes will now be used to estimatethe subjects’ cognitive state and to obtain physiological representations of planning on a single-trialbasis.

Keywords: Decision-making; Psychophysiology; Multimodal; Planning; Complex

1. Introduction

We are always making decisions in our daily life, forexample, planning a trip, preparing a coffee or as-sembling a new shelf. At each decision stage, thereare several options to choose from, which can beperformed with different levels of confidence, leadto different amounts of reward, trigger various emo-tional responses, and can be remembered for futuredecisions in similar situations. Observing correlatesof these complex internal processes, using e.g. psy-chophysiological methods, together with behavioralanalysis, could provide a more complete understat-ing of the internal states that underlie the decisions.In order to understand the decision-making problemin the context of daily life planning and relativelyunconstrained decisions, cognitive neuroscience isshifting toward more complex and dynamic experi-mental approaches.

1.1. Decision-making

Decision-making is the act of choosing one optionor course of action from a set of alternatives ac-cording to certain decision variables, i.e., variablesthat increase the likelihood of one alternative be-

ing chosen instead of another. The decision-makingprocess can be segmented into stages that involvedifferent cognitive mechanisms [1] and may be pro-cessed in parallel [2]. Examples of these include theidentification of the options, their evaluation andthe choice of an action.The most common approach used to study decision-making has been the two Alternative Force Choice(2AFC) task , which consists of making binarychoices given an input stimulus. These tasks haveestablished a groundwork for understanding the ba-sis of decision-making, especially at the level of per-ception [3]. Neural and behavioral correlates of thedecision are usually obtained and analyzed underconditions of uncertainty associated with increasedstimulus difficulty [4, 5]. These physiological andbehavioral measures are frequently fitted to mathe-matical models, such as the drift diffusion model [6]and reinforcement learning [7], that aim to explaindecision-making mechanisms.

1.2. Increasing complexity of the decision

Although binary decision-making studies have con-tributed to understand some of the basic aspects

1

of decision-making, e.g, in terms of uncertaintyand reward-seeking behavior, they are far fromrepresenting the true complex and parallel natureof decision-making involved in daily life decisions,which involves the integration of several cognitiveprocesses in a variety of situations and contexts. Itrequires the evaluation of future situations that de-pend on the present decisions as well as past ones,i.e, they involve planning. Some studies hypothe-size that planning does not involve the computationof each possible state, but instead these states areclustered into groups upon which the decisions aremade with a much less computational cost and withan hierarchical representation [8, 9]. However mod-els of human planning are less precise, due to theextra degrees of freedom they must consider.

1.3. Psychophysiology in decision-makingPsychophysiology has been extensively used in thepast few decades to assess how the body reacts togiven cognitive and emotional states [10]. Most ofthe studied physiological variables, such as heartrate (HR), heart rate variability (HRV), pupil size(PS) and electrodermal activity (EDA), are con-trolled by the Autonomic Nervous System (ANS).They have been shown to be physiological in-dexes of emotion [11], working memory[12] andattention[13]. In decision-making research, thesemethods have been used in tasks with a limitednumber of actions, such as perceptual discrimina-tion and gambling tasks, in order to obtain auto-nomic correlates of risk [14], uncertainty and errormonitoring [15]. However it is unclear how psy-chophysiology could be used to explain a subject’sinternal state during a more complex and dynamicdecision-making scenario.

1.4. ObjectivesThe aim of this masters thesis consists of studyingthe applicability of some psychophysiological vari-ables in a new complex decision-making task. Thisinvolved choosing the signals to acquire, develop-ing a multimodal acquisition setup and perform-ing a preliminary data analysis to understand thedynamics of these physiological signals in differenttime scales of the task.

2. Background

2.1. Electrodermal activityElectrodermal activity (EDA) or Galvanic skin re-sponse (GSR) is a measure of skin conductance,which increases with the amount of sweat producedby the sweat glands. This signal can be decomposedin a tonic/skin conductance level (SCL), and pha-sic/skin conductance response (SCR) activity. Thelatter is responsive to external stimuli, and is re-lated to arousal levels through its frequency andamplitude. The EDA signal is usually obtained

from the middle or distal phalanxes of the indexand middle finger, but can also be measured in thepalms and soles.

2.2. Cardiovascular measuresThe electrical activity of the heart over a period oftime, can be assessed through an electrocardiogram(ECG). This signal contains three main structures,the P wave, the QRS complex and the T wave,each corresponding to the depolarizaton of the car-diac muscle in a different stage of the cardiac cy-cle. Common features extracted from the ECG thathave been used in psychophysiology are the heartrate (HR) and heart rate variability (HRV). Theycan be both obtained from the interbeat-interval(IBI), which corresponds to the distance (in sec-onds) between adjacent R-peaks.Over a small timescale (seconds), the heart rate can be representedby the instantaneous heart rate (IHR), which is theinverse of the IBI. Both of these features are affectedby the respiration cycle, that should also be mea-sured for posterior correlation analysis. The ECGis usually measured from the chest and limb regionsusing unipolar and bipolar sensors, respectively.

2.3. PupillometryThe pupil size (PS) is a measure of the dilatationand constriction of the pupil as a response to lumi-nosity changes, lens accommodation and cognitiveprocessing. This time series (number of pupil pixelsalong time), can be obtained from the segmentationof the pupil from the images of a high-resolutioninfra-red camera, pointed to the eye region.

3. Methods for a multivariable acquisitionsetup

3.1. Behavioral taskThe behavioral task, The Spinner game, was de-signed with the aim of studying complex decision-making in a more dynamic and engaging environ-ment, rather than using 2AFC. It is a video gamewhere the subjects have to plan in order to collecttargets that grow, in size and value, from the centerof an hexagon to its vertexes within a limited pe-riod of time (Figure 1). A full ”session” of the taskis divided into 7 ”waves”, each of which is furthersubdivided into 36 trials containing between 2 and6 targets.

This task is rich in terms of analysis possibilities.Examples of these include how the subjects plantheir decisions when they have to collect multiplegrowing targets distributed across multiple vertices,how long subjects are willing to wait to collect agrowing target while risking the loss of subsequenttargets, or how optimal are the subjects’s choiceswhen compared to an optimal agent.

In this master thesis project, the main vari-ables of interest were related with the physiolog-

2

ical changes observed at the different task levels:trial, wave and session. While the session level isassociated with learning, in that subjects typicallyimprove across sessions, the wave level is related toeffort and difficulty, in that later waves have moredifficult patterns. Finally, the finest level, the trial,relates to pattern identity, since each pattern oftargets may induce a characteristic behavioral andphysiological responses.

(a) Wave 1 instance (b) Wave 6 instance (c) Wave 7 instance

Figure 1: Spinner task. Targets grow linearly fromthe center of the hexagon until the vertices, andmust be caught before reaching their maximum pos-sible size. The three figures are examples of in-stances of the task during waves 1, 6 and 7, whichhave increasing levels of difficulty determined bynumber and arrangement of targets.

3.2. Materials

A custom-made signal acquisition board, fabricatedby the Hardware Platform of Champalimaud Re-search, was used to acquire up to 11 biosignals si-multaneously, such as Electrocardiogram (ECG),Electrodermal Response (EDA), Electromyography(EMG) and motion signals (accelerometer, gyro-scope and magnetometer) with high quality andsampling frequency (maximum of 4000 Hz). This91x60 mm board contains 5 digital inputs, 6 analoginputs with a 16-bit ADC, 1 reference input, bat-tery and Bluetooth technology. The analog portsconnect to bipolar analog sensors with local filter-ing and amplification, increasing the signal-to-noiseratio. The frequency bandwidth of the EMG/ECGand EDA sensors used were 0.1-500 Hz and 0-40 Hzrespectively. A respiration sensor from Plux, Wire-less biosignals S.A., was also adapted to use withthe analog ports of the board.

The eye tracker used in this experimental setupwas developed by Pupil Labs, and contained oneadjustable IR camera below the right eye and oneworld camera fixated above the left highbrow. Bothare high-speed cameras, with a maximum acquisi-tion rate of 120 fps.

A Dualshock 3 wireless controller/gamepad wasincluded in the setup as the motor interface due dothe pressure sensors incorporated in the keys of thedevice. The use of a gamepad allows the mapping

of each motor action to an unambiguous pressurevalue, without the need of using external sensors.The disadvantage of this is that both hands are nec-essary for performing the task, thus potentially in-fluencing EDA. Two software were used in order touse this gamepad (designed for PlayStation) withthe computer, MotionJoy and Better DS3.

3.3. Experimental methods

Acquisition and synchronization of the experimentalvariables

Bonsai, an open-source visual programming frame-work [16], was the chosen interface for the acqui-sition and recording of the data from the severaldevices that were used. Synchronization is ob-tained by recording the time that each data pointis collected in Bonsai, and saving it in the formatof a timestamp. This overcomes the problem ofsome devices having irregular sampling frequencies.The data from the acquisition board, eye trackingcameras and gamepad were acquired with samplingrates of 1000, 120 and 60 samples per second, re-spectively. Due to overload problems, two differ-ent computers were used according to the schemein Figure 2a. A communication pulse was sent inorder to synchronize the two computer clocks.

(a) Acquisition and synchroniza-tion scheme

(b) Experimental spaceorganization

Figure 2: A-Acquisition and synchronizationscheme. The first computer runs Bonsai and thetask, while the second acquires only data from theacquisition board, also using Bonsai. The com-puters are synchronized with a OSC pulse; B-Experimental space organization. The green boxrepresents the acquisition board.

Configuration of sensors placement

The three bipolar ECG sensors were placed on thesubjects chest, as shown in Figure 3a. Their posi-tion was optimized to resemble the limb leads, butcloser to the heart and with a smaller distance be-tween the bipolar sensors. Two EDA sensors wereattached to each of the lower hypothenar eminencesof the palms, as shown in Figure 3b, since the mus-cles of this region dont participate in the motor ac-tion of pressing the keys. For both, adhesive gel

3

Ag/AgCl electrodes were used with previous cleanpreparation.

(a) Position of ECG sensors (b) Position of EDA sen-sors

Figure 3: Placement of ECG and EDA sensors.Each color corresponds to a pair of bipolar sensors.The black sensor corresponds to the ground

The respiration band was placed around theribcage and the position of the eye tracker cameraswas optimized to maximize the capture of the pupiland remove the regions of the eyelids and eyelashes,which interfere with pupil detection.

Experimental acquisition protocol

The experimental acquisitions took place at Cham-palimaud Centre for the Unknown, in the neuro-science department, using the experimental spaceillustrated in Figure 2b . Seven subjects (25.39 ±5.68 years old) out of eighty nine volunteers were se-lected based on their experience with video games,absence of psychiatric or neurological illness andnot taking any current medication. Subjects fa-miliarized with video games were selected based onthe premise that they would internalize the rulesof the task more quickly, and have a better motorcontrol over the gamepad [17]. This experimentalstudy received approval from the Ethics Commit-tee of Champalimaud Foundation and also from theNational Commission for Data Protection.

Before each session, the subjects perform a 2-minute relaxation period followed by a calibrationroutine for posterior gaze estimation.

3.4. Processing Methods

The acquired physiological signals are susceptible toseveral types of noise. The pre-processing of thesesignals for artifact removal and extraction of basicfeatures, was done in MATLAB [18]

EDA

All the analog signals were sampled at 1000 Hz,which is much more than what is necessary for theEDA signal, given its slowness. This signal was re-sampled to 25 Hz and filtered with a lowpass filterat 2Hz. This bandwidth excludes the effect of thepower line noise (50Hz), as well as the movementartifacts associated to key presses (only 0.0013 %

of the key press power had a frequency lower than2Hz).

Common EDA processing methods involve theseparation of the signal into tonic (SCL) and phasiccomponents (SCR), and several methods of differ-ent complexities, have been proposed to separatethem [19, 20]. While in some cases the chosenalgorithm can be important, e.g., in automatizedand real-time applications, here only basic methodswere applied to the post-acquisition EDA process-ing. The SCL was estimated using a moving averagefilter with a 10-second window, and the SCR wasdetected based on the first signal derivative and thedefinition of a threshold for minimum conductance.

ECGThe acquired ECG signals look like the ones repre-sented in Figure 4 overall, with the same color codeused in Figure 3a for the location of the sensors.

Figure 4: Acquired ECG derivations. The colorscheme corresponds to the one used in Figure 3afor sensor placement

To minimize the effects of power line interfer-ence, high-frequency noise and artifacts, baselinedrift and ECG waveforms other than the R peaks,a bandpass filter with cutoff frequencies of 10Hzand 40Hz was applied to the original signal. HRand HRV were calculated from the IBI after thedetection of the R peaks. Common methods in-clude threshold methods, wavelets and adaptive fil-ters [21]. Here the locations of the R peaks werecalculated by finding the local maxima of the nega-tive part of the filtered signal represented in red inFigure 4, in which the R peaks have a large nega-tive amplitude compared with the other waveformsof the same signal.

PSThe pupil detection algorithm used to process theimages of the eye, was developed by the creatorsof the eye tracker used. The advantage of this de-tector relative to the others, is that it is capable ofcalculating an ellipsoidal contour of the pupil, evenin the presence of eyelashes and corneal reflections[22], which are two major detection problems whenusing, e.g., binary threshold methods.

After the detection, the pupil vertical major axiscan be used as an estimate of pupil diameter, sinceit is the one that varies less with horizontal eye

4

movements relative to the position of the eye cam-era. The samples that corresponded to poor ellipsefitting were removed, as well as the ones associatedwith eye blinks . A median filter and a gaussianfilter were used to remove spike artifacts and jitteron the signal, respectively.

4. Long term signal dynamics

As described in the behavioral task section, eachsession is divided in 7 waves, with different levelsof difficulty. Difficulty depends on the number ofsimultaneous targets and their spacial distribution,and increases with wave number. Figure 5 showsthe normalized evolution of the heart rate acrosswaves. The result is normalized between 0 and 1(0 is the minimum value observed across the wavesof a session and 1 is the maximum), due to thedifferent scales observed across subjects. Averageinstantaneous heart rate was shown to be corre-lated with wave difficulty (correlation coefficient of0.952 ,p=0.001), as already observed [23]. This sug-gests that this increase in HR is a consequence ofincreased cognitive workload or a response to stress[24].

Figure 5: Evolution of the instantaneous heart rateacross waves for all the subjects. The black linerepresents the average of the normalized IHR forall the sessions of the subjects (total of 35 samples),and the gray area stands for the standard error ofthe mean

Heart rate variability didn’t show a significantcorrelation with difficulty (correlation coefficient of0.5862, p=0.1667), which suggests that HRV is notsensitive to this particular task. This is not in agree-ment with some of the existing literature [25, 26]that has observed a relation between lower HRVand higher cognitive load. Additionally, the HRVwas calculated using short time intervals (limited bythe length of waves, roughly 5 minutes), in whichits reliability remains questionable [27].

As shown in Figure 6, the average pupil size in-creases with wave difficulty (correlation coefficientof 0.879, p=0.009), which is consistent with the hy-pothesis that increasing PS is associated with work-ing memory and attention [28], two cognitive pro-cesses in which PS is very responsive to. Luminositychanges can also affect PS [29], with an increase inluminosity causing a decrease in the PS. However,in this task, luminosity increases along wave num-ber (due to the larger number of targets that have abrighter color relative to the background), as doesPS. This may be a sign that, in this task, luminos-ity is not the main modulator of PS, although itssmaller influence cannot be discarded completely.

Figure 6: Evolution of the pupil size across wavesfor all the subjects. The black line represents theaverage of the normalized PS for all the sessions ofthe subjects (total of 30 samples), and the gray arearepresents the standard error of the mean. Subject6 was excluded due to poor quality data.

In general, the SCR increasing with wave numberis consistent for all the subjects (correlation coeffi-cient of 0.9794, p=0.0001), as can be observed inFigure 7. The increasing of SCR events with dif-ficulty is in agreement with the observations madepreviously for the IHR, in terms of stress and cogni-tive workload. Nevertheless, because this index re-flects the total count, rather than the rate, of SCRevents in each wave, caution must be taken when in-terpreting these results, given that later waves arelonger. However we believe that number of trials,which are kept constant across waves, rather thanlength of time, is the main determinant of SCR oc-currence, as it will be shown next. The amplitude ofthe SCRs was negatively correlated with wave did-ficulty (correlation coefficient=-0.8019, p=0.0301),while SCL was positively correlated (correlation co-efficient=0.7979, p=0.0315). However the reliabil-ity of the SCL result has to be further investigatedgiven the occasional contact losses between the elec-trode and the skin that were observed in some cases,

5

which can affect the SCL value measured.

Figure 7: Evolution of the SCR number acrosswaves for all the subjects. The black line representsthe average of the normalized SCR number for allthe sessions of the subjects (total of 30 samples),and the gray area represents the standard error ofthe mean. Subject 7 was excluded due to poor qual-ity data.

5. Short term signal dynamics

When looking at the evolution of these signalsacross waves, we are only considering their averagevalue per wave, using only 7 points per session. Be-sides these long-term effects in the physiology, thereis a lot of variability within each wave that shouldnot be ignored, since it is a potential informationsource related with certain behavioral events. Fig-ure 8 shows an example of a full wave (5th) of onesubject, where the dashed vertical lines representthe 36 trial onsets. The physiological features rep-resented are the IHR, PS, SCR (high-pass filtered)and respiration cycle. All variables are standard-ized with Z-score.

Figure 8 allow us to see how much potential iscontained in these signals to physiologically charac-terize the different trials. In this example, it is clearthat IHR, PS and SCR are signals whose dynamicsalong time seem to be correlated with trials, pre-senting on average one peak per trial. Correlationof the trials with the respiration signal is not so ob-vious and may only be visible with further analysis,although it appears to have a rate of roughly 2 cy-cles per trial. Figure 9 shows another wave (1st)from the same subject’s session. Although thereseems to be still some periodicity in the signals, theIHR has a larger slow oscillation around the meanlevel, relative to the previous case. Additionallythe number of respiration cycles per trial is approx-imately one, which is less than the previous wave.Even more conclusive is the frequency of the SCRevents that do not align with every trial as apparent

for later waves. This supports the results obtainedin Figure 7, stating that earlier waves have fewerSCR events not because they are shorter in length,but because they are evoked by fewer trials.

Figure 8: Example of the dynamics of the signals(IHR, PS, SCR and respiration) during the trials ofa wave. This figure corresponds to a the 5th waveof the 3rd session of Subject 3. Blue dashed linesrepresent the onset of the trials and the red dashedline indicates the end of the wave.

Figure 9: Example of the dynamics of the signals(IHR, PS, SCR and respiration) during the trialsof a wave. This figure corresponds to the 1st waveof the 3rd session of Subject 3. Blue dashed linesrepresent the onset of the trials and the red dashedline indicates the end of the wave.

Figure 10 represents an average of these four sig-nals in a 10-second time window around trial onsetfor subject 5, separated by waves. The differentphase shifts are due to the fact that trials of laterwaves are longer, since they have more sequentialtargets. However they all align in a certain regiondelimited by the dashed rectangle. The points inthis region are thus invariant to the different timelengths of the trials and should be used to assessthe influence of a trial on the dynamics of the sig-nal. For example, for this subject, there is a hightendency of these signals to achieve a minimum at 3,2.7 and 6.2 seconds after the beginning of the trial,for PS, IHR and SCR respectively. For the respira-tion signal, it appears to be at roughly 6 seconds,although is not so clear.

6

(a) PS (b) IHR

(c) SCR (d) resp

Figure 10: Average of the standardized (Z-score)signals in a 10 second time window for Subject 5.The 0-second point represents the trial onset. Thesignals of different waves synchronize inside the re-gion delimited by the dashed gray rectangle, indi-cating a common dynamics in this interval of thetrial.

(a) PS (b) IHR (c) SCR

Figure 11: Summary of the dynamics of the signals(pupil size, instantaneous heart rate and skin con-ductance responses) at trial onset for the 1260 trialsof subject 5. For each plot, the y axis represents thepercentage of change relative to the baseline value,and the gray area is the standard error. The verti-cal line at 0 marks the trial onset, and the one at 5seconds is simply a reference point representing theending of the first targets of the trial.

Figure 11 summarizes the dynamics of the PS,IHR, SCR around the trial onset region. It corre-sponds to an average of the percentage of changerelative to baseline, of all the waves and sessions ofthe same example subject. In this figure, the first

targets of the trial grow within the interval from0 to 5 seconds, which is a time period common toall of the trials of different waves. Pupil size hasthe most consistent change at the onset of a trial,presenting a minimal error in the period where itdecreases after 0 seconds (by a factor of roughly11%), reaches a minimum at roughly 3 seconds andincreases after this (Figure 11a). Contrarily to PS,HR (Figure 11b) increases 1 second after the trialonset (by 1,2%) and is maximum at approximately2.7 seconds. In relation to SCR (Figure 11c), themost consistent dynamics of this signal is a decreasebetween approximately 3.5 seconds and 6.2 seconds.

Figure 12: Average percentage of change (relativeto baseline) of PS at the onset of a trial. Each linerepresents an average of the 1260 trials of each ofthe 6 subjects with valid PS signals

Figure 13: Average percentage of change (relativeto baseline) of IHR at the onset of a trial. Each linerepresents an average of the 1260 trials of each ofthe 7 subjects

A comparison between all the subjects’ signalsat trial onset are presented in the Figures 12, 13and 14 for PS, IHR and SCR respectively. The PS

7

and SCR signals are very consistent across subjects,both in amplitude and phase. This suggests that al-though inter-subject variability is present in severallevels (e.g. average values per sessions and waves),there are cases where the signals from different sub-jects demonstrate similar dynamics, as is the caseof the reaction to trial onset. However, IHR seemsto demonstrate larger variability among subjects.

Figure 14: Average percentage of change (relativeto baseline) of SCR at the onset of a trial. Eachline represents an average of the 1260 trials of eachof the 6 subjects with valid SCR signals

6. ConclusionsThe general objective of this thesis was to intro-duce psychophysiological measures in a new taskdeveloped for studying complex decision-making inhumans. One of the goals accomplished was theassembly of a muiti-modal acquisition setup involv-ing different equipments (including new instrumen-tation devices) for measuring several physiologicalvariables, namely electrocardiogram, electrodermalactivity, pupil size, eye movements, respiratory pat-terns and key pressure. The acquisition of these sig-nals with high quality is extremely important giventhe following objective of relating estimates of in-ternal states to the ongoing decision process withinshort time intervals.

The preliminary results obtained from seven sub-jects demonstrate that variations in psychophysio-logical indexes, such as HR, PS and SCR, are sen-sitive to the characteristics of this new task in longand short time scales, thus extending beyond fairlysimple experimental approaches used to study iso-lated cognitive processes. An increase in the diffi-culty level was shown to cause an increase in eachof the three signals.

In a short time scale, the signals demonstrated atendency to align with defined periods of the task,such as trials onset. This suggests that trials inducechanges in the dynamics of these signals that arereproducible across sessions and subjects.

The work presented here may be considered as astarting point from which we will increase the depthof the analysis in order to explain physiological pat-terns and their relation to the large number of be-havioral states that potentially exist in this task.Due to the multidimensional nature of this prob-lem, statistical machine learning algorithms such asprincipal component analysis and neural networksmay be needed to extract meaningful informationabout multiple dimensions of behavior, that is en-coded in the physiological signals’ dynamics. Fi-nally, one of the major interests in this research isto understand how humans use efficient shortcuts tosolve complex tasks such as planning. By lookingat these multidimensional signals at a single-triallevel, we may eventually be able to reconstruct therepresentations that subjects use when constructinga plan.

References[1] A. Rangel, C. Camerer, and P. R. Montague,

“A framework for studying the neurobiology ofvalue-based decision making,” Nature ReviewsNeuroscience, vol. 9, no. 7, pp. 545–556, 2008.

[2] P. Cisek, “Cortical mechanisms of action selec-tion: the affordance competition hypothesis,”Philosophical Transactions of the Royal Soci-ety of London B: Biological Sciences, vol. 362,no. 1485, pp. 1585–1599, 2007.

[3] H. R. Heekeren, S. Marrett, and L. G. Unger-leider, “The neural systems that mediate hu-man perceptual decision making,” Nature re-views neuroscience, vol. 9, no. 6, pp. 467–479,2008.

[4] W. T. Newsome, K. H. Britten, and J. A.Movshon, “Neuronal correlates of a perceptualdecision.,” Nature, vol. 341, pp. 52–54, 1989.

[5] H. R. Heekeren, S. Marrett, P. A. Bandettini,and L. G. Ungerleider, “A general mechanismfor perceptual decision-making in the humanbrain,” Nature, vol. 431, no. 7010, pp. 859–862,2004.

[6] R. Ratcliff and G. McKoon, “The diffusion de-cision model: theory and data for two-choicedecision tasks,” Neural computation, vol. 20,no. 4, pp. 873–922, 2008.

[7] D. Lee, H. Seo, and M. W. Jung, “Neural ba-sis of reinforcement learning and decision mak-ing,” Annual review of neuroscience, vol. 35,p. 287, 2012.

[8] J. Balaguer, H. Spiers, D. Hassabis, andC. Summerfield, “Neural mechanisms of hier-archical planning in a virtual subway network,”Neuron, vol. 90, no. 4, pp. 893–903, 2016.

8

[9] W. G. Chase and H. A. Simon, “Perceptionin chess,” Cognitive psychology, vol. 4, no. 1,pp. 55–81, 1973.

[10] D. Tranel, “Preface: Psychophysiology andcognitive neuroscience,” International Journalof Psychophysiology, vol. 61, no. 1, pp. 1–4,2006.

[11] J. Cacioppo, G. Berntson, and J. Larsen, “Thepsychophysiology of emotion,” in The Hand-book of Emotion, ch. 11, pp. 173–191, GuilfordPress, 2nd ed., 2000.

[12] J. M. Hinson, T. L. Jameson, and P. Whitney,“Somatic markers, working memory, and deci-sion making,” Cognitive, Affective, & Behav-ioral Neuroscience, vol. 2, no. 4, pp. 341–353,2002.

[13] A. L. Hansen, B. H. Johnsen, and J. F. Thayer,“Vagal influence on working memory and at-tention,” International Journal of Psychophys-iology, vol. 48, no. 3, pp. 263–274, 2003.

[14] A. Bechara, H. Damasio, D. Tranel, and A. R.Damasio, “The iowa gambling task and the so-matic marker hypothesis: some questions andanswers,” Trends in cognitive sciences, vol. 9,no. 4, pp. 159–162, 2005.

[15] E. Salvia, A. Guillot, and C. Collet, “Auto-nomic nervous system correlates to readinessstate and negative outcome during visual dis-crimination tasks.,” International Journal ofPsychophysiology, vol. 84, no. 2, pp. 211–218,2012.

[16] G. Lopes, N. Bonacchi, J. Frazao, J. P. Neto,B. V. Atallah, S. Soares, L. Moreira, S. Matias,P. M. Itskov, P. A. Correia, et al., “Bonsai: Anevent-based framework for processing and con-trolling data streams,” Frontiers in neuroin-formatics, vol. 9, p. 7, 2015.

[17] C. S. Green and D. Bavelier, “Action videogame training for cognitive enhancement,”Current Opinion in Behavioral Sciences, vol. 4,pp. 103–108, 2015.

[18] MATLAB, version 8.5.0 (R2015a). Natick,Massachusetts: The MathWorks Inc., 2015.

[19] C. L. Lim, C. Rennie, R. J. Barry, H. Bahra-mali, I. Lazzaro, B. Manor, and E. Gordon,“Decomposing skin conductance into tonic andphasic components,” International Journal ofPsychophysiology, vol. 25, no. 2, pp. 97–109,1997.

[20] M. Benedek and C. Kaernbach, “A contin-uous measure of phasic electrodermal activ-ity,” Journal of neuroscience methods, vol. 190,no. 1, pp. 80–91, 2010.

[21] B.-U. Kohler, C. Hennig, and R. Orglmeis-ter, “The principles of software qrs detection,”IEEE Engineering in Medicine and BiologyMagazine, vol. 21, no. 1, pp. 42–57, 2002.

[22] M. Kassner, W. Patera, and A. Bulling,“Pupil: an open source platform for pervasiveeye tracking and mobile gaze-based interac-tion,” in Proceedings of the 2014 ACM interna-tional joint conference on pervasive and ubiqui-tous computing: Adjunct publication, pp. 1151–1160, ACM, 2014.

[23] I. R. Gellatly and J. P. Meyer, “The effects ofgoal difficulty on physiological arousal, cogni-tion, and task performance.,” Journal of Ap-plied Psychology, vol. 77, no. 5, p. 694, 1992.

[24] H. Ursin and H. R. Eriksen, “The cognitiveactivation theory of stress,” Psychoneuroen-docrinology, vol. 29, no. 5, pp. 567–592, 2004.

[25] S. Mukherjee, R. Yadav, I. Yung, D. P. Zajdel,and B. S. Oken, “Sensitivity to mental effortand test–retest reliability of heart rate variabil-ity measures in healthy seniors,” Clinical Neu-rophysiology, vol. 122, no. 10, pp. 2059–2066,2011.

[26] C. D. B. Luft, E. Takase, and D. Darby, “Heartrate variability and cognitive function: Ef-fects of physical effort,” Biological Psychology,vol. 82, no. 2, pp. 186–191, 2009.

[27] G. R. Sandercock, P. D. Bromley, and D. A.Brodie, “The reliability of short-term measure-ments of heart rate variability,” Internationaljournal of cardiology, vol. 103, no. 3, pp. 238–247, 2005.

[28] D. Kahneman and J. Beatty, “Pupil diame-ter and load on memory,” Science, vol. 154,no. 3756, pp. 1583–1585, 1966.

[29] E. H. Hess, P. W. Beaver, and P. E. Shrout,“Brightness contrast effects in a pupillomet-ric experiment,” Perception & Psychophysics,vol. 18, no. 2, pp. 125–127, 1975.

9