and the winner is…...1 and the winner is… developing a computer program to investigate neural...
TRANSCRIPT
1
And the Winner is…
Developing a Computer Program to Investigate Neural Competition with Multimodal Stimuli
A Computer Science Project with Behavioral Applications
Presented to:
Missouri Junior Science Engineering and Humanities Symposium
University of Missouri
Columbia, Missouri
By
Caleb Jireh Martonfi
Tuscumbia High School
526 School Rd.
Tuscumbia, MO 65082
Mrs. Constance Wyrick
Science Research Advisor
Tuscumbia, MO 65082
2
Abstract Name: Caleb Martonfi School: Tuscumbia High School Sponsoring Teacher: Mrs. Constance Wyrick Project Title: And the Winner is… Developing a Computer Program to Investigate Neural Competition with Multimodal Stimuli
Neural competition occurs when two stimuli fight for representation. One form of neural competition occurs when two stimuli within a single modality compete for embodiment. A form of bimodal neural competition occurs when visual and auditory stimuli compete. The engineering goal of this project was to create a series of computer programs using an icon-based programing language to test neural competition. A Raspberry Pi was purchased on which to run the programs. Scratch, an icon-based programming language, was used to construct the programs. The engineering goal of this project was accomplished. Three different unimodal programs were created. Each of the programs tested different aspects of perception and attention. Two bimodal programs were constructed. One of the five programs was selected for further development. It could be used for the testing of bimodal neural competition with speed perception.
3
Neural Competition
Today’s society is constantly assaulted with many different stimuli. Sorting out these
sensations is difficult for many people. Often, combining two stimuli can result in unique
illusions. However, in many situations, such as driving a car, distractions from competing
stimuli- such as cell phones, radios, music, and even friends- can be hazardous. In order to
avoid tragedies such as these, neural competition must be better understood. Neural
competition occurs when two stimuli compete for representation.
One form of neural competition occurs when two stimuli within a single modality fight
for embodiment. The biased model of visual attention suggests that the visual attention selects
one stimulus over another. Change blindness and tunnel vision are two examples of single
modality neural competition.
A form of bimodal neural competition occurs when visual and auditory stimuli compete.
This can be demonstrated by examining the mechanism involved in conversation. In speech,
both linguistic and paralinguistic cues are used to decipher intent and emotion. ‘Linguistic’
refers to the actual denotation of the sentence. ‘Paralinguistic’ refers to the facial expressions,
intonation, body posture, and physical gestures involved in communication. Together, these
cues can be used to interpret the emotion behind a sentence. When these cues don’t line up,
the brain has to decide what to believe. For example, in the McGurk effect, visual and auditory
stimuli convey contrasting messages. For example, a speaker mouthing “FA” and a sound
playing “BA” would show the McGurk effect. The result is an integration of the two. However,
perception is biased towards visual stimuli [Pollack, 2005], so an onlooker would most likely
4
perceive something closer to “FA”. This is another indication that the integration of audio and
visual stimuli must occur before decision making. The Fuzzy Logical Model of Perception
suggests that bimodal perception comprises of evaluation of each individual stimulus,
integration of the stimuli, and decision making regarding which alternative is supported.
However, “…most individuals show perceptual and attentional biases towards visual stimuli”
[Pollack, 2005].
Other studies have been done regarding the speed at which processing is done.
Perceptual load theory proposes that the time that it takes perception to occur is not constant.
It depends on the perceptual demands of the task. Perceptual load refers to the attentional
demands required to perceptually recognize one thing from another. When distractors are
present, a task that is particularly strenuous leaves less room for the distractors to interfere, so
perception occurs quickly, while tasks that are not demanding leave more room for distraction,
so perception takes longer. For example, detecting color alone is easier than detecting color
and the orientation of an object.
In our technologically advancing world, there are more and more stimuli demanding our
attention. It is become increasingly evident how important it is to understand our brain’s
reaction to this neural competition. There is still much to be learned about how modalities
interact, and how distractors interfere with neural processing.
5
Programming Languages
Many programming languages are available for creating animation programs for
behavioral research and professional work. These include C++, Java/Flash, JavaScript, Python,
Unreal Engine, XNA Game Studio, Dark Basic, GameMaker, and Scratch. This research used
Scratch as its primary programming language. Scratch is an icon-based programming language
developed by the Massachusetts Institute of Technology (MIT).
6
Question Posed
Is it possible to create a series of computer programs using an icon-based programing
language to test neural competition?
Engineering Goal
The engineering goal of this project was to create a series of computer programs using
an icon-based programing language to test neural competition. The programs were developed
for testing the following types of neural competition: unimodal (within the same modality, e.g.
visual/visual) and bimodal (between two modalities, e.g. visual/auditory).
Hypotheses
The following hypotheses were formed:
1.) It would be possible to create at least one program testing the effect of
unimodal neural competition.
2.) It would be possible to create at least one program testing the effect of bimodal
neural competition.
3.) It would be possible to create at least one fully functional program. This program
should be ready for research use.
7
Engineering Process
Designing software usually involves planning, a feasibility study, product design, coding,
implementation/integration, testing, and installation/maintenance. Because the goal of this project was
to develop a series of programs that could test neural competition, not to create market-ready software,
implementation/integration and installation/maintenance were not part of the engineering process.
Testing and selection, however, were included in the design.
Planning:
The goal of this project was to create a series of computer programs using an icon-based
programing language to test neural competition. The first step to accomplishing this was the planning
phase. Many ideas were formed about what the programs should execute. In the end it was decided
that the programs would need to be able to test unimodal and bimodal neural competition. Such
programs would implement and/or examine the following: auditory stimuli with visual distractors, such
as increasing and decreasing pitches; visual stimuli with multiple visual distractors; speed perception of
auditory or visual stimuli, and memory with visual distractors. To conduct this study, an icon-based
programing platform called Scratch was used.
Feasibility:
For some researchers in the behavioral science, their options are limited to buying someone
else’s program with which to test neural competition. This becomes cost prohibitive, especially for
student researchers. In addition, the purchased program may not allow for flexibility when designing
the research. The ability to create one’s own program would be very beneficial. To examine the
feasibility of this project, various factors were examined. Availability, cost, time, and functioning were all
considered. Foremost, the availability and cost of Scratch were surveyed. Scratch was developed by the
8
Lifelong Kindergarten group at the Massachusetts Institute of Technology (MIT). Because it is available
on PC, Macintosh, and Linux operating systems, obtainability was not a limitation. Scratch is free to
everyone, so cost was not an issue either. Next, the time to develop functioning programs was
contemplated. Because the researcher had already been around Scratch for over four years, functioning
wasn’t a question. Scratch could be used to create complex programs using an icon based-platform.
Because the language was already understood, the time to create such programs was minimal.
Due to the nature of behavioral research, portability was also desired. Purchasing and using a
Raspberry Pi solved the problem. A Raspberry Pi is a credit card sized computer designed to help novice
individuals learn programing. Several programing platforms are preinstalled onto it, such as BlueJ Java,
Python 2 and 3, Mathematica, Node-RED, Sonic Pi, Wolfram, and Scratch. A Raspberry Pi only costs
about twenty-five dollars, and it can control external hardware. Because of the availability, low cost,
efficiency in time management, and reliable functioning, a Raspberry Pi was selected as the computer of
this project. A pre-owned computer monitor was purchased under ten dollars. A leftover school
keyboard and mouse were used. A wireless handheld keyboard was also purchased for its remote
feature.
Product Design:
This step was very similar to the planning phase. However, specific goals were set. The following
programs were envisioned.
1.) To test unimodal neural competition:
a.) A program with numerous visual distractors moving across the screen, to test a subject’s
attention to a specific visual tracking task with distractors;
9
b.) A program with a small, color changing dot amid changing background scenes, to investigate
change blindness;
c.) A program with alternating flashing colored objects with memory words, to investigate a
test subject’s association of color with words, to better understand visual memory.
2.) To test bimodal neural competition:
b.) A program to test the accuracy of subject’s viewing descending and ascending pitches
(auditory stimuli) conflicting with descending and ascending or lines (visual stimuli), to
investigate auditory illusions;
c.) A program to test the accuracy of a subject presented with conflicting speeds of auditory and
visual stimuli, to test visual speed perception with auditory distractors, and auditory speed
perception with visual distractors.
Coding:
In Scratch, programs are created by dragging and dropping icons that fit together based on their
functions. Individual characters are called sprites. Each sprite can be programmed to do specific things.
“Hiding” something refers to the object disappearing from view.
In the first program, a subject’s attention to a specific visual tracking task with distractors was
tested. To fashion a program to do this, five sprites were created. Figure 4. shows each of them. Three
sprites were in the form of beach balls, and were equally spaced across the screen. All but one of them
were hidden. The unhidden beach ball started at the top left of
the screen and moved downward. When it reached the bottom, it
hid itself, and broadcasted for the next beach ball to appear and
start moving down the screen. It then jumped back to the top of
the screen ready to start the process anew. Moving from left to
Figure 1.
10
right across the middle of the screen was a little blue car. Once it reached the edge, it would jump back
to the other side of the screen and immediately start moving again. After it had done that 3 times, it
would broadcast to an airplane sprite at the top of the screen to appear and start moving. The airplane
traveled across the screen once and disappeared. A subject participating in this study might be asked to
count how many times two of the five objects touched the edges of the screen. Results would be
evaluated based on the accuracy of the subjects counting. Variables that could be manipulated include
the type of object crossing the screen, the number of objects crossing the screen, or the visual
orientation of the object/s being observed. This was a unimodal test.
The second program that investigated change blindness
was created with a small color changing dot amid changing
background scenes. First, eleven different backgrounds of
random indoor/outdoor/sports/nature backgrounds were
selected. Figure 2. shows one of the possible backgrounds. They
were programmed to alternate every 0.5 seconds. A small, yellow dot was placed in the center of the
screen. Each time the backgrounds changed scenes, the small dot gradually changed color from yellow
to green to blue to purple to red to orange and back to yellow. The test subject would not be alerted to
the changing dot. They would only be asked to count the number of backgrounds. They would then be
asked to recall the order of the colors of the dot, if indeed they noticed the color change at all. This
program could test one of two things: change blindness (an individual’s inability to observe a change) or
memory with visual distractors. This was a unimodal test.
To create the third program that could investigate a test
subject’s association of color with words, to better understand visual
memory, three circular sprites were created. One circle was red,
Figure 2.
11
another was white, and the third was blue. Figure 3. shows the position of the circles. Each circle was set
to spin. Each circle was created with a list of four words. This program was coded very similar to the one
before it. One circle would appear, disappear, change the word within itself, and then broadcast the
next circle to start the process. There was a total of twelve words. To test this program, the subject
would be asked to try to remember as many words as possible. The program would show all of the
words twice, and then stop. The subject would then record the words they remembered on a sheet with
the three different colors. Results would be analyzed based on the subject’s accuracy and how many
words they recalled. This was a unimodal color-association memory test.
The fourth program that was constructed contained
ascending and descending lines and pitches, and investigated
auditory illusions. Four sprites in the shape of lines were placed
equidistance from each other, but at different elevations.
Figure 4. shows the position of the sprites. All the sprites were “hidden”. When the green flag was
pressed, “One” was broadcast to all other sprites. Sprite #1 received “One” and began executing its
code. First, it presented itself, near to top edge of the screen. It then sounded a high pitched note, and
hid itself again. “Two” was then broadcasted. When sprite #2 received “Two”, it presented itself. It had a
slightly lower y-coordinate than sprite #1. Sprite #2 sounded a pitch slightly lower than the preceding
note. It then disappeared from the screen. Next, “Three” was broadcasted. The same process was
followed for the next two sprites. In addition, if the down arrow key was pressed, it caused the program
to start on a low pitch, and ascend in pitch as the lines descended. Note, even though the ascent or
descent of the auditory stimuli varied, the visual stimuli always appeared from the top of the screen to
the bottom, and always from left to right. To use this program in a study, the subject would be asked to
listen for changes in pitch while visually observing descending lines. The auditory stimulus’ “direction”
Figure 4.
12
would be changed using the arrow keys on a wireless keyboard, to eliminate any cues that the program
had changed. Afterward, the subject would be asked to determine if the pitch was going up or down. If
the lines are going down and the pitch is going up and the subject responds that the pitch is going down,
then an auditory illusion has been created, because the visual modality overrode the auditory modality.
Studies using this program could test an individual’s ability to determine the direction of the pitch versus
another’s ability, such as a musician versus a nonmusical individual, an auditory learner versus a visual
learner, or male versus female. This was a bimodal test.
The goal of the fifth program was to test the effect conflicting speeds of auditory and visual
stimuli on rate perception. To do this, a basketball sprite and a soccer ball sprite were placed on the
screen, with the soccer ball sprite hidden. The basketball sprite speed could be adjusted manually to
affect the visual perception. The basketball would bounce and move in the opposite direction when it
hit an edge. Every time the soccer ball sprite hit a wall, it made a slapping sound. The speed of the
soccer ball sprite could be manually adjusted to affect auditory
perception. The two balls always started in the same place, so
when they were set to travel the same speed, the auditory
stimulus was synchronous with the visual stimuli. Figure 5.
shows the motion of the basketball across the screen. To carry out a study using this program, different
combinations of the speed of the ball and the speed of the clicks would be chosen. The time it takes the
basketball to cross the screen from edge to edge at different would be measured using a Scratch script.
To start, the subject would be shown a reference. The ball in this demonstration would take 1 second to
cross the screen. The subject would then be asked to record how fast they thought the ball took to
travel from one edge of the screen to the other. Results would be analyzed based on how accurately the
subject answered the questions. This was a bimodal test.
Figure 5.
13
Testing:
To test the programs, one by one they were run. Bugs were removed or fixed. Any necessary
adjustments were made.
Selection:
To select one of these programs to use for testing, Dr. Nelson Cowan; a neural-cognitive
researcher at the University of Missouri, was consulted. With his assistance, each program was analyzed
for practicality, controllability, and uniqueness.
The program with the bouncing basketball was selected as the best fit for further development.
It had the potential to be used to test neural competition between visual and auditory modalities. This
program was practical. The speed of the basketball could easily be changed. Experiments completed
using it were controllable, because the only independent variables changed are the speed of the ball and
the speed of the click, and the only dependent variable measured was the test subject’s perception of
the speed. Many of the other programs had too many distractors
and too much motion to be able to accurately pinpoint the effect
of one stimulus. Outside factors could easily affect the results.
This program is unique, because few studies have been done on
the topic of bimodal neural competition and speed perception.
Coding:
After this selection was made, several adjustments were made to aid in the actual testing of
participants. To start the modified program, the letters “G” and “O” were pressed. This displayed a
passage stating: “The first demonstration you will see is a reference. It takes the ball exactly 1 second to
Figure 6.
14
travel from one edge of the screen to the other. Use this to
help you on the questions that follow.” Figure 6. shows
that passage. The letters “N” and “X” (for next) are then
pressed to show the reference. A program was previously
used to determine how fast the ball would have to travel to
cross the screen in one second. After the ball bounces 6 times, another passage appears. “This is the
start of the first section. The ball will travel at different random speeds. Use the left and right arrow keys
to move the scale to how fast you think it took the ball to cross the screen from edge to edge. Press
space to submit your answer.” After “N” and “X” are pressed, the program chooses one out of twelve
conditions: fast ball and fast clicks, fast ball and medium clicks, fast ball and slow clicks, medium ball and
fast clicks, medium ball and medium clicks, medium ball and slow clicks, slow ball and fast clicks, slow
ball and medium clicks, slow ball and slow clicks, fast ball with no clicks, medium ball with no clicks, and
slow ball with no clicks. Fast stands for a speed of 0.5 second between wall/click, medium stands for a
speed of 1 second, and slow stands for a speed of 1.5 seconds. The condition was recorded in a list.
After the ball bounced ten times, it disappeared and a scale appeared on the screen. The user could
control a slider on the scale using the arrow keys and submit using the space button. Using an equation
to convert the x-coordinates of the slider to seconds, the user input was recorded in a list, after the
condition. Figure 7. shows the scale. A new condition was then chosen. After all of the twelve conditions
were completed, a third passage appeared. “You will now hear a reference that clicks every second.”
The clicks proceeded without a ball. A fourth passage was presented. “This is the start of the second
section. Instead of asking about the speed of the ball, you will now be asked about the speed of the
clicks. Use the scale as before.” This section tested the subject’s perception of the speed of the clicks. All
of the conditions remained the same except the conditions with the fast ball with no clicks, the medium
ball with no clicks, and the slow ball with no clicks. These were changed to no ball and fast clicks, no ball
15
and medium clicks, and no ball and slow clicks. After these twelve conditions were carried out, the
program displayed “You are finished. Thank you for participating!”, and ended. To view the list of
conditions and times, a box could be checked to display them. Testing with this program would be
simple. The subject would only have access to a keyboard. The progression of the program (e.g. “NX”)
would be controlled by the researcher using a wireless keyboard. The program would start with the
passages and explain what the subject was expected to do. The subject would use the arrow keys to
control the scale. The results would be recorded after the program finished running.
Testing:
The testing phase was repeated after modifications were made to the program. This program
could be used for the testing of neural competition.
16
Figure 1.
Conclusions
The engineering goal of this project was accomplished. A series of computer programs
to test neural competition were created successfully. Programs for testing unimodal and
bimodal neural competition were effectively developed.
The following conclusions can be drawn:
1.) The hypothesis stating “It will be possible to create at least one program testing the
effect of unimodal neural competition,” was supported. Three different unimodal
programs were created. Each of the programs tested different aspects of perception
and attention. See Figure 1.
2.) The hypothesis stating “It will be possible to create at least one program testing the
effect of bimodal neural competition,” was supported. Two bimodal programs were
constructed. See Figure 2.
17
3.) The hypothesis stating “It will be possible to create at least one fully functional program.
This program should be ready for research use,” was supported. One of the five
programs was selected for further development. This was the program that tested the
accuracy of a subject presented with conflicting speeds of auditory and visual stimuli. It could
be used for the testing of bimodal neural competition. See Figure 3.
Figure 2.
Figure 3.
18
Future Studies
The overall goal of this project was to create one fully functional program that would be ready
for research use. This was achieved with the bouncing ball program. In the future, this program will be
used by the researcher to test the effect of bimodal neural competition on speed perception.
19
Acknowledgements
I would like to thank the following people:
Dr. Nelson Cowan- Curators Distinguished Professor of Psychology, for his expertise in reviewing
the various computers developed to test neural competition.
Mrs. Constance Wyrick- my research advisor,
Steve and Robin Martonfi- my parents.
20
Works Cited
Chun-Yu Tse, G. G. (2015). Read My Lips: Brain Dynamics Associated with Audiovisual
Integration and Deviance Detection. Journal of Cognitive Neuroscience, 1723-1737.
Fogerty, K. G. (2015, September 17). Integration of Partial Information Within and Across
Modalities: Contributions to Spoken and Written Sentence Recognition. (N. Tye-Murray,
Ed.) Journal of Speech, Language, and Hearing Research, 1805-1817.
Nathan A. Parks, M. R. ( 2011). Steady-state Signatures of Visual Perceptual Load, Mulitmodal
Distractor Filtering, and Neural Competition. Jounal of Cognitive Neuroscience, 1113-
1124.
Pedram Daee, M. S. (2014, July 24). Reward Maximazation Justifies the Transition from Sensory
Selection at Childhood to Sensory Integration At Adulthood. (R. J. Beers, Ed.) PLoS ONE,
9(7), 1-13.
Pollak, J. E. (2005). Experiential Influences on Multimodal Perception of Emotion. Child
Development, 1116-1126.
Ternaux, J.-P. (2003). Synesthesia: A Multimodal Combination of Senses. Leonardo, 321-322.
21
Appendix for the Bouncing Ball Program Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 3, CPU)
22
The program starts here. Go to
page 37. Make sure that you
read all of the code before you
move on.
Go to page 32 after returning
from Words Start.
23
After Start, go to page 32
and 33.
After all twelve of the
conditions are completed,
go to page 34.
24
25
26
27
28
29
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 1, Scale)
After Start Scale, go to page 22.
30
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 2, Scale Slider)
After Start Scale, go to page 22.
31
32
33
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 4, Basketball)
After Demo, also view page 34.
Visit page 16. Then go to page
22.
After Start Basketball, go to
page 28 and 29.
34
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 5, Soccer Ball for Clicks)
After Start Basketball, go back
to page 28.
After Click Demo 2, go back to
page 34.
35
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 6, Question)
After Start Clicks, go to page 40.
After returning from Click
Demo, go to page 33.
After returning from Click
Demo 2, go to page 38.
After returning from Words
start clicks, go to page 22. If all
of the click conditions are
completed, go to page 39.
36
37
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 7, Passage #2)
38
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 8, Passage #1)
Go to back to page 21.
39
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 9, Passage #4)
After Words start clicks, go to
page 34.
40
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 10, Finale)
This is the end of the program.
41
Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 11, Passage #3)
After Click Demo, go to page 14.
42
Views of Sprites
Sprite 8, Passage #1
Sprite 7, Passage #2
43
Sprite 1 & 2, Scale and Scale Slider
Sprite 6, Question
Sprite 11, Passage #3
44
Sprite 9, Passage #4
Sprite 4, Basketball
Sprite 10, Finale