News

News Home » A Study on New Gameplay Based on Brain-Computer

A Study on New Gameplay Based on Brain-Computer

Breaking New Ground: Innovation in Games, Play, Practice and Theory. Proceedings of DiGRA 2009

© 2009Authors & Digital Games Research Association (DiGRA). Personal and educational classroom use of this paper is allowed,

commercial use requires specific permission from the author.

A Study on New Gameplay Based on Brain-Computer

Interface

Minjin Ko

Division of Media,

Ajou University

jammed@ajou.ac.kr

Kyoungwoo Bae

Division of Media,

Ajou University

rioklar@hotmail

Gyuhwan Oh

Division of Media,

Ajou University

drghoh@ajou.ac.kr

Taiyoung Ryu

Interactive Media

Division, University

of Southern

California

tryu@usc.edu

ABSTRACT

Brain-Computer Interface (BCI) is a way to control

computers by using human brain waves. As the technology

has improved, BCI devices have become smaller and

cheaper, making it possible for more individuals to buy

them. This allows BCI to be applied to new fields outside of

pure research, including entertainment. We examine

whether BCI devices can be used as a new gaming device,

approaching it from a game design perspective. We propose

game play elements that can effectively utilize BCI devices

and present a game prototype that demonstrates several of

these game play elements. Next, we use statistical data

analysis to show that using a BCI device as well as

keyboard and mouse interfaces makes the game’s control

clearer and more efficient than using the traditional input

devices. The results offer guidelines for effective game

design methodology for making BCI based games.

Author Keywords

BCI, Game Design, Intuitive, Gameplay

INTRODUCTION

A Brain-Computer Interface (BCI) is any hardware which

enables humans to interact with a computer using their

brainwaves. Currently, BCI is utilized for the healthcare

and education based on neuro-feedback technology. Neurofeedback

is a kind of bio-feedbacks using brainwaves.

Usually, bio-feedback is a medical treatment controlling the

reactions in the autonomic nervous system with blood

pressure, body temperature and pulse frequency.

Specifically, neuro-feedback is used for testing and

improving brain capacity or brain characteristic [3, 5].

Figure 1:

Neurocybernatics Inc.)

Because early BCI devices were quite large and expensive,

they were used primarily by medical institutions doing

research on brain-related subjects like analysis of

brainwaves for helping medical treatment. Figure 1 shows a

medical treatment brainwave system manufactured by

Neurocybernatics Inc. [9]. Now that BCI devices are

becoming smaller and less expensive, it’s possible to create

programs for individual users and to target broader

experiences such as gaming and multimedia.

A great deal of research has been conducted on

methodologies for utilizing BCI. Researcher Matthew

Middendorf has introduced the hands-free control system

with electroencephalography (EEG) which is performed by

The Air Force Research Laboratory. He has also shown that

users can, with practice, learn to control their brainwave

signals [8]. In his experiment, he EEG is acquired using

gold-cup electrodes located over occipital sites O1 and O2

(left mastoid as ground). The differential signal between O1

and O2 is amplified, filtered, and then processed by a lockin

amplifier system (LAS) that provides a measure of the

SSVER amplitude. The roll position of a simple flight

simulator was controlled with variances of SSVE

amplitudes over time.

2

Jaime A. Pineda, etc has also shown that users can, with

practice, learn to control their brainwave signals [12]. In his

experiment the game character’s movement was based on

two particular brainwaves, one from the left side of the

user’s brain and one from the right side of brain. If the two

signals were approximately equal the character would move

to the left and if the levels were different, the character

moved to the right. The experiment carried out twice a

week for 5 weeks, twice a week for 45 minutes. Over the

course of the experiment users became increasingly skilled

at controlling their character.

“NeuroComp System(manufactured by

Figure 2:

boards of differentf frequencies on the bottom.

Based on these results several researchers, including E.

Lalor, decided to look at whether it would be possible to

use BCI based on Steady-state Visual Evoked Potential

(SSVEP) in a game framework in real time [6]. SSVEP is a

group of signals created by the brain in response to a certain

frequency of visual stimulus. Different frequencies for each

visual stimulus will generate different SSVEPs. In the first

step of the experiment, the researchers showed users a

checkerboard with squares that alternated between white

and black at stated intervals. Researchers measured the

resulting signals generated according to the intervals. After

that, the user was asked to play a game designed for the

experiment. The game contained checkerboards that

changed at the same intervals as the checkerboard on the

first step as seen on the Figure 2. Based on SSVEP, it was

possible to determine which checkerboard the user was

looking at simply by analyzing their brainwave signals. The

game character would then tilt the selected checkerboard in

real time. The experiment showed that BCI technologies

based on SSVEP could be used to measure user responses

in real-time and control a game environment.

Bredan Z. Allison, etc evaluates the hypothesis that

overlapping stimuli can evoke changes in SSVEP activity

sufficient to control a BCI [1]. According to their results,

there was no significant correlation between EEG measures

and gender, age, substance use. Also, they explored that

subjects who have a background playing video games or

performing similar activities might be better suited to

certain types of BCIs and subjects could be trained to

perform better on tasks of visual attention. They also

conclude that training with other tasks requiring selective

attention, such as playing certain types of computer games,

may also improve performance with SSVEP.

Although a great deal of research has currently been

conducted on methodologies for utilizing BCI, most of

these are focused on methods for applying BCI to

applications or low-level technology. Research on BCI

from a game design perspective to suggest optimal game

play for BCI devices is still quite rare.

In the paper, we examine whether BCI devices can be used

as a new gaming device, approaching it from a game design

perspective. We first propose game play elements that can

effectively utilize BCI devices and then present a game

prototype that demonstrates several of such game play

elements. Finally, we use statistical data analysis to show

that using a BCI device as well as keyboard and mouse

interfaces makes the game’s control clearer and more

efficient than using the traditional input devices. The results

offer guidelines for effective game design methodology for

making BCI based games.

A screen of Mind Balance. There are two checker

GAME PLAY ELEMENTS UTILIZING BCI

In order to illustrate game play elements that take advantage

of BCI, we’ll need to first look at effective traditional game

play elements and define what kind of information can be

transferred from the user to the computer via BCI.

Definition of brainwave information transferred by BCI

Figure 3:

Comparing their prices and sizes, three portable BCI

devices are currently available or will be available soon for

applying gaming industry, NIA, Emotive EPOC, and Mind

Set. Figure 3 shows the images of the three portable BCD

devices.

These three emotions were chosen because they’re

detectable by most of the small-sized, inexpensive BCI

devices that are currently available or will be available

soon. The information perceived by typical BCI devices is

summarized in Table 1 [4, 10, 11]. Two emotions, attention

(or immersion) and meditation (relaxation) are common

human emotions and EPOC and Mind Set can also response

both of these emotions. In the paper, we will consider how

these two emotions, attention and meditation are applied for

designing a game in the following sections.

(To the left) NIA, Emotive EPOC, Mind Set

Table 1:

3

Data perceived by three portable BCI devices

BCI

Device Data Perceived by Device

NIA -Expressions : wink, frown, laughing

EPOC

-Emotions : immersion, relaxation, tense, boredom,

excitement, frustration

-Expressions :

Furrow, Smile, Clench, Smirk

Mind Set -Emotions : attention, meditation, anxiety, drowse

Blink, Wink, Look, Eyebrow,

Analysis of traditional game play elements

To utilize BCI devices as new input devices, it is necessary

to explore game play elements in existing games. Though

games employing new input devices often contain new

game play that replace traditional game play elements, new

game play elements are frequently based on older ones. For

example, the ‘Nintendo Wii’ which adopted a remote

controller mounted with a gyro sensor replaced traditional

button inputs with swinging the controller. Also, it enabled

various body motions to be applied to game play. This was

a new experience for users who hasn’t experienced that

kind of game plays before. However, it is based on a

traditional game play element that moves game characters

according to the information transferred via an input device.

One of the most frequently encountered definitions of what

constitutes a game is game designer Sid Meir’s “a series of

interesting choices.”[14] This shows that input devices play

a central role in game play because they’re the only means

players have for selected choices. These choices could

include, for example, changing a variable in the game or

influencing the game’s environment in ways that are

advantageous for the player.

Games in the racing genre provide a clear example of

variables that are changed by player choices. The speed of

the player’s car changes directly according to the player’s

choice. On the other hand, in adventure games players are

asked to solve a quiz or a puzzle using hints that are hidden

in the environment. To make the game play more exciting

or create new game play methods, designers deliberately

tweak the game’s environment to make it more challenging

for players to find the hints.

Suggestion of game play elements adopting BCI

In this chapter, game play elements adopting BCI as a

means for player’s to express choices will be dealt with

based on the examples of game play elements discussed in

the previous chapter.

Firstly, for the game play where some variables change, the

degree to which the player concentrates on the game among

all brainwave information can be used as a tool for player’s

choice. For example, in a racing game concentration could

influence the speed of the player’s car, so the speed would

go up as the player concentrated more on the game. In fact,

this has already been adopted by a gaming toolset called

‘Smart Brain Games,’ seen in Figure 4. The toolset is quite

basic. It simply replaces a few buttons on a traditional game

controller when the player plays existing games [16].

Figure 4:

For first-person shooter (FPS) games like ‘Sniper Elite’

[17] serious or ‘Call of Duty’ [2] series, the degree of

concentration affects the aiming or accuracy of the player’s

weapon. For example, as seen in Figure 5 if the player has

to use a sniper rifle, the crosshairs might be made to jitter

less as player concentration was increased, improving the

rifle’s accuracy.

Toolset of Smart Brain System

Figure 5:

In Role Playing Games (RPGs) and action games, some

particular emotions like concentration or meditation could

be used to influence the success rate of the game character’s

skill or the recovery speed of a player who had been cursed.

Secondly, for games that focus on changes to the game’s

environment, like adventure games, brainwaves could be

used as a method for selecting a choice, creating interesting

new game play. For example, in a crime scene hidden clues

might appear when a player’s concentration level gets high

enough. Alternately, if the player was relaxed or meditated

enough, he or she could receive an important hint for

solving a puzzle or escaping from a dangerous situation.

For espionage adventure games like ‘Metal Gear Solid’ [7]

series and for survival horror games like ‘Silent Hill’ [15]

4

series or ‘Resident Evil’ [13] series, the player’s tension

level would be a natural means of affecting game play. For

example, the player’s tension could influence the behavior

of non-player characters. Or the enemy could be designed

to recognize the player and begin moving towards them if

the player’s tension level rises or they become nervous.

Non-player characters could also be designed to help the

player instead when he or she is getting nervous. The most

challenging issue to consider when designing this kind of

game play is composing all the game’s elements, including

the game story, visuals and game mechanics, so that the

player naturally feels the emotion that the designer intended

for the player to feel at a certain point in order to use the

tension caused by horror or fear.

Game play image of 'Sniper Elite' [17]

DEVELOPMENT OF ACTUAL PC GAME USING BCI

Figure 6:

Title image of the game

Figure 7:

In order to test the hypothesis of this research a PC game

utilizing BCI was developed. Figure 6 shows the title image

of the game. The game is composed of two game modes.

The first one is the training mode which teaches a player

how to control attention and meditation while the player

wears a BCI headset through playing four kinds of mini

games. The second one is the story mode based on a simple

scenario: while a pretty girl is taking a nap in a park, she

falls into a strange world with her cat and an adventure

begins. The player has to find the portal to take her to home

controlling her cat and animals in the world with emotional

signals from BCI device as well as traditional input data

from keyboard and mouse to defeat monsters and solve

puzzles in the world. While the player plays the game, her

brain status is reflected into the game progress

continuously. At certain spots during the game play, the

player can do the actions mentioned above: aiming and

searching. Figure 7 shows the game flow.

Game flow

Figure 8:

Game play images

Figure 9:

left, 0 ~ 25%, 26% ~ 50%, 51% ~ 75%, 76% ~ 100%)

Visualization of attention in the game (To the

Figure 10:

left, 0 ~ 25%, 26% ~ 50%, 51% ~ 75%, 76% ~ 100%)

Figure 8 shows game play screenshots. We placed icons

and a gauge bar at the upper left corner of the screen to

show that the player either concentrates or meditates while

5

playing the game. As the value of player’s meditation

increases, the gauge is gradually filled. We quantized the

value into four and displayed icons showing four levels of

the value with ease control. Figure 9 and figure 10 show the

icons and the bar used for displaying player’s attention and

meditation while the player plays the game, respectively.

Visualization of meditation in the game (To the

Figure 11:

game utilizing the player’s meditation value (To the left, 0

~ 25%, 26% ~ 50%, 51% ~ 75%, 76% ~ 100%)

Game play images of the bulb lighting mini

Figure 12:

searching mini game utilizing the player’s attention value

(To the left, 0 ~ 25%, 26% ~ 50%, 51% ~ 75%, 76% ~

100%)

In the training mode, the player can learn how to control

her concentration and meditation level nicely by playing

four simple mini games. In the bulb lightening mini game,

the player tries to light the electric bulb by relaxing her

mind. The more the player relaxes in the game, the bulb

turns on faster with some special effects. Figure 11 shows

game play images of the mini game. In the hidden object

searching mini game, player’s concentration level affects

difficulty of finding the hidden object. Figure 12 shows the

image of the game.

In the story mode, the player plays a side-scrolling action

adventure game having a simple storyline that a main

character escorts a girl to the end of the game stage. While

playing the story mode, the player resolves puzzles and

defeats monsters by controlling her cat and a mouse with

the BCI device as well as keyboard and mouse. The game

focused on using BCI to control three elements of game

play: aiming, searching and resting. In the game, when a

player concentrates more on the game, it becomes easier for

them to shoot enemies. Sometimes, when they are

concentrating or relaxed, they can find some hidden items

or hints. Also, when players are relaxed they can take a rest

in the game and recover from any damage they’ve

sustained. When the player aims at something, the crosshair

jitters to an extent based on how much the player is

concentrating. If the player is concentrating a great deal on

the game, the jittering will be very minor, making it easy

for them to shoot enemies. When the player searches for a

hidden object or hint on the game stage, the player’s level

of concentration or meditation affects how easy it is to find

things. For resting, the speed of the player character’s

recovery varies according to the degree of the player’s

meditation. In some areas of the game, a very relaxed

player will even be able to make flowers bloom. Figure 13,

figure 14 and figure 15 show game play images of utilizing

BCI data to solve puzzles in the story mode.

Game play images of the hidden object

Figure 13:

jittering of the crosshair varies according to the player’s

attention value

Game play images of aiming. The degree of

Figure 14:

The hidden hole appears if the player’s meditation value

exceeds a certain threshold

6

Game play images of searching a hole to enter.

Figure 15:

damage to the monster increases according to the player’s

attention value

Game play images of defeating monsters. The

EVALUATION OF BCI ELEMENTS USED IN THE GAME

This research is intended to suggest a number of game play

elements adopting BCI and to verify if players can actually

experience more intuitive and exciting game play when

they play a game in which some of the game play elements

are controllable using a BCI device.

In order to evaluate the game play elements using BCI we

developed a version of the game in which all BCI related

elements were removed. The two versions of the game are

the exactly same except that in the first version players

control the game using a BCI device as well as traditional

interface devices including mouse and keyboard, while for

the second version players use only traditional interface

devices.

Subjects

15 undergraduate and graduate students (average age: 24.5)

and 5 non-students (average age: 28), for a total of 20

participants, were tested. They were divided into two

groups of 10 people each.

Conditions of the research

As described above, two different versions of the game

were used. The versions were exactly same except that one

included support for a BCI device. Both versions were

developed using XNA 2.0 and .Net Framework 2.0/2.0 SP1

and were played on PC. For the BCI device we used

NeuroSky’s Mind Set. We used brainwave information that

is commonly detectable by most small-sized BCI devices

that are currently available.

Research procedure

The research was carried out dividing the subjects into 2

groups of 5 people each. One group was asked to play the

version of the game using BCI and the other group to play

the version that did not use BCI. Neither group had any

information about what the other group played.

All subjects played Single Game from the beginning to the

middle of the second stage. After that, they played two Mini

Games adopting different game elements: aiming and

searching. The total play time was approximately 20

minutes. Afterwards they were asked to fill in a

questionnaire.

The questionnaire asked about the degree of convenience,

fun and intuitiveness in both the overall game play and the

game play using the BCI device (or not). Each category had

two or four questions. Each question had 5 choices

numerically ranged from 1 to 5: 5 (Highly), 4 (Q.Highly), 3

(Average), 2 (Q.Low), 1 (Low).

The questionnaire given to both groups was exactly the

same.

Results of the research

Based on questionnaire responses, scores by category about

game elements differentiated by using BCI or not are seen

in Table 2.

Table 2:

Scores of each category

Group Total Average

Standard

Deviatio

n

Degree

of

Freedo

m

With BCI 149 14.9 2.645 9

Without

BCI

102 10.2 4.18 9

We performed statistical t-test with the results above and it

was induced that the game version adopting BCI is

significantly better than the version without BCI in terms of

convenience, fun and intuitiveness (

t

α=0.05, t.05(18)≒2.101,≒5.605).

CONCLUSION AND DISCUSSION

This research was intended to study and suggest game play

elements that can be developed around BCI and a player’s

emotions from the perspective of game design.

In order to verify if the suggested game play elements can

provide the user with an intuitive and exciting experience,

an experimental game using several proposed game play

elements was developed and tested by users. Feedback was

collected via a questionnaire.

The results of this feedback showed that the suggested BCIbased

game play elements provide users with a more

intuitive and interesting experience than traditional non-

BCI-based game play elements. Also, it was possible to

induce that simply adding BCI to a general PC gaming

environment can make the game play more intuitive and

immersive. The result and assets of this research are

expected to be an effective guideline for developing BCIbased

games.

7

ACKNOWLEDGEMENTS

The authors world like to thank to Ian Dallas and Nahil

Sharkasi in USC(University of Southern California) who

help with the revision of our paper in English.

REFERENCES

1. Allison BZ, McFarland DJ, Schalk G, Zheng SD,

Jackson MM, Wolpaw JR, "Towards an independent braincomputer

interface using steady state visual evoked

potentials", Clinical Neurophysiology, Vol. 119, Issue 2,

pp. 399~408, 2008

2 ‘Call of Duty’ series, available at http://www.callof

duty.com

3. Choi J, "The technical trend of Brain Computer

Interface(BCI)", Report of patent trend, Korea Institute of

Patent Information, pp. 1~10, 2003.

4. Emotive System Inc. web site, available at http://emo

tiv.com

5. Kim H, Lee J, "Applications of Biofeedback in

Psychological Settings", 2007 Annual Conference, Korean

Psychological Association, pp. 94~95, 2007.

6. Lalor EC, Kelly SP, Finucane C, Burke R, Reilly RB,

McDarby G, "Brain Computer Interface based on the

Steady-state VEP for Immersive Gaming Control",

Biomedizinsche Tecknik, pp. 63~64, 2004.

7 ‘Metal Gear Solid’ series, available at http://www.ko

nami.jp/kojima_pro/english/index.html

8. Middendorf M, McMillan G, Calhoun G, and Jones KS,

"Brain-computer interfaces based on the steady-state visualevoked

response", Neural Systems and Rehabilitation

Engineering, Vol. 8, No. 2, IEEE Transactions on, pp.

211~214, 2000

9. "Neurocybernatics Inc" website, available at http://www.

neurocomp.com /

10. NeuroSky Inc. web site, available at http://neurosky.biz

11. OCZ Technology Inc. web site, available at http://

www.ocztechnology.com

12. Pineda JA, Silverman DS, Vankov A, and Hestenes J,

"Learning to Control Brain Rhythms: Making a Brain-

Computer Interface Possible", Neural Systems and

Rehabilitation Engineering, Vol. 11, No. 2, IEEE

Transactions on, pp. 181~184, 2003

13 ‘Resident Evil’ series, available at http://www.cap

com.co.jp/bio5/

14. Rollings A, Morris D, Game Architecture and Design,

New Riders Games, pp 38, 2000

15 ‘Silent Hill’ series, available at http://www.konami.com/

games/shh/

16 SmartBrain Technology web site, available at http://

www.smartbraintech.com

17 ‘Sniper Elite’ series, available at http://www.micro

ids.com/en/catalogue/28/sniper-elite-berlin-1945.html