Novel Sequential Protocols for a ERP Based BCI Mouse

M. Salvaris, C. Cinel and R. Poli[*][*]


Abstract:

Over the last few years there has been considerable activity around P300-based BCI protocols. Much of this has been focused on trying to overcome observed irregularities in ERP classification due to temporal proximity of target events. In this paper we explore three novel protocols which utilise a consistent temporal interval between targets. This was done in order to mitigate the aforementioned issue and in turn improve classification. The three protocols produced very encouraging results with the best protocol -- Random Colour protocol -- achieving a mean information transfer rate of 31 bits/min with an accuracy of over 92%. The Random Colour protocol produced a P3 component with a focus around the central parietal area indicating that it is most likely a P3b. Furthermore when compared to results obtained in a previous experiment that used the standard 'oddball' paradigm the performance was favourable.

INTRODUCTION

Brain Computer Interfaces use a variety of methods to interpret task relevant brain signal in order to achieve communication or control [1]. A well studied and often used brain potential is the P300 event-related potential (ERP). The P300 ERP has a centro-parietal focus and typically occurs around 300-600 ms after a task relevant stimulus is presented.

The most common method of achieving a P300 potential is the `oddball' paradigm. This paradigm relies on an infrequent but task-relevant stimulus being embedded in a series of frequent irrelevant stimuli [2]. Ever since its use in [3] to identify targets in a matrix speller, numerous attempts have been made to try to improve on it [4,5,6,7].

The success of BCI systems is often assessed by the information transfer rate (ITR) they provide. A logical step in improving this transfer rate is to reduce the Stimulus-Onset Asynchrony (SOA), i.e., the time interval between the beginning of two consecutive stimuli. Recently it has been shown that the classification accuracy of target events is affected by their temporal proximity to one another [8,9,10]. The cause of these effects could be attributed to the Target-to-Target Interval (TTI, i.e., the time interval between the end of a target event and the beginning of the following target event) [11], neurophysiological phenomena such as Attentional Blink or Repetition Blindness [12,10] or even more simply as overlap and refractory effects [8]. Nevertheless the issue remains that at very close temporal proximities the successful classification of the target ERPs verges on chance. This limit on the temporal proximity of targets also restricts the amount by which the SOA can be reduced.

A number of possible solutions to this problem have been proposed. These range from ensuring a minimum period between two successive targets [7], to altering the stimulation method [8], and leveraging regularities to improve classification [13]. All methods achieve relative success with modest improvements when compared to the standard protocol.

Given the point-and-click nature of most modern user interfaces, an important application of BCI is controlling 2-D pointer movements. For example, in [14] a P300-based system for the 2-D control of a cursor on a computer screen was presented. In this system four randomly-flashing squares are displayed at the margins of the screen to represent four directions of movement. However, preliminary online tests and further offline analysis showed that the mouse pointer was hard to control. So, we explored a variety of alternatives to this approach in [15] where 8 different stimuli (circles at the centre of the screen: 4 for ``up'', ``down'', ``left'' and ``right'', and 4 for the 45 degree diagonal directions) were used. A total of 8 oddball protocols were analysed which differed on the basis of SOA as well as in the features of the stimuli which were randomly varied. Results indicated that the best protocols are those where the stimuli are randomly flashed, although some protocols based on moving stimuli came very close to these. However, the information rate achievable with these oddball protocols is still insufficient to produce a responsive and accurate control over the pointer.

In order to increase the ITR in all these P300-based systems, we need to decrease the SOA while either maintaining or increasing the discriminability of the target stimulus. This could only be achieved if we overcame the aforementioned issues caused by excessively short TTIs.

Here we take novel approach to solving the problem. We have designed three novel protocols that do not utilise a random activation pattern but a predictable sequential one. This leads to more consistent target ERPs and, consequently, better classification. The first protocol (Single Colour) was expected to produce a visual evoked potential (VEP) when the target flashed. The second protocol (Imaginary Movement), inspired by previous work [16], was expected to generate a VEP as well as a motor related potentials. The third and final protocol named the Random Colour was expected to generate a VEP response as well as a P300. This paper describes the protocols and the results we obtained using them.

The paper is organised as follows. In Section II we describe the protocols, our procedures, the participants and our classification and ERP scoring strategy. In Section III we report our findings with using the three protocols. We discuss our results in Section IV and provide some conclusions and indications for future work in Section V.


METHODS


Protocols

Three protocols were investigated, all used an SOA of 100 ms ( $ \simeq6 \times 60^{-1}$) (as permitted by the 60 Hz refresh rate of the LCD monitor used) with an Inter-Stimulus Interval (ISI) of 0.00s. Eight circles were arranged at the centre of the screen with each circle having a diameter of 1.5 cm and subtending the centre of the screen by $ 1.61^{\circ}$. The background was black and the non-highlighted stimuli were grey. For the Single Colour protocol the stimuli would turn white in clockwise succession (see Fig. 1) and the subject was tasked with counting the number of times the target was highlighted. The Imaginary Movement protocol was the same as the Single Colour protocol except that the subject was required to imagine an extension movement of his/her wrist whenever the target became highlighted. In the final protocol, named the Random Colour protocol, the stimuli again followed a sequential clockwise sequence except that they were randomly highlighted red or green (see Fig. 1). This time the subject was asked to mentally name the colour of the highlighted target stimulus.

Figure 1: Visual displays in the different protocols. The top row illustrates the type of stimuli used for the Single Colour protocol and the Imaginary Movement protocol. The bottom row illustrates the Random Colour protocol.

 \includegraphics[width=0.33\columnwidth]{MouseFlash_0}  \includegraphics[width=0.33\columnwidth]{MouseFlash_1}  \includegraphics[width=0.33\columnwidth]{MouseFlash_2}
 \includegraphics[width=0.33\columnwidth]{MouseRanCol_0}  \includegraphics[width=0.33\columnwidth]{MouseRanCol_1}  \includegraphics[width=0.33\columnwidth]{MouseRanCol_2}
 Identify Target  First Flash  Second Flash


Participants

Data were collected from 4 participants (average age: 28). All subjects carried out the 3 protocols. The order in which the subjects carried out the protocols was randomised.


Stimuli and Procedure

In each session participants were presented with visual displays showing 8 small circles arranged around an imaginary circle at the centre of the display as shown in Fig. 1. Each session was divided into direction epochs.

Each participant carried out 16 direction epochs for each of the three protocols. This meant that the 8 possible directions were covered twice over. At the beginning of each direction epochs the participant was greeted by a blank screen. After which the stimuli appeared near the centre of the screen. A red arrow then appeared for 1 second pointing to the target. Subjects were instructed to carry out the set task every time the target was highlighted. After 2 seconds the sequential activation of the stimuli started. After 20 to 24 trials this would stop and the subject was requested to verbally communicate the number of times the target stimulus changed state or for the Random Colour protocol the colour of the target after the final stimulation. Each trial consisted of the individual activation of each of the 8 stimuli.

Participants were seated comfortably at approximately 80 cm from an LCD screen. Data were collected from 64 electrode sites using a BioSemi ActiveTwo EEG system. The EEG channels were referenced to the mean of the electrodes placed on either earlobe. The data were sampled at 2048 Hz.


Classification

Classification was carried out using a linear SVM. The data were filtered between 0.15 and 30 Hz and downsampled to 128 Hz. Then, from each channel an 800 ms epoch was extracted and further decimated to 32 Hz. All 64 EEG channels were used.

The classification results were estimated using 16 fold cross-validation, with each direction epoch being a cross-validation fold. The penalty parameter for the SVM was determined by an inner cross-validation loop for each of the outer cross-validation loops.


RESULTS

To assess the performance associated with the different protocols we used the area under the curve (or AUC) of the receiver operating characteristics of the SVM output,[*] the ITR and the classification accuracy.

The AUC results can be seen in Table I. The Random Colour protocol achieves better classification than either of the two other protocols across all subjects. A paired one-way ANOVA on the AUC results shows that the differences between the protocols would be statistically significant ($ p=0.012$) if AUC distributions were approximately normal. However, given the small sample available we could not confirm if this is true. So, we also performed multiple pair-wise one-sided Kolmogorov-Smirnov tests to compare the AUC distributions obtained with different protocols. The $ p$ values recorded in the comparisons are presented in Table II. Even after adjusting such values for multiple dependent tests, the results for the Random Colour protocol are clearly statistically superior to the others tested.

The accuracy and information-transfer rates of the protocols tested can be seen in Fig. 2. Accuracy was computed based on the target direction for each direction epoch and the direction predicted using the formula    predicted direction$ = \operatornamewithlimits{argmax}_{\mathrm{dir}} {\mathrm{SVM}_J(\mathrm{dir})}
$, where $ \mathrm{dir}\in\{0,1,2,3,4,5,6,7\}$ is one of the 8 possible directions and $ \mathrm{SVM}_J(\mathrm{dir})$ is the sum of the raw outputs produced by the SVM in the first $ J$ flashes in direction $ \mathrm{dir}$. In other words, $ \mathrm{SVM}_J(\mathrm{dir}) = \sum_{j=0}^{J-1}
\mathrm{SVM}(x_{j\times 8 + \mathrm{dir}})$, $ x_t$ being the feature vector associated with the $ t$-th flash in a direction epoch.

As Fig. 2 shows, the Random colour protocol achieves an average ITR of around 31 bits/min with an accuracy of 92% which is a reasonable performance considering that the protocol utilises 8 possible targets. The other two protocols provide reasonable results but are not as good as the results obtained using the Random Colour protocol. By looking at the ERP averages in Fig. 3 it is easy to see that the Random Colour protocol induces a P300 response where as the others do not, or at least a very diminished one. From the average squared correlation coefficient ($ r^2$) of the three midline channels (see Table III) we can see that the Random Colour protocol provides results nearly an order of magnitude greater than the other two protocols.


Table I: Area under the curve values for the protocols tested.
  Random Colour Imaginary Movement Single Colour
 S01 0.863 0.779 0.762
  S02 0.877 0.675 0.761
  S03 0.946 0.626 0.725
  S04 0.908 0.820 0.808
  Mean 0.898+/-0.032 0.725+/-0.078 0.764+/-0.029
 
Table II: Results of pair-wise one-sided Kolmogorov-Smirnov tests for the protocols under test. Adjusted $ p$ values were obtained via the Hochberg correction for multiple dependent tests.
Comparison $ p$ value Adj. $ p$ value
Random Colour vs Imaginary Movement 0.018 0.037
Random Colour vs Single Colour 0.018 0.037
Imaginary Movement vs Single Colour 0.779 0.779

Figure 2: Accuracy (Acc) and information transfer rate (ITR) of the three protocols tested.

[c][c][0.5][0]Trials [c][c][0.5][0]Accuracy & ITR(bits/min)

Image AccandITR

Figure 3: Target ERP averages across subjects for the three protocols in Fz (top), Cz (middle) and Pz (bottom).
[c][c][0.5][0]Amplitude ($ \mu$V) [c][c][0.5][0]Time (s)
 Image Fz_ERP_average  
 Image Cz_ERP_average  
 Image Pz_ERP_average  


Table III: Average squared correlation coefficient ($ r^2$) for each protocol from channels Fz, Cz and Pz.
  Random Colour Imaginary Movement Single Colour
Fz 0.00358 0.00050 0.00081
Cz 0.00557 0.00059 0.00082
Pz 0.00415 0.00064 0.00049


DISCUSSION

The aim of the work was to try novel protocols that utilised a sequential and predictable stimulation sequence rather than the random sequence often used in P300 based BCIs. The results were encouraging demonstrating that all three protocols provide ERP data that is at least classifiable. The Random Colour protocol provided unequivocally the best results. For example, at four repetitions the protocol achieves a mean accuracy of 92% with an ITR of 31 bits/min. On the contrary, achieving approximately the same accuracy requires at least 10 repetitions for the Imaginary Movement and the Single Colour protocols.

We compared the results of the Random Colour protocol with the results we obtained with a standard oddball paradigm with the same SOA (reported in [15]). The mean AUC scores obtained with the former (see Table I) are significantly better the mean AUC obtained with the latter (0.831). As verified with a 2-sample, one-tailed Kolmogorov-Smirnov test, the differences are almost statistically significant ($ p=0.058$), which is encouraging considering the small sample sizes and the unpaired nature of the samples.

The results obtained using the Single Colour protocol were not surprising: with predictable sequences it has been shown that the P300 is severely diminished or even indiscernible, so the majority of components observed in the ERPs would mainly be visual evoked potentials. The Imaginary Movement results were disappointing, especially when compared to the results obtained in [16]. Of course this experimental protocol differed considerably from the one used in [16]. For instance in the previous experiment subjects had to carry out 8 direction epochs of real movement before they carried out any imaginary movement recordings. Although in this experiment subjects were given some time to practice it may not have been enough. The difference between subject performance in the AUC scores shown in Table I may be indicative of this. Also, the SOA in this protocol was considerably shorter and the TTI was half that of the shortest TTI in the previous experiment. These effects may have compounded to reduce the discriminability of the ERP.


CONCLUSIONS AND FUTURE WORK

In summary, the results obtained using our sequential protocols were encouraging. Furthermore we demonstrated that the standard 'oddball' paradigm is not necessary in order to generate a highly discriminable ERP: sequential stimuli can be used as well. The results obtained using the Random Colour protocol are effectively significantly better than what can be produced using the standard 'oddball', although a larger (ideally, pairwise) comparison will need to be performed at some point in the future to more precisely quantify the benefits of the new approach.

The other two protocols presented here produce reasonable results but they do not perform as well when compared to the standard oddball results in [15].

In future we plan to test the new protocol online within a BCI Mouse system. We will also seek to incorporate the Random Colour protocol into the standard matrix speller.

{\bibname}

url@rmstyle =0pt 4 =2plus 3minus 4 [2]l@#1 =l@#1 #2

1
J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, ``Brain-computer interfaces for communication and control,'' Clinical Neurophysiology, vol. 113, no. 6, pp. 767-91, June 2002.

2
J. Polich, ``Updating P300: an integrative theory of P3a and P3b.'' Clinical neurophysiology, vol. 118, no. 10, pp. 2128-2148, Oct 2007.

3
L. A. Farwell and E. Donchin, ``Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials.'' Electroencephalography and clinical neurophysiology, vol. 70, no. 6, pp. 510-523, Dec 1988.

4
B. Z. Allison and J. A. Pineda, ``Effects of SOA and flash pattern manipulations on ERPs, performance, and preference: implications for a BCI system.'' International journal of psychophysiology, vol. 59, no. 2, pp. 127-140, Feb 2006.

5
B. Hong, F. Guo, T. Liu, X. Gao, and S. Gao, ``N200-speller using motion-onset visual response.'' Clinical neurophysiology, vol. 120, no. 9, pp. 1658-1666, Sep 2009.

6
K. Takano, T. Komatsu, N. Hata, Y. Nakajima, and K. Kansaku, ``Visual stimuli for the P300 brain-computer interface: A comparison of white/gray and green/blue flicker matrices,'' Clinical neurophysiology, June 2009.

7
G. Townsend, B. K. LaPallo, C. B. Boulay, D. J. Krusienski, G. E. Frye, C. K. Hauser, N. E. Schwartz, T. M. Vaughan, J. R. Wolpaw, and E. W. Sellers, ``A novel P300-based brain-computer interface stimulus presentation paradigm: Moving beyond rows and columns,'' Clinical Neurophysiology, vol. 121, no. 7, pp. 1109-1120, July 2010.

8
S. M. M. Martens, N. J. Hill, J. Farquhar, and B. Schölkopf, ``Overlap and refractory effects in a brain-computer interface speller based on the visual P300 event-related potential.'' Journal of neural engineering, vol. 6, no. 2, p. 026003, Apr 2009.

9
L. Citi, R. Poli, and C. Cinel, ``Exploiting P300 amplitude variations can improve classification accuracy in Donchin's BCI speller,'' in 4th International IEEE EMBS Conference on Neural Engineering, Antalya, May 2009, pp. 478-481.

10
M. Salvaris and F. Sepulveda, ``Perceptual errors in the Farwell and Donchin matrix speller,'' in 4th International IEEE EMBS Conference on Neural Engineering, Antalya, Turkey, 2009, pp. 275 - 278.

11
C. L. Gonsalvez and J. Polich, ``P300 amplitude is determined by target-to-target interval.'' Psychophysiology, vol. 39, no. 3, pp. 388-396, May 2002.

12
C. Cinel, R. Poli, and L. Citi, ``Possible sources of perceptual errors in P300-based speller paradigm,'' Biomedizinische technik, vol. 49, pp. 39-40, 2004, Proceedings of 2nd International BCI workshop and Training Course.

13
L. Citi, R. Poli, and C. Cinel, ``Documenting, modelling and exploiting P300 amplitude changes due to variable target delays in donchin's speller,'' Journal of Neural Engineering, vol. 7, no. 5, p. 056006, 2010.

14
L. Citi, R. Poli, C. Cinel, and F. Sepulveda, ``P300-based BCI mouse with genetically-optimized analogue control,'' IEEE transactions on neural systems and rehabilitation engineering, vol. 16, no. 1, pp. 51-61, Feb. 2008.

15
M. Salvaris, C. Cinel, R. Poli, L. Citi, and F. Sepulveda, ``Exploring multiple protocols for a brain-computer interface mouse,'' in Proceedings of 32nd IEEE EMBS Conference, Buenos Aires, September 2010, pp. 4189-4192.

16
M. Salvaris and F. Sepulveda, ``Classification effects of real and imaginary movement selective attention tasks on a P300-based brain–computer interface,'' Journal of Neural Engineering, vol. 7, no. 5, p. 056004, 2010.

17
D. J. Hand and R. J. Till, ``A simple generalisation of the area under the ROC curve for multiple class classification problems,'' Machine Learning, vol. 45, pp. 171-186, 2001.

{\infopagename}

Novel Sequential Protocols for a ERP Based BCI Mouse

This document was generated using the LaTeX2HTML translator Version 2008 (1.71)

Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The command line arguments were:
latex2html -split 1 EMBS2011NER.tex

The translation was initiated by Riccardo Poli on 2011-08-01

Riccardo Poli 2011-08-01