Exploring Multiple Protocols for a Brain-Computer Interface Mouse

Mathew Salvaris, Caterina Cinel, Riccardo Poli, Luca Citi, and Francisco Sepulveda School of Computer Science and Electronic Engineering
University of Essex
CO4 3SQ, Colchester, UK


Abstract:

In recent years, various visual protocols have been explored for P300-based BCI. In stimulus-driven BCI paradigms such as P300 BCIs it is vital to optimise the stimulation protocol as much as possible in order to achieve the best performance. Due to the inherent variability between subjects and the complex nature of the brain it is unlikely that an optimal protocol will be identified through a single iteration of random exploration. That is why in this paper we explore 8 different visual protocol configurations based on recent literature, in the hope of identifying key features that can later be used to create further improved protocols. Results indicate that luminosity changes, the standard method of stimulation used in visual P300 BCI protocols, do provide the best performance of the variations presented here.


Introduction

Over the last few years there has been a healthy interest in trying to improve P300-based BCIs through modifications of the protocol used [1,2,3]. These modification come from trying to mitigate the temporal limitations of the P300 signal [1] or simply by observing the possible effects of the alterations to the visual makeup of the protocol [2,3]. For example, in [3] a number of modifications were made to the visual properties of a P300 matrix speller, including, the size of the letters, the distance between the letters and the colours used. Gonsalvez et al. [4] explored the effects of the Target to Target Interval (TTI) on the P300 and found a direct correlation between TTIs and P300 amplitudes. Martens et al. [1], Salvaris and Sepulveda [5] and Citi et al. [6] explored the effects of very short TTIs in P300-based BCIs and concluded that temporally proximal target events are harder to detect than target events separated by greater temporal distance. Although the reason for this effect is not entirely clear, a possible explanation is repetition blindness and attentional blink [7]. In [5] the possibility of repetition blindness was explored and, although the effects on accuracy did match those of repetition blindness, it was not enough to unequivocally commit to that explanation. In [8] one of the modifications shown to mitigate the effects of repetition blindness was saliency between the two target events. In [1] it was shown that the repetition blindness-like effect present in their data could be mitigated by simply using a differing visual protocol and they attributed this difference to the apparent motion of the stimuli used. Others [9,10] have limited the temporal proximity of possible target events thereby avoiding these effects altogether.

In this paper we explore a number of novel visual protocols within the framework of a P300-based BCI mouse [11]. The changes made concerned the SOA, the duration of the change and type of stimulation. The various visual protocols created were intended to investigate the effects of different TTI values, apparent motion and target event saliency on epoch classification.


The Multiple Protocols

In total 8 different visual protocols were produced by varying three different factors (see Figure 1). The first is the Stimulus Onset Asynchrony (SOA) which had values of 200 ms ( $ \simeq12 \times 60^{-1}$) and 100 ms ( $ \simeq6
\times 60^{-1}$) (as permitted by the 60 Hz refresh rate of the LCD monitor used). The second factor was whether the colour or the position of stimuli changed. When colour was changed, we used black for the background, grey for the neutral stimuli and white for the highlighted stimuli. When position was changed, stimuli, which were white on a black background (and did not vary in colour), subtended from the centre of the screen by $ 1.61^{\circ}$ when stationary and $ 2.33^{\circ}$ when moved. The third and final factor was the type of state change. For the flash state change the stimulus entered the active state for the prescribed SOA and then returned to its neutral state (as per most oddball BCI protocols). For the persistent state, the active stimulus changed state and remained in that state until changed again. In the Flash colour visual protocol the stimulus changed from grey to white and then back again. In the persistent colour protocol the stimulus changed from grey to white or vice versa and remained in the new state until the stimulus was activated again. In essence this meant that the stimulus change in the persistent protocol would last as long as the minimum TTI of the protocol. Also, at each point in time the static visual configuration of the protocol was different, providing greater visual saliency between two successive target events. The protocols used an Inter-Stimulus Interval (ISI) of 0.00s, so in order to avoid undesired visual effects, adjacent stimuli to a previously flashed stimuli were not allowed to activate within 1 SOA. Furthermore, the target stimulus was not allowed to be active twice in succession. This meant that the minimum TTI between two target events for the slow protocols (SOA $ =$ 200 ms) was $ 2 \times$   SOA$ = 400$ ms while for the fast ones (SOA $ =$ 100 ms) the minimum TTI was $ 2 \times$   SOA$ = 200$ ms.

Figure 1: Visual displays in the different protocols. Each type of display was used with SOAs of 200 ms and 100 ms.
 \includegraphics[width=0.5\columnwidth,viewport=70 50 370 300,clip]{Colour_blank}  \includegraphics[width=0.5\columnwidth,viewport=70 50 370 300,clip]{Flash_move}    
 Flash Colour  Flash Distance    
 \includegraphics[width=0.5\columnwidth,viewport=70 50 370 300,clip]{Colour_persist}  \includegraphics[width=0.5\columnwidth,viewport=70 50 370 300,clip]{Persist_move_t}    
 Persistent Colour  Persistent Distance    


Methods


Participants

Data were collected from 8 participants with an average age of 26.5. Subjects 1 and 2 carried out all 8 visual protocols. After that, however, it was determined that the experiment lasted too long and that fatigue might prevent other subjects from completing the experiment. Therefore, a Latin square approach of experimental design was used to split the 8 experiments into 4 experiments which would allow for the assessment of up to second order interactions.


Stimuli and Procedure

In each session participants were presented with visual displays showing 8 circles (with a diameter of 1.5cm) arranged around an imaginary circle at the centre of the display as in Figure 1. Each session was divided into runs, which we will call direction epochs.

Each participant carried out 16 direction epochs for each of the chosen visual protocol configurations. Participants 1 and 2 carried out 8 visual protocol configurations while the rest carried out 4 visual protocol configurations. With 16 direction epochs, the 8 possible directions were covered twice over. Within an experiment the direction epochs and visual protocols were randomised, the only constraint being that, within 4 or 8 direction epochs (depending on the number of visual protocols conducted by the subject), each of the possible visual protocols be conducted only once.

During a direction epoch, the participant was greeted by a blank screen and after a predefined period the stimuli appeared near the centre of the screen. A red arrow then appeared for 1 second pointing to the target. Subjects were instructed to count the number of state changes for that target. After 2 seconds the random activation of the stimuli started. This stopped after 20 to 24 trials, with a trial consisting of the individual activation of each of the 8 stimuli. After the direction epoch had been completed, the subject was requested to verbally communicate the number of times the target stimulus changed state.

Participants were seated comfortably at approximately 80 cm from an LCD screen, their neck supported by a C-shaped inflatable travel pillow to reduce muscular artifacts. Data were collected from 64 electrode sites using a BioSemi ActiveTwo EEG system. The EEG channels were referenced to the mean of the electrodes placed on either earlobe. The data were initially sampled at 2048 Hz.


Classification

Classification was carried out using an ensemble of 6 linear SVMs, with each SVM trained on a subset of the collected data across all the channels. The classification method is similar to the one used by Rakotomamonjy and Guigue [12] except that no channel selection was carried out and all 64 channels were used instead. The data were filtered between 0.15 and 30 Hz and initially downsampled to 128 Hz. Then, from each channel an 800 ms epoch was extracted and further decimated to 32 Hz.


Results

The classification results were estimated using 16 fold cross-validation, with each direction epoch being a cross-validation fold. The data were initially compared in two groups. One group consisted of the experiments carried out using the slower SOA and the other one was for the faster SOA.[*] In Figures 2 and 3 the corresponding Receiver Operating Characteristic (ROC) curves of the experiments across all the subject are shown. The classification accuracy values in these plots, like all others presented in the paper, were calculated based on single-epoch classification, i.e., on whether the data (presented to the classifier from all the channels in an 800 ms post-event epoch) contained a P300 or not. From Figure 2 it is evident that the best protocol is the standard flash protocol, while the worst protocol is the persistent colour protocol. These observations are also reflected in the Area Under the Curve (AUC) values shown in Table I. A similar conclusion can be drawn from observing the results for the faster protocols (see Figure 3 and Table II).

Figure 2: Receiver operating characteristic curve of the four slow visual protocols.

Image slow_ROC

Figure 3: Receiver operating characteristic curve of the four fast visual protocols

Image fast_ROC


Table I: Area under the curve values for the slow protocols.
  Flash Colour Flash Distance Pers. Colour Pers. Distance
 S01 0.883 0.861 0.830 0.854
  S02 0.885 0.839 0.764 0.846
  S03 0.900 - - 0.871
  S04 - 0.837 0.809 -
  S05 0.923 - - 0.855
  S06 - 0.845 0.799 -
  S07 0.878 - - 0.838
  S08 - 0.854 0.716 -
  Mean 0.894$ \pm$0.016 0.847$ \pm$0.009 0.784$ \pm$0.040 0.853$ \pm$0.011
 

Table II: Area under the curve values for the fast protocols.
  Flash Colour Flash Distance Pers. Colour Pers. Distance
 S01 0.820 0.804 0.801 0.828
  S02 0.792 0.824 0.674 0.792
  S03 - 0.814 0.825 -
  S04 0.897 - - 0.806
  S05 - 0.868 0.785 -
  S06 0.835 - - 0.788
  S07 - 0.831 0.717 -
  S08 0.811 - - 0.791
  Mean 0.831$ \pm$0.036 0.828$ \pm$0.022 0.760$ \pm$0.056 0.801$ \pm$0.015
 


Table III: Results of the Kolmogorov-Smirnov test for slow protocols.
  Flash Col. Flash Dist. Pers. Col. Pers. Dist.
 Flash Col. - 0.00794 0.00794 0.00794
  Flash Dist. 0.00794 - 0.00794 0.87302
  Pers. Col. 0.00794 0.00794 - 0.00794
  Pers. Dist. 0.00794 0.87302 0.00794 -
 

Table IV: Results of the Kolmogorov-Smirnov test for fast protocols.
  Flash Col. Flash Dist. Pers. Col. Pers. Dist.
 Flash Col. - 1.00000 0.35714 0.35714
  Flash Dist. 1.00000 - 0.07937 0.35714
  Pers. Col. 0.35714 0.07937 - 0.35714
  Pers. Dist. 0.35714 0.35714 0.35714 -
 

The results from a multiple-comparison Kolmogorov-Smirnov test can be seen in Tables III and IV for the slow and fast protocols, respectively. From the p-values presented in Table III it is apparent that the two protocols that differ the most from the rest are the flash colour and the persistent colour, the flash colour protocol being the best performing and the persistent colour protocol being the worst performing, according to the AUC values presented in Table I. A similar conclusion cannot be drawn for the fast protocols based on the results of the Kolmogorov-Smirnov test (see Table IV).

The top two slow protocols (based on AUC) were then compared to the top two fast protocols. Of course, this is a little unfair since in theory, due to the effects of TTI, the slower protocols should always out-compete faster protocols in single epoch classification, as a result of the increased amplitude in the P300. A quick observation of the AUC-results' tables confirms this. Evidently, the faster protocols with an SOA half that of the slower ones can carry out twice as many events within the same time period. So, the AUC scores for the top two fast protocols were recalculated by obtaining the mean score of two temporally adjacent epochs of the same stimulus. This was done on the score output of the previously trained classifiers and did not involve retraining the classifiers on the combined data. These new AUC results can be seen in Table V and the results of the Kolmogorov-Smirnov in Table VI.


Table V: AUC values of top two slow and fast protocols.
  Fast Flash Col. Fast Flash Dist. Slow Flash Col. Slow Pers. Dist.
 S01 0.894 0.886 0.883 0.854
  S02 0.871 0.910 0.885 0.846
  S03 - 0.883 0.900 0.871
  S04 0.960 - - -
  S05 - 0.934 0.923 0.855
  S06 0.904 - - -
  S07 - 0.902 0.878 0.838
  S08 0.890 - - -
  Mean 0.904$ \pm$0.030 0.903$ \pm$0.019 0.894$ \pm$0.016 0.853$ \pm$0.011
 


Table VI: Results of Kolmogorov-Smirnov test between top two fast and slow protocols.
  F. Flash Col. F. Flash Dist S. Flash Col. S. Pers. Dist
 F. Flash Col. - 1.00000 0.87302 0.07937
  F. Flash Dist. 1.00000 - 0.87302 0.00794
 S. Flash Col. 0.87302 0.87302 - 0.00794
  S. Pers. Dist. 0.07937 0.00794 0.00794 -
 

Discussion

The results seem to confirm the hypothesis that the slower protocols with the greater TTI range do indeed provide greater separability between the target and non-target epochs. What is interesting, however, is that if we compensate for this, by taking the mean of the score of two epochs (of the same stimulus event) in the faster protocols for every one in the slower one, we end up with comparable results for the flash colour protocols. For the faster protocols the top two performing protocols provide very similar results. The same cannot be said for the top two slow protocols, with the slow persistent distance protocol being significantly different from the fast flash distance protocol and the slow flash colour protocol. Overall the faster protocols perform better than their slower counterparts within the same time period, probably due to the error correcting effects of combining multiple trials. Another observation between the fast and slow protocols is that the fast protocols show a greater amount of variance between subject performance than their slower counterparts, possibly indicating that the optimum SOA might be subject-specific.

The protocols that utilised apparent motion did not seem to provide any advantage over using luminosity changes in terms of the AUC values. It is possible that the range of movement chosen for the apparent movement was not large enough and that greater displacement may provide better results, although the apparent motion protocols seem to result in less inter subject variance.

The persistent protocols generally seem to provide far worse results than the flash protocols. This could possibly be attributed to subject confusion, since each subject carries out a mixture of flash and persistent protocols and at very short intervals the persistent stimuli often appear as flash stimuli (two successive persistent state changes appear as a single momentary state change). This in turn may increase memory load, adversely affecting the amplitude of the P300. In future experiments we will explore the advantages and disadvantages of having more than two states, thereby decreasing the possibility of confusion with the flash protocols and increasing inter-target saliency. Another interesting possibility may be to test persistent protocols in isolation to flash protocols on naive subjects, removing the possibility of this confound.


Conclusion

Overall the results indicate that the basic luminosity change does provide some of the best results in this protocol. It is surprising that the apparent motion did not provide better results, although there are numerous factors that may have affected it. Of course, the number of subjects is quite small and the extent of the variations limited. This in turn limits the power of the statistical tests and the certainty with which protocol effects can be measured. Nevertheless the aforementioned visual protocols have further elucidated the possible effect these modifications may have on P300-based BCIs.

Acknowledgements

We would like to thank the UK Engineering and Physical Sciences Research Council (grant EP/F033818/1) for financial support.

Bibliography

url@samestyle

1
S. M. M. Martens, N. J. Hill, J. Farquhar, and B. Schölkopf, ``Overlap and refractory effects in a brain-computer interface speller based on the visual P300 event-related potential.'' Journal of neural engineering, vol. 6, no. 2, p. 026003, Apr 2009.

2
K. Takano, T. Komatsu, N. Hata, Y. Nakajima, and K. Kansaku, ``Visual stimuli for the P300 brain-computer interface: A comparison of white/gray and green/blue flicker matrices,'' Clinical neurophysiology, Jun. 2009.

3
M. Salvaris and F. Sepulveda, ``Visual modifications on the P300 speller BCI paradigm.'' Journal of neural engineering, vol. 6, no. 4, p. 46011, Jul 2009.

4
C. J. Gonsalvez, R. J. Barry, J. A. Rushby, and J. Polich, ``Target-to-target interval, intensity, and P300 from an auditory single-stimulus task,'' Psychophysiology, vol. 44, no. 2, pp. 245-50, Mar. 2007.

5
M. Salvaris and F. Sepulveda, ``Perceptual errors in the Farwell and Donchin matrix speller,'' in 4th International IEEE EMBS Conference on Neural Engineering, Antalya, Turkey, 2009, pp. 275 - 278.

6
L. Citi, R. Poli, and C. Cinel, ``Exploiting P300 amplitude variations can improve classification accuracy in Donchin's BCI speller,'' in 4th International IEEE EMBS Conference on Neural Engineering, Antalya, May 2009, pp. 478-481.

7
C. Cinel, R. Poli, and L. Citi, ``Possible sources of perceptual errors in P300-based speller paradigm,'' Biomedizinische technik, vol. 49, pp. 39-40, 2004, Proceedings of 2nd International BCI workshop and Training Course.

8
M. M. Chun, ``Types and tokens in visual processing: a double dissociation between the attentional blink and repetition blindness,'' Journal of Experimental Psychology. Human Perception and Performance, vol. 23, no. 3, pp. 738-55, Jun. 1997.

9
G. Townsend, B. K. LaPallo, C. B. Boulay, D. J. Krusienski, G. E. Frye, C. K. Hauser, N. E. Schwartz, T. M. Vaughan, J. R. Wolpaw, and E. W. Sellers, ``A novel P300-based brain-computer interface stimulus presentation paradigm: Moving beyond rows and columns,'' Clinical Neurophysiology, vol. 121, no. 7, pp. 1109-1120, Jul. 2010.

10
H. Serby, E. Yom-Tov, and G. F. Inbar, ``An improved p300-based brain-computer interface,'' IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 13, no. 1, pp. 89-98, Mar. 2005.

11
L. Citi, R. Poli, C. Cinel, and F. Sepulveda, ``P300-based BCI mouse with genetically-optimized analogue control,'' IEEE transactions on neural systems and rehabilitation engineering, vol. 16, no. 1, pp. 51-61, Feb. 2008.

12
A. Rakotomamonjy and V. Guigue, ``BCI competition III: dataset II- ensemble of SVMs for BCI P300 speller,'' IEEE transactions on bio-medical engineering, vol. 55, no. 3, pp. 1147-1154, Mar 2008.

About this document ...

Exploring Multiple Protocols for a Brain-Computer Interface Mouse

This document was generated using the LaTeX2HTML translator Version 2008 (1.71)

Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The command line arguments were:
latex2html -split 1 MultiProtocol.tex

The translation was initiated by Riccardo Poli on 2011-08-01

Riccardo Poli 2011-08-01