Analogue Evolutionary Brain-Computer Interfaces
Objectives and ApproachThe keyboard and mouse provide us with reliable, but unnatural forms of input, being primitive transducer of muscular movement. Wouldn't it be nice some day to be able to replace them with Brain-Computer Interfaces (BCIs) capable of directly interpreting the intentions of computer users?
The objective of this project was the development of the technology and related scientific basis for the creation of usable hands-free BCI mice.
Our guiding vision was that brains are analogue devices that should communicate whenever possible without the restrictions imposed by traditional BCIs, which rely on a discrete classification. Instead, our BCI mice are logically analogue, contrary to previous BCI design wisdom. So, the 2-D motion of the mouse pointer is not in discrete steps, but gradual ones which are proportional to the amplitude of particular electrical waves, known as P300s, produced by the brain.
P300 waves are only produced if a user sees particular patterns appear on the computer screen and his/her attention is fully devoted to those stimuli. Nobody knows what visual patterns are best for generating P300s and what's the best way of keeping users interested in such patterns. So, developing a BCI mouse did not only mean solving signal processing and machine learning problems: importantly it also meant to find the best way of interacting with the mind of users. This required an interdisciplinary approach where technical solutions were compatible or even exploited the cognitive and perceptual limits of the human mind.
Summary of ResultsSince no design techniques existed for this type of system, in the project we combined knowledge from signal processing and psychology with machine-learning techniques, including innovative evolutionary algorithms and support-vector machines. However, this was not enough to overcome the extremely difficult challenges involving in eliciting and interpreting brain signals. We also had do carry out fundamental research to understand how the shape and amplitude of P300 waves varies depending on the type and timing of the visual stimuli used. In turn, this required developing better ways of recording and averaging such waves.
Results with this holistic approach were extremely promising. In particular, in the exploration of visual stimuli, we found a totally innovative protocol, which uses periodic, and thus predictable, sequences of stimuli instead of the traditional random sequences. This goes against all the psychophysiology literature on the generation of P300s, yet, with an appropriate mental task for users, we found that periodic sequences produced much stronger and more widely distributed P300s than traditional sequences. Correspondingly, this produced a marked increase in performance in our mouse.
In using evolutionary algorithms to aid the design of our BCI mice, we discovered that they devoted much attention to minimising the effects of muscular artifacts (such as eye blinks or swallowing) on the mouse trajectories. When we combined our new periodic stimulation patterns with the best machine learning technology and the best evolved system for artifact rejection, we obtained excellent results. Subjects were able to use the mouse in both controlled conditions and in a standard Windows environment with very good accuracy only minutes after wearing the electrode cap and with no previous training.
Our analougue mouse is currently the best P300-based BCI mouse in the literature by far, performing one movement every 100ms. Watch a video of our BCI mouse in action here.
In addition, we were able to exploit the newly acquired knowledge on P300 variability beyond our BCI mouse, building a matrix speller that provides a significant improvement in accuracy over the top-performing algorithm in the literature to date. Finally, while developing our analogue systems, we stumbled onto a couple of other useful results as reported here.
While further research is required to make our BCI mouse and speller competitive for able-bodied people, performance and reliability appear to be now sufficient for people suffering from severe muscular disorder or locked-in syndrome.
Publications Resulting from the Project
Journal ArticlesRiccardo Poli, Luca Citi, Francisco Sepulveda, and Caterina Cinel, Analogue Evolutionary Brain Computer Interfaces, IEEE Computational Intelligence Magazine, 4(4):27-31, November 2009.
Riccardo Poli, Caterina Cinel, Luca Citi and Francisco Sepulveda, Reaction-time Binning: a Simple Method for Increasing the Resolving Power of ERP Averages, Psychophysiology, Volume 47, issue 3, pages 467-485, Jan 2010.
L. Citi, R. Poli, and C. Cinel, Documenting, modelling and exploiting P300 amplitude changes due to variable target delays in Donchin's speller, Journal of Neural Engineering, vol. 7, Oct. 2010.
R. Poli and M. Salvaris, Comment on "Fast attainment of computer cursor control with noninvasively acquired brain signals", Journal of Neural Engineering, August, 2011.
R. Poli, M. Salvaris and C. Cinel, A Genetic Programming Approach to the Evolution of Brain-Computer Interfaces for 2-D Mouse-Pointer Control, Genetic Programming and Evolvable Machines, invited article.
Refereed Conference PapersLuca Citi, Riccardo Poli, and Caterina Cinel, Exploiting P300 Amplitude Variations Can Improve Classification Accuracy in Donchin's BCI Speller, 4th International IEEE Engineering in Medicine and Biology Society Conference on Neural Engineering, Antalya, Turkey, 2009, pp. 478-481.
Luca Citi, Riccardo Poli, and Caterina Cinel, High-significance Averages of Event-related Potential via Genetic Programming, Genetic Programming Theory and Practice VII, Chapter 9, pages 135-157, Springer, 2009.
Poli, R., Citi, L., Salvaris, M., Cinel, C., & Sepulveda, F. Eigenbrains: the Free Vibrational Modes of the Brain as a New Representation for EEG. 32nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society August 31 - September 4, 2010 Buenos Aires Sheraton Hotel, Buenos Aires, Argentina, pp. 6011-6014.
Salvaris, M., Cinel, C., Poli, R., Citi, L., & Sepulveda, F. Exploring Multiple Protocols for a Brain-Computer Interface Mouse. 32nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society August 31 - September 4, 2010 Buenos Aires Sheraton Hotel, Buenos Aires, Argentina, pp. 4189-4192.
Riccardo Poli, Mathew Salvaris and Caterina Cinel, Evolution of a Brain-Computer Interface Mouse via Genetic Programming, Proceedings of Genetic Programming - 14th European Conference, EuroGP, Torino, Italy, April 27-29, 2011, Lecture Notes in Computer Science 6621, Springer, pp. 203-214.
Riccardo Poli, Mathew Salvaris and Caterina Cinel, Evolutionary Synthesis of a Trajectory Integrator for an Analogue Brain-Computer Interface Mouse, Proceedings of Applications of Evolutionary Computation (EvoApplications), Torino, Italy, April 27-29, 2011, Lecture Notes in Computer Science 6624, Springer, pp. 214-223.
Riccardo Poli, Mathew Salvaris, and Caterina Cinel, Evolution of an effective brain-computer interface mouse via genetic programming with adaptive Tarpeian bloat control, Genetic Programming Theory and Practice IX, forthcoming, 2011.
M. Salvaris, C. Cinel and R. Poli , Novel Sequential Protocols for a ERP Based BCI Mouse, 5th International IEEE EMBS Neural Engineering Conference, forthcoming, 2011.
Riccardo Poli, Caterina Cinel, Luca Citi and Mathew Salvaris, A Genetic Programming Approach to Detecting Artifact-generating Eye Movements from EEG in the Absence of Electro-oculogram, 5th International IEEE EMBS Neural Engineering Conference, forthcoming, 2011.