Research Task 2, Part 2 (Isabelle Greenberg)

Brain-Activity-Driven Real-Time Music Emotive Control:
Abstract:

• Current active music systems allow a user to control aspects like the playback and volume.
• This project will use the Emotiv EPOC headset to receive EEG data and map it to emotional states. Then, the music will transform based on the user's mood

Introduction:

• This description is very similar to the abstract

Background:

• "GoTo" classifies music systems by their ability to control playback, touchup, retrieval, and browsing
• This system will have two parts; a real time system and a system able to adapt based on the real time data

2.1 Emotion Detection

• Methods of emotion detection are voice and expression, skin conductance, heart rate, and pupil dilation
• By measuring the alpha and beta activity on the prefrontal lobe, indicators for arousal and valence can be measured
• This can classify emotions such as happiness, anger, sadness, and calm

2.2 Active Music Listening

• Interactive systems allow the listener to control the music like a conductor using hand gestures
• The parameters are set from high to low, and they can effect the music and the influence of the gestures being made

2.3 Expressive Music Performance

• The KTH music system has a set of thirty rules that control different aspects of expressive music
• The magnitude is controlled by parameter "k", and different combinations of k parameters can create different styles, stylistic conventions or emotional intention
• An example of this situation was shown in 2006 with an arousal/valence plane and 7 of the rules

3 Methodology

• First, the process would begin with the Emotiv Epoch headset getting data in the prefrontal cortex (the sensors are F3, F4, AF3, AF4)
• That cortex regulates emotion and deals with our experience when we are conscious
• Affective states in emotions result from activation or deactivation (this is arousals) and pleasure or displeasure (this is valence)
• The results are happiness, anger, relaxation, and sadness and each has a different combination of arousal and valence

3.1 Signal Reprocessing

• Alpha waves are dominant in relaxed and awake states of mind
• Beta waves are indicators of excited mind states
• The signal has to first be filtered to get these waves separated
• The sample waves are put through a logarithmic equation that I do not presently completely understand but will look into further. It finds the band power.

3.2 Arousal and Valence Calculation

• The arousals are found by adding the beta band power of F3 and F4 and dividing it by the sum of the alpha band power of the same electrodes. The valence is found with a different equation

3.4 Synthesis

• The use of the rules comes in after the pDM program is used to analyze the values found so far

3.4 Experiments

• Two types of experiments were performed, one while sitting and not moving and one while playing an instrument
• The subject was asked to try to change their emotion and then the results from the software would be checked to see whether it was correct

4 Results

• The EEG signal came through and the arousals were mapped out on a graph, but the signal was not very smooth

5 Conclusions

• The opportunities for EEG data to control music is possible, because the arousals and valence values do change in accordance to emotion
• However, the signal is not smooth and more experiments would have to be done for emotions other than happy or sad

Algorithm Notes:

Decision Trees:

• Different patterns are divided into single pattern subsamples in a decision tree
• When there is only one pattern, the decision tree has a leaf
• Threshold: Purity measure of each node to improve feature selection
• The gain is the information value of all the features minus the information value of the nodes nodes
• sample sets are divided into two sets and the threshold can affect how the tree functions
• The entropy is low, or technically zero, when there is all elements of the same class. This means a higher gain
• At present, this math is confusing. The logs are understandable but how the equation is set up is unclear
• The leaf does not need to be processed any further

Self Organizing Maps:

• The inputs are in a table separated by lines and governed by different maximum and minimums 
• The values are updated on a graph for all the input data and a longer process is gone through
• The neurons are displaced?

K-Means Algorithms:

• Proper to compact clusters, sensitive to outliers and noise, and only works with numerical attributes
• The iteration is repeated again and again by increasing a counter each time it runs
• If the classification of one scenario is less than the convergence, it will repeat from an earlier step
• The K is given a  value and it will find the same amount of clusters as in that value
• Uses the random defined means to find all the elements based on the elements nearby to them
• The different means shift around the elements, but it still classifies the elements

Artificial Neural Network Algorithm:

• The inputs are 1 or 0
• The middle is hidden and two neurons are the first hidden layer and another neuron composes the second
• Propagate is to find the value
• The step function in the neuron is a simple graph measured by an equation
• Same inputs can have different weights

Support Vector Machine Algorithm:

• Research hyperplanes
• The larger margin for a hyperplane makes that hyperplane better
• research weight vectors

Random Forest Algorithm

• The combination of learning models increases the classification accuracy
• This idea is behind "bagging" which is to average noisy and unbiased models to create a model with low variance
• Random Forest: Large collection of decorrelated decision trees
• Research "matrix"
• this involves many features
• random subsets using the different matrix to create class prediction

Hopefully more basic research will allow me to process these videos better.

Research Task 2, Part 1 (Isabelle Greenberg)

Mind-Controlled Keyboard:

Introduction:

• Although the subjects for this project are relatively immobile, their eye movements may help to give more possibilities for the research. In the previous papers, extra movements like blinking would move the the next slide and mean something for the program.
• This will rely more on neurophysiologic signals as an access method, or the neuron system in general
• Factors that have to be considered are the users, the environment, and the algorithm behind the process
• There are many ways to make an EEG controlled keyboard and one of them is the RSVP (Rapid Serial Visual Presentation) system. It allows the user to study options one after another by showing the alphabet one letter at a time until the user chooses one
• The headgear must recognize the user's intent to select one of the signals with some form of EEG data
• One of the waves is called P300 (this is probably described more later)
• The language will be in Thai, and the target group is the disabled

2.1 Brain-Computer Interface

• The BCI connects directly between the human brain and the computer, so it will be useful for people with locked in syndrome or ALS
• The user's habits, homes, and their environments will be considered, and they may or may not be considered in our project(the environment may be the most useful in our situation)

2.2 Mind-Controlled Keyboard Interfaces

• Rapid Serial Visual Presentation: Shows one letter at a time and when the expected letter arrives, the positive intent value stands out in comparison to the other letters.
• Matrix Speller: Letters are in rows and columns like a chess board, and moves over the columns one by one. When the expected column arrives, the positive intent signal will distinguish it, and then the speller goes letter by letter in that column
• Hex-o-Spell: This is more visual with groupings of letters in circles. It highlights each circle and loops until the positive intent signal, with which it further breaks down that circle.
• The same idea is used repeatedly but with different setups and appearances.

2.3 Emotiv EPOC

• 14 sensors that track the user's EEG
• Accurately detects when given the user's gender, age, handedness, intentional control, vividness of visual imagery, and mental rotation ability (the ability to move a 2D or 3D image in your mind)
• Test bench is going to be used mostly with this project, and I believe was described more thoroughly in an earlier paper

2.4 EEG Data

• There is a constantly changing electric field on the scalp because of the signals fired by neurons in the brain
• There is 4 types of EEG data:
• P300:
• Detectable peak in activity that occurs 300 ms after some stimulus is presented. This signal helps choose the specific highlighted value when the computer hovers over it
• Slow Cortical Potential:
• Shown by changing voltage in the brain which can be controlled after a long period of time
• Sensorimotor Rhythms:
• Rhythms detected when relaxing while not thinking about movement.
• Steady State visual evoked potential:
• If flashing stimulus is presented to the user, brainwave modulations of the same frequency as the flashing rate of stimulus is detected in the visual cortex.

4.1 Objective and Outputs

• They want to study the signal activities in the brain and get useful info from it, make software with the headset, create new algorithms, and make a new method to the keyboard system
• The outputs will be a program, a report, a new algorithm, and the result of the experiments.
• The new algorithm has to give the fastest solution and the most accurate result

4.2 Benefits

• People with these project can communicate in a new way that's better than before, and the headset will make the setup relatively inexpensive
• In practice the technology we have is insufficient, slow, and sometimes not reliable

Literature Review

• Brain Computer Interfaces is a possible assistive technology (AT) for the disabled and it is classified as an augmentative and alternative communication (AAC) possibility.
• Previous AAC tech has been the joystick, mouse, binary switch, eye gaze and head control, however this does not help a good portion of the disabled
• BCI research for this purpose has 5 components
• Input modalities for the device
• Processing demand of the device
• Language representation
• Output modalities
• Functional gain of the device
• BCI technology has various components
• Stimulus presentation paradigm (check 2.4)
• Signal acquisition (Info received from the headset)
• Preprocessing (filtering out the noise)
• Dimensionality (reducing random variables)
• EEG evidence
• COntextual Evidence
• Joint interference

Available Tech

• Matrix Speller
• first goes column by column then by each value
• Works as a loop until the sentence is completed
• The matrix can be rearranged like how a keyboard is set up in a specific way
• 7.8 characters per minute with 80% accuracy
• Study Case for the Matrix Speller
• This is called the Brain-Computer Interface Virtual Keyboard for accessibility
• 95 keys that had groups then sections then the values themselves, using the drill-down approach
• The accuracy was 61.25%, thus showing how a massive amount of variation in the matrix makes users spend more time selecting characters
• Ways to improve: many groups with the same type of keys inside each group, thus there is less variation
• Rapid Serial Visual Presentation (RSVP)
• A symbol is presented one at a time in the center of the screen rapidly and seemingly randomly (it is not actually random, it just appears to be)
• Depends less on eye gaze control like the previous, but only shows one character at a time
• 5 characters a minute, which cannot be used for conversations
• Must be improved
• A good filter in the program to take out the extra noise is essential (but how will we make a good filter?)
• Symbol selection can increase more after use
• Balanced-Tree Visual Presentation
• Groups in circles balanced in probability according to a Huffman tree
• When the group with the desired symbol is selected the symbols distribute themselves, and there is a "back" symbol in case a mistake is made
• The matrix speller was chosen for this project, but now the algorithm suggests probable words after a word is started.

6.1 Approach

• The approach begins with the input which is obtained with the Emotiv EPOC headset in this situation. There are 14 channels that are communicated over bluetooth to the computer
• The denoise is the first part of the software. The data received has to be cleared of the clutter that affects signal quality.
• Data mining then detects the user's intention, but many instances of the intention have to be recorded so that a relative pattern is found. Afterwards, the pattern triggers the command of the virtual keyboard in real time
• the GUI is made to suit people with disability in Thai and has a word guess function
• The output is shown and the user decides if it is okay, or decides to pass

6.2 Tools and Techniques

• They used a compiler called Eclipse and Java because they are more suited with it
• The testbench and control panel were a major help

The last portion of the paper were examples of previous works and images of those projects.

The next two sets of notes will be posted tomorrow, my internet isn't working that well right now.

Sorry!