Friday, June 19, 2020

Augmented Emotional Intelligence

I am a Maker.

 
But what does it mean to be a Maker, and how does being a Maker impact one's life?

 
My path to becoming a Maker involved learning electronics, simple mechanical engineering, machine learning, 3D printing, robotics, machining and milling, vacuum molding, parametric modeling and simulation, PCB design, sewing, laser cutting, carpentry, and also first aid unfortunately. But it was not becoming proficient in any one of these skills that caused me to think of myself as a Maker. Somewhere on the path of learning all of these skills I realized that I had started to look at the world in a different way; I had begun to see the world as a continuous series of opportunities to create solutions for problems and challenges in my own and other people's lives, and that the only thing preventing me from solving any given problem was my own imagination. I also realized that this wasn't only constrained to problems that had solutions in the domains of the aforementioned skills, but rather, all the problems and challenges I faced in all aspects of my life. It is this attitude of being willing to attempt to solve any problem, with all the skills that one can bring to bear, or with new skills that you may need to learn, and seeing every challenge as on opportunity to create a solution, that makes one a Maker.


This Maker ethos has inspired me to make many gadgets, toys, tools, programs and processes; from the ultimate 3D-printed back-scratcher, to a machine learning-based pan, tilt and pedestal, face-tracking gimbal for a web camera. However, the project I am most proud of, and invested in, is an Augmented Emotional Intelligence wearable for children and adults with Autism Spectrum Disorder (ASD).


ASD has impacted the lives of several people who I  care dearly about, and this has motivated me to become very involved in Neurodiversity advocacy. The challenges that those people, and all neuro-atypical people, face daily, has inspired me to apply my Making mindset to reducing the stress and anxiety of Autistic people. One of the challenges that many people "on the Spectrum" face is that they are not able to automatically and instinctively read the emotions of the people around them. There is a misconception that Autistic people have low empathy. Autistic people in fact generally have high emotional empathy, which means that they are very sensitive to the emotions of others. Unfortunately, they also tend to have low cognitive empathy, which means that they are not able to accurately identify emotions, nor construct accurate causal narratives for why someone might be feeling a specific emotion. This lack of ability to correctly interpret the emotional states of others puts them at higher risk than neurotypical people; particularly when those emotions are frustration, anger and disgust.


I decided to attempt to build a wearable device that would be able to recognize the emotions of the people around the wearer, and give the user discreet feedback so that they were able to recognize and appropriately respond to those emotions. I coined the term "Augmented Emotional Intelligence" to describe the essential function of the device. I chose to use a haptic device to give the user feedback, and employ Affective Haptics and Emotion Elicitation to indicate which emotions were being detected, and the intensity of those emotions. The device would detect emotions that might cause the wearer stress or anxiety, or potentially expose them to harm from others, primarily frustration, anger and disgust. Not only would the device give the wearer feedback but could also automatically call for help from a caregiver, in the face of intense negative emotions, with a location, and a summary of the detected emotions. The device could also include sensors to detect the wearer's emotions, though this was not part of the scope of my initial project.

 
Machine learning technology has come a long way in a very short time, and Deep Neural Networks as they have been applied to Computer Vision, have given us the ability to computationally efficiently recognize and classify Faces in images and video. One of the sub-domains that this is being applied to is Human Emotion Detection and Recognition. Neural Networks that use Facial Coding features, which break the face into several groups of muscles, can relatively accurately recognize primary emotions, though they are not yet able to efficiently measure more subtle emotions or interpret micro-expressions. Voice and speech Recognition, which have also hugely benefited from the invention of Deep Neural Networks, can also be used to recognize emotions, either on their own, or combined with video data, but these significantly increase the overall computational costs of the models or systems.


My initial, and unfortunately naïve goal was to use software-based machine learning on a microcomputer to detect micro-expressions captured from an attached camera. My initial attempt used a Raspberry Pi 3B, OpenMV, and Python. I very quickly discovered that the Pi was woefully computationally inadequate for running even the simplest ML models for primary emotion detection, let alone micro-expression detection, which have durations of less than 500ms. I resorted to using the embedded camera and microcomputer for no more than facial detection and then once detected, sending the image of the face to Microsoft’s Cognitive Services face/emotion detection service. Though the results, even from the cloud service, were not adequate for a real product, they were good enough to demonstrate what a real product might look like.


In the evolution of this project I have developed versions of the device based on multiple versions of the Kendryte K210 based Sipeed MAIX, which includes a Neural Network Accelerator; multiple versions of NVIDIA’s Jetson platform, including the Jetson TX1, TX2 and Nano; and the Google Coral TPU. I have also tried several pre-trained emotion detection models, and models that I have trained myself using existing datasets. This project has also motivated me to collaborate with researchers at Microsoft Research who are working in similar areas, primarily in Accessibility. Unfortunately, the technology is not yet at the point that a device with the required specifications could be manufactured. Despite all the software advances that have been made in Machine Learning, and the hardware advances that have brought ML to embedded devices, the detection of micro-expressions on a System- or Module-on-a-Chip is still not possible, and most sophisticated emotion detection models will use 100% of the CPU, GPU and memory of a modern, high-end engineering workstation.


But the software and hardware continue to improve at an ever-accelerating rate. Models are now being trained on truly massive data sets, and new techniques are being discovered that dramatically improve detection, recognition and computational efficiency in general. The Internet of Things, Edge AI, robotics, and autonomous vehicle research and development are driving substantial hardware innovation in embedded AI.


At some point in the not-too-distant future my vision of an Augmented Emotional Intelligence wearable that will meet the performance, reliability, predictability and durability requirements for a device that will protect vulnerable neurodiverse adults and children, will be realized. Until that day I will continue to evolve my prototypes and apply my Maker mindset to the problem.

 
Even if this product is never brought to market, it will continue to give me a platform to talk about Neurodiversity and be an advocate and ally for people with ASD. And that is a cause worth pursuing in its own right.