MATIA - multisensory optic system for the blind (Artificial Intelligence)
Have you ever thought how it is to be blind? Do you know that almost 80% of information from the outside world come to us through our eyes. Including 63 thousand Polish citizens more than 45 mln people in the world live without eyes. There is more and more visually impaired people and that is the problem we would like to solve.
Living means processing information. We are surrounded by the growing avalanche of visual information. Paradoxically, while the world is developing the problem of missing eyesight is becoming more and more serious. The number of sight disorders (like diabetes) and the number of blind people are growing constantly. The problem is that we are not able to find on market an effective antidote for the missing organ of sight. For over a year we were contacting with the blind and with the foundations such as Audidescrypcja. They confirmed our conclusion. According to their suggestions I created an artificial eye for the blind. This system of vision attempts to solve the problem of sight in global way.
The core of MATIA, a multisensory optic system for the blind, demented and those diagnosed with Alzheimer’s disease, is a small and mobile optic module which could be strapped to almost any part of the body. This has been equipped with two forward facing cameras, ultrasound sensors, and ten other sensors, which of whom use the central system to send data to the paired smartphone.
The main function of the system is the ability to describe the users surrounding with words and music. It will describe the location, size, and traits of certain obstacles in front of the user. The use of deep neural networks allowed for precise audio description of many objects at once. The musical trait is used as a secondary channel to warn the user of the difference between certain obstacles, for example a street pole will make a different sound than a trash can. An important trait of the system is the ability to recognize faces and their emotional status. In an emergency, MATIA automatically will call for assistance and connect with the users guardian. In the case of a low battery, the system will auto activate an emergency status that will allow safety on user's way home. There are several more functions that are used by the blind today, such as text reading, pedestrian crossing allocation, dangerous situation alert, store product presentation, stoplight color reading, currency recognition, step counter and many other functions.
An important part of MATIA is the virtual control hub, that allows full control of the paired smartphone. This is thanks to gesture control, which happens by either gesturing in front of the optical sensors, or by using a special gesture control ring. With the use of the hud, the risk of phone theft is reduced as the users phone could safely sit inside the pocket.
MATIA works with special but optional accessories that improve the overall abilities. These accessories include a vibrational belt that presents the surrounding area by pressure on the body, and a hat with bone audio converters that sends audio signals to the head without the use of headphones.
In device I used artificial intelligence methods (including, but not limited too, Convolutional Neural Networks, Long / Short Term Memory).
In comparison to other devices, MATIA stands out because:
- It combines most of the necessary functions into one module,
- Implementation of image analysis and artificial intelligence to describe the surrounding objects, through voice, music and touch,
- A system that warns in dangerous situations and of dangerous objects,
- Description based on deep neural networks,
- Automatic help signal in critical situations,
- Human recognition with the ability to read emotions,
- Image recognition based on onboard memory banks,
- Virtual HUD that controls the system, as well as the paired smartphone,
- Minimal costs.
Expected effects in the case of mass use:
- Greater overall safety of the blind in public places,
- Overall rise of cellphone users within the blind community,
- Overall help to the people with dementia and Alzheimer disease.
During testing
PCB fragment
A part of the circuit diagram
The main part of the project is a special application, working on the ANDROID device. It do more sophisticated signal processing than it is in embedded systems (STM32F429).