- Carolina Mehret
- Alexander Ortenburger
- Daniel Rexin
Alex' girlfriend trains guide dogs for blind people, so he has pretty often contact to blind people and realized the need for a cheap and easy way to participate in everyday activities like going grocery shopping or eating out. There are about 100k blind people in Germany and about 170k profound visually impaired which can not participate in everyday's activities independently. Right now they still rely on social workers who help them to master the daily grind. We do have the technology and the algorithms to provide them significant help and improve their quality of life - let's tackle this problem!
What it does
The app speakingEye is the personal speech assistant for visually impaired people. speakingEye uses the smartphone camera to capture and detect objects and texts to support visually impaired people in their daily life. It can be completely controlled by voice commands or gestures and uses text to speech to guides and supports the user through the world. It can be used for several use cases. The use case we implement at opencodes is a grocery shopping assistant. One of the biggest struggles of visually impaired people is shopping for groceries. They have no ability to compare prices of products. One workaround for them is, that they use a text to speech engine to "listen" to an online shopping portal (i.e. dm.de, real.de). This is not applicable in an local "offline" store. Using computer vision and machine learning algorithms the app detects the products and their prices in the shelf and communicates them to the user, so the user can compare the prices by themselves.
Challenges we ran into
Accomplishments that we are proud of
Submitting our project on Devpost
What we have learned
Sleep is for the weak
Use CTRL+A, CTRL+C before submitting on Devpost
What's next for speakingEye
Implement more use cases: We want to provide even more use cases. I.e. reading the menu in a restaurant, describing colors, integrating into other services like yelp or google maps for sightseeing, bus schedule, ...
Adding a hardware camera: To make it even more convenient to use, we want to design a hardware camera which is similar to google glass without the screen but with speakers.
Get feedback for the app from visually impaired people: We want to share the app with a few visually impaired people to get feedback. With that we follow the agile developing paradigm to improve the user experience.
Cooperate with CBM (Christoffel-Blindenmission): To reach and support people in third world countries we want to cooperate with CBM.
Utilizing NLP (natural language processing): One of the most requested features by visually impaired people of speaking assistants is to optimize voice output. We want to use NLP algorithm to provide the same information using fewer words.
Benefit from the digitalization in the retail sector: The retail sector is moving quickly towards digitalization. We still have barcodes on products but no RFID-tags, but maye in a few years. We benefit from this movement. We might even provide in store navigation for our users.
Sell localized price data to retailers: speakingEye recognizes a product with its price inside a local store. We store the gathered information inside a database. Local prices are extremely valuable for retailers, so they can automatically observe and monitor the prices of their competitors.
Freemium app model: The free app is fully functional and has the same features. The freemium model lets you just cutomize the app for your personal needs.
Crowdfunding the hardware: Our goal is to sell an affordable hardware camera. To achieve that we want to crowdfund the needed money for research and development of the hardware camera.
- OrCam: All-in-one hardware solution. Priced around 3.500$ US.
- Barcode Scanners (i.e. Scandit): Can just scan barcodes, but have no image recognition.
- image recognition (i.e. Google goggles): Does not provide the pricing information.
- Germany: 100k blind people and 165k profound visually impaired (below 5% eyesight)
- Worldwide: about 161 mil profound visually impaired people in 2002
At least one third of them is a potential user of speakingEye because they are below the age of 65 and therefore most likely a smartphone user. source: WHO (http://www.who.int/bulletin/volumes/82/11/en/844.pdf)
You need to have node.js, ionic framework,cordova, ng-cordova, android sdk, x code builder, ios deploy. To install these check their documentation.
- run 'ng install' in the speakingEye folder
- run 'ionic serve' in the speakingEye folder to test live
- run 'ionic platform add ios' and 'ionic build ios' in the speakingEye folder to build for ios or 'ionic platform add android' and 'ionic build android' to build for android
https://southcentralus.api.cognitive.microsoft.com/customvision/v1.1/Prediction/3af68434-e1d6-4139-9f9a-d669bc54ca46/image?iterationId=61927395-10f0-4e3a-b89a-7fbb9cfd80ab Prediction-Key: d3c0118f34a0484b83dae3fad50cbd05
set body to <image file>