Blindsist – Assisting Visually Impaired People using IBM Bluemix Visual Recognition

This blog post will cover how I developed Blindsist app for Xhacknight.

The app was developed using Xamarin.Forms, targeting 3 major platforms (iOS, Android and Windows Phone) same time.

Blindsist is a simple one page application. Objective of the application is taking a photo using Smartphone, The photo is then uploaded to IBM Bluemix using REST Api. IBM Bluemix returns the data in Json format, which will be parsed and spoken out using Native Text to Speech services.

Open a new Xamarin.Forms Portable Class Project.

Right click on the Project Blindsist (Portable), click Add new Item, Select Forms Xaml Page, name it ScanPage.xaml

Update the XAML content in ScanPage as below

What it basically does is create a Button Called “ScanButton” which on click trigger event handler called “ScanButtonClicked”.

Let us look to ScanPage.xaml.cs for C# code!

In the constructor ScanPage() write following Code.

For this to work, we have to add a awesome library developed by JamesMontemagno.

It can be found here https://www.nuget.org/packages/Xam.Plugins.TextToSpeech/

To install, right click project, Manage Nuget Package, Search for Xam.Plugins.TextToSpeech, install it in every individual projects.

So when the app starts up, It starts speaking introduction to visually impaired user.

Next is adding Button Clicked event. Just copy this code in your ScanPage.xaml.cs

Now right click your Portable class project, right click, add new item. Add new class file, name it Library.cs

Add following data members.

Now add a Member function GetImage() for Getting Image from Camera, The specialty of coming code is that it works in all 3 platforms with single code. Thanks to another library, which is available in Nuget. Search for Xam.Plugin.Media and install in every projects.

It will check whether Camera is available, if not available return a null object with error message embedded.

Else it will create a file, takes a photo from Camera and store as byte[] format, if file is OK, it returns the Image, with Image Url, and Image in byte[] format.

Now add a member function ProcessImage() to process the image by sending to Bluemix Watson Service.

Each line is commented for understanding what happens there. Atlast we get string bb with Json formatted data from IBM Bluemix with possible results in order of priority.

Now revisit ScanPage.xaml.cs, We have to add “GetPicture()” function on button click. Copy the following code.

Now string temp has the json data, we have to parse it to get the results. Add new class file called Classify.cs in PCL project.

Using http://json2csharp.com we can generate class required for parsing json data from IBM Bluemix service.

To parse json data, again there is a good library called NetwonSoft.Json, obtain it from Nuget, and add it to all projects.

Now we need a simple function to perform deserializing json data.

Create a static class Operations, and a function GetMatch()

Now back to ScanPage.xaml.cs, We have temp ready with Json string. Just pass it to Operations.GetMatch() to get the required data back. Add following codes to where you stopped earlier.

That’s all.. Try Building and running the project. If you face any errors it might be due to not adding Nuget Package mentioned in this post for all projects, or some other errors. Feel free to ask. Together we can build this project to perfection.

The project is in github, https://github.com/muhaym/Xamarin.Forms-Blind-Assist

Feel free to fork, modify, report errors, ask doubts and do whatever you want with the project for the sake of learning.

Please assist me to add Image Compression, because the image upload to Watson Service and getting response back is taking hell of time now.

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.