Windows Phone – Abdul Muhaymin Arif

Blindsist – Assisting Visually Impaired People using IBM Bluemix Visual Recognition

This blog post will cover how I developed Blindsist app for Xhacknight.

The app was developed using Xamarin.Forms, targeting 3 major platforms (iOS, Android and Windows Phone) same time.

Blindsist is a simple one page application. Objective of the application is taking a photo using Smartphone, The photo is then uploaded to IBM Bluemix using REST Api. IBM Bluemix returns the data in Json format, which will be parsed and spoken out using Native Text to Speech services.

Open a new Xamarin.Forms Portable Class Project.

Right click on the Project Blindsist (Portable), click Add new Item, Select Forms Xaml Page, name it ScanPage.xaml

Update the XAML content in ScanPage as below

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"
xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
x:Class="Blindist.ScanPage">
<Grid BackgroundColor="#dbdef0"
Padding="0"
RowSpacing="0"
ColumnSpacing="0"
VerticalOptions="FillAndExpand">
<Grid.RowDefinitions>
<RowDefinition Height="150" />
<RowDefinition Height="*" />
</Grid.RowDefinitions>
<Image Source="logo.png" Grid.Row="0" BackgroundColor="#101536"/>
<StackLayout Grid.Row="1">
<Button x:Name="ScanButton" HeightRequest="650" Text="Scan the environment" Clicked="ScanButtonClicked" BackgroundColor="#27327e"></Button>
<Image WidthRequest="50" HeightRequest="50"  x:Name="image"/>
</StackLayout>
</Grid>
</ContentPage>

<?xml version="1.0" encoding="utf-8" ?>

<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"

xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"

x:Class="Blindist.ScanPage">

<Grid BackgroundColor="#dbdef0"

Padding="0"

RowSpacing="0"

ColumnSpacing="0"

VerticalOptions="FillAndExpand">

<Grid.RowDefinitions>

</Grid.RowDefinitions>

</StackLayout>

</Grid>

</ContentPage>

What it basically does is create a Button Called “ScanButton” which on click trigger event handler called “ScanButtonClicked”.

Let us look to ScanPage.xaml.cs for C# code!

In the constructor ScanPage() write following Code.

CrossTextToSpeech.Current.Speak("Hello there, I am here to help you, Please click the screen two times with 5 second gap for knowing what is in front of you. Point your Smartphone camera to the direction required.");

1	CrossTextToSpeech.Current.Speak("Hello there, I am here to help you, Please click the screen two times with 5 second gap for knowing what is in front of you. Point your Smartphone camera to the direction required.");

For this to work, we have to add a awesome library developed by JamesMontemagno.

It can be found here https://www.nuget.org/packages/Xam.Plugins.TextToSpeech/

To install, right click project, Manage Nuget Package, Search for Xam.Plugins.TextToSpeech, install it in every individual projects.

So when the app starts up, It starts speaking introduction to visually impaired user.

Next is adding Button Clicked event. Just copy this code in your ScanPage.xaml.cs

async void ScanButtonClicked(object sender, EventArgs args)
{
//will update soon
}

async void ScanButtonClicked(object sender, EventArgs args)

{

//will update soon

}

Now right click your Portable class project, right click, add new item. Add new class file, name it Library.cs

Add following data members.

public string error { get; set; } // to update error in fronted 
public  byte[] imgraw { get; set; } // contains Byte[] format Image to be sent to Watson
public ImageSource img { get; set; }

private readonly string url = "https://gateway.watsonplatform.net/visual-recognition-beta/api";
//Create Bluemix account for Username/password
private readonly string username = "your-user-name-here";
private readonly string password = "your-password-here";

public string error { get; set; } // to update error in fronted

public byte[] imgraw { get; set; } // contains Byte[] format Image to be sent to Watson

public ImageSource img { get; set; }

private readonly string url = "https://gateway.watsonplatform.net/visual-recognition-beta/api";

//Create Bluemix account for Username/password

private readonly string username = "your-user-name-here";

private readonly string password = "your-password-here";

Now add a Member function GetImage() for Getting Image from Camera, The specialty of coming code is that it works in all 3 platforms with single code. Thanks to another library, which is available in Nuget. Search for Xam.Plugin.Media and install in every projects.

public async Task<ImageProcess> GetImage()
{
if (!CrossMedia.Current.IsCameraAvailable || !CrossMedia.Current.IsPickPhotoSupported)
{
ImageProcess ob = new ImageProcess()
{
error = "Oh No ! No Camera.. I can't access your camera",
imgraw = null,
img = null
};
return ob;
}
var file = await CrossMedia.Current.TakePhotoAsync(new Media.Plugin.Abstractions.StoreCameraMediaOptions
{
Directory = "Sample",
Name = "test.jpg"
});
if (file == null)
return null;
var  imgrawtemp = new BinaryReader(file.GetStream()).ReadBytes((int)file.GetStream().Length);
var image = ImageSource.FromStream(() =>
{
var stream = file.GetStream();
file.Dispose();
return stream;
});
ImageProcess temp = new ImageProcess()
{   imgraw = imgrawtemp,
error = "Success",
img = image
};
return temp;
}

public async Task<ImageProcess> GetImage()

{

if (!CrossMedia.Current.IsCameraAvailable || !CrossMedia.Current.IsPickPhotoSupported)

{

ImageProcess ob = new ImageProcess()

{

error = "Oh No ! No Camera.. I can't access your camera",

imgraw = null,

img = null

};

return ob;

}

var file = await CrossMedia.Current.TakePhotoAsync(new Media.Plugin.Abstractions.StoreCameraMediaOptions

{

Directory = "Sample",

Name = "test.jpg"

});

if (file == null)

return null;

var imgrawtemp = new BinaryReader(file.GetStream()).ReadBytes((int)file.GetStream().Length);

var image = ImageSource.FromStream(() =>

{

var stream = file.GetStream();

file.Dispose();

return stream;

});

ImageProcess temp = new ImageProcess()

{ imgraw = imgrawtemp,

error = "Success",

img = image

};

return temp;

}

It will check whether Camera is available, if not available return a null object with error message embedded.

Else it will create a file, takes a photo from Camera and store as byte[] format, if file is OK, it returns the Image, with Image Url, and Image in byte[] format.

Now add a member function ProcessImage() to process the image by sending to Bluemix Watson Service.

public async Task<string> ProcessImage(byte[] img)
{
CrossTextToSpeech.Current.Speak("Please Wait while we process your image!");
try
{
var multiPartContent = new MultipartFormDataContent();
var byteArrayContent = new ByteArrayContent(img);
byteArrayContent.Headers.Add("Content-Type", "image/jpeg");
multiPartContent.Add(byteArrayContent, "img_File", "temp.jpg"); // Embed the Image in byte[] format, with "img_File" format with temp.jpg file name.
using (var client = new HttpClient())
{
CrossTextToSpeech.Current.Speak("Please Wait while we process your image!");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", GetBase64CredentialString());

var response = await client.PostAsync("https://gateway.watsonplatform.net/visual-recognition-beta/api/v1/tag/recognize", multiPartContent);
CrossTextToSpeech.Current.Speak("Please Wait while we process your image!");
if (response.IsSuccessStatusCode == true)
{
return await HandleResponseAsync(response);
}
else
{
return null;
}

}
}
catch (Exception ex)
{
return null;
}
}
private async Task<string> HandleResponseAsync(HttpResponseMessage response)
{
string bb = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
return bb; // this bb will contain string in json format!
}
private string GetBase64CredentialString()
{
var auth = string.Format("{0}:{1}", username,password);
return Convert.ToBase64String(Encoding.UTF8.GetBytes(auth));
}

public async Task<string> ProcessImage(byte[] img)

{

CrossTextToSpeech.Current.Speak("Please Wait while we process your image!");

try

{

var multiPartContent = new MultipartFormDataContent();

var byteArrayContent = new ByteArrayContent(img);

byteArrayContent.Headers.Add("Content-Type", "image/jpeg");

multiPartContent.Add(byteArrayContent, "img_File", "temp.jpg"); // Embed the Image in byte[] format, with "img_File" format with temp.jpg file name.

using (var client = new HttpClient())

{

CrossTextToSpeech.Current.Speak("Please Wait while we process your image!");

client.DefaultRequestHeaders.Accept.Clear();

client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", GetBase64CredentialString());

var response = await client.PostAsync("https://gateway.watsonplatform.net/visual-recognition-beta/api/v1/tag/recognize", multiPartContent);

CrossTextToSpeech.Current.Speak("Please Wait while we process your image!");

if (response.IsSuccessStatusCode == true)

{

return await HandleResponseAsync(response);

}

else

{

return null;

}

catch (Exception ex)

{

return null;

}

private async Task<string> HandleResponseAsync(HttpResponseMessage response)

{

string bb = await response.Content.ReadAsStringAsync().ConfigureAwait(false);

return bb; // this bb will contain string in json format!

}

private string GetBase64CredentialString()

{

var auth = string.Format("{0}:{1}", username,password);

return Convert.ToBase64String(Encoding.UTF8.GetBytes(auth));

}

Each line is commented for understanding what happens there. Atlast we get string bb with Json formatted data from IBM Bluemix with possible results in order of priority.

Now revisit ScanPage.xaml.cs, We have to add “GetPicture()” function on button click. Copy the following code.

       ImageProcess imgtemp = new ImageProcess(); // creates imgttemp object of ImageProcess Class.
            imgtemp =  await imgtemp.GetImage(); // Gets image from Camera to imgtemp

            if(imgtemp !=null) //if there is something in imgtemp
            {
                if(imgtemp.img!=null) // checks whether the image source is empty!
                {
                    image.Source = imgtemp.img; //Set Image in front end as taken image.
                  
                    string temp = await imgtemp.ProcessImage(img.imgraw); // passes the image in byte[] format to be processed, returns json formatted data of possible outcomes
                    if (temp == null)
                    {
                        CrossTextToSpeech.Current.Speak("I can't connect at the moment, try again!");
                        return;
                    }
//Now temp has json data, we have to parse it

                 }
                else
                {
                    await DisplayAlert("Error", imgtemp.error, "Ok");
                    ScanButton.IsEnabled = true;
                }
            }

ImageProcess imgtemp = new ImageProcess(); // creates imgttemp object of ImageProcess Class.

imgtemp = await imgtemp.GetImage(); // Gets image from Camera to imgtemp

if(imgtemp !=null) //if there is something in imgtemp

{

if(imgtemp.img!=null) // checks whether the image source is empty!

{

image.Source = imgtemp.img; //Set Image in front end as taken image.

string temp = await imgtemp.ProcessImage(img.imgraw); // passes the image in byte[] format to be processed, returns json formatted data of possible outcomes

if (temp == null)

{

CrossTextToSpeech.Current.Speak("I can't connect at the moment, try again!");

return;

}

//Now temp has json data, we have to parse it

}

else

{

await DisplayAlert("Error", imgtemp.error, "Ok");

ScanButton.IsEnabled = true;

}

Now string temp has the json data, we have to parse it to get the results. Add new class file called Classify.cs in PCL project.

Using http://json2csharp.com we can generate class required for parsing json data from IBM Bluemix service.

public class Label
{
public string label_name { get; set; }
public string label_score { get; set; }
}
public class Image
{
public string image_id { get; set; }
public string image_name { get; set; }
public List<Label> labels { get; set; }
}

public class RootObject
{
public List<Image> images { get; set; }
}

public class Label

{

public string label_name { get; set; }

public string label_score { get; set; }

}

public class Image

{

public string image_id { get; set; }

public string image_name { get; set; }

public List<Label> labels { get; set; }

}

public class RootObject

{

public List<Image> images { get; set; }

}

To parse json data, again there is a good library called NetwonSoft.Json, obtain it from Nuget, and add it to all projects.

Now we need a simple function to perform deserializing json data.

Create a static class Operations, and a function GetMatch()

  static public class Operations
{
static public List<Label> GetMatch(string json)
{
var result = JsonConvert.DeserializeObject<RootObject>(json); // Will deserialize the json data to RootObject type
return result.images[0].labels; Return required data to our Button Event Handler.
}

}

static public class Operations

{

static public List<Label> GetMatch(string json)

{

var result = JsonConvert.DeserializeObject<RootObject>(json); // Will deserialize the json data to RootObject type

return result.images[0].labels; Return required data to our Button Event Handler.

}

Now back to ScanPage.xaml.cs, We have temp ready with Json string. Just pass it to Operations.GetMatch() to get the required data back. Add following codes to where you stopped earlier.

var labels =  Blindist1.Operations.GetMatch(temp);
labels = labels.OrderByDescending(x => x.label_score).ToList();

CrossTextToSpeech.Current.Speak("I guess there is " + labels[0].label_name + " in front of you" + "Also there might be " + labels[1].label_name + " in front of you");
ScanButton.IsEnabled = true;

var labels = Blindist1.Operations.GetMatch(temp);

labels = labels.OrderByDescending(x => x.label_score).ToList();

CrossTextToSpeech.Current.Speak("I guess there is " + labels[0].label_name + " in front of you" + "Also there might be " + labels[1].label_name + " in front of you");

ScanButton.IsEnabled = true;

That’s all.. Try Building and running the project. If you face any errors it might be due to not adding Nuget Package mentioned in this post for all projects, or some other errors. Feel free to ask. Together we can build this project to perfection.

The project is in github, https://github.com/muhaym/Xamarin.Forms-Blind-Assist

Feel free to fork, modify, report errors, ask doubts and do whatever you want with the project for the sake of learning.

Please assist me to add Image Compression, because the image upload to Watson Service and getting response back is taking hell of time now.

How to develop a Xamarin.Forms Hello World Application – Xamarin.Forms Series Part 1

Xamarin.Forms a quick Introduction

Today, we will learn how to develop a cross platform Hello World application using Xamarin.Forms.

Xamarin.Forms is not the final word for Cross platform Mobile app development. It is best for getting started with Xamarin, and targetting 3 major platform by sharing 99% Code.

Xamarin.Android or Xamarin.iOS is best for application dealing with high device specific API usage like geolocation, and other such capabilities. Through Xamarin.Android or Xamarin.iOS a C# developer can share the business logic among Projects and design Native UI’s and make use of Device specific capabilities for 3 platforms independently.

Xamarin.Forms on other hand can be used for simple application where there is less Platform specific API usage.

Xamarin.Forms is great for prototyping or making a quick prototype application. And after you’ve done that, you might just find that you can continue using Xamarin.Forms features to build the entire application.

Hello World using Xamarin.Forms

To get started you must have Installed latest version of Xamarin, I am using Visual Studio for developing the Hello World application so that I can target 3 platform. But I wont be showing iOS screenshots because I don’t have an Apple device in my network. But don’t worry, all the implementations we do here will hopefully work in iOS device.

Hello World! The first application you probably create when you learn new language or start using new development tool.

I am not covering everything in detail, like the basic C# concepts, or Xamarin.Forms complete capabilities. This tutorial is intended for Getting started with Xamarin.Forms, everything in detail can be learned from Creating Mobile Apps with Xamarin.Forms Book Preview 2 by Charles Petzold http://www.microsoftvirtualacademy.com/ebooks#9780735697232

Open Visual studio, and Create new project.

Select Xamarin.Forms Portable Class Library Project (PCL). As Xamarin Documentation suggests to go for shared if our Project contains XAML (Extensible Markup language, usually pronounced Zammel), in which we will be developing from now on.

For Xamarin.Forms solutions that contain XAML, we recommend using the Xamarin.Forms Portable Class Library (PCL) template rather than using Shared Asset Projects (SAP). The PCL approach isolates Xamarin.Forms code in a separate DLL, while the SAP shares code directly among the three platforms. Before using XAML with the Shared Asset Project (SAP) template read the details below on enabling experimental XAML support in SAPs.

Create Xamarin.Forms PCL New Project in VS2015.

The Xamarin.Forms Blank template will set up projects for iOS, Android and Windows Phone 8.0. Feel free to delete any project if you are not going to target it.

Right click the solution, click Build Solutions and Run the project in platform of your choice by setting startup project.

Hint for beginners: Startup Project can be changed by right clicking desired Project in Solution, and Selecting “Set as startup project”.

You can see that app is generated as above screenshot.

Now let us dig deep, so we can understand what is happening.

Each Projects in the solution has

iOS – AppDelegate.cs
Android – MainActivity.cs
Windows Phone – MainPage.xaml.cs

These three files are the starting points of respective platforms. All these files contain a line

LoadApplication (new HelloWorld.App ());

1	LoadApplication (new HelloWorld.App ());

Here HelloWorld is our Project Shared, App is the entry point (App.cs) in our PCL Project.

This line invokes the App.cs in the main HelloWorld Project which is where we write code to Share among three platform, which contains

            MainPage = new ContentPage
            {
                Content = new StackLayout
                {
                    VerticalOptions = LayoutOptions.Center,
                    Children = {
                        new Label {
                            XAlign = TextAlignment.Center,
                            Text = "Welcome to Xamarin Forms!"
                        }
                    }
                }
            };

MainPage = new ContentPage

{

Content = new StackLayout

{

VerticalOptions = LayoutOptions.Center,

Children = {

new Label {

XAlign = TextAlignment.Center,

Text = "Welcome to Xamarin Forms!"

}

};

MainPage is the entry point of your app, the first page displayed in app. Why I love programming with help of XAML is that I can write the complex parts like the new ContentPage{ … part in above code in more understandable form.

So We just want to create a new entry page. Right Click HelloWorld (Portable) Project, Click Add new Item.

Select Forms Xaml Page, and give it a name FirstPage.xaml. We will use this project as continuation for next tutorial in the series, navigation of pages in Xamarin.Forms.

In FirstPage.xaml, write the code as below

<?xml version=”1.0″ encoding=”utf-8″ ?>
<ContentPage xmlns=”http://xamarin.com/schemas/2014/forms”
xmlns:x=”http://schemas.microsoft.com/winfx/2009/xaml”
x:Class=”HelloWorld.FirstPage”>
<StackLayout VerticalOptions=”Center” HorizontalOptions=”Center” >
<Label Text=”I love XAML” />
</StackLayout>
</ContentPage>

Now we will compare with the first set of code we wrote (or was there in default in App.cs).

new ContentPage { is a main node <ContentPage> in xml format

new StackLayout { is a child node <StackLayout> under <ContentPage>

new Label { is again a child node <Label> under <StackLayout>.

In my view this is simpler than coding it in C#. You can code in the way you want.

Now we want this FirstPage.xaml loaded in startup. Just a small changes in App.cs will get it done.

MainPage = new FirstPage();

1	MainPage = new FirstPage();

and comment out the rest of part so that you can compare and study.

Now Build the project and run again.

Voila! The new Xaml Page loads in startup.

So in our First Series, we have learned bit more than a simple Hello World app.

We covered

Creating Xaml.Forms New PCL Project.
Building it and Running.
Coding UI in C# and XAML.
Loading any Xamarin.Form Page as MainPage.

Shout out your suggestions, feedbacks and doubts in comment section below. To be frank this is the first time I am writing tutorial, so your suggestions will help me improve my self.

The HelloWorld Solution can be downloaded from my Github (not a good Project to share, still you guys can fork it and start building on it in case of any issues)

https://github.com/muhaym/Xamarin.Forms.HelloWorld

Xamarin Students Offer

Hey Guys,

I have been long since I have blogged. Sorry for being away. I think I am the laziest person in this world.

Today I have something to share with you guys.

Hope you all know about Xamarin, Xamarin is a cross platform mobile application development tool, Which lets you create applications for major platforms such as iOS, Android and Windows Phone. I will be blogging about developing cross platform application using Xamarin. So this tool is sooooooo costly that we students, or startup can’t afford.

Wait, But what if they want you to develop something great? Yea.. Xamarin is free for students, you can apply for a Students license which will let you develop iOS, Android application using Visual Studio or Xamarin Studio.

It is very easy to apply for Xamarin Student License. Just mail your student ID card scanned copy or some document to verify that you are student to student – at – xamarin dot com.

For more details visit http://xamarin.com/student

Sit tight and we will start developing cross platform application from next post onward.