What is Azure Cognitive Service? Complete understanding with Face API example.

I would recommend you to read the articles mentioned below-

In this article, I will walk through step-by-step of exploring FACE and EMOTION API’s of Microsoft Cognitive Services. Henceforth, this article will cover below three parts and I would suggest you to go through this article in the below given order only:-

So, below are the prerequisites to work with Microsoft Cognitive Services API:-

Part 2

FACE API using Azure Cognitive Service

Face API, provided by Microsoft Cognitive Services, helps to detect, analyse and organize the faces in a given image. We can also tag faces in any given photo.

Step 2 – Once you click on “AI + Cognitive Services”, you will see list of API’s available in Cognitive services:-

 

Step 3 – Choose Face API to subscribe to Microsoft Cognitive Services Face API and proceed further with subscription steps. After clicking on FACE API, it will show the legal page, read it carefully and then click on Create.

Below are the generated Keys and Endpoint URL, in my case:-

1.       using Microsoft.ProjectOxford.Face;

2.       using Microsoft.ProjectOxford.Face.Contract;

To add the above references, browse for two dll’s and click on install:-

1.       Newtonsoft.JSON

2.       Microsoft.ProjectOxford.Face

 

Once you will add these two dll’s, these will be shown in the solution as follows:-

 

Step 8 – Add lines of code given below to click event of Browse button

 

private async void BrowseButton_Click(object sender, RoutedEventArgs e)         {             // Get the image file to scan from the user.             var openDlg = new Microsoft.Win32.OpenFileDialog();             openDlg.Filter = “JPEG Image(*.jpg)|*.jpg”;             bool? result = openDlg.ShowDialog(this);             // Return if canceled.             if (!(bool)result)             {                 return;             }             // Display the image file.             string filePath = openDlg.FileName;             Uri fileUri = new Uri(filePath);             BitmapImage bitmapSource = new BitmapImage();             bitmapSource.BeginInit();             bitmapSource.CacheOption = BitmapCacheOption.None;             bitmapSource.UriSource = fileUri;             bitmapSource.EndInit();             FacePhoto.Source = bitmapSource;             // Detect any faces in the image.             Title = “Detecting…”;             faces = await UploadAndDetectFaces(filePath);             Title = String.Format(“Detection Finished. {0} face(s) detected”, faces.Length);             if(faces.Length > 0)             {                 // Prepare to draw rectangles around the faces.                 DrawingVisual visual = new DrawingVisual();                 DrawingContext drawingContext = visual.RenderOpen();                 drawingContext.DrawImage(bitmapSource, new Rect(0, 0, bitmapSource.Width, bitmapSource.Height));                 double dpi = bitmapSource.DpiX;                 resizeFactor = 96 / dpi;                 faceDescriptions = new String[faces.Length];                 for (int i = 0; i < faces.Length; ++i)                 {                     Face face = faces[i];                     // Draw a rectangle on the face.                     drawingContext.DrawRectangle(                         Brushes.Transparent,                         new Pen(Brushes.Red, 2),                         new Rect(                             face.FaceRectangle.Left * resizeFactor,                             face.FaceRectangle.Top * resizeFactor,                             face.FaceRectangle.Width * resizeFactor,                             face.FaceRectangle.Height * resizeFactor                             )                     );                     // Store the face description.                     faceDescriptions[i] = FaceDescription(face);                 }                 drawingContext.Close();                 // Display the image with the rectangle around the face.                 RenderTargetBitmap faceWithRectBitmap = new RenderTargetBitmap(                     (int)(bitmapSource.PixelWidth * resizeFactor),                     (int)(bitmapSource.PixelHeight * resizeFactor),                     96,                     96,                     PixelFormats.Pbgra32);                 faceWithRectBitmap.Render(visual);                 FacePhoto.Source = faceWithRectBitmap;                 // Set the status bar text.                 faceDescriptionStatusBar.Text = “Place the mouse pointer over a face to see the face description.”;             }         }

 

Add a new await function named UploadAndDetectFaces(), which accepts imageFilePath as an object parameter.

 

// Uploads the image file and calls Detect Faces.         private async Task<Face[]> UploadAndDetectFaces(string imageFilePath)         {             // The list of Face attributes to return.             IEnumerable<FaceAttributeType> faceAttributes =                 new FaceAttributeType[] { FaceAttributeType.Gender, FaceAttributeType.Age, FaceAttributeType.Smile, FaceAttributeType.Emotion, FaceAttributeType.Glasses, FaceAttributeType.Hair };             // Call the Face API.             try             {                 using (Stream imageFileStream = File.OpenRead(imageFilePath))                 {                     Face[] faces = await faceServiceClient.DetectAsync(imageFileStream, returnFaceId: true, returnFaceLandmarks: false, returnFaceAttributes: faceAttributes);                     return faces;                 }             }             // Catch and display Face API errors.             catch (FaceAPIException f)             {                 MessageBox.Show(f.ErrorMessage, f.ErrorCode);                 return new Face[0];             }             // Catch and display all other errors.             catch (Exception e)             {                 MessageBox.Show(e.Message, “Error”);                 return new Face[0];             }         }

 

Step 9 – Build the Application and Run. You will see the output as below:-

 

 

Summary

Therefore, in this part we saw that by writing very less code, we can use Microsoft Cognitive Services FACE API.

Part 3

EMOTION API in Azure Cognitive Service

Emotion API provides the advanced Emotion algorithms and it has two main functions:-

1.       Emotion Recognition

a.       It takes image as an input and returns the confidence of the emotions for each face in the image.

b.      The emotions detected are happiness, sadness, surprise, anger, fear, contempt, disgust or neutral.

c.       Emotion scores are normalized to sum to one.

2.       Emotion in Video

a.       It takes video as an input and returns the confidence across a set of emotions for the group of faces in the image over a period.

b.      The emotions detected are happiness, sadness, surprise, anger, fear, contempt, disgust or neutral.

c.       It returns two types of aggregates:

                                                               i.      windowMeanScores gives a mean score for all of the faces detected in a frame for each emotion.

                                                             ii.      windowFaceDistribution gives the distribution of faces with each emotion as the dominant emotion for that face.

Now, let’s create a C# Console Application to test Microsoft Cognitive Services Emotion API.

Step 1 – We already created the Endpoint URL and subscription keys in Azure for Emotion API. We will create a Console Application in Visual Studio 2015.

Step 2 – Replace the Program.cs with the following code:-

using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; using System.IO; using System.Net.Http; using System.Net.Http.Headers; namespace Emotion_API_Tutorial {     class Program     {         static void Main(string[] args)         {             Console.WriteLine(“Enter the path to a JPEG image file:”);             string imageFilePath = Console.ReadLine();                         MakeRequest(imageFilePath);             Console.WriteLine(“nnnWait for the result below, then hit ENTER to exit…nnn”);             Console.ReadLine();         }         static byte[] GetImageAsByteArray(string imageFilePath)         {             FileStream fileStream = new FileStream(imageFilePath, FileMode.Open, FileAccess.Read);             BinaryReader binaryReader = new BinaryReader(fileStream);             return binaryReader.ReadBytes((int)fileStream.Length);         }         static async void MakeRequest(string imageFilePath)         {             try             {                 var client = new HttpClient();                 // Request headers                 client.DefaultRequestHeaders.Add(“Ocp-Apim-Subscription-Key”, “<Enter Your Key Value>”);                 //Endpoint URL                 string uri = “https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognize?”;                 HttpResponseMessage response;                 string responseContent;                 // Request body.                 byte[] byteData = GetImageAsByteArray(imageFilePath);                 using (var content = new ByteArrayContent(byteData))                 {                     // This example uses content type “application/octet-stream”.                     // The other content types you can use are “application/json” and “multipart/form-data” and application/octet-stream.                     content.Headers.ContentType = new MediaTypeHeaderValue(“application/octet-stream”);                     response = await client.PostAsync(uri, content);                     responseContent = response.Content.ReadAsStringAsync().Result;                 }                 //A peak at the JSON response.                 Console.WriteLine(responseContent);             }             catch (Exception ex)             {                 Console.WriteLine(“Error occured:= “ + ex.Message + ex.StackTrace);                 Console.ReadLine();             }         }     } }

Step 3 – Build and Run the Application. A successful call will return an array of face entries and their emotion scores. An empty response indicates that no faces were detected. An emotion entry contains the following fields:

·         faceRectangle – Rectangle location of face in the image.

·         scores – Emotion scores for each face in the image.

Below is the output of my trial run. Provide the path of an image from your local folder and hit Enter:-

 

JSON Response:- [   {     “faceRectangle”: {       “height”: 44,       “left”: 62,       “top”: 36,       “width”: 44     },     “scores”: {       “anger”: 0.000009850864,       “contempt”: 1.073325e-8,       “disgust”: 0.00000230705427,       “fear”: 1.63113334e-9,       “happiness”: 0.9999875,       “neutral”: 1.00619431e-7,       “sadness”: 1.13927945e-9,       “surprise”: 2.365794e-7     }   } ]