Image to Text PCF Control

PowerApps Component Framework is becoming a gold standard to infuse AI in the business apps. PowerApps team released an experimental preview of PCF for Canvas Apps and its truly a game changer.

Here is one example where the user can take an image or pick an already existing image on their mobile devices and the PCF control automatically parses the image to recognize the text and prints them on the screen.

Computer Vision Cognitive Service contains many APIs – detect object, describe image, get thumbnail, tag image, extract text to name a few. This control leverages the Computer Vision’s Extract Text API in order to parse the text from the image.

The Extract Text API uses 2 API calls to parse the text.

  • Batch Read API call – The first api call is to the Batch Read method that takes the image and runs asynchronously. This method returns a URI (in response) in the value “Operation-Location” response header field.
  • Read Operation Result API call – This “Operation-Location” value forms the basis for the URL to issue the next API call. This URI represents the “Read Operation Result” method to both check the status and return of the results of the Batch Read API call.

A point to highlight is that the Batch Read API call runs asynchronously, so you may need to wait before issuing the 2nd API call (Read Operation Result). If the document is expected to be huge, then the wait has to be adjusted appropriately.

Another important point to highlight is that the image has to be converted from base64 to binary raw blob to pass it to the Batch Read API.

This PCF control allows the user to take a picture on their device and process them, but the above API calls can be modified in order to pass an URL of the image if you happen to store them remotely and extract the text. This API call accepts both the handwritten and the printed text seamlessly without any need for additional configuration.

This PCF control can also be embedded on the Model Driven apps, but only works on Phone and Tablet with “Dynamics 365 App” on the mobile App Store.

The full source code can be found in github.

Thanks for reading and hope this is useful!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s