Forms Recognizer PCF Control

AI framework is not just to make the customers’ lives easier, but also to make the internal users’ lives easier.

We are heading 2020 and the its been 25 years since the internet took off, but sadly, we still have some paper processes to deal with. But fortunately with AI services these days, we can read through the unstructured papers, parse out the details in a way that online systems can understand. Imagine the productivity boost this service is going to provide your staff and empower them to do more with their valuable time.

In this blog, lets dive in on how to build a PCF control that leverages the Forms Recognizer Cognitive Service to extract the data as key/value pairs from an unstructured PDF form.

What is Forms Recognizer service?

Azure Form Recognizer is a cognitive service that uses machine learning technology to identify and extract key/value pairs and table data from form documents. It then outputs structured data that includes the relationships in the original file. To get started, you just need a few filled-in form documents of the same type as your input material. You quickly get accurate results that are tailored to your specific content without heavy manual intervention or extensive data science expertise.

I encourage you to review this article to understand a bit more about this service.

Forms Recognizer service is currently available in a limited-access preview. To get access to the preview, fill out and submit the Form Recognizer access request form.

Create Forms Recognizer service

Follow this article (only this section) to create the Forms Recognizer Service.

Once the service is created, go to Keys and copy the content in Key 1. This is your subscription key, which you will use it in the code further.

Create Azure Storage Blob

Create a Azure Blob Storage

Go to Storage Explorer and upload the sample documents for training the model

I have attached the sample documents that I have used to get you upto speed.

Go to Shared Access Signature blade on the Storage and click Generate SAS and connection string and you should see the response as below. The value of interest is SAS token.

Your should get the response and the value of interest is the Query string.

Train API

I have used Postman to call the Train API to train the model with the uploaded documents and the Postman configurations are:

  • Request: POST
  • URL: https://<endpoint&gt;.cognitiveservices.azure.com/<endpoint>/v1.0-preview/custom/train

You can find the endpoint in Azure:

  • Headers: Content-Type – application/json, Ocp-Apim-Subscription-Key – The value copied from Key 1 from the Forms Recognizer service.
  • Body

{
"source": "https://<storage account>.blob.core.windows.net/<container name>?<SAS value>"
}

The SAS Value is the Shared Access Signature that you copied on the step above.

The response from the Train api on Postman should look like below:


That’s all. You have successfully trained the model. Grab the model id from the above response, which you will need it to analyze the document that you are going to upload.

Analyze Api

Now, we are ready to call the Analyze api that will take the PDF document as input and extract the content as key/value pairs.

Construct the Analyze Model Url to call the Analyze api.

https://<endpoint&gt;.cognitiveservices.azure.com/<endpointname>/v1.0-preview/custom/models/<modelId>/analyze.

  • Follow the above step to update the endpoint in the url.
  • Replace modelId with the modelId obtained from the above JSON response from Train API.

Here is the source code of the Forms Recognizer PCF control. Download the source code and use the PCF CLI to build the Dynamics 365 solution. If you haven’t used the CLI to build the solution, follow this article.

Once the PCF solution is successfully imported in Dynamics 365, open to the Contact entity’s Contact for Interactive Experience form and add a multiline control. If you do not have it already, feel free to create on.

Go to the multiline field’s property and navigate to Controls tab and add Forms Recognizer Control.

Associate the relevant fields to the bound controls. Add the subscription key copied (from Key 1 field in Azure) and the Analyze Model Url prepared above.

Here is how it should work.

I understand this could use a few more enhancements such as showing the form that is being uploaded below the “Recognize” button and uploading the PDF form automatically as a Note attachment and can even take any types instead of just PDFs. I guess I need to table those for the future.

It is really interesting how we can integrate various cognitive services and slowly imbibe AI into your Dynamics 365 business processes using PCF.

Hope this is useful and have fun!

One thought on “Forms Recognizer PCF Control

  1. amazing stuff, finally some interesting use case rather than just a prototype for the sake of it. I’d love to see what you can do with it to generalise it a bit more (as you said yourself, different file types and entity/field mapping should at the very least be configurable entities).

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s