City of Wisdom – The Bot in Action

In the previous post, we learnt how to setup a Bot and deploy it to Facebook Messenger using MS Bot Framework.

In this post, let’s take it one step further and find the type of request the citizen intended to submit by analyzing the uploaded image and finally, store the service request in Dynamics 365.

Firstly, let’s deconstruct the idea in order to understand all the moving parts involved.

  • We need to send a greeting message to the citizen when they start to interact with our bot.
  • When the citizen uploads an image, we parse all the tags received from the Cognitive Services API and find out the intention of the citizen.
  • We need to request the location where the citizen spotted the problem and when they share, we need to parse the location data.
  • There is a value in storing the data that the Messenger platform sends us. This data contains information to uniquely identify the citizen on a particular platform. We will use this data to associate the service request that the citizen submits.
  • Finally, we create a service request based on the details gathered so far in the Dynamics 365.

Hope it is clear as to what we are going to do in this post.

Let’s get started!

I have githubbed the source code for your reference.

Greet the citizen

We need to greet the citizen when they interact with our bot for the very first time. But wait, how do we know that the citizen engaged with the bot on Messenger for the first time?

We need some kind of a hook from the the Messenger indicating that the citizen interacted with our bot for the first time. We will take this opportunity to show a quick message (you can call it as “greeting”) as to how they can interact with the bot.

This documentation explains the necessary steps in order to setup a web hook so that our bot code receives a message when the citizen interacts with our bot for the very first time.

To simplify, here is the command you will run.

curl -X POST -H "Content-Type: application/json" -d '{ "get_started": {"payload": "<postback_payload>"}}' "https://graph.facebook.com/v2.6/me/messenger_profile?access_token=<PAGE_ACCESS_TOKEN>"

Before running the command, replace “postback_payload” with “requestWelcome” and <PAGE_ACCESS_TOKEN> with your corresponding page access token.

To retrieve the Page Access Token, open your application from Facebook developer portal and navigate to the Messenger -> Settings -> Select a page.

page acces token

You can use git bash (https://gitforwindows.org/) in order to run the above command.

cURL Request success

When the citizen adds “City of Wisdom” bot in the Messenger, the Messenger platform sends a payload with an event “requestWelcome” which will enable the bot to send the greeting message.

That’s all.

Service Request Type

We could ask the citizen to mention the service request type, which would make our life super easy. But, what fun is that!

What if we find the service request type that the citizen intended to submit based on the image they upload? Sounds like a plan.

But, how do we know which type of request that the citizen intended to submit from the uploaded image? This is where the image recognition integration that we setup in our previous post comes handy.

We send the uploaded image to the image recognition API which returns the list of relevant tags. We are going to take a calculative guess on the service request type from the list of tags returned.

Bingo! Once deduced, we pass the service request type to the RootDialog.cs calling method.

Currently, we support only “graffiti” and “pothole” service request types. You can enhance the code for other service request types as required.

If you review the code, you will notice that we have utilized Dialogs to model the conversation.

In a bot, everything begins with the root dialog. The root dialog invokes the location dialog. At that point, the location dialog takes control of the conversation and remains in control until it either closes or invokes other dialogs. If the location dialog closes, control of the conversation is returned back to the root dialog.

There are two piece of information that we gather from the user.

  1. Image upload
  2. Location data.

As you would have noted, RootDialog.cs contains the flow to manage the image upload aspect of the conversation and LocationDialog.cs manages the location data aspect of the conversation.

Gather problem location

Now, the field staff who generally goes out to the field to resolve the issue will obviously need an address as well. For this purpose we are going to prompt the citizen to share the location one-time and store it on the service request record.

Quick info about user privacy: Obviously, the idea here is not to track the citizen’s location until they renounce their digital life. We request for the problem’s location data so that we can send the field agents onsite for the resolution. Originally, the idea was to find the location from image’s EXIF data. But for very obvious reasons, Facebook decided not to publish EXIF data of the image. So, we need to request the citizen to share the location every time.

When the citizen sends the location data, Facebook strips all the details but the longitude and latitude. We are going to parse the longitude and latitude from the payload and pass on to the RootDialog.cs.

Great! Let’s assume that this information is enough for us to store the data in Dynamics 365 and start deploying the agent onsite to resolve the issue.

Bot Channel Profile

Before creating the service request record in D365, I think it is valuable to understand which citizen submitted the request on what channel and associate them with the Service Request record, so that we can send out a quick acknowledgement once the request has been resolved (Sending acknowledgement to the citizen will be covered in the future post).

If you watch the “activity” object that is passed to the bot, it contains all the properties like ConversationId, ChannelId, From, Receiver etc., with which we can construct the profile for the citizen and associate with the Service Request.

activity

Access Dynamics 365

CRMOperation.cs contains all the methods that we require in order to establish the connection to the D365 and store the data appropriately.

We are going to use server-to-server (S2S) authentication to securely and seamlessly communicate with Dynamics 365 from our bot application.

In order to setup S2S authentication, we need two steps to be done.

  1. Register an application in Azure Active Directory.
  2. Once the application is registered, create the application user in Dynamics 365.

This blog post provides all the steps necessary in order to register an application and create an appliaction user in D365. Follow until the heading that says “Setting Up the Application User in CRM”. Once complete, assign the security role that has all the accesses to “Bot Channel Profile” and “Service Request” entities.

Update the Dynamics 365 URL in the below web.config setting.

<add key=”ServiceUrl” value=”YourD365Environment.crm.dynamics.com” />

Obtain the necessary data from the Azure App Registration and update in the web.config.

<add key=”ClientId” value=”Application ID of the Azure App Registration” />
<add key=”RedirectUrl” value=”Home page URL of Azure App Registration” />
<add key=”Key” value=”Key of the Azure App Registration”/>
<add key=”AuthEndpoint” value=”AUTH 2.0 Authorization Endpoint found in Azure App Registration”/>

Now, publish the source code to the Azure as per the steps in the post.

When the citizen interacts with the bot for the very first time, we create a bot channel profile record in the D365 and then associate the service requests submitted by the citizen with this channel profile.

Create Service Request

We aren’t bothered about getting the auto generated service request number back from the D365 and showing it to the user. As an end user, I don’t care too much about the acknowledgement number and all I care is to report an incident blazing fast and carry on with my day (of course I expect someone to resolve it!). Having said that, we can look at enhancing the code to retrieve the service request number in the future if we’d like.

I have already created a D365 solution and configured a few entities. You can find ServiceRequestsBot_1_0.zip in the source code location.

The solution contains two entities:
1. Service Request – This is where we store the request that the citizen submits.
2. Bot Channel Profile – A channel profile is created for each of the citizen in order to associate with their service requests for the future follow ups.

Import this solution to your Dynamics 365 trial instance.

You are all set.

Now, launch the Messenger app and search for the City of Wisdom or the name that you have come up with for your Messenger app and click on “Get Started” button. You will now see the greeting message from the bot.

You can upload an image (I have included a sample graffiti image in the source code “Images” folder) and share the location. The bot should create a “Graffiti” service request record and store the latitude and longitude data.

Messenger Service Request.gif
Bot channel profile record in D365
Bot Channel Profile
Service Request record in D365
Service Request

For the scope of this post, we aren’t going to reverse geocode the latitude and longitude and store the address in D365. This is something you will have to do for a production grade implementation (Didn’t I say our city is imaginary?).

Also, we aren’t uploading the image in the Service Request record. We can look at enhancing our solution in the future post.

Hope all the moving parts involved makes sense now.

So far we have seen how MS Bot framework natively supports the Messenger channel and abstracts all the implementation details from us. We just need to understand the Bot SDK and it takes care of the rest. I really have to underline the fact that bot backed by the rich cognitive services such as Image Recognition API will make it much more powerful, as you can imagine.

In the future posts, we can make our bot even more powerful by proactively responding to the citizens.

Stay tuned.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s