Standard is not always easy
Modelling a very complex business process flow in Dynamics 365 using the standard functionality can be quite challenging.
For example, the user first needs to search for and find the record in the system that they would like to process. Just this step can be prone to error if they select an incorrect record.
Then the user may need to create a new record, for example an “Order”, and submit this order to the Enterprise Resource Planning (ERP) system. There may be a delay between when the “Order is submitted” and when it is processed by the ERP system, preventing the user from moving directly onto the next step in the process.
All this manual processing of records in Dynamics 365 can be very time consuming for the user and lead to bottlenecks in the business process. That’s why we came up with an automation solution which would increase the throughput of the system.
This automation needed to be able to:
- Be initiated via an Application Programming Interface (API)
- Automate their existing manual processes
- Cut down on user mistakes by validating all inputs
- Run unattended until the process completes
- This could take up to 10 minutes, sometimes you need to wait for records to be processed in the ERP system
In this post I will explain the business requirement and provide an overview of the finished solution.
I would like to acknowledge the contribution of my colleague Michel Gueli, who was instrumental in the technical development of the solution. Together we were able to take the requirement, which was only an idea, give it shape and come up with a solution.
Background and Process
Our Cloud4Commodities solution, available in the AppSource, can be used to facilitate the capture of Trading Deals between two parties. This is a function completed by their Traders in the Front office.
The Logistical Settlement process of these deals is a function completed by Operators in the Middle Office.
There are two main processes which the Middle Office users perform in settling a Deal:
- Receival of Inventory
- Matching of Inventory to Sales Deliveries
For both processes Dynamics 365 is integrated with ERP system:
- Purchase Orders are generated, and received Inventory is booked into the system
- Sales Orders are Invoiced out to customers and Inventory is booked out
As you can imagine, accurately tracking all the Receipts and Matching these to Deliveries is a time-consuming job for the Middle Office.
To find a solution which helps automate these processes for the Middle Office, we came up with two ideas:
- Since the Receival of Inventory is first registered on another external system, let us call it the “Inventory Register system”, we developed an API which is called to facilitate the automated receival of inventory.
- This API also needed to support the matching of Inventory to Sales Deliveries. This would make sure there’s the ability to match Inventory to Sales Deliveries outside of the system and send in a request to the API to perform the actual process.
Our solution
Above is a high-level diagram of the final developed solution. Here I will describe each component of the solution.
1. API for Process Engine
As mentioned, a requirement of the solution was to expose the “Business Process Automation” functionality via an API so that a process could be initiated from outside of Dynamics 365.
This requirement was met by developing a RESTful web service, which runs in Microsoft Azure as an “App Service”.
Some notable methods supported by this service include:
- Executing a Process – This operation will initiate a new Process
- Retrieving the Process Status – This Operation will check the status of a Process
2. Validate the Process Inputs
Each Process that is submitted to the system is first validated. This means that the inputs are evaluated for correctness, preventing bad requests from being accepted and processed.
3. Create Process Record and store Input Process Values
Every process in the system starts with the creation of a new Process entity record in Dynamics 365.
The screenshot above contains details such as:
- The type of process (I.e. Receipt or Match)
- The status of the process (I.e. Active or Finished)
- Process values passed as input or created during the process E.g. The Deal ID against which the Stock is being Received
- A log of all completed process steps
4. Scheduled Job to run the Process Engine
We make use of a product from our “Products” business, called the Dynamic Scheduler, which is used to execute our “Process Engine” according to a configurable schedule/time frame.
The “Process Engine” will pick up active Process records and process these.
5. Validate the Process Step
Every Process Step is first validated. If the Step passes validation, then it is executed.
E.g. If a Purchase Order Number has not yet been passed back from the ERP system, the next step responsible for adding the Stock to Inventory cannot execute.
6. Execute Process Step and Store generated Process Values
Initially the “Process Engine”, will execute the first step of an active Process.
E.g. Create a Purchase Order for a Buy Deal.
When a Process Step action is executed, this action may lead to the creation of a record in the system. The system will then store the details of this generated record in a “Process Values” entity related to the Process.
E.g. We would store the PO Id of the Purchase Order created above.
7. Get Next Step in the Process
The “Process Engine” is also responsible for advancing to the next defined step in the process or marking the Process as “Finished” once all steps have been executed.
What did we solve?
By adhering to our approach and the design principles which we adopted, we managed to solve the following challenges:
- We were able to create a Process Engine which can run unattended over long running processes (More than 10 minutes even)
- The design of this process engine allowed us to already add a 3rd process to the solution
- The solution validates all inputs and process steps maintaining the data integrity of the system
- The Process engine can be triggered by our API
- Due to the component-based design of the solution we have also been able to add a feature to start a process from within Dynamics 365
The result is an extensible best practise solution and one which will save you money in the long run when it comes to future development on the system.