Event-Driven Process Orchestration: A Practitioner’s ViewPoint

Modern customers are looking for services that are capable, adaptable, versatile, and customizable. They want services that are unrestricted and uncomplicated, with effortless access and seamless experiences. Efficiency and accuracy are key factors in determining customer satisfaction and loyalty for businesses today. As a result, around 85% of medium to large enterprises have embraced the microservices architecture approach. This architecture allows for independent, decentralized components within applications that are resilient to failures and can be maintained and updated separately. This promotes self-sufficiency, scalability, reliability, and streamlined service delivery for enterprises.

While microservices architecture prepares an application for flexible maintenance, it is important to note that the overall customer experience is not solely determined by the decoupling of application components. Rather, it relies on the seamless automation of every step in a customer success workflow to ensure their satisfaction. This becomes especially crucial as businesses expand and add more components, as without proper orchestration, cohesion chaos can occur and undermine the supposed advantages of microservices. Therefore, keeping the end goal of customer delight in mind is essential when creating a logical flow within business processes.

By utilizing an event streaming platform like Kafka, microservices applications can be easily clustered and their process flows can be orchestrated. This orchestration can be managed and governed by a BPM or Integration engine, such as RHPAM, Camunda, or MuleSoft. These platforms offer seamless co-existence between API led architecture and events-based architecture. With this type of architecture, each microservice listens for user actions within the event stream and triggers corresponding services based on the logical process flow defined. In this way, each service is self-sufficient and responds according to their specific trigger point, embodying the true essence of event-based orchestration.

During my discussions with different businesses from different regions and industries, clients often try out the concept of using a service such as Kafka for event streaming or utilizing a BPMN engine like RHPAM to centrally manage their services. However, each option has its own advantages and disadvantages. Recently, the combination of using both a BPMN engine for centralized orchestration and coordinating with worker services through an event stream has become increasingly popular. Additionally, the emergence of major Enterprise Integration companies like MuleSoft, which offer support for event-driven architecture in addition to traditional API-led integration, has made the range of solution choices even more intriguing for customers.

The Mechanism at play in Bob’s ordering of a Ridola:

Bob will go through the following microservices when ordering a Ridola, assuming he is signing in for the first time: Customer Profile Service, Location Service, Cab and Driver Management, Trip Management, and Payments.

In simple terms, the value chain goes like this: Bob opens the app and registers by providing his profile information. Then, he selects his starting and ending locations for his journey. Ridola searches and suggests nearby cabs and drivers with their corresponding rates. Once Bob chooses a cab, the trip management service takes over to guide the driver and arrange for Bob’s preferred vehicle. The trip begins from Amsterdam to Den Haag. After the trip is completed, Bob is prompted to make payment and a bill will be sent to his email address upon receipt of payment.

Options available for Ridola to provide a well-orchestrated service to Bob:

1. BPMN driven Centralized Orchestration: In this approach, Ridola integrates business workflow logic into a centralized BPM Engine, such as RHPAM or Camunda. These technologies utilize the standard Business Process Modeling Notation (BPMN) for process modeling. To facilitate integration between the Passenger UI and the application, REST is used. Upon Bob’s login to the application, commands are triggered by the centralized engine (the “brain” of the business workflow) and sent to worker services, with wait time for their response. These commands can be issued through a Java delegate or a platform like AWS Lambda. The “cab ordering service” itself is constructed as a Springboot microservice.

After Bob logs in, the centralized engine will issue the first command to the Customer Profile Service, which will pop up and ask him to sign in. After this step has been completed, the centralized engine will command the location service to kick in, inquiring about Bob’s current location, origin, and destination. After that, the cab and driver management services are triggered centrally, and so on.

Figure: BPMN based orchestration architecture

This approach is moot because the central orchestration engine triggers the actions on the worker services. As all this is centrally managed through BPMN standards, it can be easily maintained, supported, and upgraded. The worker services do not pass the commands to each other. As a result, the BPM engine can provide a single repository for transaction updates, service state, etc., and can therefore serve as a source of Observability.

That said, while it may make sense for a smaller enterprise with minimal transactions, larger companies like Ridola should avoid this type of integration to prevent tightly coupling their landscape and utilizing point-to-point integration. Unfortunately, in Ridola’s case, this approach could cause the orchestration engine to overheat and ultimately harm the customer experience.

2. Event Driven Orchestration: Vis-à-vis the centralized approach, many customers are opting for event streaming platform-led orchestration. This involves utilizing technologies like Apache Kafka, IBM Data Streams, Amazon Kinesis, and Confluent. It is a decentralized approach in which the business logic is embedded into each microservice that Bob encounters while using his Ridola cab service. Each of these services (such as customer profiles, location, cab and driver management, trip management, and payment) is connected to a central event stream (like Kafka or Confluent) and listens for relevant topics related to their function. These topics are triggered by either Bob’s actions (such as signing into the app) or by the preceding service’s actions (like the customer profile service). They serve as cues for the next service (such as location) to prompt Bob for his current location and desired destination. This allows each service to be aware of their role and responsibilities in an orderly, “event driven” manner throughout the cab ordering process.

Figure: Event Streaming based orchestration

Although this approach improves on the lack of “loose coupling” in the first approach, it also introduces challenges in maintaining and managing services during changes to the overall business process. Additionally, there is no centralized observability for performance, requiring us to refer to individual service logs and traces when troubleshooting. This can be time-consuming and inefficient.

Nevertheless, a business owner needs to evaluate the scenarios and take a call if the process value chain and services comprising the process are well established.

3. Hybrid Approach- BPMN led Event Orchestration: Using both BPMN and event orchestration is gaining popularity among customers. Many are realizing the power of this combination and opting to first test it through a proof of value, followed by full implementation. In this setup, the centralized BPM engine (RHPAM or Camunda) contains the business logic, while communication with downstream worker services is not done directly. Instead, an event broker is used to establish this communication, ensuring loose coupling between services. This approach can also be referred to as event streaming based orchestration.

Figure: BPMN led Event Orchestration

As seen above, when Bob logs into the application, the centralized engine issues commands to the worker services via event streams and awaits their responses.

As a result of this approach, enterprises are able to gain central governance and observability benefits without making their ecosystems tightly coupled and difficult to maintain. This is a very good model for large enterprises and is gaining wide acceptance.

4. MuleSoft + Event Driven Orchestration: In any major corporation, like Ridola, API-led integration is a standard practice. Yet, there are situations where asynchronous communication is crucial for business success, even in an API-led architecture that operates synchronously. Here, Event Driven architecture serves as a complement to API-led architecture and together they can effectively prevent internal architectural constraints from hindering customers like Bob.

This marriage of synchronous (API led) and asynchronous (Event led) can be seen as plausible in the following scenarios:

Asynchronous Backend Updates: MuleSoft’s architecture is layered, with Experience APIs on top serving customers in multiple channels, Process APIs at the bottom which are the pipelines that process the actual task at hand and pass on the outcome to the Experience APIs as well as System APIs that are the repository of enterprise data that are used by Process APIs to tailor the solution contextually.

When an avalanche of customer requests come in, the Process APIs may be overwhelmed by the repeated need to fetch data from the System APIs, which can add to the latency of the Experience APIs.

Figure: Event Stream layer between System and Process APIs helps faster processing

This scenario (as shown above) illustrates how an event broker can act as a repository of the most requested information (such as customer information) and serve as a “one stop shop” source for this information for the Process APIs. In this way, there will be no unnecessary, repetitive calls to the CRM system for every Process API request, since the appropriate systems will be updated asynchronously with the needed information.

Connectors in MuleSoft can capture data changes and publish them as events to the event broker. The upstream applications can then subscribe to these events and update the end-systems.

As a result of system overload and acknowledgement of customer requests, processing can be delayed: When the system layer fails after reaching its peak capacity, the back and forth between Experience APIs, Process APIs, and System APIs would continue in a futile manner, causing the system to become increasingly overloaded. Additionally, it is possible that the system is down for maintenance over the weekend, but requests are still being received.

In this situation, MuleSoft-powered applications have the ability to produce uncomplicated events for each request made at either the Experience or Process layers. These events can then be saved in an event broker until the receiving system is prepared to handle the request. Upon acknowledgment of the customer’s request, any requests that can be fulfilled using existing Process or System APIs will be promptly processed. For those requests lacking sufficient information, a notification will be sent to alert the customer of a possible delay and prevent any potential issues.

In order to ensure seamless customer service, API-led and event-driven architectures are emerging as key themes with modern day customers.

Conclusion:

As businesses navigate the “experience economy,” gaining customer trust is crucial for success. Combining BPMN led Event Orchestration with a balance of API and Event led architectures can achieve system resilience, practical processes, and an enjoyable customer experience. Enterprises should now consider how these methods can benefit their specific industries and optimize their business goals. These approaches each have their strengths and weaknesses, which depend on factors like an enterprise’s process maturity, frequency and scale of changes, transaction volume, and customer reach.

Many enterprises worldwide are partnering with the leading System Integrators in the ecosystem because making the right decision and choosing the right option for the right use case can be challenging and may require careful due diligence. Therefore, if you are considering embarking on Event orchestration, there are many partners ready to guide you and walk you through the journey. Get started today!