You have multiple disparate systems, and each of those systems is part of an overall business function. For example, a business function such as processing an incoming order may require the participation of the customer management system, the inventory system, the shipping system, and one or more financial systems. The business could operate more efficiently if the systems could be integrated so that the business function could be completed automatically.
How do you coordinate the execution of a long-running business function that spans multiple disparate applications?
To solve this problem, you have to balance the following considerations and forces:
Define a business process model that describes the individual steps that make up the complex business function. Create a separate process manager component that can interpret multiple concurrent instances of this model and that can interact with the existing applications to perform the individual steps of the process.
Figure 1. Process Integration with a process manager component directing applications according to a process model
For each incoming request for the business function, the process manager creates a new process instance based on the process model. Each instance maintains the current state of the process plus any additional information that is needed for the business process to continue. Creating individual process instances allows the process manager to execute many business functions in parallel.
After one application completes its business function, the process manager determines which function to execute next based on the state of the process instance. Therefore, each participating application can operate individually without any knowledge of the sequence of steps defined in the process model.
The process manager maintains this state even if a specific application is temporarily unavailable. As a result, the overall execution of the business function is not stopped if a specific application is temporarily unavailable. Instead, the business function can continue after the application is back online.
The process manager interacts with the individual applications by way of Data Integration, Functional Integration, or Presentation Integration, depending on the nature of the interaction and the architecture of the application. Therefore, Process Integration resembles a composite application built on top of existing business functions that reside in existing applications.
Process Integration provides a clean separation between the definition of the process in the process model, the execution of the process in the process manager, and the implementation of the individual functions in the applications. This separation allows the application functions to be reused in many different processes.
The separation of process model and process manager also allows the process manager to be domain independent because it only has to interpret the basic constructs that form a process model. These constructs manage the parallel execution of multiple steps, and they typically comprise sequences, decision points, forks, and joins. Using these constructs, enterprises can create a model at the business level without having to understand the internals of the process manager or the individual applications.
Process Integration involves collaboration between the components that are described in Table 1.
Table 1: Process Integration Components
The process manager typically exposes an external interface so that the business process defined by the process model can be initiated by a user, by business partners, or by another application. Correspondingly, the interface can take the form of a user interface or the form of an application programming interface (API) that can be made available to other applications by using Functional Integration, making it essentially indistinguishable from a simple application. Therefore, this business process can be used as part of another overarching business process that is defined using a higher-level process model. As a result, process managers and applications can be linked in a recursive fashion. Figure 2 shows a hierarchy of process managers and applications.
Figure 2. Hierarchy of process managers and applications
At first glance, the process manager resembles a traditional application that implements business logic. The only difference is that the process manager takes advantage of functions implemented in existing systems. However, the fact that the execution of the business function can take hours or days places specific requirements on the process manager. Specifically, a process manager typically must be able to:
Correlating Messages and Process Instances
Orchestration of processes involves messages sent to external systems and received from external systems. These external systems implement the actions that make up the business process. At any time, there are likely to be many instances of the business process running at once, and any message that is received must be correlated with the correct instance of the business process that it was intended for.
Transactions provide encapsulation of individual atomic actions. A transaction is an action or a series of actions that transform a system from one consistent state to another. When dealing with long-running business functions, it is useful to distinguish between two kinds of transactions: atomic transactions and long-running transactions. When you use Process Integration, your design must support both kinds of transactions.
Atomic transactions are the same kind of transactions as those found in databases. They adhere to a set of properties, commonly called the Atomicity, Consistency, Isolation, and Durability (ACID) properties. This type of transaction requires transactional resources, such as a database update or the sending of a message, that permit you to automatically roll back changes if the transaction is stopped. Atomic transactions are suited only for short-running portions of the overall process.
Long-running transactions are used when one or more of the following conditions are true:
For long-running transactions, you must allow the user to stop the process mid-stream. When the process stops, the system will persist data regarding the completed actions and the intermediate states. After the process restarts, the application will reload this intermediate state to allow the process to continue from the point where it stopped.
Handling Exceptions and Compensating Transactions
When external applications are invoked to implement a specific action, a variety of errors can occur. These errors cause error codes to be sent in messages that are returned by the external systems, and these errors cause exceptions to be thrown.
Status code errors can be handled by conditional logic inside the process model. This approach is natural, but can lead to an unwieldy control flow if all possible error conditions have to be covered. Errors in the form of exceptions can be handled by exception handlers attached to the scope in which the exception occurred. This scope can be the entire process model, or it can be a specific subsection. If an exception occurs within this scope, the exception handler is automatically invoked without the process designer having to explicitly model each individual error condition.
Some of these errors require the application to revert the transaction state to the state before the long-running transaction started. When these errors occur, the application issues a compensating transaction.
Process Integration is commonly used to streamline the execution of a sequence of tasks. One popular application in the financial industry is the notion of straight-through processing (STP). STP describes the automated end-to-end processing of transactions such as trades from inception to settlement.
As described in the solution section, Process Integration can also be used to provide an aggregate service to other applications. For example, consider a service that makes concurrent updates to multiple data sources as defined in Entity Aggregation. The implementation of such a service requires Process Integration internally to manage transactional integrity, partial failure situations, and compensation actions.
Process Integration is such a common need that standards committees and working groups are defining standard languages to describe process models. Examples of such languages are:
Process Integration results in the following benefits and liabilities:
Separating the process definition from the functions provided by the applications allows you to test each function individually by creating a simple test driver that only calls one specific function and that verifies the results, as shown in Figure 3.
Figure 3. Test driver
Likewise, you can test the process manager by creating placeholder implementations, or stubs, of the actual services while providing test triggers to the process manager (see Figure 4).
Figure 4. Using stubs to test the process manager
For more information, see the Process Manager pattern [Hohpe04]. Process Manager describes the internal function and implementation of the process manager component.
[Hohpe04] Hohpe, Gregor; Bobby Woolf, Enterprise Integration Patterns: Designing, Building and Deploying Messaging Solutions. Addison-Wesley, 2004.
[Ruh01] Ruh, William. Enterprise Application Integration. A Wiley Tech Brief. Wiley, 2001.