Implementing Process Integration with BizTalk Server 2004
You are developing new solutions that reuse functionality provided by existing applications and systems in your enterprise. The individual mechanisms to integrate with each of these applications have already been defined, so now you use Microsoft BizTalk Server 2004 to implement Process Integration to coordinate and manage the overall interaction between all these systems.
Global Bank is developing new banking solutions, including an electronic bill payment system. Global Bank wants to enable customers to schedule payments from their accounts. These scheduled payments will be batch-processed daily by the bank's systems.
The application to be built, which processes the customer's scheduled payments (Execute Scheduled Payments use case), will reuse functionality supplied by various bank systems including the Microsoft SQL Server database, various Web services, and the Microsoft Host Integration Server interface to the bank mainframe (see Implementing Gateway with Host Integration Server 2004). The Execute Scheduled Payments application will also interact with internal, external, and manual payment systems.
Global Bank has already defined the individual integration mechanisms for each of these applications and systems. Now Global Bank wants to use BizTalk Server 2004 orchestration to manage interactions across all the systems that constitute the composite electronic bill payment business process.
BizTalk Server orchestration is a good choice for describing complex process models. Orchestration is a .NET Framework language, just like C# and Microsoft Visual Basic® .NET. Orchestration provides programming constructs such as loops, decision statements, program scope, transactions, and exception processing semantics. Orchestrations are compiled to produce .NET Framework common language runtime (CLR) libraries that are hosted by the BizTalk Server runtime. The BizTalk Server runtime essentially acts as an application server for long-running business applications.
Global Bank developers have already implemented the individual integration mechanisms required to interact with each of the systems that are involved in the Execute Scheduled Payments solution. You now define a process model that describes the sequence of steps that make up the overall application. This process model is defined and implemented as a BizTalk Server orchestration. Figure 1 shows the relationship between the orchestration (the process model), the existing systems, and the BizTalk Server runtime (the process manager).
Figure 1. BizTalk Server orchestration directing applications according to a process model
To develop the Execute Scheduled Payments application, you use BizTalk Orchestration Designer to define the process model. Orchestration Designer presents a visual design and development environment that defines the process flow separately from the implementation of the individual steps in the process. You then link each action in the process flow with the implementation of that action. The implementation is usually represented as an interaction with an application that is external to the orchestration itself.
Each incoming request to execute a payment creates a new instance of the Execute Scheduled Payments orchestration. These orchestration instances run in parallel, maintain their own state, and maintain any additional information that is needed for execution. The orchestration engine uses the orchestration to determine the external applications to invoke and the sequence in which to call them. The external applications are invoked synchronously or asynchronously depending on the nature of the interaction and the architecture of the external application. For example, an external application is invoked synchronously when updating status information in the database and asynchronously when sending payment requests to external systems.
To completely model a process, an orchestration must do more than invoke each of the external applications in sequence. The orchestration must be able to do the following:
BizTalk Server orchestration provides functionality to support each of these requirements, as described in the following sections.
Correlating Messages and Process Instances
Orchestration of processes such as Execute Scheduled Payments involves messages sent to external systems and received from external systems. These external systems implement the actions that make up the business process. At any time, there are likely to be many instances of the business process running at the same time. Each instance maintains its own state, and any message that is received must be correlated with the correct instance of the business process that it was intended for. For example, Execute Scheduled Payments receives a payment acknowledgment message from a payment authority to indicate that a specific payment has been processed. For more information, see, "Step 6: Sending the Payment and Receiving an Acknowledgment," in the "Example" section.
To set up correlation within an orchestration, you start by defining a correlation type that specifies one or more message properties. The combination of these properties is guaranteed to be unique across all instances of the business process. These properties map to fields in the outgoing and incoming messages. For example, SSN could be a property that occurs in both the outgoing message and the incoming message, albeit in a different location in each message.
To map the properties to fields in the outgoing message, you create an instance of this correlation type called a correlation set. This correlation set is then initialized (unique values are assigned to the properties in the correlation type) by specifying this correlation set as the Initializing Correlation Set for a Send shape in the orchestration. This means that when a message is transmitted by way of this Send shape, the correlation set will be initialized using the correct fields in the message (SSN, for example).
Lastly, you configure the matching Receive shape to use this correlation set as the Following Correlation Set. This means that when a message is sent back to the orchestration, the message will be passed to the instance of the orchestration that has a correlation set with properties that match the properties in the message.
A long-running transaction contains persistence points that are also restart points if a failure occurs. BizTalk Server 2004 implements transactions, compensation, and exception handling within a Scope shape. The Scope shape supports both atomic and long-running transactions. For more information about atomic and long-running transactions, see "Transactions" in the Process Integration pattern.
The Execute Scheduled Payments orchestration uses long-running transactions to handle long-lived interactions with external systems that may fail or time out. For more information, see "Step 6: Sending the Payment and Receiving an Acknowledgment" in the "Example" section.
Handling Exceptions and Compensating Transactions
When external applications are invoked to implement a specific action, a variety of errors can occur. These errors may cause exceptions to be thrown. For example, when an orchestration calls a Web service, the Web service can raise an exception that is passed back to the BizTalk orchestration process. Exceptions are handled by exception handlers that are attached to the scope within which the exception occurred. The outermost scope in a BizTalk orchestration is that of the orchestration itself. Additional scopes can be nested within an orchestration by using the Scope shape.
Exception handling logic can be defined for the entire orchestration as a separate execution flow within the Orchestration Designer. It is also possible to encapsulate various action shapes within a Scope shape and to define an exception handling boundary around these shapes so that the shape has its own unique exception handling properties. An example of this is shown in "Step 4: Handling Exceptions."
When an error occurs, an orchestration may need to reverse the effects of transactions that occurred earlier in the orchestration, and that have already been committed. To achieve this, the orchestration executes a compensation block, which specifies how the transaction can be reversed. Compensation also reverses actions that are not transactional, such as sending a message.
For example, an orchestration contains a scope with an atomic transaction that commits and writes changes to a database. This first scope is followed by another scope with a long-running transaction. When an error occurs in this second scope, the error aborts the long-running transaction and then starts exception handling logic for the long-running transaction. However, the effects of the first atomic transaction cannot be rolled back, because they have already been committed. Instead, a compensating transaction (series of actions which compensate for the original transaction) is required to remove the changes from the database.
Implementing the Execute Scheduled Payments Orchestration
The following steps form the general guidelines for implementing and deploying an orchestration like the Execute Scheduled Payments orchestration:
This example follows a Global Bank developer through the process of creating the Execute Scheduled Payments scenario and describes the specific configurations that are necessary to meet Global Bank's requirements.
Figure 2. High-level view of the Execute Scheduled Payments business process
To implement the Execute Scheduled Payments process, the Global Bank developer starts by designing the process model that describes the solution. Figure 2 shows a high-level sketch of the process flow for this solution. The developer uses this process model design to develop the Execute Scheduled Payments orchestration in the BizTalk Server Orchestration Designer. Initially, the developer just specifies the process flow in the orchestration and then links the actions in the orchestration to the implementation of those actions. For example, actions may be implemented by a call to an external Web method.
The following steps describe in detail how the Execute Scheduled Payments orchestration implements the process model. They also describe how the completed orchestration is compiled and deployed in the Global Bank environment.
Note Although the following steps adhere to the general implementation and deployment guidelines presented earlier in "Implementing the Execute Scheduled Payments Orchestration," the order and number of the steps vary slightly. The sample steps that follow reflect both the order in which the developer creates the orchestration in BizTalk Server and the order in which BizTalk Server presents the orchestration in the user interface. The developer also performs some steps such as error handling multiple times, according to accepted practices.
Step 1: Receiving a Payment Request
The first shape in the orchestration is a Receive shape, which receives payment messages (requests). This Receive shape is linked to an orchestration port called ReceivePaymentRequest (see Figure 3).
Figure 3. The first stage in the orchestration is receiving the Payment message
Whenever a payment request message is received, a new instance of the orchestration is created and the message is received by this port. For information about how these messages are generated, see "Step 10: Setting Up the SQL Server Adapter."
Step 2: Designing the Payment Message Schema
The developer needs to specify an XML schema that describes the payment message so that it can be processed by the orchestration. Normally, a developer would use the BizTalk Schema Editor to design a schema. However, in this case, the payment messages are generated by using a SQL query to create an XML record set (see "Step 10: Setting Up the SQL Server Adapter"). The developer can use the SQL Server Adapter Wizard to automatically generate a schema that is appropriate for this query. Figure 4 shows the Global Bank payment schema as viewed in the BizTalk Schema Editor.
Figure 4. The generated payment schema in the BizTalk Schema Editor
When the SQL query is executed, multiple payment records are typically returned. The payment records must be split into separate records so that each payment can be handled individually. The developer uses the BizTalk Schema Editor to create another schema that is known as an envelope schema. The envelope schema represents the set of payments. When the receive port is configured to use these schemas, the XML recordset that is generated is automatically split into individual XML records.
Step 3: Calling a Web Service to Debit the Account
Next, the customer's account is checked to ensure that it has sufficient funds to pay the bill. The appropriate amount is then debited from the account. The functionality for these steps exists on the mainframe and has been exposed as a Web service through Host Integration Server (see Implementing Gateway with Host Integration Server 2004).
To implement this part of the orchestration, the developer uses the Add Web Reference Wizard to add a Web reference to the project, just as he or she would add a Web reference to any other .NET Framework application. Adding the Web reference creates an orchestration port that is bound to the Web service. Figure 5 shows the results of adding a Web reference to an orchestration; the DebitPort is the port bound to the Web service, and CheckBalanceAndDebit and CreditAccount represent some of the Web methods that this Web service implements.
In addition, the Add Web Reference Wizard generates XML schemas that represent the request-response messages for each of the Web methods exposed by the Web service. The Add Web Reference Wizard also creates corresponding orchestration message types. The developer then creates instances of the orchestration message types CheckBalanceAndDebit_request and CheckBalanceAndDebit_response. These instances are called DebitRequest and DebitResponse respectively.
The DebitRequest message is initialized using a Transform shape called TransformPayment2DebitRequest (see Figure 6). This shape applies an Extensible Stylesheet Language for Transformations (XSLT) transformation to the incoming Payment message and initializes the DebitRequest message.
The DebitRequest message is then sent to the Web method CheckBalanceAndDebit by the SendDebitRequest shape, which ensures the account has sufficient funds. The appropriate amount is then debited from the account. The Web method returns the DebitResponse message to the ReceiveDebitResponse shape. This message has fields that contain the new account balance and the status of the debit action.
Step 4: Handling Exceptions
Under some circumstances, the CheckBalanceAndDebit Web method may return an exception. For example, an exception is generated if a request to CheckBalanceAndDebit specifies an invalid account number. The orchestration must handle this exception by providing an exception handler that specifies an appropriate action under these conditions.
Exception handling can be set at various levels of scope within the orchestration. In this instance, the Web method request and response are enclosed by a Scope shape. The Scope shape has specific exception handling code attached (see Catch SOAP Exception in Figure 5).
In this example, the exception handling is simple. The code in the Set Error Status Expression shape sets the value of the errorStatus orchestration variable to 1, to indicate that a Web service exception has occurred. This value is checked later in the orchestration (see "Step 5: Checking for Errors").
Note This example assumes that if an exception is generated, no debit occurs. Therefore, no compensation logic is required in this case. For an example that requires compensation logic, see "Step 7: Compensating for Payment Errors."
Step 5: Checking for Errors
Step 4 described how the Web method can throw exceptions, and how these exceptions are caught and handled. Additionally, the Web method response message (DebitResponse) returns a status code that may also indicate an error such as insufficient funds in the account.
Exceptions and error status messages differ in their recoverability. An exception is reserved for situations where a fundamental error has occurred that is not likely to be recoverable. Step 4 provides an example of an exception being thrown for an invalid account number. This situation may indicate a significant problem (a data error) that is not likely to be recoverable.
An error status message is reserved for an error status that is returned by a Web method and that indicates a business error occurred where the business error is likely to be recoverable. An example of a business error that is likely to be recoverable is an attempt to execute a payment when there are insufficient funds in the account.
When the Web method returns a response message, the response message contains a Boolean status field. The orchestration checks the value of this field to determine if the Web method call was successful. In this case, the Web method call is successful if the appropriate amount was debited from the account. To check the value of the field, a Decide shape is used to test the value of the status field as shown in Figure 6.
Figure 6. Checking the status in the Web method response
The following is the expression in the Decide shape.
(DebitResponse(ExecutePaymentSchemas.success) == true) && (errorNo == 0)
Notice that the Decide shape also checks the value of the global orchestration variable, errorStatus, which is set to 1 if an exception occurs. Assuming that both of these conditions are met, no error has occurred, so the orchestration proceeds to "Step 6: Sending the Payment and Receiving an Acknowledgment."
If either of the conditions is not met, an error occurs, and the orchestration writes status information back to the database. Specifically, the orchestration writes status information to the Status field of the appropriate record in the Payment table. The orchestration then terminates by using the Terminate shape. The Terminate shape sets an error message that which appears in the operations view of the BizTalk Server Health and Activity Tracking (HAT) console.
Step 6: Sending the Payment and Receiving an Acknowledgment
After the account has been checked for sufficient funds and the funds have been debited, a payment message is sent to the appropriate payment authority based on the value that is specified in the fulfillmentTypeId field in the payment. To simplify this process and to avoid having to change the Execute Scheduled Payments business process whenever the payment authorities make changes, the payment message is sent to a Message Broker. Message Broker is implemented with BizTalk Server also, For additional information, see Implementing Message Broker with BizTalk Server 2004.
Message Broker selects the appropriate payment authority based on the fulfillmentTypeId field, transforms the message to the appropriate format for that payment authority, and then sends the message. This series of events is shown in Figure 7.
Figure 7. Sending the payment request and receiving a response
After the payment message is sent to the payment authority through the Message Broker, the payment authority sends an acknowledgement back to the ReceivePaymentResponse port. When these acknowledgement responses arrive, they need to be correlated to the correct instance of the payment orchestration for which they are intended.
To correlate the response message with the orchestration instance, the developer creates a correlation type that specifies certain fields in the message. These fields are guaranteed to be unique across all instances of the orchestration; this example uses the PaymentId field. The developer creates an instance of this correlation type called a correlation set. The correlation set is be initialized with the actual value of the PaymentId from the PaymentRequest message when that message is sent out the SendPaymentRequest orchestration port. Lastly, the developer configures the Receive Payee Response shape to wait for an acknowledgment message that matches this correlation set.
Now, when the payment authority sends an acknowledgment back, the acknowledgment includes the PaymentId. The BizTalk orchestration engine automatically passes the acknowledgment to the instance of the ReceivePaymentResponse port with a correlation set that equates to this PaymentId.
Step 7: Compensating for Payment Errors
At this stage in the orchestration, the following errors could have occurred:
Exceptions and errors are handled in the same way as in steps 4 and 5. One or more exception handlers catch exceptions, and a Decide shape checks the status fields in the response messages.
Note: The Send Payment Request Tx scope shape can have a timeout attached so that if no response is received within a certain time, the orchestration automatically generates an exception.
Compensation processing is also needed at this point. Compensation processing reverses the effects of earlier transactions that have occurred in the business process. In this example, the customer account is debited before the payment request is sent to the payment authority. It is not possible to roll back this transaction because it has already been committed and the money has been deducted from the customer's account balance. Because the bill has not been paid, the compensation processing must reverse the effects of this transaction by crediting the money back to the account.
Figure 8 shows how the compensation is added to the Scope shape that sends the payment request message. The compensation logic transforms the payment message to create a Web method request that can be sent to the CreditAccount Web method.
Note A real-world scenario would require additional error handling around the compensation code, but this has been excluded in this example for simplicity.
Step 8: Deploying the Orchestration
The orchestration, schemas, and maps now must be compiled to a .NET Framework assembly, and then the assembly must be deployed to the BizTalk Server Configuration database and to the global assembly cache. To deploy to the global assembly cache, the assembly must be signed with a strong name.
Before building the assembly, create a key file by using the following command at the command prompt.
Copy the file to the project folder, and then enter the file name in the Assembly Key File name property in the project properties. Next, build and deploy the solution.
Note The developer can also use the scripts that ship with the BizTalk Server 2004 Software Development Kit (SDK) to perform these actions.
Step 9: Setting Up Send and Receive Ports
As well as the two sets of SOAP ports that are automatically negated when the orchestration is deployed, the orchestration has the following ports:
A physical BizTalk receive or send port is created for each of these and is bound to the appropriate orchestration port. The developer uses BizTalk Explorer inside Visual Studio .NET to create these ports. After the physical ports are set up, the orchestration is bound to these ports in BizTalk Explorer. The developer can also use the BTSdeploy command line utility and a binding file. The binding file is created by using the Export BizTalk assembly binding to file option in BizTalk Deployment Wizard to bind ports.
Note The ReceivePaymentRequest port is configured to use the SQL Server Adapter to periodically execute a SQL query. This is covered in more detail in the next section, "Step 10: Setting Up the SQL Server Adapter."
Step 10: Setting Up the SQL Server Adapter
In this step, the developer creates a BizTalk Server receive port that is configured with the SQL Server Adapter. The SQL Server Adapter is configured to periodically execute a query that returns all payments to be processed as an XML recordset. Figure 9 shows how the SQL Server Adapter is configured to periodically call this query and to return the individual XML records.
Figure 9. Configuring the SQL Server receive adapter
The following query returns all payments that need to be run now as an XML recordset that is based on scheduled time (payment.dateToMake) and status (payment.status).
SELECT payment.MFtransactionId, payment.amount, payment.dateMade, payment.dateToMake, payment.frequency, payment.status, Payee.MFPayeeId, Payee.fulfillmentTypeId, Payee.payeeName, Payee.accountNumber, customerAccount.customerAccountId, customerAccount.MFaccountNumber, customer.firstName, customer.middleName, customer.lastName FROM payment, Payee, customerAccount, customer WHERE payment.payeeId = Payee.PayeeId AND payment.customerAccountId=customerAccount.customerAccountId AND customerAccount.customerId=customer.customerId AND payment.status='sc' AND payment.dateToMake > getdate() FOR XML AUTO, ELEMENTS
As described in steps 1 and 2, the record set returned is split into individual payment messages by using an XML envelope schema, and each of these messages creates a new instance of the orchestration.
Step 11: Starting the Orchestration
Finally, the developer uses BizTalk Explorer to bind the physical receive and send ports created in steps 9 and 10 to the ports in the orchestration. The developer then starts the orchestration. Now, the SQL Server Adapter should execute the SQL query every six hours, and one instance of the orchestration should be created for each payment record that is returned. These orchestrations execute the scheduled payment as described and terminate upon completion.
This implementation of Process Integration results in the following benefits and liabilities.
Separating the process definition from the functions provided by the applications means that you can test each business function (service) individually by creating simple test drivers that call the functions and verify the results. For example, you can test the Gateway Web service by using a.NET Framework application to call the Web service, by supplying various test parameters, and by validating the responses.
The advantage of using a commercial process manager such as BizTalk Server 2004 is that it is designed to handle complex issues that are related to throughput, threading, transactions, locking, and performance bottlenecks. This means that you only need to perform minimal testing of these aspects of the application. This also means that more testing effort can concentrate on the unit tests of the individual steps of the business process and on the integration of these steps into a complete business process.
Applications built using BizTalk Server orchestration typically access multiple business systems that encapsulate the individual steps of the business process. These business systems often provide their own authentication mechanism, usually a user name and password. A common requirement when using BizTalk orchestration to access these systems is for the orchestration to supply credentials for the user who is represented by this instance of the business process.
For example, a business process may be initiated by many users filling out a Microsoft Office InfoPath™ form and sending it in a SOAP message to BizTalk Server. The messages initiate a new instance of the orchestration for each user. These instances then access business systems such as an Enterprise Resource Planning (ERP) system by using the user's credentials for that system. Because it is not practical to request the credentials from the user, credentials are either encoded in the message itself or accessed by BizTalk from a secure store.
BizTalk Server provides the enterprise single sign-on (SSO) system to store credential information. SSO provides a secured store, which is based on the user's security context, and stores security and state information that is pertinent to that security context. In the earlier example, the security context of the user filling out the InfoPath form can be used to retrieve the user name and password for the ERP system from SSO. The orchestration can then access the ERP system on that user's behalf.
BizTalk Server orchestrations are bound to a BizTalk Server host. A BizTalk Server host is an engine for messaging and orchestration. An instance of a BizTalk Server host can be created on each BizTalk Server computer in the BizTalk Server group, which results in the creation of a process on a computer through a Windows service. The process then executes instances of the orchestration.
You can use the BizTalk Server Administration console to create additional hosts. You can then bind the additional hosts to orchestrations and map to servers in the BizTalk Server group. This allows orchestrations to be deployed to any combination of servers in the BizTalk Server group. Therefore, the business application can be distributed across the servers in your enterprise as needed.
Process Manager [Hohpe04] describes the internal function and implementation of the process manager component.
[Hohpe04] Hohpe, Gregor; Bobby Woolf, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley, 2004.
[Chapell03] David Chappell, Chappell & Associates; Understanding BizTalk Server 2004. Microsoft Corporation, 2003 (http://go.microsoft.com/fwlink/?LinkId=21313).