Enterprise information systems contain a variety of applications that provide levels of interaction that range from simple to complex. The functions that are included in these systems offer a variety of capabilities and levels of access, from undocumented functions and silo applications to fully developed composite applications. In addition, the mechanism for invoking a function differs from one system to another.
How do you integrate information systems that were not designed to work together?
To correctly solve this problem, you need to consider the following forces:
Integrate applications at the business logic layer by allowing the business function in one application (the source) to be accessed by other applications (the target), as shown in Figure 1.
Figure 1. Integrating applications at the business logic layer
For an external application to integrate with the source application through Functional Integration, the following two conditions must be met:
If the desired business function is not available, you have to modify the source application to make the function available. If modifying the source application is not feasible, you should add the new function outside the source application and then communicate with the application through another form of integration, such as Data Integration or Presentation Integration. This approach has fewer side effects and is generally available for most types of applications.
Many applications only expose their business functions as a local API that is dependent on a specific programming language such as C++ or C#. In those cases, a local adapter or middleware interface has to be created that translates incoming messages from other applications into local API calls. Likewise, results from API calls are translated back into messages. In most cases, such an adapter can be generic enough to support a variety of different functions without having to be modified for each individual function that you want to make available externally.
Functional Integration is very versatile because many different operations can be performed though an API. A functional interface can retrieve data, update data entities (change an address), or perform business functions (validate a credit card). In fact, one common use of Functional Integration is to ensure data consistency between applications [Ruh01].
Functional Integration is based on the interaction between the components that are described in Table 1.
Table 1: Functional Integration Components
After you decide to use Functional Integration, you must choose a particular kind of integration that is appropriate for your situation. Your choices are summarized by the following patterns:
Distributed Object Integration
Distributed Object Integration is also known as instance-based collaboration because it extends the model of object-oriented computing to distributed solutions. Objects inside one application interact with objects in another remote application in the same way that they would interact locally with another object. This implies that the interaction occurs with a specific object instance and that the client application often manages the lifetime of the object it is accessing. This type of interaction usually seems natural to application developers, but it can result in a complex and tightly-coupled interaction model between the components. This tight coupling is not a problem as long as the components are part of a single distributed application. However, it is generally not a good choice when integrating multiple stand-alone applications.
Great examples of distributed component middleware are technologies such as .NET remoting, COM+, or CORBA. For more information, see the Remote Procedure Invocation pattern [Hohpe04] or the "Distributed Systems" chapter in Enterprise Solution Patterns Using Microsoft .NET [Trowbridge03].
Message-Oriented Middleware Integration
Message-Oriented Middleware Integration connects systems by using asynchronous message queues that are based on proprietary message-oriented middleware. The connected systems then communicate by using messages that contain small packets of data. Because the communication is asynchronous and durable, there is little chance of the messages being lost during network or system failure.
To share request/response type functionality, the consuming system must create a request message and send it by way of the message queue to the system that provides the functionality. The provider then takes the message from the queue, interprets it as a request, and processes it. Upon completion, the provider creates a response message and sends it back to the functional consumer by way of the message queue. Of course, not all functionality is shared by using a request/response style collaboration, but similar principles apply. For more information, see the Messaging pattern [Hohpe04].
Service-Oriented Integration connects systems by enabling them to consume and provide XML-based Web services. The interfaces to these systems are described through Web Services Definition Language (WDSL) contracts. Systems interact with each other by using SOAP messages. SOAP messages are usually conveyed through HTTP by using XML serialization.
Figure 2. Using a service gateway and service interface to connect a Web service consumer and provider
Using Service-Oriented Integration increases interoperability by using XML and XML Schema as the basis of message exchange and by using SOAP as an extensible messaging framework. XML Schema provides for a type system that is portable between disparate technical architectures. In contrast, SOAP can be bound to a number of different transport mechanisms. For more information, see Service-Oriented Integration.
Choosing Between Alternatives
There are many factors to consider when choosing the kind of Functional Integration that is best for your particular requirements. Some of these factors include:
Choosing Distributed Objects
If your team is proficient with object-oriented development and if you use a platform infrastructure that offers a Broker such as .NET Framework remoting, Distributed-Object Integration can be fairly simple to implement. Although the remote interface is almost as easy to manipulate as the local interface, you must always consider network latency, network failure, and distributed systems failures. You must develop a significant amount of error detection and correction logic to anticipate these impacts. With high-latency network round trips, you want to avoid a large number of fine-grained method invocations. Therefore, use Remote Facade [Fowler03] and Data Transfer Object [Trowbridge03] to optimize call granularity and payload size.
This kind of integration works best when the systems that you want to connect are located in the same data center and when those systems have fairly reliable and high-speed connections. This kind of integration does not work well across slow and unreliable connections, including any connection that uses the Internet. This kind of integration is fragile if incompatible updates are introduced to any of the participants. Of course, if the target system you want to integrate with only exposes object-based APIs, you must use this method or an adapter to connect.
Choosing Message-Oriented Middleware Integration
Message-Oriented Middleware Integration is an excellent choice when systems are not reliably connected because there is the potential for network failure or distributed systems failure. Because Message-Oriented Middleware Integration works by placing messages asynchronously into a durable queue, there is little chance that the messages could be lost during a system or network failure. In addition, using asynchronous communication decouples the sender from the receiver. The application that sends the messages will continue to function even if the receiver fails, and the failed message receiver will continue processing messages from the queue after the receiver is restored.
The sender and the receivers are also decoupled from each other because they use a common message format. This makes it easier to separate the message from the set of intended receivers in the enterprise. After a message is sent from a source, you can design an integration network that uses the Message Broker pattern, the Message Bus pattern, the Publish-Subscribe pattern, or a host of other message receivers without modifying the original message sender. This allows the source, distribution, and consumption of the message to vary independently, thus improving flexibility.
Of course, there are tradeoffs involved with using Message-Oriented Middleware. Although the programming model is fairly simple for Distributed-Object Integration, the programming model becomes much more complex for Message-Oriented Middleware. Messages arriving at endpoints become events; therefore, the programming model is event driven. In addition, these events arrive across networks that may be unreliable and that do not have implicit correlation between them. The order of message arrival is not necessarily guaranteed and there may be duplicate messages. Sometimes, processing duplicate messages (also known as idempotent messages) does not adversely affect your system. Other times, processing duplicate messages is a serious flaw, for example, when you are transferring money. The security context of the message must be established, policies must be applied, and trust boundaries must be established. Your particular requirements must anticipate and justify this additional complexity. In addition, using proprietary message-oriented integration binds you to a particular message-oriented middleware implementation. It must be installed on the endpoint you want to communicate with, which may not always be the case inside your enterprise or between trading partners.
Most enterprises have heterogeneous technical architectures and want to take advantage of system-based collaboration over the Internet. The most competitive enterprises want flexible and automated business processes that may require technology independence. All of these factors contribute to an urgent requirement for interoperability between different technical architectures. The best way to design for interoperability is by using Service-Oriented Integration.
Like Message-Oriented Middleware Integration, Service-Oriented Integration uses messages to communicate between senders and receivers. Unlike Message-Oriented Middleware, Service-Oriented Integration uses XML Schema and SOAP to transport and resolve messages. These two standards provide a portable type system and an extensible messaging framework that is not coupled to any proprietary implementation or transport. In addition, Service-Oriented Integration recommends the Web Services Integration (WS-I) Basic Profile to ensure interoperability between endpoints.
Service-Oriented Integration enables interoperability and allows you to send both synchronous and asynchronous messages (for more information, see the Service-Oriented Integration pattern). As a result, you can have the same kind of complex programming model as Message-Oriented Middleware. In addition, you have the complexity of a new and still emerging set of standards to understand and comply with. Therefore, make sure your requirements justify this level of complexity.
After you build these interoperable systems, there is one more tradeoff to consider: performance. Using XML incurs the cost of serializing, deserializing, and parsing XML documents. In addition, XML documents are often much larger than their binary equivalents because they contain metadata. This can increase the size of the payload that must be processed during message exchanges. However, because processing power is relatively inexpensive and because processors get faster every year, the issue of payload size can be addressed though hardware. Also, you can selectively trade interoperability for performance by using binary encoding inside your SOAP message as needed. Finally, given the support of major vendors, it is likely that new platform infrastructure will evolve in a way that will optimize Web services–based performance.
Combining Distributed Objects, Message-Oriented Middleware, and Services
It is likely you will use some combination of all three types of integration in your enterprise. Start by identifying services at a level of granularity that is meaningful to the business, such as steps within a business process. This will help you define service boundaries where interoperability is important. For interactions across these boundaries, use Service-Oriented Integration. For an example of service identification, see Chapter 9, "Integration Examples." To implement a service within these boundaries, you may want to use Distributed-Object Integration, Message-Oriented Middleware Integration, or Service-Oriented Integration. If you need to do two-phase commits across distributed databases, using Distributed-Object Integration permits you to take advantage of platform infrastructure that supports two-phase commits across distributed databases. Just ensure that you keep these transactions inside your service boundaries.
Any time you connect with systems and networks that you consider to be unreliable, consider using Message-Oriented Middleware to provide a durable buffer between connected systems. Using messages as your means of communication, either through Message-Oriented Middleware or Service-Oriented Integration, permits you to construct more advanced integration architectures that use Message Broker and Message Bus to communicate. As needed, you can always connect these integration architectures by using object-based APIs as described by the Adapter pattern [Gamma95].
Regardless of the kind of Functional Integration you use, you will encounter the following benefits and liabilities.
Functional Integration is easier to test than other integration approaches for the following reasons:
However, you must consider the following considerations. Your ability to test depends on how well the functional interface of the existing application is structured. If the interface exposes functions that are well-defined and that are free of side effects free, testing should be relatively easy. If the functional interface consists of a blur of poorly defined functions that are full of side effects, then testing will be difficult.
The richer semantics of Functional Integration allow for a more finely grained security model. Many applications enforce security rules based on user identity, and they control permissions on an object level. For example, a user is only allowed to access the accounts that belong to that user and to access related accounts. This type of permission checking is typically not available with Data Integration.
The disadvantage of this more complex security model is that the integration solution has to acquire the proper user credentials to access data or to invoke functions. Because other applications are generally unaware of the security context used in another application, the security context has to be established by the middleware. You can establish the security context either by using a fixed user or by using an explicit translation of security contexts between applications.
[Hohpe04] Hohpe, Gregor, Bobby Woolf, Enterprise Integration Patterns. Addison-Wesley, 2004.
[Linthicum04] Linthicum, David. Next Generation Application Integration. Addison-Wesley, 2004.
[Ruh01] Ruh, William. Enterprise Application Integration. A Wiley Tech Brief. Wiley, 2001.
[Trowbridge03] Trowbridge, David, et al. Enterprise Solution Patterns Using Microsoft .NET. Microsoft Press, 2003. Also available on the MSDN Architecture Center at: http://msdn.microsoft.com/architecture/patterns/default.aspx?pull=/library/en-us/dnpatterns/html/Esp.asp.
[W3C04] "Web Services Architecture W3C Working Draft 11 February 2004." Available on the W3C Web site at: http://www.w3.org/TR/2004/NOTE-ws-arch-20040211/