coarsegrained: Pluggable Services in the Web Services Bus

December 03, 2003

Pluggable Services in the Web Services Bus

In previous posts discussing the concept of a "web services bus", I've referred to the concept of pluggable "integration utility services". What exactly am I referring to here? And just what is the pluggability mechanism?

The Value of Pluggable Services>
As we've mentioned, the web services bus allows endpoint applications to an easier way to connect to other applications as web services. They simply drop messages onto the bus. The messages are then sent to their recipients by the bus itself, either to an explicit logically named recipient or to a direct physical address. Alternately no recipients are specified and either content-based routing is used to deliver the message to interested parties, or the sender can publish the message to a topic on the bus. All well and good: buses make it easy to deliver messages.

But there are other aspects of the XML and web services-based integration puzzle that an intermediary can assist with. One example is content transformation. End applications can drop off messages onto the bus and a content transformation service can be registered for all messages (or just some document types) to assist in transforming those messages to the format that particular recipients can support. The mappings need to developed and entered into the registry. But, just like the endpoint application programmer no longer needs to be worried about the address and protocols used by other endpoints, the transformation service allows them to no longer be concerned about the XML document types that they send. Making the service available as part of the services of the bus allows the endpoint to be truly unaware of the transformation capability and how it is configured. The maintenance and administration of the service (e.g. storage and updates of the mapping rules) is performed by the intermediary - the end applications and web services just reap the benefits.

There are other parts of the web services integration process that currently involve configuring endpoint services that can instead be moved into a capability of the bus. Web services management vendors (the likes of Amberpoint, Confluent, Actional and others) all provide some way of summarizing the performance, reliability and throughput of individual web services. Today, they do this in one of a few ways:
- modify the endpoint application server to intercept the http traffic (often referred to as deploying an "agent" to the endpoint)
- direct all web services traffic through a proxy server which they then use to intercept traffic and forward it on to its true destination
- deploy an IP "sniffer" and filter it for "web service" traffic

None of these solutions are particularly satisfying. Either they require touching each endpoint web service deployed in the enterprise (an approach which is very tedious as organizations deploy thousands of web services) or they create serious performance bottlenecks in the web service traffic. They also require a extensive development commitment on the part of the vendors. However if web service traffic is passing through a mediating bus, the bus can simply give the management reporting service a chance to look at each message as it passes. All of the work of getting at the web service traffic is avoided - the bus hands the reporting service each message as it goes by.

Potential Pluggable Services
There are several other services that exist now that can be more easily developed and deployed assuming a bus that can give them access to web service traffic. This includes "business activity monitoring" or "business intelligence on XML content" services that aggregate information about the XML document payloads of web service traffic (one realtime, the other usually asynchronously against offline stores). One of the attributes of XML document-based integration that web services vendors and corporate IT personnel are noticing is that they contain information about the business event in denormalized human readable form. Capabilities are emerging from management vendors such as AmberPoint and Service Integrity to aggregate and analyze this content level information. We think traditional business intelligence vendors will start noticing the same opportunity and provide similar capabilities. But, for sure, a general Business Intelligence vendor would not want to have to build their own agent or proxy server capabilities just to get access to the XML document (it is way too far afield from their core business). Such XML content analysis services can simply register their interest in certain classes of XML documents and they will be handed to them.

Other possible services include a variety of security services ( performing encryption, transformation of tokens, policy negotiation and enforcement), business process management, transaction management (of the loosely coupled, long running transaction flavor), and XML document validation. One particularly important pluggable service is of course content routing itself, which enables one of the most important value propositions of the bus, the ability to send messages without specifically identifying recipients or explicitly publishing to a topic (we'll talk more about content routing later).

The Pluggability Mechanism
OK, so these pluggable integration utility services provide value. Just how do we go about "plugging" such services into the bus? When an integration service is deployed, the administrator registers what document types (potentially all of them) the service is interested in the registry that any bus needs (we at Systinet would use a UDDI-based reigstry). Registering a utility service involves implementing a WSDL-defined service that exposes operations that act as "callbacks" to notify the utility service of a relevant document. It also involves identifying when the utility service should be notified. This can be global (every document), a set of specific document types, or XPath expressions that restrict use to only some subset of documents.

Is This Just Content-Based Routing?
You could say that this is similar to what is used to set up content routing to have documents sent. However, the usage pattern is different for the administrator setting up a set of utility services to participate in web service integration flow. First of all, for the administrator of the bus itself, utility service configuration involves setting up many different services and, usually, having very simple rules to define what documents the utility is interested in. It will be very common for utility services to simply be applied to ALL documents or documents of a set of document types: without necessarily defining XPath expressions to filter the traffic. For example, management reporting services typically are interested in looking at every XML document flowing by. The administrator of an endpoint application typically manages just a very few content-based routing rules to identify documents they are interested in receiving. And, by contrast to the administrator setting up utility services, the criteria that they use to establish their interest is almost always at the "business level" of the content of the document. For example, "send me all purchase orders for products in the durables category", stated as an XPath expression. Also, at runtime and later, when analyzing the traffic (such as the management tools we've mentioned do such a good job at), administrators (of both endpoint apps and the bus itself) will want to make a clear distinction between documents sent to endpoints and the document processing done by integration utility services. An endpoint app processing a document has a very real business meaning, but the intercession of a utility service in document flow should not be considered meaningful at a business level. This is not to say that content-based routing can't be helpful in handing off traffic to utility services. Its just that the administration preparatory configuration and posthoc analysis usage scenarios (and by extension the ideal user interfaces) are quite different.

Dependencies, Priorities and Pipelines
Also during the registration process, any dependencies on other utility services that a specific utility service may have are identified. A priority can also be specified to affect the order that integration services are invoked. Note that this is an "implied pipeline" of services. The administrator does not have to configure an explicit pipeline of services, which would be difficult to do since different services subscribe to differing document types. An example of such a dependency/priority hierarchy of services would be: an encryption service specifying a dependency on a transformation service to insure that it is invoked only after any content manipulation is performed, or a management service specifying itself as low priority, insuring that any multiplication of messages through transformation happens before it reports on the traffic.

What Happens At Runtime
At runtime the web services bus then queries the registry for any integration utility services that need to look at or modify a message that is coming through. It notifies the registered service by invoking WSDL-defined contracts that those services have implemented. Another WSDL-defined API allows those services to retrieve portions of the message (individual headers or parts of the payload or the whole document) from the bus to perform their work. If they are registered "read only" services they simply inform the bus when they are complete by invoking another WSDL-defined operation. If they need to modify the document, they post either the entire document or modify individual portions of it via another WSDL-defined operatoin.

December 03, 2003 | Permalink

Verify your Comment

Previewing your Comment

Posted by: |

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:

Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Sun	Mon	Tue	Wed	Thu	Fri	Sat
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30

coarsegrained

thoughts and observations from Adam Blum on web services and XML document-oriented development approaches and tools

December 03, 2003

Pluggable Services in the Web Services Bus

Comments

Verify your Comment

Previewing your Comment

Post a comment

About

September 2005

Recent Posts

Categories

Blogs

Specs

Companies

Archives