coarsegrained

Web Services Buses, Part II: The Web Services Bus Itself

In our previous post we discussed what requirements might exist for web services message buses. I have gotten several questions about this. Am I talking about Enterprise Service Buses? ESBs are a very useful term to hang many interrelated capabilities around. To quote from one early ESB entrant:

An ESB is a standards-based, service-oriented backbone capable of connecting hundreds of application endpoints. ESBs combine messaging, Web services, XML, data transformation and management to reliably connect and coordinate application interaction. The ESB deployment model is an integrated network of collaborating service nodes, deployed in service containers.

All well and good. I am very excited about the ESB category in general and the opportunities it creates for web services infrastructure (for a good recent posting about ESBs in general, see Phil Wainewright's October 30 entry on http://looselycoupled.com/blog)

But the interesting question is just what the messaging bus within this definition is. What are its communications capabilities? What are the APIs, ideally, for interacting with it? Where do the communications capabilities execute? Are they on servers somewhere centrally, replete with failover for any data containers (queues, topics) that are located there? Or is most of the work and data distributed (as some legacy EAI buses, such as Tibco's Rendezvous) are? If the interfaces to the buses facilities are new and, presumably, web service-oriented, does that mean the underlying bus must be new as well? Do companies have to rip and replace their existing infrastructure? Do we need a world of web services that are all loosely-coupled and supporting WS-ReliableMessaging and WS-Security before we can start getting any use out of this bus?

I'd like to put some stakes in the ground of some attributes what the ideal web services bus could look like, if it were available today:

• has set of communications capabilities that are create to support web services communication. In other words: every communication model ever supported by a bus is not necessarily here. But certainly: publication/subscription and other mechanisms of broadcasting to multiple recipients, as well as one-way or two-way asynchronous communication (of various forms), would be supported.
  
• the capabilities supplied need to be particularly robust in support asynchronous loosely coupled web services (which drives the scenarios for using a bus architecture in the first place)


• all bus features (communications capabilities such as individual message send and receive, and management functionality) are invocable via modern (WS-*) OR simple web services standards (plain SOAP)
  
• can leverage existing infrastructure that provides reliable messaging (from endpoint perspective sending individual messages). It is not necessary to rip out existing middleware buses that the company already has in use
  
• can still also be deployed on top of unreliable transport (e.g. http) and use web service standards such WS-ReliableMessaging to provide reliable messaging, where no such existing bus is in place
  
• can be deployed to provide fault tolerance for all data stored and all service invocations, whether this is provided on explicit servers hosting the service, or in a distributed deployment
  
• enables easy plugin of not just application participants but also "integration facility participants", that provide many capabilities to ease the integration process, including a standards-based registry and a standardized "utility service plugin interfaces"
  
• although thriving on loosely coupled coarse grained asynchronous integration expands the universe of potential integration by providing extensive bridging capabilities (as such facilitation services) for "simple endpoints" (everything from providing the reliable messaging capabilities for simple SOAP invokers to making simple synchronous services look like asynch services to providing security services for negotiating and enforcing security policies)
  
• in factoring out what capabilities are "the communications bus itself" versus what capabilities might be provided by facilitation services, respect the boundaries of transport and envelope versus payload. That is, content routing (in push model), transformation, and content-level subscription are all content-related and are naturally done as "integration facilitation services". Various flavors of asynch communication for one or multiple recipients are all at the communication and transport level and should be provide by the core bus.
  
• It needs to be easy to manage and monitor the state of the bus and its collective services and plugged in applications at any time: what endpoints are plugged in, what integration utility services are registered to process what documents. A standards-based registry of this information will go a long way toward that goal
  

What I haven't done here is describe why each of these items is required for a good "web services bus". I also haven't described really what such a bus would look like: what are its components, interfaces and value-added services. I'll be posting on this topic fairly soon.

XML Schemas to represent data types

Jon Udell did an interesting post today on XML Schemas and data types, specifically with reference to the WinFS data model. He notes the

remarkable synergy that InfoPath, most notably, makes possible. The message payloads exchanged on the Web services network, and the documents read and written by people, can be the same texts, governed by the same datatype and structure definitions. And those texts are universal in scope, not tied to any platform or framework.

Of course this only works if the information conveyed over the wire is document-oriented. InfoPath is not going to do much with RPC-like method calls delivered over the wire. Yes, it could display one or two fields to be entered into a method call - but this would certainly be a case of "way too big a hammer". Indeed, given a good job in designing the XML document content, InfoPath's capabilities are quite exciting. I maxed out my Microsoft alumni quota buying copies of Office 2003 Professional for my colleagues, so that we could all have InfoPath available.

The important part of the post is Jon's comments on Dare Obasanjo's statement that

W3C XML Schema is a way for describing the structure and contents of an XML document while a WinFS schema describes types and relationships of items stored in WinFS.

Jon basically says that there is nothing particularly special about the data types in WinFS, and that they could have all been defined by XML Schema. I agree. I have my own separate reservations about just how much XML Schema can validate. But this has nothing to do with why WinFS needed a different type definition method. Anyway, here's where Jon's post gets interesting.

It seems to me there is lots of precedent for a third approach: use XSD for base datatype/structure definition, and advance a new layered standard (if needed) for relationship definition.

OK, now we're getting somewhere. Its this ability to have reusably defined types and easily assemble them into XML document definitions that we're looking for. Also, it would be great to have software that let you "harvest" the types present in XML document schemas and WSDLs to encourage type reuse, or for better searching amidst a sea of interface definitions. The use of XML Schema to define components to be used in larger documents and software is a fascinating opportunity. There is room for a lot of innovation in the space. This includes Jon's suggestion of of "relationship objects" to tie underlying types together. It can also include "XML schema type repositories" and other software to facilitate management and composition of such building block types. I will posting more on this topic later.

The Web Services Bus

I've discussed in earlier posts why its important to facilitate building loosely coupled, coarse-grained document-oriented web services. I've more to say on that subject as time goes on. But I've also gotten feedback in various forms from customers, partners and attendees of talks I've done to the following effect:

Thats great, I already agree that this is the best way to build web services. We're already doing that: defining common shared libraries common XML document schemas that the whole enterprise uses. But once we have these shared vocabularies does it really make sense to keep building out peer to peer SOAP connections? If we have shared document models, can't we use that to enable some form of hub-based communication model?

Of course the way people say this tends to vary. Sometimes the feedback is "hmmm, shared object models. Sounds like the old days of EAI". In effect the answer to all of these statements and questions is "yes". And the Systinet toolset can be used in these scenarios. We have JMS support so you can "build your own web services hub" with Systinet on top of JMS. The question really is, is there more that can be done? Can we use the tools and APIs available for web services protocols to work on top of common messaging buses (JMS, proprietary buses, and, with the advent of WS-ReliableMessaging, SOAP over http) to provide a hub for web services communication.

The MOM market has a number of common paradigms that it supports in synchronous and asynchronous messaging, for two party and multiparty scenarios. For two party asynchronous communication, some of the scenarios include:

• Asynchronous request-response. The bus infrastructure can create correlation variables to help manage this scenario.
• One way message to a web service from the client
• Solicit response – services send requests to clients. The bus creates correlation variable and client sends back response
• Notification – web service sends a message to the client

For multiparty asynchronous communications, a messaging bus provides great value in assisting with simultaneous communication with multiple other endpoints. The two primary methods of performing this are:
• Multicast - which sends identical messages and content to multiple named parties where each parties status and response is tracked individually. Value-added services such as content transformation or routing can plug into multicast events to optimize the communication to individual parties. The necessary correlation IDs are generated to be able to track each participants status and responses
• Broadcast – messages are sent to some form of aggregated list such as a topic or a group. Responses are not tracked.

For long running transactions (for two or more parties), the bus can supply support for and management of long-running web services conversations. It can generate and manage the conversation IDs that are necessary for tracking these conversations. Many endpoints may not have the capability of generating conversation IDs. The bus can provide calls to generate the conversation IDs transparently and to allow endpoints to check on the status of the conversation at any time. It can support several methods of communicating the conversation meta-information including WS-SecureConversation.

Finally, even though the bus is primarily built to support loosely coupled integration, it can still add value to the world of simple synchronous request/response. And such support may be necessary when sync communication is used in a longer conversation being managed by the bus. That is, if synchronous communication is all that is desired for some particular integration a MOM may not even be used. But often a long running conversation may include one or two synchronous web services invocations to facilitate the process. The bus can support this scenario and even provide value-add in providing error handling and failover for such scenarios that a simple synchronous web service client might not have the time or capability of providing. Even more interesting: the bus (or a service on the bus) can provide the capability for endpoints that invoke simple synchronous services to treat those as if they were asynchronous. Endpoints can fire and forget and communications services offered by the bus will call them back when the synchronous execution is complete and provide the data in the response.

The question really is: which of these are important in the web services world? Even if you build a web services layer on top of very full-featured MOMs, do you need all of the possible methods. I'm very interested in feedback from people doing web services-based integration (and even application integration using older technologies) about which of these capabilities continue to be important.