BizTalk Server and the ESB SOA design pattern

In January this year I participated in a SOA architect workshop and after 5 exams I officially became certified SOA architect. The certification was issued by SOA schools which is an initiative of the well known SOA author Thomas Erl.

It was a very interesting course which discussed concepts I already knew, but wasn’t really aware of. During the course I more and more was able to map the concepts of SOA to Microsoft technology. One of the SOA characteristics is vendor neutrality so the course didn’t really discuss implementations.

Two course modules discussed design patterns (http://www.soapatterns.org/) and one of them is the ESB compound design pattern. It is called a compound pattern because it is build of other design patterns.

I started to recognize and map parts of BizTalk Server onto the ESB design pattern, but I didn’t have the full picture yet. In this blog post I take the ESB design pattern as a starting point to find out how and if BizTalk Server can be called an ESB based on this design pattern.

 

The ESB compound design pattern consists of the base pattern with these design patterns:

  • Asynchronous Queuing
  • Service Broker
    • Data Model Transformation
    • Data Format Transformation
    • Protocol Bridging
  • Intermediate Routing

and the extended version is:

  • Reliable Messaging
  • Policy Centralization
  • Rules Centralization
  • Event-Driven Messaging

BizTalk Server (together with the ESB Toolkit) is Microsofts product for implementing an ESB. To what extend does the vendor neutral design pattern map to the functionality in BizTalk Server?

Let’s start with the base pattern.

Asynchronous Queuing

In the SOA world this means that a service should be able to exchange messages with its consumers via an intermediary buffer, allowing service and consumers to process messages independently by remaining temporally decoupled.

BizTalk Server is asynchronous by nature, because all messages flowing through BizTalk are stored in the SQL Server message box thereby disconnecting consumers from services.

This SOA design pattern is explained here.

Service Broker

The service broker design pattern covers some of the most basic concepts necessary for application integration. This is about conversion of messages and the capability to receive messages via various types of protocols.

  • Data Model Transformation
    • A data transformation technology should be incorporated to convert data between disparate schema structures.
  • Data Format Transformation
    • Intermediary data format transformation logic needs to be introduced in order to dynamically translate one data format into another
  • Protocol Bridging
    • Services using different communication protocols or different versions of the same protocol should be able to exchange data.

Both types of transformations are solved in BizTalk Server using maps. Model transformation is about transforming one XML Schema to another and format transformation is transforming one message format to another, for example flat file or CSV to an XML Schema.

Protocol bridging is implemented in BizTalk Server using adapters and is about the ability of communication between different communication protocols like different versions of SOAP or something totally different like FTP.

Intermediate Routing

From a SOA perspective this is about the ability to dynamically determine message paths though the use of routing logic. Intermediate routing is typically implemented using service agents. A service agent is an event-driven program that doesn’t require explicit invocation, it just intercepts the messages. There are two types of service agents, active and passive. Active service agents not only read the message but also modify the message contents. Passive service agents only read the message and will or will not take action on the message content but doesn’t change it.

In BizTalk Server intermediate routing is implemented by means of a message agent. This agent is monitoring the message box to handle subscriptions to messages. In BizTalk everything is routed via the message box, so it can be implemented with only promoted properties or there can be complete routing logic in orchestrations or via business rules.

Reliable Messaging

This is about assuring that a message is actually received to the service. In a SOA world this is a technology concept to enable reliable communication with services. Commonly this is solved by a combination of the WS-ReliableMessaging standard and guaranteed delivery extensions like a persistent repository.

In BizTalk Server this is called guaranteed delivery, which assures to the caller that once a message is received in BizTalk Server it never will be lost. This is the persistent repository in the section above. The WS-ReliableMessaging standard is not supported by BizTalk due to its architecture.

Policy Centralization

In the context of SOA a service contract can be comprised of one or more XML Schemas, the WSDL, one or more Policies and an SLA. A policy adds information to the service commonly applied using the WS-Policy standard. This information can for example consist of the service window of a service or that it is only capable of handling so many requests an hour. Policy centralization is about applying centrally maintained policies to services.

Looking at BizTalk Server it seems like the ESB Toolkit supports some kind of policy centralization, but that appears to be related to itinerary centralization. Policy centralization probably should be maintained by a 3rd party product and cannot be maintained in BizTalk Server or with the ESB Toolkit.

Rules Centralization

Business rules are critical to companies and having the rules spread across a company is not a sustainable solution. The rules centralization design pattern is about centrally accessing, storing and managing business rules.

Business rules are part of BizTalk Server by means of the Business Rules Engine(BRE). This is even a separate component that can be used without other BizTalk components. Rules can be centrally accessed, stored and maintained in there. In a SOA approach the BRE would be wrapped into a service so it’s functionality is accessible in a standardized way from other technologies besides BizTalk Server. Wrapping such a component into a service raises other issues like introducing a single point of failure and capacity issues. These issues can be solved by applying other SOA design patterns

Event-Driven Messaging

From a SOA perspective this is about notifying customers of events. Commonly this is solved by subscribing customers to service events via one-way and asynchronous data transfer.

This is one of the most known concepts in BizTalk Server, because it refers to the publish-subscribe pattern as it is used in BizTalk Servers messaging architecture with the SQL Server message box as the heart of the system.

      Conclusion

      In this blog post we’ve been discussing the official ESB compound design pattern as defined by Thomas Erl. At least one of the requirements cannot be fulfilled by BizTalk Server, and that is Policy Centralization. So officially BizTalk Server is not an ESB product according to these requirements.

      On the other hand, policy centralization is one of the least used requirements in an ESB implementation at the moment. And besides that, BizTalk Server implements many more design patterns than just the ones mentioned in the ESB design pattern.

      On of the most recognizable design patterns for BizTalk Server is the Orchestration compound design pattern.

      Although this compound pattern is targeted at synchronous processing, it does consist of interesting design patterns we can also find in BizTalk Server:

      • Process Abstraction
        • This is about abstracting logic into a parent processes (task services). In BizTalk Server processes can be constructed of combined (agnostic) services using orchestrations. An orchestration is an abstraction of such parent processes which can be called by services without knowledge of the underlying logic.
      • Process Centralization
        • This is to prevent process logic to be spread across the company. All processes in BizTalk Server are centralized into one repository thereby fulfilling this requirement.
      • State Repository
        • This is about offloading state to a repository to release resources when they are not necessary at that time. BizTalk Server supports this by means of hydration and dehydration of processes.
      • Compensating Service Transaction
        • To prevent system locks during long running processes, compensating measures can be taken in case of an exception. BizTalk Server supports compensating service transactions as part of orchestration building blocks.
      • Atomic Service Transaction
        • To be able to rollback changes if one of the services in a chain of service calls fail, atomic service transactions can be used. BizTalk Server supports atomic service transactions in it’s WCF adapter.
      • Rules Centralization
        • Is also part of the ESB compound design pattern
      • Data Model Transformation
        • Is also part of the ESB compound design pattern

      Didago IT Consultancy

      Azure Service Bus EAI/EDI April 2012 Refresh, What’s new?

      Back in December 2011 Microsoft released the first CTP of the Azure Service Bus EAI/EDI. It showed what direction Microsoft is heading with bringing BizTalk like functionality to the cloud. Or to quote Microsoft:

      “Windows Azure Service Bus EAI and EDI Labs provides integration capabilities for the Windows Azure Platform to extend on-premises applications to the cloud, provides rich messaging endpoints on the cloud to process and transform the messages, and helps organizations integrate with disparate applications, both on cloud and on-premises. In other words, Service Bus EAI and EDI Labs provides common integration capabilities (e.g. bridges, transforms, B2B messaging) on Windows Azure Service Bus”

      If you want to read more about this, please take a look at my two previous blog posts about this topic:

      BizTalk in the Cloud, one step closer (part 1)

      BizTalk in the Cloud, one step closer (part 2)

      Now Microsoft has released the second preview:

      Service Bus EAI and EDI Labs – April 2012 Release

      Download the April 2012 Release

      So what’s new in this April 2012 release?

      Regarding the EAI Bridges, we’ve got the following enhancements. BizTalk developers will recognize parts that currently only are available in BizTalk….

      • FTP Source Component. In the December 2011 release it was already possible to connect WCF endpoints via the publish/subscribe pattern. In this refresh another endpoint type has been added: FTP, which can be used as a source component on the configuration bridge. In the BizTalk world this would be the FTP adapter, but so far it is reading only.
      • Flat File Support. This is also a familiar concept in the BizTalk world. Combined with the FTP source component, flat files for example can be read and converted into XML now. But also via HTTP flat files can be read and processed. The available flat file wizard looks quite the same as currently can be found in BizTalk Server.
      • Schema Editor. Previously only available in BizTalk, now also part of the Service Bus EAI/EDI release. Used to create and edit XSD and flat file schemas
      • Service Consuming Wizard. Also straight from BizTalk Server, the possibility to generate the necessary artifacts to be able to consume WCF services. The difference with EAI is that it only generates an XSD and no other output files like binding info and an orchestration.
      • Operational Tracking of messages processed by the bridge. This is also something from BizTalk Server. Within a bridge, a message undergoes processing under various stages and can be routed to configured endpoints. Specific details of the message such as transport properties, message properties, etc. can be tracked and queried separately by the bridge developers to keep a track of message processing.

      For Service Bus Connect, there are the following enhancements.

      • Support for Windows Authentication while connecting to on-premise SQL Servers
      • User Interface enhancements
      • Support for SSL

      The transforms also have been improved.

      • New map operations. In BizTalk this is called ‘Functoids’ but are named operations here. Added are:
        • Number format operation, to format an input number using a given format
        • Date/Time operation, to manipulate date/time
        • Generate ID operation, to generate a unique ID of a given length
      • Runtime behavior configuration. This is also interesting for BizTalk developers, because like Microsoft says:”In this release, users can configure the runtime behavior for how the transform handles exceptions, errors, null values, and empty data input. This configuration can be done for each transform”. This sounds exciting and probably something we’ll be seeing in the next version of BizTalk Server.

      Finally the EDI or Business to Business Messaging, which now has support for:

      • Processing batched EDI messages. You can now use the TPM portal to configure agreements that can process batched messages coming from trading partners
      • Tracking and archiving EDI messages. You can now use the TPM portal to track messages as they are processed using the agreements

      I’m looking forward dive into the details! BizTalk Server in the cloud again one step closer!

      Didago IT Consultancy