Abstract: This article is geared to help CIOs, IT managers and architects understand: when to use BizTalk Server, SQL Server, Windows Server AppFabric; when to use Azure Service Bus; when to use hybrid integration architecture; the Federated Enterprise Service Bus (ESB); integration of PaaS and SaaS applications; and how to design future proof integration patterns.
Integration is key in a best-of-breed application landscape, B2B (business-to-business) scenarios or service-oriented architecture (SOA). Applications, functions and services need to exchange data in order to participate in business processes. In the early days, integration was done by creating peer-to-peer connections between applications using direct database interaction, exchanging proprietary import/export files or executing API (application programming interface) calls. Over time, this resulted in unmanageable and badly performing applications due to the well-known “spaghetti integration” dilemma.
Figure 1 – Spaghetti Integration Dilemma.
With the arrival of the enterprise application integration (EAI) server, integration got a more sophisticated approach. However, very often it still results in peer-to-peer connection spaghetti, but now with a broken (hub) in the middle.
Figure 2 – Hub and Spoke Integration.
Especially with the acceptance of SOA, the ESB integration pattern was born: loosely coupled integration through an Enterprise Service Bus, preferably based on the principles of service design by Thomas Erl [REF-1], thereby creating a manageable and flexible abstraction layer. But, the risk of creating unmanageable and non-performing integration architectures is still there; it is said that the greatest competitor of integration middleware is the developer.
Figure 3 – Enterprise Service Bus.
Traditionally, integration middleware (the common name for the group of products facilitating ESB, EAI, B2B) is hosted and operated by the company that also hosts the back-end applications that are used to support the company business processes. For some scenarios, mainly B2B, external providers are sometimes used to integrate the third parties outside the company firewalls. This is where you currently see VAN (Value Added Network) providers morph into cloud-based B2B integration service providers.
Depending on the integration scenarios, patterns have evolved over the years to become standard integration patterns. A great resource that still applies is Enterprise Integration Patterns [REF-2], which contains all possible integration patterns that are integration middleware agnostic.
The paradigm shift to cloud computing, brought by Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS), will be faced by many companies bringing in new challenges, such as:
- How do I integrate my back-end systems and services with multi-tenant SaaS applications?
- How do I create composite services that are orchestrated services provided by multiple SaaS applications or even SaaS vendors
- Should I use integration middleware in the cloud (PaaS) to integrate my on-premise applications?
- What does hybrid integration architecture look like and how to I manage such an environment?
With the move to mass-market, multi-tenant SaaS solutions, integration will become even more important than it is today because now companies can start to deliver their value add to customers not only by building tailor-made solutions anymore, but by leveraging smart combinations of cloud based offerings.
This article will provide high level guidance on when to choose specific Microsoft integration solutions, how to combine them and what integration architecture should be applied to implement the new scenarios around PaaS and SaaS so you can create the appropriate future-proof architecture today.
On-Premise Integration Middleware (BizTalk, SQL Server, Server AppFabric)
Integration software that is hosted on a company’s own servers or on servers managed by a dedicated hosting provider are called on-premise integration middleware. Microsoft mainly provides three products in that category, each used for different integration needs: BizTalk Server, SQL Server and Windows Server AppFabric.
Microsoft BizTalk Server is a very mature integration middleware product that has been around since 2000. Today, with the 2010 release, it provides a robust platform to implement an EAI or B2B integration layer and it is also used very often as an ESB at the core of service-oriented architectures. The product contains facilities for transformation, routing, business process orchestration, business activity monitoring (BAM), and business rules; all of the ingredients to create a scalable integration layer are there.
More than 12.000 enterprises world-wide use the product today, among which there are very large implementations handling several million transactions a day. BizTalk is also used quite often as the middleware layer between SharePoint front-ends and multiple back-end systems and it functions to provide service composition, transactions, scalability, and monitor abilities in Enterprise Portal scenarios.
Although there have been rumors saying otherwise, it is important to note that BizTalk Server has a future roadmap, the next release (called BizTalk Server 2010 R2) will be available 6 months after the release of Windows 8 Server. The reason why the release date is this “fuzzy” is that one of the main development topics for any new BizTalk (or for that matter, SharePoint, Dynamics, etc.) release is “platform alignment”, meaning compatibility with and leveraging the new features of the latest versions of core Microsoft platform products such as Windows, SQL Server and Visual Studio must be done.
SQL Server, more specifically SQL Server Integration Services (SSIS) is often used for data integration services, such as ETL (Extract, Transform and Load). For batch-oriented data level integration this is perfectly suited.
Apart from providing data integration services, SQL Server is also a very important piece of BizTalk Server and SharePoint:
- For BizTalk it is the main persistence engine for publish & subscribe, guaranteed delivery and long running business processes paradigms
- For SharePoint it is the main persistence engine for storing documents and workflow states
By means of SQL Server Analysis Services (SSAS) and SQL Server Reporting Services (SSRS), in combination with the BAM facilities in BizTalk and tooling such as Excel and PowerPivot surfaced in SharePoint, it provides a very powerful operational BI stack that can help improve business processes. The latest version of SQL Server is 2012, released in March.
Windows Server AppFabric
For light-weight on-premise integration it is possible to use Windows Server AppFabric together with WF (Windows Workflow) and WCF (Windows Communication Foundation), which are basically all standard components in the .NET framework. In combination with the BizTalk LOB Adapter Pack, it is possible to integrate back-end systems without installing and operating BizTalk Server. Of course this is not on-premise integration middleware architecture as sophisticated in terms of monitor ability and scalability like BizTalk Server, but for scenarios where there are no requirements for such Quality of Service attributes (i.e. non-functional requirements), it suffices. With the introduction of Windows Server AppFabric, Microsoft has already started the trend of embedding integration middleware in the Windows platform.
It is important to note that all of the BizTalk Server, SQL Server and Windows Server AppFabric (including WCF and WF) integration solutions are built using Visual Studio based tooling, a familiar environment for developers in the Microsoft world. By introducing Windows Server AppFabric, Microsoft has already made a little bit of a start with providing integration services as part of the operating system (on-premise only in this version).
Cloud Integration Middleware (Azure Service Bus)
Integration software that is provided as a (multi-tenant) service in the cloud, or running in a single-tenant fashion dedicated to one company, is called cloud integration middleware. Microsoft mainly provides two solutions for integration in the cloud: Windows Azure Service Bus and your own virtual machines (most probably with BizTalk Server installed) running in the cloud on the IaaS platform (Infrastructure-as-a-Service, available in H2 2012).
Windows Azure Service Bus
With the arrival of Microsoft’s cloud platform, Windows Azure, it has also become possible to build and deploy cloud integration solutions. Windows Azure is a PaaS offering and “PaaS is the architectural center of the cloud” [REF-3]. Windows Azure is mainly an application platform in the cloud that can be used to host tailor-made applications at cloud scale. This means that companies using this technology will build their own solutions to be deployed and run in the cloud. The company is not only responsible for architecture, design and development of the applications, but also for functional and technical maintenance and health.
Providing a platform in the cloud also means that it needs to incorporate integration capabilities. Enter Azure Service Bus (previously named Windows Azure AppFabric Service Bus). This Integration-as-a-Service offering (or “Cloud service integration and composition” [REF-3]) has become an integral part of Windows Azure, making it possible to host and manage integration services. Currently, in the customer-ready release it is possible to deploy relatively simple integrations only, using WCF-based communication endpoints, messaging (queuing) and routing mechanisms. Service Bus can therefore be used as an abstraction layer between (cloud) applications and services, providing REST, SOAP and WS-* connection capabilities.
Since the December 2011 CTP (customer technology preview) release of “Service Bus EAI and EDI labs” it is also possible (in a lab environment only, until official release) to leverage functionalities for processing B2B-specific scenarios such as EDI (Electronic Data Interchange). Features include: Transform (for creating transformations between XML instances), TPM (Trading Partner Management) and bridges for specific communication and exchange protocols such as AS2 (secure exchange of B2B transactions over HTTP), FTP and X12 (EDIFACT support will follow in a later release).
This clearly means that within a couple of releases from now (and the release cycles in the cloud realm are much shorter than for on-premise software; typically 3-6 months), Azure will provide sufficient integration capabilities (including workflow for process orchestration and composition) to cater to production-ready, full-featured integration solutions for cloud-cloud integration and cloud-on-premise integration. All of this will be set up as part of the platform as it should be.
Hybrid Integration Solutions (Mixing On-Premise and Cloud Architecture)
Many companies will keep on hosting parts of the application infrastructure in-house (or with a dedicated hosting provider) and some parts of it in the cloud. Cloud applications (multi-tenant SaaS or homegrown and deployed, single-tenant in PaaS) will be used for their (elastic) scalability, time-to-market and ease of management features. And let’s not forget the very important advantage of having Opex (operational expense) instead of Capex (capital expense) because you don’t own the software; it’s pay-per-use. The on-premise applications will either be legacy applications that cannot be ported to the cloud because of their (lack of) architecture or applications that have very high security or privacy requirements. This means that integration between on-premise and in-the-cloud applications will most of the times be necessary. Enter Hybrid integration solutions.
Integration of on-premise applications and services with cloud applications and services can leverage the facilities provided by BizTalk Server or Windows Server AppFabric handling the on-premise integration requirements and Windows Azure Service Bus handling the cloud integrations. A bridge between the two domains is provided by Windows Azure Connect, using the relay services in the cloud and ACS (Access Control Service) for the security between the two worlds. By using Azure Connect from within the on-premise integration layer, the on-premise layer creates a secure “gateway” with Azure Service Bus without undermining the DMZ (demilitarized zone) or other security rules.
Probably the most important integration pattern for hybrid integration solutions is the Federated ESB pattern, where two (or more) ESBs are deployed – one on-premise and one in the cloud – thereby serving the two domains with the same approach. These ESBs are connected by either implementing a brokered ESB architecture (where one “master” ESB determines where to route requests, thereby offering bridge services) or a connected ESB architecture (each ESB having its own service registry, visible to all ESBs, thereby reflecting each service’s affinity for one of the ESBs). Either way, each ESB is responsible for connectivity in its own domain and routes service requests to other ESBs if these services are not hosted in their own domains. IBM has said the following about these patterns: “Service specification and implementation are governed at the domain level, with a degree of centralized governance to enable cross-domain service reuse and interoperability” [REF-5].
Figure 4 – Federated ESB. Click here for larger image.
What Will the Future of Integration with Microsoft Look Like?
Microsoft is dedicated to merging the integration features provided by BizTalk Server, Windows Server AppFabric, WCF, WF, and Windows Azure Service Bus into an architecture that has the following key capabilities:
- One workflow engine (instead of two: X/LANG in BizTalk and WF in .Net)
- One toolset for development, deployment and management of integration solutions
- One hosting “container”, available in public cloud, private cloud and on-premise
- Providing for low latency and high throughput / volume solutions
- Alignment of Service Bus and ESB across platforms
- Provide BAM and Business Rule Engine (BRE) across platforms
This will be a multi-year development effort, and in the meantime the cloud-based integration middleware will further evolve to incorporate new features and functionalities that are most often based on technology or capabilities already available in the BizTalk Server or Windows Server AppFabric platform. These features will be made available as reusable .NET components that can be mixed and matched to create multi-tenant integration solutions for customers. The toolbox in the Visual Studio development environment will be extended with integration specific capabilities, such as: Transform, Route, Messaging, Identity & Access, and Cache. The runtime container that will host the integration solutions built with these components will provide built-in features for multi-tenancy, scalability, resilience, and management & monitoring. It will not matter if the solutions will be deployed in the public cloud, a private cloud or on-premise; the tools are the same.
By doing this, Microsoft will provide an attractive integration middleware platform that appeals to integration architects, specialists and the .NET developer community.
Future Proof Integration Architecture Guidance
Gartner has said the following related to this topic: “Users that delay the adoption of PaaS to sometime in the future, when the standards, leading providers and best practices are better established, should invest now in building expertise in service-oriented architecture (SOA), including its event-driven form. SOA is a bridge from the traditional computing in the enterprise data center to the hybrid model of computing, engaging enterprise data center and cloud resources” [REF-3].
SOA is the architecture paradigm that makes integration of disparate applications, functions and services possible by adhering to the following design principles (see [REF-6]):
- Reusability – Services contain and express agnostic logic and can be positioned as reusable enterprise resources (think about the future uses and make them generic)
- Autonomy – Services exercise a high level of control over their underlying runtime execution environment (there are no other services making use of the underlying components)
- Discoverability – Services are supplemented with communicative meta data by which they can be effectively discovered and interpreted (if nobody knows about the existence of services, they will never be re-used)
- Composability – Services are effective composition participants regardless of the size and complexity of the composition (break-up larger components into smaller, more manageable and composed ones)
- Loose coupled – Service contracts impose low consumer coupling requirements and are themselves decoupled from their surrounding environment (contract-first development of services; don’t generate contracts)
- Abstractness – Service contracts only contain essential information and information about services is limited to what is published in service contracts (the contract does not contain system level information on underlying systems)
- Standardized Service Contract – Services within the same service inventory are in compliance with the same contract design standards (by using a uniform language and applying standards notations for creating contracts, discoverability, reliability and application of governance of services will improve)
- Statelessness – Services minimize resource consumption by deferring the management of state information when necessary (keeping excessive state can undermine the availability and scalability of services)
When designing and developing new software, make sure you do it in a service-oriented way and comply with the above principles whenever feasible. This will help build solutions that are highly usable, composable and portable.
Integration middleware can also be used to wrap functions in legacy applications in such a way that they can adhere (as good as possible) to the above principles and thereby become candidates to publish their “services” on the on-premise ESB for consumption by composition services or other consumers, on-premise or through a federated ESB in the cloud. This way, investments in legacy keep on delivering value.
Furthermore, by applying standard integration patterns, as described in Enterprise Integration Patterns (see Introduction), the chance that a manageable and, even more important, portable integration architecture will be created increases significantly. It is also vital to implement as little custom (coded) components as possible in order to make compatible solutions for future versions of integration middleware, such as BizTalk Server or Azure Service Bus.
Last but not least, Microsoft will make migration of artifacts between intermediate integration solution products and versions possible. Schemas used currently in BizTalk Server can most of the times be migrated to Azure Service Bus without conversion. And for transformations, a migration tool for converting BizTalk transformation maps to the Azure- based Transforms has been made available earlier this year.
The most important rule that applies though, remains: make sure your integration solutions indeed have an architecture based on standard patterns and best practices; this will make the transition to a federated ESB much easier.
Integration architecture is a complex topic. With the arrival of the cloud computing era, companies today are faced with even more difficult architecture choices than ever before. Even in a “Microsoft unless” environment, there are still many questions to ask, such as: how to integrate SaaS with on-premise applications, should I use cloud integration middleware to provide connectivity between my on-premise applications and functions, should I still invest in creating on-premise integration solutions, is service oriented architecture the right path to follow, will an enterprise service bus help me address all requirements?
This article provides you (CIO, IT Manager or Architect) the most up-to-date facts on the availability and roadmap of Microsoft integration products and it provides you with high level guidance on how to design future proof integration solutions with Microsoft technology. Hopefully it has helped to organize your thoughts on integration architecture in today’s “confusing” application landscapes and will prove useful now and in the future.
[REF-1] Erl, Thomas, “SOA Principles of Service Design,” Prentice Hall, 2007. www.soabooks.com/psd
[REF-2] Hohpe, Gregor and Woolf, Bobby, “Enterprise Integration Patterns,” Addison-Wesley Professional, 2003.
[REF-3] “Hype Cycle for Cloud Computing 2011,” Gartner, 2011. www.gartner.com/id=1753115
[REF-4] “Introducing new Messaging features in Windows Azure AppFabric Service Bus,” Microsoft, 2011. download.microsoft.com/download/7/1/1/71170C7C-6217-469F-8071-E8B04866E8A3/ServiceBusCTPUserGuide.docx
[REF-5] “Choose an ESB topology to fit your business model,” IBM. www.ibm.com/developerworks/webservices/library/ws-soa-esbtop/
[REF-6] Erl, Thomas, “SOA Principles: An Introduction to the Service-Orientation Paradigm,” www.soaprinciples.com