This is the second in a multi-part article series. The first part is published at Empowering the Discipline of Cloud Integration – Part I.
Describing a Service-Oriented Integration (SOI) Framework
Service-Oriented Architecture (SOA), the architectural style and scheme for the enterprise IT, is creating strategically significant opportunities for the worldwide IT industries
as well as business organizations. SOA has impacted service providers, consulting companies and product vendors alike in planning and picking up new avenues for extra and sustainable
revenues. Enterprise executives are equally optimistic about the success of the SOA paradigm towards transitioning theirs to on-demand and resilient in their operations, dealings,
and deliveries to partners, employees, vendors, and customers. SOA guarantees business flexibility and continuity, whereas business optimization and transformation are being given
a fresh start with the acceptance of SOA.
Service is the construction component for producing mission-critical and enterprise-scale service-oriented systems. Furthermore, services, not like other building-blocks,
show glimpses of the appropriate application integration, modernization, and composition entity. Enterprise IT has benefited from SOA in streamlining and simplifying the hard-to-crack
tasks such as legacy-enablement and enterprise integration. The advantage of SOA is that it guarantees realization of composite processes, data, services and applications.
SOA Integration Solution Framework
As mentioned before, integration has been a difficult project; however, SOA-based integration has recently emerged as the standard for successful IT integration. Attempts to
solve the integration challenges using EAI techniques and other slow approaches have not produced tangible business results. SOA offers a strategic alternative for the integration
problem. SOA helps to drive IT innovation, keeps things flexible while reducing risks and cutting costs. SOA transforms IT applications and data sources into highly flexible,
valuable and reusable services. This shift helps in applications interacting with one another on a on need basis to propagate state changes across systems in real-time, to
feed data to different data stores automatically, to aggregate distributed data into a single entity, etc. Redundancy disappears, consolidation becomes easier and utilization
rate goes up, a synchronized and unified view of data is possible, distributed transactions would become smooth, control and data flows become dynamic, etc.
SOA integration capitalizes on the capabilities of service orientation to make better and more innovative use of IT assets by transforming them into accessible and consumable
services. SOA integration provides a framework for enabling services, data and events besides connecting them to better align with business requirements. It supports key integration
patterns that allow architects to collective, arrange, and mediate these services. This feature can accelerate IT responsiveness to unpredictable changes in business needs.
Companies are into leveraging common, standards-based and reusable service components to rapidly create integrated systems. Componentization is a big thing in software
engineering and there is a compact segregation of all the common things and they are implemented as QoS-compliant, configurable, and sharable services. For example, for
integration purposes, we need a growing array of generic services such as transformation, mediation, brokering, discovery, access, mapping, security, connectivity, and support.
Each shared component may require specializations in tooling and runtime optimizations in order to be usable in most of the integration requirements. For example, the runtime
optimization needed for managing a data query in a data service is different from the optimization necessary for managing a stateless message flow.
Considering the facts above, any standard SOA integration solution framework should comprise a multitude of integration-specific components for automating various activities
such as process integration, service orchestration, data services, and a unified tooling. Each has to work together to provide the necessary integration capabilities in all
kinds of situations.
A. Enterprise Service Bus (ESB)
An ESB is a brokering service that facilitates communication between services. For example, there is a service that only communicates via TCP on port 1000 and there is a
client that can only make requests over HTTP due to a corporate firewall. An ESB can broker such communication. For an ESB to be successful, it must be flexible. It needs
to support many different protocols, be able to communicate synchronously and asynchronously, manage security, translate messages between services, and be programming
language-agnostic. Therefore, ESB is the central integration backbone fulfilling the various integration patterns. Enterprise service bus, being the essential and core
infrastructure for application integration, ought to be versatile, adaptive, high-performing and assuring, highly available and scalable. In order to accomplish the
much-acclaimed process integration, service orchestration capability has to be bestowed with ESBs. The importance of orchestration as a shared component gives ESBs
the flexibility and the power towards the success demanded and desired.
B. Enterprise Data Bus (EDB)
This is the most common infrastructure and the preferred mechanism for the task of data integration. As discussed in the beginning of this chapter, any enterprise IT has
to integrate a multitude of distributed and dissimilar data sources. EII-based information as a service paradigm has emerged as the viable option. Data services, being the
well-adopted implementation method, are a collection of well-defined data-specific services such as data access, modeling, analysis, profiling, mapping, cleansing, transformation,
and security. These shared components collectively find, link and synchronize a number of disparate data sources via the centralized EDB layer.
C. Connectivity Module
Artifact-based development has become the mainstream method. Having a repository of enterprise assets such as service components, utilities, widgets, process diagrams and
descriptions, platform independent models (PIMs), design patterns, implementation methodologies, best practices, and evaluation metrics benefits your business. Maintaining
validated and verified integration modules assist with arriving to solutions. There is a need for a service component that ensures an efficient, secure and reliable flow of
data into and out of multiple sources for solving the integration needs effectively. This is the key driver for shared, plug & play, and reusable connectivity components,
which work across integration solutions.
D. Complex Event Processing (CEP)
Business and technical events happen millions of times, almost every minute for large industries such as financial services companies. Events must be indicative, instructive
and informative for enterprise IT to be anticipative, adaptive, and responsive. They have become an essential abstraction unit for business IT and appropriate infrastructure
to be in place to make use of flowing and streaming events in a proactive fashion. The automated capture, consumption, and processing of events, besides generating actionable
insights, have become mandatory for concerns to be competitively ahead especially in this economy. CEP engine is the event driven architecture (EDA)-compliant platform for
events processing. Any SOA integration solution framework has to have the CEP as its vital module. Events unite decoupled services and applications through the 'publish & subscribe'
E. Legacy Enablement Gateway
Legacy systems can be service-enabled by defining a legacy service gateway (in WSDL) between software systems such as databases, Java EE, CORBA, .NET, Mainframe (e.g.: CICS, IMS),
and packaged applications, by providing SOAP applications that can receive SOAP messages and convert them into message-level or API-level invocations of the legacy systems. When
a service consumer needs legacy functionality, it sends SOAP messages over HTTP to a legacy service gateway, which in turn, converts the SOAP messages into the corresponding
native format expected by the legacy application. The legacy service gateway sends the native messages to the legacy application through a standard-based connector (.NET, JMS,
JCA, ODBC, JDBC, etc). The legacy service gateway is also responsible for converting any responses from the legacy system into SOAP messages and routing them back to the service
consumer. It is important to point out that service-enabling legacy functionality may also provide a clear way to phase out legacy systems in favor of newer custom-built
implementations or package applications.
We have seen the most contributive and constructive components in any standard SOA integration solution framework. As new techniques and technologies appear, there will be new
shared module to be constructed and attached with the framework. Industry leaders prescribe and describe a number of new-generation characteristics for next-generation SOA,
which promises a number of commendable paradigm-shifts in software engineering. The first and foremost is the arrival and adoption of powerful composites. Composition happens
at various levels: process, service, data, view etc. Composites can be dynamically found and loosely linked for formulating creative applications. Composites are more suitable
for establishing on-demand and adaptive businesses. Furthermore, all kinds of unplanned changes occurring at different levels and layers (technology, process, requirements,
platforms, context, people etc.) could be transparently accommodated within systems due to the inspiring ability of system-building composites.
Envisioning "IT as a Service"
ItaaS is a new concept. With the rise of service orientation, every single IT resource, activity and infrastructure is being viewed and visualized as a service that sets the tone
for the unfolding of the dreamt service era. Systems are engineered as a collection of interacting services. Infrastructures are service-enabled to be active, exchanging and
collaborative. The rejected delivery aspect has gone through several transformations and today the whole world has solidly settled for the paradigm 'IT as a service'. This is
accentuated due to the pervasive and affordable Internet. We are bombarded with countless implementation technologies and methodologies. Clouds are the most visible and viable
infrastructure for realizing ITaaS. Another influential factor is the maturity obtained in consumption-based metering and billing capability. HP announces this evolving trend as
'everything as a service'.
Software as a Service (SaaS)
SaaS is the first IT resource being delivered as a service is the growing array of software services and applications. Any desktop, utility, and business application modules,
such as office suite, report generation, etc. and enterprise packages such as SCM, CRM, ERP, and KM, can be hosted in cloud infrastructures, delivered over the web and accessed
via browsers by one or more users concurrently. It is described as a web-enablement of conventional applications. This would enable a single instance of any application through
the much-acclaimed multi-tenancy characteristic to reach out to millions of users at global level on.
SaaS is a new software application hosting and delivering concepts in which a software vendor develops, hosts, and operates the applications which are used by its global customers,
over a network (usually the Internet). Users do not need to pay anything for the software packages, and in this case, the total cost of ownership (TCO) is zero. Clients are instead
required to pay for the duration or the amount of software usage just as people pay for their everyday utilities such as water, gas, electricity and phone services. A number of
IT services and solutions are being taken to clouds in order to gain the benefits out of this paradigm. This is an example of the benefits for technical people, business users
and executives derived out of this transition.
Data Quality on Demand
A data quality SaaS interface enables business users to submit their quality requirements via the web. For example, a business user downloads a customer file of customers who live
in a designated metro area. He wants it cleansed, duplicates removed, and enriched, to know which of these customers have purchased or leased automobiles, and what model and year
the automobiles are. He submits the request through his browser and sends the file to an external SaaS data quality provider. The provider then processes the data—including matching
and de-duplicating customer records, cleansing and correcting address data, and augmenting with the requested data. The provider then returns the newly-cleansed customer list, which
serves as the basis for a new campaign. This process involves no more than a point-and-click via an easy-to-use and web-based browser.
The SaaS model has gained good traction for a variety of reasons: faster deployment, lower risks, improved operational efficiencies, simpler upgrades, and effective dealing with IT
resource constraints. SaaS has invaded the enterprise (small, medium and large) in a big way. According to McKinsey's report, the cost to a customer to develop, deploy and integrate
a SaaS-based application costs only 20-30 cents on the dollar, a 50%-70% savings over on-premise platforms. It is no wonder then that 9 out of 10 companies surveyed for a recent
Gartner report expect to increase their usage of SaaS applications and many plan to replace on-premise software with SaaS applications to reduce their TCO. IDC reports that 79%
of enterprises are currently purchasing or evaluating SaaS solutions, and Gartner Group projects say that a third of all software revenues will eventually be derived from SaaS
Evolution of SaaS
Newer resources and activities are being consistently readied to be delivered as a service (Figure 1). Experts are in agreement that cloud is the best possible infrastructural s
olution for effective service delivery. There are several ways clouds can be leveraged for diverse IT problems. The following list is a representation, newer means and methods via
clouds will be considered for existing as well as emerging requirements in the future.
- Storage as a Service is the ability to leverage storage servers or appliances that physically exist remotely, but logically as a local storage resource to any application
that requires storage. Amazon simple storage service (S3) is the prominent on-demand storage.
- Database as a Service provides the ability to leverage the unique services of a remote, multi-tenant and hosted database. Amazon DB is the well-known hosted and on-demand DB service.
- Information as a Service refers to the ability of applications or users to consume any type of information retrieved and aggregated from one or more distributed data
sources through a well-defined interface. Data federation technique through the EII technology is the driver for this sort of service.
- Business Process as a Service (BPaaS) refers to designing, composing, integrating, validating and optimizing business processes using hosted BPM suites and BPEL
designers and delivering as a service over the Internet. A business process typically binds many distributed resources (deployed in different clouds) together to achieve
one or more business functionalities. Processes are more affable and open for modifications than the applications originated from the processes. There are hosted process
engines for simulating processes for verifying their ability and limitations. Another possibility is that with the development of service orientation, which is fully
process-centric, the value of formal business processes getting modeled, maintained and managed digitally has gone up significantly.
Figure 1 - The Evolution of SaaS
- Platform as a Service (PaaS) is a very recent development. Integrated development environments (IDEs), rapid application development (RAD) toolkits, modeling and CASE
products are increasingly hosted on clouds in order to facilitate architects, designers and developers to deliver complete IT solutions compactly and cheaply. Based upon
the traditional timesharing model, modern platform providers provide the ability to create enterprise-class applications in clouds for a small subscription price or for free.
- Infrastructure as a Service refers to the numerous platforms being taken to clouds. These platforms need underlying infrastructures to delegate their obligations perfectly.
Besides the base operating system, other popular infrastructures include virtual machines, middleware, hubs, and buses.
- Integration as a Service (IaaS) is the capability of clouds to fulfill the business integration requirements. Business applications are deployed in clouds to reap a
variety of business and technical benefits. On the other hand, there are innumerable applications locally stationed and sustained. The question is how to create a seamless
connectivity between hosted and on-premise applications to empower them to work together. IaaS overcomes these challenges by utilizing the time-tested business-to-business (B2B)
integration technology as the bridge between the SaaS solution and in-house business applications. B2B systems are capable of driving this new on-demand integration model
because they are traditionally employed to automate business processes between manufacturers and their trading partners. That means they provide application-to-application
connectivity along with functionality that is crucial for linking internal and external software securely. Unlike pure EAI solutions designed only for internal data sharing,
B2B platforms have the ability to encrypt files for safe passage across the public network, manage large data volumes, transfer batch files, convert disparate file formats,
and guarantee data delivery. IaaS imitates this established communication model to create reliable and durable linkage for ensuring smooth data passage between hosted and
The IaaS model also leverages the adapter libraries developed by B2B vendors to provide rapid integration with various business systems. The use of Hub & Spoke (H&S)
architecture further simplifies implementation and avoids placing an excessive processing burden on the customer side. The Hub is installed at the SaaS provider's data
center to do the heavy lifting such as reformatting files. A Spoke unit at each user site typically acts as basic data transfer utilities. With these pieces in place,
SaaS providers can offer integration services under the same subscription-based pricing model as their core offerings. A few integration service providers are
- Blue Wolf Group - a system integrator with an IaaS offering
- Boomi - a SaaS integration layer services company
- Cast Iron Systems - a software based integration appliance (i.e. hardware)
- Informatica - a data integration provider with a multi-tenant, cross-enterprise data integration on-demand service
- Magic Software - a business and process integration provider
- Pervasive - a data integration provider with an IaaS offering
- Snap Logic - an open source data integration tool provider
- Security as a Service is the ability to deliver core security solutions and services remotely over the Internet.
There are management platforms and governance solutions now residing in clouds. Testing for ensuring the quality of services and solutions being rendered is being accomplished via
cloud infrastructures and platforms. Data Quality on Demand is a new cloud service being implemented. With the growth of cloud computing, enterprises may soon be having conversations
about compliance as a service as they seek to deal with the legislative and compliance requirements around protecting customer and corporate data.
The point enforced here is not only IT resources but also IT activities are also being exported to clouds. In the days to come, almost all kinds of services and solutions will be
delivered over the Internet.
Part III of this article series will cover Infrastructure as a Service and Integration as a Service in more detail.