> Archive > Issue LI: June 2011 > Empowering the Discipline of Cloud Integration - Part II
Pethuru Cheliah

Pethuru Cheliah


Dr. Pethuru Raj has been working as a TOGAF-certified enterprise architecture (EA) consultant in Wipro Technologies, Bangalore. On the educational front, armed with the competitive UGC research fellowship, he could proceed with his research activities and was awarded the prestigious PhD degree by Anna University, Chennai, India. He then could acquire the meritorious CSIR fellowship to work as a postdoctoral researcher in the Department of Computer Science and Automation (CSA), Indian Institute of Science (IISc), Bangalore. Thereafter, he was granted a couple of international research fellowships (JSPS and JST) to work as a research scientist for 3 years in two leading Japanese universities. Dr. Raj also had a fruitful stint as a lead architect in the corporate research (CR) division of Robert Bosch, India, for 1.5 years.

Dr. Raj has more than 12 years of IT industry experience. Primarily, he has been a technical architect and currently he is providing technology advisory services for worldwide business behemoths on the transformation capabilities of enterprise architecture (EA) in synchronization with some of the emerging technologies such as the Internet of Things (IoT) / Cyber Physical Systems (CPS) / Machine-to-Machine (M2M) Integration, Big Data, Cloud and Service Computing paradigms, Real-time Analytics of Big data using Cloud-based NoSQL databases, Hadoop framework, etc. and Mobility. He has made use of the opportunities that came on his way to focus on a few business domains, including telecommunication, retail, government, energy, and health care.

Dr. Raj has contributed book chapters for a number of technology books that were edited by internationally acclaimed professors and published by leading publishing houses. Currently he is writing a comprehensive book with the title "The Internet of Things (IoT) Technologies for the Envisioned Smarter Planet" for a world-leading book house. The CRC Press, USA has just released his book on "Cloud Enterprise Architecture" and you can find the book details in the page


rss  subscribe to this author


Empowering the Discipline of Cloud Integration – Part II

Published: June 17, 2011 • Service Technology Magazine Issue LI PDF

This is the second in a multi-part article series. The first part is published at Empowering the Discipline of Cloud Integration – Part I.

Describing a Service-Oriented Integration (SOI) Framework

Service-Oriented Architecture (SOA), the architectural style and scheme for the enterprise IT, is creating strategically significant opportunities for the worldwide IT industries as well as business organizations. SOA has impacted service providers, consulting companies and product vendors alike in planning and picking up new avenues for extra and sustainable revenues. Enterprise executives are equally optimistic about the success of the SOA paradigm towards transitioning theirs to on-demand and resilient in their operations, dealings, and deliveries to partners, employees, vendors, and customers. SOA guarantees business flexibility and continuity, whereas business optimization and transformation are being given a fresh start with the acceptance of SOA.

Service is the construction component for producing mission-critical and enterprise-scale service-oriented systems. Furthermore, services, not like other building-blocks, show glimpses of the appropriate application integration, modernization, and composition entity. Enterprise IT has benefited from SOA in streamlining and simplifying the hard-to-crack tasks such as legacy-enablement and enterprise integration. The advantage of SOA is that it guarantees realization of composite processes, data, services and applications.

SOA Integration Solution Framework

As mentioned before, integration has been a difficult project; however, SOA-based integration has recently emerged as the standard for successful IT integration. Attempts to solve the integration challenges using EAI techniques and other slow approaches have not produced tangible business results. SOA offers a strategic alternative for the integration problem. SOA helps to drive IT innovation, keeps things flexible while reducing risks and cutting costs. SOA transforms IT applications and data sources into highly flexible, valuable and reusable services. This shift helps in applications interacting with one another on a on need basis to propagate state changes across systems in real-time, to feed data to different data stores automatically, to aggregate distributed data into a single entity, etc. Redundancy disappears, consolidation becomes easier and utilization rate goes up, a synchronized and unified view of data is possible, distributed transactions would become smooth, control and data flows become dynamic, etc.

SOA integration capitalizes on the capabilities of service orientation to make better and more innovative use of IT assets by transforming them into accessible and consumable services. SOA integration provides a framework for enabling services, data and events besides connecting them to better align with business requirements. It supports key integration patterns that allow architects to collective, arrange, and mediate these services. This feature can accelerate IT responsiveness to unpredictable changes in business needs.

Companies are into leveraging common, standards-based and reusable service components to rapidly create integrated systems. Componentization is a big thing in software engineering and there is a compact segregation of all the common things and they are implemented as QoS-compliant, configurable, and sharable services. For example, for integration purposes, we need a growing array of generic services such as transformation, mediation, brokering, discovery, access, mapping, security, connectivity, and support. Each shared component may require specializations in tooling and runtime optimizations in order to be usable in most of the integration requirements. For example, the runtime optimization needed for managing a data query in a data service is different from the optimization necessary for managing a stateless message flow.

Considering the facts above, any standard SOA integration solution framework should comprise a multitude of integration-specific components for automating various activities such as process integration, service orchestration, data services, and a unified tooling. Each has to work together to provide the necessary integration capabilities in all kinds of situations.

A. Enterprise Service Bus (ESB)

An ESB is a brokering service that facilitates communication between services. For example, there is a service that only communicates via TCP on port 1000 and there is a client that can only make requests over HTTP due to a corporate firewall. An ESB can broker such communication. For an ESB to be successful, it must be flexible. It needs to support many different protocols, be able to communicate synchronously and asynchronously, manage security, translate messages between services, and be programming language-agnostic. Therefore, ESB is the central integration backbone fulfilling the various integration patterns. Enterprise service bus, being the essential and core infrastructure for application integration, ought to be versatile, adaptive, high-performing and assuring, highly available and scalable. In order to accomplish the much-acclaimed process integration, service orchestration capability has to be bestowed with ESBs. The importance of orchestration as a shared component gives ESBs the flexibility and the power towards the success demanded and desired.

B. Enterprise Data Bus (EDB)

This is the most common infrastructure and the preferred mechanism for the task of data integration. As discussed in the beginning of this chapter, any enterprise IT has to integrate a multitude of distributed and dissimilar data sources. EII-based information as a service paradigm has emerged as the viable option. Data services, being the well-adopted implementation method, are a collection of well-defined data-specific services such as data access, modeling, analysis, profiling, mapping, cleansing, transformation, and security. These shared components collectively find, link and synchronize a number of disparate data sources via the centralized EDB layer.

C. Connectivity Module

Artifact-based development has become the mainstream method. Having a repository of enterprise assets such as service components, utilities, widgets, process diagrams and descriptions, platform independent models (PIMs), design patterns, implementation methodologies, best practices, and evaluation metrics benefits your business. Maintaining validated and verified integration modules assist with arriving to solutions. There is a need for a service component that ensures an efficient, secure and reliable flow of data into and out of multiple sources for solving the integration needs effectively. This is the key driver for shared, plug & play, and reusable connectivity components, which work across integration solutions.

D. Complex Event Processing (CEP)

Business and technical events happen millions of times, almost every minute for large industries such as financial services companies. Events must be indicative, instructive and informative for enterprise IT to be anticipative, adaptive, and responsive. They have become an essential abstraction unit for business IT and appropriate infrastructure to be in place to make use of flowing and streaming events in a proactive fashion. The automated capture, consumption, and processing of events, besides generating actionable insights, have become mandatory for concerns to be competitively ahead especially in this economy. CEP engine is the event driven architecture (EDA)-compliant platform for events processing. Any SOA integration solution framework has to have the CEP as its vital module. Events unite decoupled services and applications through the 'publish & subscribe' feature.

E. Legacy Enablement Gateway

Legacy systems can be service-enabled by defining a legacy service gateway (in WSDL) between software systems such as databases, Java EE, CORBA, .NET, Mainframe (e.g.: CICS, IMS), and packaged applications, by providing SOAP applications that can receive SOAP messages and convert them into message-level or API-level invocations of the legacy systems. When a service consumer needs legacy functionality, it sends SOAP messages over HTTP to a legacy service gateway, which in turn, converts the SOAP messages into the corresponding native format expected by the legacy application. The legacy service gateway sends the native messages to the legacy application through a standard-based connector (.NET, JMS, JCA, ODBC, JDBC, etc). The legacy service gateway is also responsible for converting any responses from the legacy system into SOAP messages and routing them back to the service consumer. It is important to point out that service-enabling legacy functionality may also provide a clear way to phase out legacy systems in favor of newer custom-built implementations or package applications.

We have seen the most contributive and constructive components in any standard SOA integration solution framework. As new techniques and technologies appear, there will be new shared module to be constructed and attached with the framework. Industry leaders prescribe and describe a number of new-generation characteristics for next-generation SOA, which promises a number of commendable paradigm-shifts in software engineering. The first and foremost is the arrival and adoption of powerful composites. Composition happens at various levels: process, service, data, view etc. Composites can be dynamically found and loosely linked for formulating creative applications. Composites are more suitable for establishing on-demand and adaptive businesses. Furthermore, all kinds of unplanned changes occurring at different levels and layers (technology, process, requirements, platforms, context, people etc.) could be transparently accommodated within systems due to the inspiring ability of system-building composites.

Envisioning "IT as a Service"

ItaaS is a new concept. With the rise of service orientation, every single IT resource, activity and infrastructure is being viewed and visualized as a service that sets the tone for the unfolding of the dreamt service era. Systems are engineered as a collection of interacting services. Infrastructures are service-enabled to be active, exchanging and collaborative. The rejected delivery aspect has gone through several transformations and today the whole world has solidly settled for the paradigm 'IT as a service'. This is accentuated due to the pervasive and affordable Internet. We are bombarded with countless implementation technologies and methodologies. Clouds are the most visible and viable infrastructure for realizing ITaaS. Another influential factor is the maturity obtained in consumption-based metering and billing capability. HP announces this evolving trend as 'everything as a service'.

Software as a Service (SaaS)

SaaS is the first IT resource being delivered as a service is the growing array of software services and applications. Any desktop, utility, and business application modules, such as office suite, report generation, etc. and enterprise packages such as SCM, CRM, ERP, and KM, can be hosted in cloud infrastructures, delivered over the web and accessed via browsers by one or more users concurrently. It is described as a web-enablement of conventional applications. This would enable a single instance of any application through the much-acclaimed multi-tenancy characteristic to reach out to millions of users at global level on.

SaaS is a new software application hosting and delivering concepts in which a software vendor develops, hosts, and operates the applications which are used by its global customers, over a network (usually the Internet). Users do not need to pay anything for the software packages, and in this case, the total cost of ownership (TCO) is zero. Clients are instead required to pay for the duration or the amount of software usage just as people pay for their everyday utilities such as water, gas, electricity and phone services. A number of IT services and solutions are being taken to clouds in order to gain the benefits out of this paradigm. This is an example of the benefits for technical people, business users and executives derived out of this transition.

Data Quality on Demand

A data quality SaaS interface enables business users to submit their quality requirements via the web. For example, a business user downloads a customer file of customers who live in a designated metro area. He wants it cleansed, duplicates removed, and enriched, to know which of these customers have purchased or leased automobiles, and what model and year the automobiles are. He submits the request through his browser and sends the file to an external SaaS data quality provider. The provider then processes the data—including matching and de-duplicating customer records, cleansing and correcting address data, and augmenting with the requested data. The provider then returns the newly-cleansed customer list, which serves as the basis for a new campaign. This process involves no more than a point-and-click via an easy-to-use and web-based browser.

The SaaS model has gained good traction for a variety of reasons: faster deployment, lower risks, improved operational efficiencies, simpler upgrades, and effective dealing with IT resource constraints. SaaS has invaded the enterprise (small, medium and large) in a big way. According to McKinsey's report, the cost to a customer to develop, deploy and integrate a SaaS-based application costs only 20-30 cents on the dollar, a 50%-70% savings over on-premise platforms. It is no wonder then that 9 out of 10 companies surveyed for a recent Gartner report expect to increase their usage of SaaS applications and many plan to replace on-premise software with SaaS applications to reduce their TCO. IDC reports that 79% of enterprises are currently purchasing or evaluating SaaS solutions, and Gartner Group projects say that a third of all software revenues will eventually be derived from SaaS offerings.

Evolution of SaaS

Newer resources and activities are being consistently readied to be delivered as a service (Figure 1). Experts are in agreement that cloud is the best possible infrastructural s olution for effective service delivery. There are several ways clouds can be leveraged for diverse IT problems. The following list is a representation, newer means and methods via clouds will be considered for existing as well as emerging requirements in the future.

  • Storage as a Service is the ability to leverage storage servers or appliances that physically exist remotely, but logically as a local storage resource to any application that requires storage. Amazon simple storage service (S3) is the prominent on-demand storage.
  • Database as a Service provides the ability to leverage the unique services of a remote, multi-tenant and hosted database. Amazon DB is the well-known hosted and on-demand DB service.
  • Information as a Service refers to the ability of applications or users to consume any type of information retrieved and aggregated from one or more distributed data sources through a well-defined interface. Data federation technique through the EII technology is the driver for this sort of service.
  • Business Process as a Service (BPaaS) refers to designing, composing, integrating, validating and optimizing business processes using hosted BPM suites and BPEL designers and delivering as a service over the Internet. A business process typically binds many distributed resources (deployed in different clouds) together to achieve one or more business functionalities. Processes are more affable and open for modifications than the applications originated from the processes. There are hosted process engines for simulating processes for verifying their ability and limitations. Another possibility is that with the development of service orientation, which is fully process-centric, the value of formal business processes getting modeled, maintained and managed digitally has gone up significantly.

Figure 1 - The Evolution of SaaS
  • Platform as a Service (PaaS) is a very recent development. Integrated development environments (IDEs), rapid application development (RAD) toolkits, modeling and CASE products are increasingly hosted on clouds in order to facilitate architects, designers and developers to deliver complete IT solutions compactly and cheaply. Based upon the traditional timesharing model, modern platform providers provide the ability to create enterprise-class applications in clouds for a small subscription price or for free.
  • Infrastructure as a Service refers to the numerous platforms being taken to clouds. These platforms need underlying infrastructures to delegate their obligations perfectly. Besides the base operating system, other popular infrastructures include virtual machines, middleware, hubs, and buses.
  • Integration as a Service (IaaS) is the capability of clouds to fulfill the business integration requirements. Business applications are deployed in clouds to reap a variety of business and technical benefits. On the other hand, there are innumerable applications locally stationed and sustained. The question is how to create a seamless connectivity between hosted and on-premise applications to empower them to work together. IaaS overcomes these challenges by utilizing the time-tested business-to-business (B2B) integration technology as the bridge between the SaaS solution and in-house business applications. B2B systems are capable of driving this new on-demand integration model because they are traditionally employed to automate business processes between manufacturers and their trading partners. That means they provide application-to-application connectivity along with functionality that is crucial for linking internal and external software securely. Unlike pure EAI solutions designed only for internal data sharing, B2B platforms have the ability to encrypt files for safe passage across the public network, manage large data volumes, transfer batch files, convert disparate file formats, and guarantee data delivery. IaaS imitates this established communication model to create reliable and durable linkage for ensuring smooth data passage between hosted and installed applications.

    The IaaS model also leverages the adapter libraries developed by B2B vendors to provide rapid integration with various business systems. The use of Hub & Spoke (H&S) architecture further simplifies implementation and avoids placing an excessive processing burden on the customer side. The Hub is installed at the SaaS provider's data center to do the heavy lifting such as reformatting files. A Spoke unit at each user site typically acts as basic data transfer utilities. With these pieces in place, SaaS providers can offer integration services under the same subscription-based pricing model as their core offerings. A few integration service providers are listed below:

  • Blue Wolf Group - a system integrator with an IaaS offering
  • Boomi - a SaaS integration layer services company
  • Cast Iron Systems - a software based integration appliance (i.e. hardware)
  • Informatica - a data integration provider with a multi-tenant, cross-enterprise data integration on-demand service
  • Magic Software - a business and process integration provider
  • Pervasive - a data integration provider with an IaaS offering
  • Snap Logic - an open source data integration tool provider
  • Security as a Service is the ability to deliver core security solutions and services remotely over the Internet.

There are management platforms and governance solutions now residing in clouds. Testing for ensuring the quality of services and solutions being rendered is being accomplished via cloud infrastructures and platforms. Data Quality on Demand is a new cloud service being implemented. With the growth of cloud computing, enterprises may soon be having conversations about compliance as a service as they seek to deal with the legislative and compliance requirements around protecting customer and corporate data.

The point enforced here is not only IT resources but also IT activities are also being exported to clouds. In the days to come, almost all kinds of services and solutions will be delivered over the Internet.


Part III of this article series will cover Infrastructure as a Service and Integration as a Service in more detail.