> Issue XLVIII: March 2011 > DEmpowering the Discipline of Cloud Integration - Part I
Pethuru Cheliah

Pethuru Cheliah


Dr. Pethuru Raj has been working as a TOGAF-certified enterprise architecture (EA) consultant in Wipro Technologies, Bangalore. On the educational front, armed with the competitive UGC research fellowship, he could proceed with his research activities and was awarded the prestigious PhD degree by Anna University, Chennai, India. He then could acquire the meritorious CSIR fellowship to work as a postdoctoral researcher in the Department of Computer Science and Automation (CSA), Indian Institute of Science (IISc), Bangalore. Thereafter, he was granted a couple of international research fellowships (JSPS and JST) to work as a research scientist for 3 years in two leading Japanese universities. Dr. Raj also had a fruitful stint as a lead architect in the corporate research (CR) division of Robert Bosch, India, for 1.5 years.

Dr. Raj has more than 12 years of IT industry experience. Primarily, he has been a technical architect and currently he is providing technology advisory services for worldwide business behemoths on the transformation capabilities of enterprise architecture (EA) in synchronization with some of the emerging technologies such as the Internet of Things (IoT) / Cyber Physical Systems (CPS) / Machine-to-Machine (M2M) Integration, Big Data, Cloud and Service Computing paradigms, Real-time Analytics of Big data using Cloud-based NoSQL databases, Hadoop framework, etc. and Mobility. He has made use of the opportunities that came on his way to focus on a few business domains, including telecommunication, retail, government, energy, and health care.

Dr. Raj has contributed book chapters for a number of technology books that were edited by internationally acclaimed professors and published by leading publishing houses. Currently he is writing a comprehensive book with the title "The Internet of Things (IoT) Technologies for the Envisioned Smarter Planet" for a world-leading book house. The CRC Press, USA has just released his book on "Cloud Enterprise Architecture" and you can find the book details in the page


rss  subscribe to this author


Empowering the Discipline of Cloud Integration – Part I

Published: March 09, 2011 • SOA Magazine Issue XLVIII PDF


As the web is becoming more pervasive and persuasive, there is a rush to position customer-facing business applications on remote and reliable servers. This allows it to be accessed by hundreds of thousands of users simultaneously at global scale. Another transformative trend is the huge and unprecedented success of software as a service (SaaS) model. This model generates fresh ideas and innovations for the struggling software engineering community. SaaS paradigm allows companies to meet their customers' expectations without investing on additional IT infrastructures and their maintenance. The much-hoped SaaS model, the new-generation application delivery approach, enables companies to do a kind of twisting and tinkering on their existing applications to embrace the ubiquitous SaaS model. The prevailing trend is that organizations with one or more SaaS applications running usually have numerous enterprise applications, packages and data sources in their back-end systems on their premises. In a nutshell, besides establishing a larger number of on-demand, hosted, and online systems in clouds, the elastic infrastructures develops, deploys, manages, and maintains SaaS applications and services. This allows worldwide firms maintain a large number of strategic software assets and store core information systems in their own onsite infrastructures. As a result, the key to realizing and revitalizing the full potential of SaaS applications is through creating a safe and seamless route for the SaaS systems to talk with the company's back-end systems for data and service mashups creation to realize integrated views. The main point being, business processes now span to clouds for customer applications and data, linking middleware, and more.

Traditional software vendors and IT departments frequently attempt to position integration as the Achilles heel of the evolving SaaS model by arguing that SaaS solutions lack enterprise application integration (EAI) capabilities. SaaS providers, having understood the simmering challenges, are now responding with new approaches and product suites that enable enterprises' legacy applications and packages to be integrated with hosted software containing the same delivery and cost benefits of the SaaS model itself. This emerging service model is referred to as 'Integration as a Service (IaaS)' that treats integration as a fully hosted service, eliminating the complexity, lengthy deployment cycles and upfront expense of custom integration. IaaS providers come out with a multitude of generic as well as specific adapters, connectors, engines, and drivers for smoothly integrating hosted and onsite systems. IaaS style also offers a flexible, pay-as-you-go and subscription-based pricing model that scales back and forth based on the level and complexity of integration required. Cloud infrastructures are being strengthened as the competent, cost-effective and elastic containers for IaaS products. With the convergence of SaaS and cloud computing concepts, integration as a service (IaaS) paradigm is growing convincingly and comprehensively in the forthcoming service and cloud era. This article is all about the optimized processes, participating services, middleware, enabling clouds and their beneficial partnerships for simplifying and streamlining the well-intended integration goals.

Delving on the Aspect of Enterprise Integration

IT, being the prominent and dominant business enabler, is being embraced in a big way across industries. Not only is this for automating business processes but also for insightfully refining business processes through a host of optimizations and improvisations. In spite of all the noteworthy advancements, challenges refuse to go away. Due to the multiplicity, heterogeneity, and proprietary nature, today's enterprise IT is forced to live with application and data islands. On the other hand, due to the brisk pace in globalization, decentralization, distribution and digitalization activities, businesses are expanding fast and opening up branches in different places across the world to reach out more people at global level. The other dimension is that businesses are increasingly interacting with their trade partners, suppliers, and retailers outside to fulfill soaring expectations of clients, customers, and consumers. The fallout is that burgeoning software applications need to interact within and outside too. In other words, it is intra-enterprise as well as inter-enterprise integration. However, to make applications link, talk and synchronize with one another in real-time dynamically has now become the question. Stalwarts and visionaries have carved out a range of paradigms and principles for achieving enterprise integration through point-to-point (P2P) and hub and spoke (H&S) architectures.

The Importance of Integration

Technically disintegration brings in innumerable limitations and inflexibilities for the entire IT ecosystem of any company. Besides redundancy, software modules are neither interoperable nor reusable, thereby are incapable of proactively finding, accessing, or binding in order collaboratively function, which is the most ideal and likable thing for any enterprise IT. The real-time propagation and presentation of state changes across the enterprise systems cannot be accomplished. Therefore the much-wanted composition of distributed and disparate data, services, applications and processes could not be realized. Infrastructure teaming, resource sharing and dynamic composition of dissimilar data do not exist and evolve in any disintegrated ecosystem. Besides technical inhibitions, businesses are too severely constrained by the IT fragmentation. Businesses ought to be agile and adaptive, and these propositions can be met only in seamlessly integrated environments. In short, integration enforces a staggering variety of innovations at various levels to produce creative processes and delivery, pricing, and interaction models. A unified and real-time view of enterprise data is the major attraction of integration. There are integration policies, patterns, platforms, procedures, prototypes, practices, and processes abounding. In the ensuing sections, we are to discuss about integration technologies, infrastructures and methodologies in detail.

SOA for the Rescue

In the recent past, the service orientation concept has come out powerfully to extend a helping hand to the struggling integration community. SOA has proven that enterprise integration, a very complicated and time-consuming assignment, could be easily analyzed and accomplished without much hassle. Service oriented integration (SOI) is the new jargon gaining tremendous acceptance and appreciation from integration experts. The development community is overwhelmingly leaning towards the methods of asset / artifact-based system construction, service-oriented modeling, development, composition, and governance, causing them to strive. Service is the fundamental building-block for SOA, which is constantly emitting newer paradigms and propositions. SaaS is one incredible derivative of the buzzing service-orientation. Every single IT resource is getting deployed in clusters, fabrics, grids and clouds to be exposed and presented as a service that can be dynamically located, matched and leveraged over the cheap and common Internet infrastructure by global users concurrently. Another inspiring trend is that every software engineering and integration activity could be performed in the cloud infrastructure. So activity done so far that has been successfully deployed in on-premise infrastructures can be gradually moved to be hosted on remote systems. This transition in turn, will bring in a lot of technical as well as economic benefits to enterprises.

The lingering issue is how to enable SaaS applications to talk to each other as well as to create a seamless route for SaaS applications to interact with onsite corporate applications and data sources. All along professionals have been pondering and postulating for a simple way or mean of achieving integrated systems from within. Now with the faster emergence of SaaS applications being deployed in scalable servers and delivered over the supple web, the integration landscape has dramatically become big and challenging. Product vendors are tirelessly working towards an array of methods and mechanisms to create a flawless connectivity among traditional and SaaS applications. The overwhelming view is to take the integration pieces to the cloud itself to reap the unrivaled SaaS advantages. This is the solid foundation for coining new jargon 'Integration as a Service (IaaS)' [REF-11, 12].

In this article, we are to discuss how the smart combination of the service and cloud computing paradigms fulfills the grand objectives behind IaaS. Further on, there is a write-up about the economic benefits of depositing integration hubs, message queues, and service buses in clouds. The ultimate aim is to come out with a lean and flexible integration methodology in order to simplify the overly complex integration tasks. Precisely speaking, integration will be an on-demand utility so that users can pay only for the integration services rendered or realized.

Digging into Enterprise Integration Types and Methods

Integration has been an acute and astute challenge for both business as well as IT communities. Technocrats have devised an assortment of integration-simplification techniques and tools, but due to rising business complexity, nothing seems to be working for the lively and money-spinning integration space. As generally accepted and adopted, integration is to make dissimilar and distributed applications connect and work together. And interface is the leading technique for integration of applications that were never intended to work together.

Oracle sometimes argued that interfaces are expensive and there are innumerable problems with the rising complexity and on managing the scope and the integrity across multiple data sources. Especially for point-to-point (P2P) interfaces, data must be constantly cleansed and harmonized across multiple data sources. Any changes to one system can lead to costly re-testing or even re-design and coding of the interfaces. Therefore an early type of integration approach widely advertised by Oracle is to have all relevant data getting aligned with a single data model and stored only once in a single place. The implication is that we could place all of our data for a set of business processes inside the Oracle E-Business Suite (OBS) and this significantly reduces the total cost of ownership (TCO). This much-acclaimed proposition could not hold water (due to some well-known reasons). Although the Oracle way of integration is not completely dead, enterprises mostly focus on integration through one or other forms of interfacing.

The prominent types of integration through interfacing are

  • User Interface (UI) Integration
  • Process Integration
  • Application Integration
  • Data / Information Integration
  • Business-to-business (B2B) Integration

Presenting integrated and dynamic view for users has been an ongoing activity and is the key goal of UI integration. In the recent past, the most enigmatic mashup technology has revolutionized the UI integration discipline. Composition is the mainstream technique and data mashups is actually creating and supplying composite and real-time data dynamically of distributed and disparate data. Portlet is another worthwhile technology in this space and assists immensely in producing portal applications that are unbelievably customizable. Other integration types such as process, application, service and data integrations ultimately contribute in realizing integrated delivery and display of data.

Process Integration

Process steps within a process generally are tied up with different applications, services and data stores. This integration is termed as intra-process integration. Within an organization, there would be several processes. Integrating them is called as inter-process integration. These require complex business logic, involve extensive human-machine interaction, and need native connectivity to data sources and legacy applications. For example, a credit approval process involves multiple steps of approval and human interaction. It requires complicated reasoning to determine the creditworthiness of a customer and needs information from multiple systems to determine an answer. Data and service integrations lead to process integration, which in turn fulfills the compact automation of end-to-end processes.

Data Integration [REF-11]

The top contributor for the goal of business integration has been an increasingly strategic endeavor. Precisely speaking, business integration ultimately leads to responsive, cost-effective, and smart enterprises. As corporate data volume surges, its value too subsequently shoots up if it gets leveraged proactively and positively through time-tested processes and tools. Actionable insights are being derived out of the data heap of every company who empowers them to go for extra miles to entice and enable customers, partners and decision-makers. Information is power, a strategic asset and above all, a great value-addition for organizations to think ahead and to execute well. Due to its perpetual significance for business transformation, several vendors came forward enthusiastically to unearth new-generation data integration technologies and platforms.

With the deeper pervasiveness of messaging middleware, many services and solutions are seeing the reality these days. This includes service oriented architecture (SOA)-aware enterprise, service bus and enterprise data bus (EDB), complex event processing (CEP) engines, enterprise information integration (EII), enterprise application integration (EAI), composite data services, and mashup runtimes including 'near real-time'/ 'real-time' data integration. The term 'near real-time' is used to describe target data that has a low latency of a few minutes or may be a few hours. Data with zero latency is known as real-time data. Real-time sensing, processing, notification and actuation are increasingly desired by end-users. IT professionals are sincerely plotting the ways and means of realizing real-time IT technologies and infrastructures. Batch data integration is the prominent one at this point of time and in future. According to industry visionaries, there is a swelling popularity for real-time data integration.

SOA-aware Data Services

As we all know, SOA caused a major paradigm shift for enterprises. Data integration is getting tightly embedded within the SOA style. As a result, data services will likely follow the mainstream adoption. It is believed that data services will solidify the data abstraction layer and will evolve into multiple types: enterprise search, reporting, and the single view of the truth. Data services will move ahead of simple data federation into multi-source data-access services. As federation becomes useful for business intelligence (BI) and master data management (MDM) applications, the value of data services is bound to rise sharply. In the long term, data services will most likely be a part of overall data integration and data management strategies.

On concluding, data have to be gathered from different sources, synchronized, and analyzed in totality in order to create actionable insights that in turn empower enterprises to be ahead of their competitors. Data integration is an indispensable cog. It is a long and complex process of transitioning data into information and knowledge that is not at all simple or easy.

Application Integration

Similar to data islands, there are application silos in every growing enterprise. It is essential to seamlessly link differing applications to ensure dynamic application interaction. In the beginning, point-to-point (P2P) integration of applications was done in an ad hoc basis. Due to the increasing complexity and inflexibility resulting out of P2P integration, hub and spoke (H & S) approach became very popular. EAI technologies use a strategy or framework by which an organization centralizes and optimizes its application integration, usually through some form of data replication or message brokering mechanisms. In other words, EAI integrates applications by allowing them to communicate and exchange transactions and data messages with each other using standard interfaces. EAI is usually employed for real-time interaction. The most popular EAI infrastructures include hubs, buses, fabrics, etc. and this helps to extend integration to heterogeneous sources (mainframe systems, enterprise packages, XML databases, enterprise information systems, CORBA applications etc.). Primarily EAI middleware or integration backbones are supposed to do message content transformation / translation, routing, brokering, filtering, mediation and delivery in a reliable and secure manner. Messages can be transmitted in both synchronous and asynchronous communication.

Centralized Vs Federated Hubs

Typically hub & spoke architecture uses a centralized broker (hub) and adapters (spokes). Spokes help to connect diverse applications to the centralized or federated hub by converting application data format to a format, which the hub understands and vice versa. Hub on the other hand manages all messages and takes care of content transformation/translation of the incoming message into the format the destination system understands and routing the message accordingly. Having a single hub makes system with this architecture easy to manage but scalability takes a hit. At some point in time, as the number of incoming as well as outgoing messages will increase drastically, so the federated hub and spoke architectural approach is recommended instead of tackling this debilitating issue at hardware level. The concept of federated hub and spoke architecture in which multiple hubs can be present, each hub would have local metadata and rules as well as global metadata. Changes to global rules and metadata are automatically propagated to other hubs. Federated architecture alleviates the scalability issue while central management of multiple hubs makes this method easy to manage and brings down support cost.

Java Enterprise Edition (JEE)

JEE could be looked as a comprehensive integration framework. JEE-compliant products are capable of smoothly integrating Java modules with different components implemented in divergent technologies. Remote method invocation (RMI) ensures connectivity among distributed Java applications. Enterprise JavaBeans (EJBs) leverage compliant containers in order to integrate distributed software components. Java connector architecture (JCA) guarantees direct connectivity between Java applications with enterprise information systems (EISs) and packages. Similarly Java messaging service (JMS) supports loose and light coupling among software components. Java database connectivity (JDBC), Java naming and directory interface (JNDI) etc. help Java applications to spotlessly interact with backend databases, directory and identity servers. Geographically distributed applications could talk and leverage each other through distributed transaction, which is intrinsically a part and parcel of JEE. Specialized connectors, drivers, adapters, and engines are being attached to empower Java applications to fuse with other software components dynamically. Similarly Microsoft .NET framework supports linking distributed and decentralized windows applications

Business-to-Business (B2B) Integration

With the Internet leads the pack as the most common business platform, novel applications such as business to consumer (B2C) e-commerce, consumer to consumer (C2C) e-market, business to business (B2B) e-business etc. are being conceived and deployed to the delight of web users. For B2B e-business, independent companies often execute electronic transactions with each other. B2B connectivity is all about the passing of data (not business process logic) through agreed-upon implementations such as electronic data interchange (EDI), extensible markup language (XML), etc. Since business process logic is not shared, B2B connectivity is sometimes called data-centric interfacing. For B2B connectivity, several features are required when interacting with external organizations; e.g. community management, trading partner profile management, sophisticated security mechanisms, and support for industry standards, such as open buying over the internet (OBI), XML, and EDI. B2B connectivity could support any number of business requirements, such as sharing information with trading partners to support a supply chain or collaborating on a product design. Most of the leading infrastructure solutions providers do have B2B integration modules in their kitty.


Part II of this article series, will be dedicated to describing a service-oriented cloud computing integration framework.


[REF-11] SaaS Data Integration Solutions.
[REF-12] SaaS Integration Solutions.