Abstract: SOA and agile processes as a whole. Nowadays agility processes and agile architectures are critical success factors for many companies whose business is driven and determined by continuous changes. Business uses IT as a tool to automate processes and store data in order to streamline processes. However, IT often represents a rigid machine that evolves and reacts to changes independently and differently from the business. Similarly, an IT framework that is reactive to change is not enough for it to be considered fully agile in support of the business, if changes are not driven by agile processes. Although not much has been written so far on this topic, this article describes the benefits and the variables SOA brings to agile processes; in particular it describes the features in support of the "Business-IT alignment" paradigm in an agile environment.
Service-oriented architecture (SOA) and agile methodologies are individually affirmed within the IT world. However they are also becoming more interesting when synergically adopted. Even though they are fundamentally different, and covers different areas, (one being a set of architectural principles, while the other a set of methods and practices), one of the key goals and benefits they share is to enable changes to keep up with business evolution and needs.
Changes are inevitable and continuous in the software lifecycle. They can also be of different types. For example, new business requirements, changes to improving software performance, to re-engineering systems, to adapting the software to a new environment or just fixes and so on. Software agile development methodologies have been identified to simplify development processes and to accommodate changes to be implemented in a timely manner.
However, the questions we try to analyze here are:
- Would an agile process by itself be able to increase efficiency and productivity, lower maintenance costs, and keep business needs aligned with IT roadmaps?
- Would an agile process be able to ensure quality in artifacts and processes?
It is estimated that 80% of IT budgets are spent for maintenance. Mostly due to unstructured not modularized systems, redundant code, ad-hoc interfaces, lack of standardization/up to date documentation, and so on. Although agile processes lean on the lifecycle bureaucracy, this alone does not equal to systems becoming agile.
This is because they do not say anything about technical processes. Changes can be treated agile during their identification, but their implementation may not be so lean if the underlying IT architecture does not evolve in alignment with agile concepts. The result is a non-optimized agile lifecycle.
Drawback of Waterfall Processes
Implementation of changes in the waterfall methodologies need several serialized phases: analyze the problem, map it to specific service(s) and interface(s), create an implementation plan, document the specification and test-cases, architect and design a solution, and finally to develop, test and release it. A waterfall model defines sequential phases and although simple, it falls into the category of processes whereby the client might have to wait for months (2/3 in typical applications with tens of interfaces or hundreds of thousands LOC), in order to have the changes go live. Only then can a business smile and see what had been required few times back (and likely to be subject to a new change).
This is acceptable in some industries, as it may be the case for manufacturing or law institutions. However, it is much less acceptable in domains where events require immediate and continuous changes, such as the financial and regulatory domains. With its pros and cons, the waterfall model has been criticized in the last 10 to 15 years in favor of more flexible processes. As a result, in 2001, 17 people coined the term Agile Software Development and "The Agile Manifesto" was born [REF-2].
Agile Architecture Basic Requirement
Agile stresses the idea that everything changes, and that software development teams must be able to recognize and respond to these changes frequently. However, it does not say anything about technical procedures.
The point is that if agile methodologies and IT frameworks could fit together, then a fully agile lifecycle would be set. As an IT framework, SOA is similar to agile methods in that SOA recognizes that change is inevitable and that organizations need to effectively cope with those changes.
To implement change in an agile environment, teams need to work on simple architecture models. The models have to be easy to understand, modularized and each component has to be segregated from others to provide a clear unique purpose service. It is also necessary that the architecture is open in a way that it is not resistant against modifications. Therefore, the question is whether SOA is able to fit into this context. To do that we need to understand in further detail, the goals and benefits an SOA solution implies.
SOA: Agile Features and Constraints
From SOAGlossary.com: "Service-oriented architecture represents an architectural model that aims to enhance the agility and cost-effectiveness of an enterprise while reducing the overall burden of IT on an organization [...]" [REF-5]
Service-oriented architecture (SOA) get its origins from EAI, Object-Oriented, BPM, Web Services and others concepts used to unify and (still) evolve their respective best practices.
SOA mandates that systems are modularized, proprietary details encapsulated and/or hidden so that service reusability, interoperability, standardization, abstraction and segregation of duties concepts along with governance procedures are leveraged to increase ROI, organization agility and reduce IT costs. This results in the collateral goal to align business strategies with IT roadmaps.
SOA is comprised of a set of design patterns [REF-1], principles [REF-4] and governance precepts [REF-6] addressed to achieve strategic goals and benefits relevant to IT and business.
In the context of SOA, one application should be never seen as a standalone application. Applications expose various services. Services need to interact with each other to fulfill front to back business processes. There are various inter-application dependencies that can be described as service dependencies in a service model. Any service should be considered for potential reuse by other applications. Thus, service applications should be seen as reusable products. In effect, this means that agility becomes more important, because the number of consumers to the applications increases, as well as the business processes. Proper design targeted to leverage reusability, abstraction, composability and loose coupling are key factors to achieve prompt agile integrations. Effective changes are soon localized and mapped onto services and interfaces which evolve by definition.
However, it is not difficult to recognize that an effective service-oriented adoption successfully works when a common strategy is set up within an enterprise or at least between "related" functional units. Segregated silo departments integrated on an ad-hoc basis are just tactical solutions. They are difficult and expensive to reuse or extend due to different systems strategies. Their integration processes indeed require customized re-work anytime new requirements come in. This is what service-oriented architecture aims to avoid, by offering different options.
For example, standardization in technologies, interfaces and design eases the communication between or within teams to make it more streamlined and simpler. To say that in the idea of the Conway's law, it also means that the quality of the interfaces and their mutual dependencies is also reflected as an immediate result of a leaner analysis cycle, which supports teams to identify high quality solutions quicker.
Another example is that through service re-composition, agility becomes easier to manage. You can capture a change in business processes by changing the service collaboration profile. A change in or a creation of a new service collaboration is easier and faster to achieve than a refactoring in the implementation. And the most peculiar part is where an SOA framework maximizes ROI by reducing the development costs and the delivery time by 75% greater [REF-7]. Not to mention that in support of this, the market offers plenty of products, tools and IDEs specialized to streamline and automate the production of service artifacts, often without the need to write code.
SOA leverages business entities and business processes to model, design and maintain software components in a controlled environment. It supports changes relating to control and discipline, in order to prevent IT from bloating with complexity and TCO (total costs of ownership). SOA provides discipline to discover, abstract or reuse services that need to be in place. SOA defines design strategies, change/versioning management and governance precepts. Another point to consider is when agile processes are applied without an IT control strategy. Changes can be subjectively interpreted and applied as per the wish of developers, technical leads or project managers, with the end result going in many divergent directions!
To give an idea on what this means, here is a simplified implementation task checklist to apply a new requirement in a controlled environment:
- Discover the affected service
- Extend/reuse the service (eventually reassembling the composition to create a new process)
- Applying versioning if necessary
You can note that the steps are repeatable and streamlined, and the concept reusability and composability are leveraged so that no new development effort is needed. After going through this list, you can then ask yourself, "from a process point of view, does this look agile?"
However, if the service it is not yet part of the inventory, then we have missed a few activities:
- Analyze the business service model
- Analyze the service inventory
- Define the interfaces, apply patterns and standards
- Implement the logic (eventually composing other logic if this is a task service)
- Add the service to the inventory
- Applying versioning if necessary
A mature-filled and well-designed inventory makes the majority of the domain services available.
These and other multitudes of SOA "lean" features can be put in place in support of agile processes, to ultimately increase ROI, organizational agility and reduce IT costs.
SOA Constraints for an Agile Adoption
Although combining SOA and agile gives you a lot of abilities to deploy IT services reactively and provide high quality systems, they do not imply each other. They can be pursued independently or concurrently. For example, there are successful agile projects that are supported by waterfall approaches.
Sometimes, SOA is not adopted due to various reasons: e.g. no SOA culture spread within the company, no IT strategic or common visions, no interest in IT investments beyond the "basic functions", changes financed by and for business only, no SOA education/awareness at management level, fear to change, etc... The paradox is that a lot of money is still spent around IT infrastructure to make business live.
It is possible to have agile processes applied on legacy systems or primitive EAI architectures with no SOA concepts at all. You will need to be agile in gathering requirements, fragmenting deployment in chunks of deliverables that business can touch earlier. However if you aim to use agile to strategically increase productivity, reduce costs and increase quality then you may be on the wrong path. Bear in mind, productivity can be associated with agility when it comes to the number of changes implemented in the delivery or release time.
That being said, SOA adoption does not come for free. Awareness at management level, skills and expertise are needed within your company. Also, if your company lacks cross-functional units and teams communication, then you may have serious problems when pursuing a SOA strategy. SOA performs naturally over the boundaries of a single application. Its artifacts model business domains that generally does not operate isolated but it fosters to streamline applications' integrations (see interoperability). So just be prepared to welcome SOA.
Although SOA requires departments or business-units to establish cross-functional communication and common integration strategies, SOA projects can adopt the bottom-up approach: starting from modeling a single functional unit to subsequently leverage ESB (enterprise service bus) or middleware infrastructures to alleviate the disparity between cross-functional inventories.
Furthermore, SOA promotes the divide and conquer method. Big requirements or projects are fragmented and possibly assigned to different teams and/or professional roles under the supervision of architects or other SOA specialists. On the other hand, agility does not discriminate within the team and does not segregate teams. It aims to keep a team cross-functional. Although the two methodologies seem to clash, the incompatibility is marginal since the up-front business services and inventories analysis, application of design patterns, contract-centric approaches and standardizations represent the "common understanding" between the teams. Again, technical procedures do not represent a deterrent for changes but they simply aim to streamline production processes and generate higher quality.
Also, agile methodologies promote an incremental and iterative approach, with visibility of the impact of change given through a quicker and chunked deployment. This is because change is less costly if fragmented and implemented earlier. While SOA establishes well-defined service interfaces through its contracts, the interfaces should be kept as stable as possible since they are the only external visible part. That is one of the major reasons why SOA defines governance and change management processes: to track and control each and every deliverable. A versioning mechanism is needed for the evolution of contracts, and therefore a migration process is also required. Versioning is a critical success factor to keep a consistent layer of services while they evolve.
So, if the development of services and service-consumers are in-line with agile, the design of service interfaces is not that natural. On the other hand an uncontrolled incremental approach would cause serious compatibility problems, especially with remote (third party) consumers. Just think, if you are consuming services under uncontrolled agile process whose contracts evolve and change continually, then there is a need to establish an IT framework to control change and their consistent application. This is where SOA processes come reasonably into the game.
Moreover business entities evolve but rarely change disruptively. Service functional, capability and data granularity degree [REF-4] plays an important role on the level of flexibility a service has to accommodate changes. For example, accurate service design to create backward compatible contracts or increase the amount of data exchanged by a service (document-centric messages) should be also considered to build a service "open" to modifications. Plenty of design patterns exist [REF-3] to support the design of adaptable services in a way that the margin to accommodate apparently-disruptive changes is not null. If you cannot achieve that, it may likely be because the business process you are modeling is not well understood. In that case, you will need to review the solution or gain a better understanding. However an incremental implementation should be pursued by trading off the granularity level (coarse vs finer grained) against the level of security required. Although a coarse grain interface is loosed and more open to backward compatibility changes it may open security breaches in your services; however there are specialized security design patterns to mitigate these issues (security patterns [REF-7]).
SOA offers a lot of tools in support of agile methods and issues. However, accurate planning is required to adopt them in conjunction. This is especially so during ramp-up phases where you are likely building the domain services from scratch, refactoring applications or filling half-empty inventories. This will have an impact on IT productivity and efficiency. But that lost will be gained along with all the interests during the maturity period (maximization of ROI).
When agile processes and service-oriented architecture are combined together to support the whole IT-business service development lifecycle, you can get the most out of them. SOA introduces a controlled environment in which changes are accommodated in support of agile processes, where quality, efficiency and productivity are increased through the appliance of design patterns, standards and governance procedures. Design patterns like service reusability, composability and abstraction, to cite a few [REF-4], are leveraged to provide flexible and adaptable ecosystems.
Agile methods also enable the lifecycle to be more incremental and interactive, allowing the business to get/give faster feedback from/to IT. They both support the continuous business-IT cycle that is needed to allow businesses to set up strategies aligned with IT.
Agile and SOA share similar principles to achieve common benefits. Although SOA establishes a controlled technical environment, they can be used at the same time in the same lifecycle without incompatible overlapping. However, a strategy in support of incremental changes has to be planned.
It is possible that you can run agile on legacy applications or ad-hoc integration architecture, however that just means fragmenting the requirement backlog into chunks of deliverables that the business side can run into earlier. With this method, you have to expect high development efforts along with an increased IT burden and TCO costs.
Be aware that SOA does not come for free: discipline, skills, expertise and awareness at management level are necessary. A significant initial investment and true support from high management are required. Although these might represent "entry barriers", the ultimate goal is sustaining a long-term ROI.
If the objective of your customer is to run agile processes to support of their changes in a timely fashion and lower development costs, then it is worth investigating if the underlying IT architecture is able to smoothly accommodate them in terms of responsiveness and costs. Otherwise, you may end up achieving only marginal benefits.
[REF-1] SOA Design Patterns, ISBN: 9780136135166, Prentice Hall, www.soabooks.com/patterns
[REF-3] SOA Patterns Catalog, www.soapatterns.org; SOA Principles of Service Design, ISBN: 9780132344821, Prentice Hall, www.soabooks.com/psd
[REF-4] SOA Principles, www.soaprinciples.com
[REF-5] SOA Glossary, www.soaglossary.com
[REF-6] SOA Books Governance, www.soabooks.com/governance
[REF-7] SOA Security http://soaschool.com/certifications/securityspecialist
[REF-8] Increased ROI, http://www.whatissoa.com/increased_roi.php