> Archive > Issue XXXVIII: April 2010 > Effective Top-down SOA Management in an Efficient Bottom-up Agile World - Part I


Effective Top-Down SOA Management In An Efficient Bottom-Up Agile World - Part I

Published: April 8, 2010 • SOA Magazine Issue XXXVIII
Nor certitude, nor peace, nor help for pain;
And we are here as on a darkling plain
Swept with confused alarms of struggle and flight,
Where ignorant armies clash by night.

"Dover Beach", Matthew Arnold (1867)


As with politics and religion, information technology has its armies that fight for great truths in the name of half-truths. And one great half truth is that Agile and Service-Oriented Architecture (SOA) are incompatible.

On the face of it, that seems true. Service-oriented architecture must be top-down in conception and execution for it to be effective. Agile is a bottom-up systems development methodology that emerges from self-organizing collectives. SOA and Agile have both demonstrated their value and have firmly established themselves in the marketplace. And yet the experience of more than a decade at many hundreds of firms has exposed flaws in how SOA and Agile are practiced in the real world where vast resources are at stake.

For purposes of this article, "top-down" means a rule-based management structure of command and control. "Bottom-up" means a consensus-based development structure. We are talking about tendencies rather than absolutes. I acknowledge that Agile has planning and a waterfall SDLC is somewhat team-driven.

The purpose of this article is to reconcile these two paradigms and to suggest that the weaknesses that SOA and Agile have present opportunities for integrating SOA and Agile into a complementary partnership. SOAG-- Agile that informs SOA-- can have a decisive, positive, and profitable organizational impact. My goal is not so much to communicate specific technical solutions to SOA-implementation but to put the question of SOA management into the broader context of organizational transformation.

Systems development failure and success rarely has its roots in the choice of language, platforms, methodology or the number and quality of its staff. Rather, they are the result of how managers deploy those resources. In the pages that follow, I will explore how managers can successfully orchestrate SOA and Agile.


Here is what I hope to show:

  1. SOA implementation must be top-down.
  2. Agile SDLC is bottom-up.
  3. To implement an effective SOA, we can combine Agile in a top-down SOA waterfall SDLC.

How to marry SOA to Agile is the subject of the last part of this article. In this section, I discuss the following:

  1. SOAG analysis and problem-solving.
  3. SOAG and the human factor

In a topic as broad as this, more can be said and much has been left out. Some of this may be the basis for future articles as well as discussion by others in the SOA community.

Why SOA Is Top-Down

SOA is not a new name for an API or Web services or a marketing label. And nor is it magic, easy, or cheap. It is, however, a potential business facilitator and the wave of the future. SOA is based on open standards, supports vendor diversity, fosters interoperability, promotes discovery, and supports incremental implementation. The essence of SOA is to decompose business processes into discrete functional units or services and group existing or new business functions into business services. Consumers can then discover and invoke services that the provider publishes.

Top-down management is plan-based, hierarchical, and totalitarian. Bottom-up management is humane, consensual, and egalitarian. Effective SOA implementation—in fact any kind of effective business transformation—must be top-down. SOA management is an imposed vision because we can design and implement SOA only by considering a company’s business processes in their totality from the highest level along with the tools and governance structure that is needed as bedrock for that architecture. And that can only be done with championship from the highest levels of the organization. It also means looking at the organization not as a collection of technologies but as a collection of business processes. [REF-3]

“The bottom-up strategy is really not a strategy at all”, Thomas Erl writes. “Nor is it a valid approach in achieving contemporary SOA. This is a realization that will hit many organizations as they begin to take service-orientation, as an architectural model, more seriously. Although the bottom-up design allows for the efficient creation of Web services as required by applications, implementing a proper SOA at a later point can result in a great deal of retro-fitting or even the introduction of new standardized service layers positioned over the top of the non-standardized services produced by this approach.” [REF-4]

This is not to say that SOA cannot be rolled out incrementally. Of course it should. Nor should a top-down management structure be blind to the experiences, thoughts, and feelings of all of those who may be impacted by those changes. SOA will fail if it is not managed as a top-down effort. SOA will also fail if it is only managed as a top-down effort. But, as a question of architecture and strategy, a populist approach is like building a new house around its appliances. A bottom-up approach introduces needless complexities into already complex systems that are at cross purposes with the company’s umbrella strategy.

A big bang SOA deployment is not sensible. The most prudent approach is to strive to understand the full business context and then undertake a strategy of deployment from opportunistic projects to enterprise level business solutions, taking into consideration that SOA enterprise is a journey, filled with ambiguity, missteps, back steps, and discovery. It is choreography—determining how processes are linked together across the enterprise. And it is orchestration—determining how to mesh those processes underlying business applications so that they can be exposed and linked to processes into other applications.

We can get insight into why SOA execution must be top-down by considering ineffective and effective organizations. SOA attempts to solve the corporation's strategic goals by replacing an ineffective organization with effective organization.

Ineffective Organization: The Silo and the Cog

The silo and the cog are models of dysfunctional organizations

Figure 1 - Echo Chambers

Figure 2 - Static, Expendable
Wikimedia Commons

In the case of the silo, functions, such as marketing, finance, and operations, evolve independently of each other into departments or divisions, each sometimes containing duplicated IT resources and a knitting ball of interrelationships. Silos can occur at the team or even the individual level, where information is husbanded to build personal power rather than to promote the corporate good. A more accurate model might fill these silos with hives having barely sentient winterized bees. Not only is there paralysis of communication cross-functionally but also with the department, sometimes consisting of a myriad of echo chambers.

The cog model views employees as static resources. The half truth that employees need to smoothly mesh with each other is extended into a lie—that Mike and Jane are incapable of change and are thusly expendable. Thus, an employee is branded as a “Java programmer”, for example, rather than as a problem solver who knows Java. So, when the department’s technology changes to .NET, management will replace that cog with a Microsoft-branded cog. Of course, it is only a matter of time when those same managers are replaced by yet other cogs.

Effective Organization: The Factory and The Guild

A more promising model is a hybrid of the factory’s assembly line and the guild from the Middle Ages.

Figure 3 - Factory Time to market good, innovative flexibility poor

Figure 4 - Guild Time to market poor, innovative flexibility good
Wikimedia Commons

Under the factory system, tasks are decomposed and executed methodically, predictably, quickly, and cheaply. Under this model, time to market is good, while innovative flexibility is poor. The guild system gave us architectural gems of Western civilization, such as Canterbury, Notre Dame and other cathedrals with their stonework and stained glass windows. Under this system, beauty and craftsmanship was paramount with scant regard to timeliness or cost-effectiveness.

Our challenge is to integrate the best of both models to give us the time to market of the factory system combined with the innovation and quality from the guild system.

Effective Organization: The Pyramid

The pyramid, a three-dimensional triangle, is the most-enduring man-made structure and an appropriate model for a strong, stable business enterprise.

Figure 5 - The Pyramids of Giza, Built c. 2560 BCE
Wikimedia Commons

Figure 6 - Roadmap

The pyramid is replicated as the most successful government, religious, and corporate structures, and we only need to reflect on organizations such as the governments of the United States or the People’s Republic of China, the Catholic Church or the Church of Latter Day Saints, and IBM or Toyota to recognize this. However, such structures also carry within them the seeds of self-sabotage. Consider the Toyota Production System. Toyota's kaizen doctrine of continual process put the authority to stop a production line in the hands of line personnel. But such customer-driven drive for quality hit a speed bump when cars with defective accelerators resulted in deaths and lawsuits. This may have been a case where the process was so good that it seemed inconceivable to Toyota's salary-men that a car that rolled off the assembly line was defective.

As a bureaucratic organism, the corporate pyramid is constantly mutating—blocks moving up, down, or out, and the pyramid expanding or contracting. The strongest organizations have relatively few layers, and such paucity of layers allows critical information to pass up and down the chain of command efficiently.

One approach to implementing SOA is to conceptualize it as a triad of governance, common services, and platforms and tooling.

Figure 7 - Governance Model

SOA platforms and tooling, consisting of SOA-enabling middleware, provides the backbone of a future distributed and interoperable computing environment.

Figure 8 - Adapted From US Department of Treasury Presentation [REF-5]

Commons services require an iterative governance SDLC.

Figure 9 - Adapted From US Department of Treasury Presentation

Why Agile is Bottom-up

To understand the bottom-up nature of Agile, we must reflect on the paradigm that it displaced-- the waterfall SDLC.

Figure 10 - A Top-Down Flow

Figure 11 - It Starts With BDUF
Wikimedia Commons

The Waterfall Model

The waterfall model grew out of the monolithic corporations that first funded equally monolithic data centers in the 1950s through the 1980s. Such data centers often had hundreds of programmers punching out cards for projects that were managed with PERT or Gantt charts. Under this model, developers could not go in their project from one step to another until there was closure, usually indicated by signoffs. Upon getting those signoffs, the team surfed to the next level in the waterfall.

Figure 12 - Waterfall Model

Agile acolytes mock the waterfall model for its BDUF—big design up front—that makes for inflexibility to embrace downstream requirements. But the waterfall model is effective. It built code that put men on the moon, synthesized life-saving drugs, and launched aircraft carriers. This model’s singular advantage is that it encouraged rigor and minimized re-work by enforcing a forward development path. "A requirements defect that is left undetected until construction or maintenance will cost (up to) 200 times as much to fix the same bug later on in the process." [REF-6] The promise of the waterfall method is that carefully analyzing all requirements using BDUF would squeeze out vagueness and downstream errors while enforcing a structure of accountability. This promise was not always realized in its rigidity to meet dynamic customer requirements and to implement large projects on time and within budget.

The Promise of Agile

The dominance of big iron receded with the advent of the web, object orientation coding, distributed objects, the ubiquity of computers, the plunge in the price of storage and chip prices, the Y2K hysteria, the dot com boom and bust, and offshoring. These pressures forced developer to re-think lifecycle development theory. One reaction to heavyweight SDLC was Agile. Its basic principles were encapsulated in 2001 in the Agile Manifesto [REF-7] by representatives of new methodologies such as Extreme Programming, Scrum, and Pragmatic Programming.

The Practice of Agile

Some skepticism as to the merits of Agile is needed. Instead of BDUF, we sometimes have BPOC—big person on campus— relentless egomaniacs that hijack and sometimes destroy projects and reputations. Agile can become a cover for cowboy coding—a flight from team interactions and customer corroboration as Agile advertises. It can become an excuse not to have any process, documentation, e-mails, or audit trail. Tribal knowledge comes out of happenstance of ability and availability. What some see as flexibility and humanity is often chaos, shoddiness, and indolence with ten year-old systems with no documentation and implementations without regard to impacts across the enterprise.

Agile can work. Much rests on the good will, energy, and collaborative temperament of the team. But it often breaks down under the weight of the cussedness of human nature. The Agile premise of dialectical truth—that the best truth comes out of the litigation of clearly-articulated, strongly-held contrary points of view—is valid, so long as there is honest, open debate where persuasion is not tied to titles or personalities but to evidence.

It is not sufficient to say that when Agile fails, it is because its disciples do not understand Agile. They often understand it all too well, but cannot to make it succeed, due not to defects in its theory but to defects in personalities that struggle to realize its theory. The denial that Agile has within it the potential for bad practices falls into the No True Scotsman fallacy, an argument that takes this form:

    Argument: "No Scotsman puts sugar on his porridge."

    Reply: "But Angus likes sugar with his porridge."

    Rebuttal: "Ah yes, but no true Scotsman puts sugar on his porridge."

Consider a parallel argument:

    Argument: "No Agile implementer is a cowboy coder."

    Reply:"But Mark is a cowboy coder."

    Rebuttal: "Ah yes, but no true Agile implementer is a cowboy coder."

The problem with this argument is that it derives an unproved predicate ("puts sugar on porridge", “cowboy coder”) from the subject ("Scotsman", “Agile implementer”). Now, sometimes the argument is valid as when the predicate derives from the subject, as in "no true vegetarians eat beef.” But there is nothing inherent in Agile that will prevent someone from being a cowboy coder, and the cowboy who codes under the banner of Agile will be the very flower of Agile. Or so he or she will believe.

I believe that Agile and Agile-like methods such as Scrum are integral to the success of a SOA implementation. But to ensure that Agile does succeed, it is important to come to grips with how and why Agile can fail. And the reason Agile can fail is because it is at the bottom a personality-driven process that demands exceptional individual discipline and a zeal for collaboration.


The second part to this article will be published in the May 2010 issue of The SOA Magazine.




[REF-3] Hurwitz, Bloor, Kaufman, Halper, Service-Oriented Architecture For Dummies, (Wiley Publishing, 2009), 357. "If you relegate SOA to IT, we the authors have failed miserably. We throw up our hands. SOA must be a joint endeavor between business and IT. You have everything to gain if you make SOA a joint venture – and everything to lose if you position SOA in your company as a new cool project for an IT development team."

[REF-4] Thomas Erl, Service-Oriented Architecture: Concepts, Technology, and Design, (Prentice-Hall, 2005), 369-370.

[REF-5] Adapted from Ari Bender, "Service-Oriented Architecture Discussion", Department of Treasury, Internal Revenue Service, February, 2007 presentation.

[REF-6] Steve McConnel, Software Estimation: Demystifying the Black Art, (Microsoft, 2006), 72


[REF-8] Adapted from "SOA Security", OWASP Foundation, December 7, 2007, presentation.

[REF-9] Adapted from "How To Leverage SOA Investments With Agile Methods",, 2001-9, 5.

[REF-10] "Who Were History's Great Leaders?", TIME, July 15, 1974, 26. University of Chicago philosopher Mortimer Adler's answer: Pericles of Athens, Winston Churchill, and almost any of the founding fathers.

[REF-11] John P. Kotter, "What Effective General Managers Really Do", Harvard Business Review, November-December, 1982. "It is vital to recognize the importance of political activities such as developing and maintain personal relationship, building team spirit and commitment, developing support for our ideas and programs, sustaining visibility for your projects, lobbying for priority and resources, and discovering the currents and undercurrents of business."

[REF-12] Robert Townsend, Up The Organization: How To Stop The Corporation From Stifling People and Strangling Profits, (Jossey-Bass, 1970), 77. Townsend says that a company's Devil's Advocate "must have a loud voice, no fear, and a passionate hatred for institutions and their practices."

[REF-13] I owe this idea to comedian Yakov Smirnoff, from his seminar "The DNA of Happily Ever Laughter", December, 2009. The seminar was his effort to salvage from the wreckage of his own divorce insights on how to build enduring relationships. DNA stands for discovering how we are sometimes performers or audience in a relationship, noticing each other's needs ("Happiness is when our needs are met"), and agreeing on a way forward.