> Archive > Issue XXXIX: May 2010 > Effective Top-Down SOA Management In An Efficient Bottom-Up Agile World - Part II


Effective Top-Down SOA Management In An Efficient Bottom-Up Agile World - Part II

Published: May 12, 2010 • SOA Magazine Issue XXXIX

SOAG: A SOA and Agile Synthesis

System building is a search for truth, and SOAG is no exception. We must establish SOAG on a foundation of sound epistemology. SOAG must be above all a rational process, a process that binds us to principles of empirical adequacy and rational coherency without respect to what I believe or to what the team believes. Empirical adequacy requires that the concept under question be amendable to empirical verification and asks the question: does it meet the evidence? Rational coherency requires that the concept should be consistent with other concepts that were arrived at rationally. It asks the question: is it consistent within itself?

SOAG asks us to think for ourselves. But, in contrast to Agile, SOAG makes the radical claim that we should not think for ourselves. "It is a profoundly erroneous truism, repeated by all copybooks and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing," writes Alfred North Whitehead. "The precise opposite is the case. Civilization advances by extending the number of important operations that we can perform without thinking about them. Operations of thought are like cavalry charges in a battle-they are strictly limited in number, they require fresh horses and must only be made at decisive moments." Agile is more personality rather than plan-driven. Hence, it is a more subjective process that takes its success not so much from charts, plans, and rules and as it does from the team's group dynamics and mutual agreement. There is no need to think or to rely on group thought when we have processes in place that will do our thinking for us. SOAG advocates that plans and structures are in place so that the kind of singular craftsmanship that flourished in the guilds of the Middle Ages is unnecessary.

Figure 1 - SOA Agile Triangle

SOAG Analysis and Problem-Resolution

René Descartes (1596-1650), who is generally regarded as one of the greatest Western thinkers of the past few centuries, wrote his Discourse on Method in 1627. But his analytical principles still live on today as we approach the question of how to think about SOA with its multi-dimensional questions with answers that have far-reaching implications.

 1.   The first was never to accept anything for true which I did not clearly know to be such; that is to say,
       carefully to avoid precipitancy and prejudice, and to comprise nothing more in my judgment than
       what was presented to my mind so clearly and distinctly as to exclude all ground of doubt.

Descartes asks us to do something that is exceptionally difficult: To clear our mind of presuppositions. For example, we must empty our brain-pans of any kind of attachment to a particular vendor. The SOA paradigm assumes technology agnosticism, in which we can code a presentation layer while still maintaining, for example, heterogeneous databases. We must approach SOA generally as it is decomposed into a myriad of clustering sub-problems with neutrality. We must resist jumping to conclusions. This suggests skepticism as to vendor claims and accepts that as a rule that all vendor documentation is flawed.

 2.   The second, to divide each of the difficulties under examination into as many parts as possible, and
       as might be necessary for its adequate solution.

To break down into its smallest components is to analyze. Keep turning the problem around looking for different ways to atomize business processes. How do processes relate to the whole? Are there other ways to decompose and relate processes to each other and the whole?

 3.   The third, to conduct my thoughts in such order that, by commencing with objects the simplest and
       easiest to know, I might ascend by little and little, and, as it were, step by step, to the knowledge of
       the more complex; assigning in thought a certain order even to those objects which in their own
       nature do not stand in a relation of antecedence and sequence.

Look for analogues from what is known to what is unknown. Use vendor relationships to get case studies of SOA implementations in your line of business, and use those studies as a benchmark for best practices and lessons learned. Work from the simplest and easiest issue to the most complex issue or application. This ordering is the essence of the process we need as SOA is rolled out with more complex and more mature common services. With the SOA, the caveat is that we need to address complex issues early in the process. These issues include security, Web services optimization, distributed databases, and getting buy-in from the business side.

Figure 2 - From opportunistic, project level to enterprise level bommon services
(Adapted From a US Department of Treasury presentation)

 4.   And the last, in every case to make enumerations so complete, and reviews so general that I might
       be assured that nothing was omitted.

Agile dogma is that we can discount upfront design, and that is a fundamental mistake in any large project. That is especially true with SOA, where the goal of change is not evolution but revolution-a paradigm shift no less dramatic as from the mainframe to e-commerce. Descartes asks us to be detail oriented, but also to integrate those details into meaningful patterns.

While SOAG agrees with Agile that "the most efficient and effective method of conveying information to and within a development team is face-to-face conversation" but insists that there must be more. We must take ideas out of our mind into the minds of others using the written word. Landlords find out soon enough that they cannot rely on a tenant's goodwill or memory to formulate agreement. This is especially true if the effort needs to be coordinated with co-located team members or teams.

The mutability of speech is such that unintended ambiguities slow the process with misunderstandings. It is the distinction between the metaphysical ("what is time?") and the practical ("what is the time?"), the comedic ("Tom fell on the ice") and the critical ("Tom fell through the ice"). Not only do developers need common values, they also need a common vocabulary and grammar to establish common meaning. Planning must allow for ambiguity, and it is not necessary that all things be known in all levels of the pyramid. However, where things must be known, sometimes they need to be known explicitly, such as on questions of capacity and hardware. In systems development, using the wrong words or undefined words have consequences, ranging from waste to failure. The written word can be imprecise, but that imprecision is open for the world to see and for the world to correct. Conversely, the written word sometimes conveys more certitude than is intended, especially in the earliest stages of the project. And, in the effort to be precise, analysts publish fifty-page documents that sometimes give a false impression of certitude even when they are read. There is a middle ground, and as a general proposition, writing rather than speech should guide development to drive the project.

Apply information engineering or unified modeling language notation for use cases, sequence, collaboration, statechart, activity, class and object, component, and deployment diagrams, extension mechanisms, conceptual, logical, and physical models, and business cases. Use as much symbolism that is needed but no more than is needed. The Franciscan monk William of Ockham (1285-1349) gave us the principle that "plurality should not be posited without necessity". His Principle of Parsimony states that we should not make any more assumptions then the minimum needed, as eliminating the superfluous reveals or advances its resolution. That entities should not be multiplied save out of necessity predicates Lean and Agile software development. There is a qualitative difference between no documentation and some documentation. But that documentation in many cases need not be more than a single page.


The Manifesto for Agile Software Development is identical to SOAG with one exception. We should change "Responding to change over following a plan" to "Following a plan over responding to change." The original sentence implies a false dichotomy. A response to change is reaction rather than proaction. Responding to change rather than planning for change shows a lack of forward thinking. Instead of BDUF, we should architect a SDUF-small design up front. Push decisions as low as possible. So long as we can establish simple but sound scaffolding, we must extend trust to others will fill in the gaps when the time comes.

The design should be small, but the plan should be big-BPUF. "Make no little plans," advises Daniel Burnham, the greatest American architect of the early 20th century. "They have no magic to stir men's blood, and probably themselves will not be realized. Make big plans, aim high, and hope, and work, remembering that a noble, logical diagram once recorded will never die, but long after we are gone will be a living thing, asserting itself with growing intensity."

Rigorous planning need not be the opening wedge for a plethora of red-tape. Common sense, periodic reviews, and sunset policies need to constrain process. A few rules that are rigidly enforced are better than many rules that are poorly enforced.

The Latin phrase quo vardis?-to where?-seems appropriate. We should ask: What is the point of this document or this process? Is the information duplicated anywhere else, for example, in meeting minutes, an application lifecycle management (ALM) system, or in the code itself? And there should be good answers. "That it is the way we have done things before" or "that it is part of CMMI" are poor answers. Standards for documentation have a way of freezing into a process that has to do more with cementing personal power than with driving the project, and a governance structure needs to evaluates whether such process leverages results. Documentation might consist of nothing more than an e-mail that has been pasted into an ALM system. The modernist architectural and industrial principle that form follows function suggests that so long as the function is met-in this case, communicating and storing process-related documentation-the form is not important.

The SOAG SDLC model is based on principles of planning, adaptation, iteration on a foundation of governance:

Figure 3 - Business-Driven Development Methodology (OWASP)

SOAG is built around the following phases:

  • Model: Use modeling tools to define the business process at a business function level, and model the actual services that will be part of an assembled, composite application.
  • Assemble: Assemble the individual services and write the code that is needed to implement the business rules for the application.
  • Deploy: Deploy the services to runtime environments. Use integration components, primarily an enterprise service bus (ESB), to link together the various services needed for the composite application. Manage: Implement the management infrastructure for monitoring and managing the services and the service infrastructure. This includes not only information technology management tools, but also business management and monitoring tools to measure business activities.
  • Governance underpins all the life cycle stages, providing guidance and oversight for the entire service-oriented environment [REF-7]. Here is the impact is the SOAG lifecycle impact [REF-8]:

  Traditional SOA SOAG
Funding Upfront IT funding Funded by business
Definition of Service Architecture Mainly upfront Incrementally as composites are deployed
Definition of Master Data Models Upfront Incrementally
Enterprise Alignment Required upfront Acquired naturally through proven results
Time To ROI 6-12 months 6-12 weeks

SOAG and the Human Factor

A corporation is a community of people with its unique psycho structure. Some companies are confident and courageous while others are diffident and dysfunctional. Corporations take on the best and worst traits that emerge from its people and culture. Governance is needed to do to build winning teams to transform the organization into a SOA enterprise. Governance relates to the nuances of its culture in addition to formally-established rules. It is a company's system of interrelationships between its leaders, managers, and followers.

Figure 4 - Governance Triangle

Management is the deployment of capital, process, and people to achieve a goal.

Figure 5 - Management Triangle

Leadership, that most mysterious and rare of human qualities, derives in Aristotlean terms from ethos, pathos, and logos. Ethos is moral character, the source of his or her ability to persuade. Pathos is his or her ability to touch feelings. Logos is his or her ability to give sound reasons and to move people intellectually [REF-9].

Figure 6 - Collaboration Triangle

The model for a leader is less that of a president than a psychologist. Few of us have the authority to compel anyone to do anything, and even managers who do have that authority struggle to figure out how to make the wheels of their chariot turn. But effectiveness does relate to how well we can influence. We must believe firstly that we can make a difference and reject the fatalism that "it is what it is". Our effectiveness comes out of our expertise but also out of our ability to build positive relationships so that we build winning teams [REF-10].

Followership is no less important than leadership, and is the confluence between collaboration, trust, and doubt.

Figure 7 - Followership Triangle

The doubters and naysayers within the company, as disagreeable as they may sometimes be, perform an essential service and often care more deeply about the company than do the more placid and pleasant. For hundreds of years, the Catholic Church institutionalized a position of a person whose job it was to challenge the qualifications of a potential saint called the Devil's Advocate. Businesses would also do well to have such a Devil's Advocate-someone who could not be fired unless they were not showing enough courage in challenging conventional wisdom. [REF-11]

In a collaborative model, each role must assume as circumstances warrant a leadership and a followership position to be effective.

Figure 8 - Collaboration Triangle

The relationship between followers and leaders is akin to that between bar magnets [REF-12]. A bar magnet consists of a north and south pole. The north pole of a magnet repels the north pole of another magnet. A bar magnet will only attract another bar magnet if the north pole is aligned to its opposite, the south pole. In the same way, an unhealthy team relationship has pockets of repulsion-staff and managers butting heads- or impotence-where neither managers nor subordinates care for each other and so nothing much gets done. A healthy organization dynamic connects strong to weak to strong to weak up and down the chain of command and across the pyramid of the corporation. This means having the flexibility and courage to assume the role of leadership and followership as circumstances demand. The organizational-level transformation that SOA demands will require the best kind of leaders, of which Lao Tzu (6th century BCE) spoke: "The best of leaders, when the job is done, when the task is accomplished, the people will say we have done it ourselves."

Figure 9 - Leadership/followership relationship.

Figure 10 - A healthy organizational dynamic. No weak to weak relationships where there is impotence or strong to strong relationships where there is repulsion.


SOA is not business technology. It is business architecture. And, like the architecture of bricks and mortar, top-down thought, discourse, and direction need to guide a corporation's strategic transformation, as well as deep tactical understanding that can only come from all team members. The promise of SOA is increased business responsiveness, broader service offerings, reduced risk, reduced time to market and vendor dependence, and increase flexibility, innovation, reliability, scalability, and internationalization. All of the conditions exist for everything to remain exactly as they are. To change an organization for the better, we must adjust or create conditions that foster that change. Those conditions include governance, agility, and a culture of communication and collaboration. Organize for success by promoting the business value of SOA rather than SOA, by getting leadership championship on broad strategy, by evangelizing the vision throughout the enterprise, by leveraging SOA-related lessons learned and business cases from vendors, by taking a politically astute and non-intrusive approach, and by considering the human factor. The deficiencies of Agile and the classic waterfall lifecycle are compensated by their strengths when these two paradigms merge into one. The success of a SOA rollout will largely depend on a structure of governance to administer a SOAG lifecycle. To paraphrase Alexander Pope (1688-1744)

"For forms of life cycles let fools contest
Whate'er is best administered is best."




[REF-3] Hurwitz, Bloor, Kaufman, Halper, Service-Oriented Architecture For Dummies, (Wiley Publishing, 2009), 357. "If you relegate SOA to IT, we the authors have failed miserably. We throw up our hands. SOA must be a joint endeavor between business and IT. You have everything to gain if you make SOA a joint venture – and everything to lose if you position SOA in your company as a new cool project for an IT development team."

[REF-4] Thomas Erl, Service-Oriented Architecture: Concepts, Technology, and Design, (Prentice-Hall, 2005), 369-370.

[REF-5] Adapted from Ari Bender, "Service-Oriented Architecture Discussion", Department of Treasury, Internal Revenue Service, February, 2007 presentation.

[REF-6] Steve McConnel, Software Estimation: Demystifying the Black Art, (Microsoft, 2006), 72


[REF-8] Adapted from "SOA Security", OWASP Foundation, December 7, 2007, presentation.

[REF-9] Adapted from "How To Leverage SOA Investments With Agile Methods",, 2001-9, 5.

[REF-10] "Who Were History's Great Leaders?", TIME, July 15, 1974, 26. University of Chicago philosopher Mortimer Adler's answer: Pericles of Athens, Winston Churchill, and almost any of the founding fathers.

[REF-11] John P. Kotter, "What Effective General Managers Really Do", Harvard Business Review, November-December, 1982. "It is vital to recognize the importance of political activities such as developing and maintain personal relationship, building team spirit and commitment, developing support for our ideas and programs, sustaining visibility for your projects, lobbying for priority and resources, and discovering the currents and undercurrents of business."

[REF-12] Robert Townsend, Up The Organization: How To Stop The Corporation From Stifling People and Strangling Profits, (Jossey-Bass, 1970), 77. Townsend says that a company's Devil's Advocate "must have a loud voice, no fear, and a passionate hatred for institutions and their practices."

[REF-13] I owe this idea to comedian Yakov Smirnoff, from his seminar "The DNA of Happily Ever Laughter", December, 2009. The seminar was his effort to salvage from the wreckage of his own divorce insights on how to build enduring relationships. DNA stands for discovering how we are sometimes performers or audience in a relationship, noticing each other's needs ("Happiness is when our needs are met"), and agreeing on a way forward.


I wish to thank the following individuals who reviewed and critiqued my paper: Larry Gorman (Senior Director/Systems Development, Choice Hotels International), Tony Pallas (Systems Architect, Choice Hotels International), and Marc-Thomas Schmidt (Distinguished Engineer & Chief Architect SOA Connectivity, IBM).