Abstract: This two-part article series discusses real-world challenges of applying service-oriented principles and patterns. Informed by case studies of SOA wins and losses, this article contrasts the SOA ideal to its realization and suggests that strong leadership must drive a successful SOA. It proposes a new way to represent SOA principles and patterns using symbolic notation. This article supplements two textbooks by Thomas Erl, "SOA Principles of Service Design" and "SOA Design Patterns", which expand on that synergy between principles and patterns.
Between the desire
And the spasm
Between the potency
And the existence
Between the essence
And the descent
Falls the Shadow
T.S. Eliot, "The Hollow Men" (1925)
Introduction: Bridging the Ideal and the Real
This article discusses real world challenges of working with SOA principles and design patterns based on personal project experience and published cases [REF-1]. We'll try to understand the shadow between funding SOA and the spasm of its failure. "This is the way the world ends," Eliot says in his poem. "Not with a bang but a whimper." A company's SOA can also end with a whimper, not failing but bringing little value to the company. An architecture that seems good in theory but fails in practice is a bad theory. Thus, a theory for how we craft an SOA must align with road-tested practice.
In the first article in this series, we'll contrast the SOA ideal to its realization and suggest a way to represent SOA patterns with symbolic notation. In the second article, we'll apply two principles to an example of a claims service using thirteen patterns.
The Ideal SOA
Service-oriented architecture (SOA) tries to solve the problem of poorly planned systems and fragile chains of dependency. SOA provides a two-fold benefit to our enterprise. First, SOA requires a process of deep scrutiny that exposes waste and obsolescence. Secondly, it provides an innovative platform to meet strategic goals. The promise of SOA is that it creates conditions for cutting costs, increasing profits, and responding to the marketplace. We build SOA using a framework, an agile life cycle, principles, and patterns [REF-2].
The Real SOA
Most architects view SOA as a standard for deploying flexible, market-driven solutions. We see success in every sector of the economy throughout the world. AstraZeneca, Independence Blue Cross, Gaylord Hotels, Thomson Reuters, Avnet, Bell Aliant, and many other organizations from startups to multinationals have successfully deployed enterprise-level SOA projects. But other companies have struggled to turn their SOA into a platform for profits.
What is "real SOA"? We walk into an office with people trying to build an SOA. Everything we see is real in the sense that it's tangible. But it's also transient. People, buildings, and technologies come and go. What's real is what lasts. And what lasts are the SOA principles of standardized service contract, loose coupling, abstraction, reusability, autonomy, statelessness, discoverability, and composability.
What also lasts is the full range of the human spirit that makes and uses that SOA. That means looking at our landscape as clearly as possible, realizing that fear and insecurity can motivate our colleagues and that they're self-interested (although they can be blind to those interests). Our goal as an SOA realist is to look at those we depend on in totality, sometimes questioning their motives, but also appreciating and harnessing all of their positive qualities.
Real SOA is similar to Realpolitik, a theory of power politics that stressed practical application rather than conduct that is driven by ideology. The paradox is that seeing people as they are, in a harsh light, is often the best way to reach our SOA ideals. "Because I know what I want and what others are capable of," said the 18th century diplomat Prince Klemens von Metternich, "I am completely prepared." And part of our preparation means considering the relationship of leadership to a successful SOA.
Although SOA is going into its second decade as an established architectural paradigm, it still demands considerable innovation and forward thinking. For this to happen, the enterprise requires a balance between the leaders and the technologists. The former have the resources to create an entrepreneurial culture and the latter respond with innovative solutions.
What are the facts? In a 2008 study, The Burton Group found that SOA projects in half of the 20 companies that took part in the study failed. On the other hand, some projects were "wildly successful", significantly contributing to the company's profile and profits. Another 30% were neither successful nor wholly failed. Failure was a tipping point where the company spent money and time with little return on investment (ROI). Technical success sometimes had no payoff. "Some users had executed nearly perfectly in terms of doing SOA on the IT side, but the initiative had not yielded increased agility, quicker time to market, or project savings because the business remained completely oblivious to the initiative" [REF-3].
A failure rate of 50% for SOA projects compared to other large information technology projects is high. Complexity notwithstanding, there shouldn't be a difference in success between an SOA project and any other enterprise-wide project. That there's a difference suggests that there's more in play than in building services. One reason may be fear. And the most fearful people are those with the most to lose. They are C-level executives and vice presidents who cling to the oxymoron of riskless capitalism. They may see their job like the children's game of chutes and ladders where one false step sends them down the chute to early retirement. SOA is a threat, a tool for the creative destruction of systems, roles, and departments. A measure of SOA success is the extent to which it does destroy the old order. A failed SOA is an SOA that works like a watch but still preserves the existing enterprise as if that SOA never existed.
Fear is both a gift and an illusion. It's a phantom that challenges us to be resourceful and flexible in the fires of refining change. Those who succeed in our profession are those who take to change in the same way that surfboarders take to the roll of an ocean wave. The velocity of change is always increasing. We can expect to learn at the minimum one new language and master five new tools each year that we work. If we see ourselves as cogs in a machine, we shouldn't be shocked if other cogs replace us and other machines replace that machine. That premise will prevent us from growing and adapting to new conditions, including the loss of a job. It comes down to first principles. If our premise is that man isn't a machine but a being, bound by mental and emotional limitations but that is also able to transcend those limitations through new experiences, then we'll see that the end of a job is not the end of the world but the start of countless new ones.
SOA is at the apex of that trend of change over the last half century, from a low-level assembler of moving bit strings from one register to another register to high-level integration of moving services into clouds. "Give me a place to stand on," the Greek mathematician Archimedes said, "and I will move the Earth." SOA promises to move enterprises in a better direction. But if change is the changeless reality of our lives, where should we stand? Where is our security? Machines become obsolete because they cannot learn, but we can learn. Basic suitcase skills of reading, writing, analyzing, speaking, and interacting that let people excel in the Renaissance and the Industrial Revolution are the same skills that we need today and tomorrow. And, as we learn, we'll find that seeking security in a firm or in wages or in a position is a mirage and that the only real security is that which lies within.
The shadow of fear is conservatism. "Watch out for people who are threatened because you are changing the way things are done (even when it is for the better), or because your solution will be replacing one of their legacy applications, or because they are territorial about their 'sandbox'. You want to fix the enterprise spaghetti, but don't forget that some people have their tribal knowledge and make their living by keeping the corporate data nervous system tangled" [REF-4].
The orthodox impulse has its place as a check against half-baked notions. "A wise man ought always to follow the paths beaten by great men, and to imitate those who have been supreme, so that if his ability does not equal theirs, at least it will savor of it," said Niccolò Machiavelli, the 16th century political theorist. Despite better paths, the corporation's majority will prefer inertia to progress. It's to the shame of most corporations that among that majority is the top management. "It ought to be remembered that there is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things," Machiavelli says. "The innovator has for enemies all those who have done well under the old conditions and lukewarm defenders in those who may do well under the new." The chasm between "have done well" and "may do well" threatens the status quo. Culture trumps strategy. The best strategies cannot prevail if they conflict with a company's cultural habits, values, and traditions.
What the "eyes can see the wits can solve" can only take us only so far. We need candor from experts with the knowledge to see beneath the surface of things, a frankness that won't come if they think that their job is on the line. From managers to subordinates and from subordinates to managers, we may get equivocation, where managers and subordinates don't lie but neither do they tell the truth. In a company of poker players, an SOA is impossible because there's only deceit, and thus no truth. It's dangerous to be right when the company is wrong. A dad who spanks his child for telling the truth or a firm that fires an employee for telling the truth will have children and employees who won't tell the truth. This climate accounts for the jelly that runs some corporations.
Mid-level managers who fear what SOA can do to their place and power will act predictably according to the law of the jungle. If they see lukewarm support for an SOA from the top and ardent support for an SOA from their peers and their reports, they'll marginalize or purge those who ardently support an SOA. They may act with subtlety or with ferocity, but as surely as night follows day they will act. The earliest stages of SOA planning are the most critical. It's a time when the reptilian brain can out maneuver the rational brain. A forceful mid-level manager can exploit the diffidence of top leadership by putting his boot on the tender plant of SOA. A weak chief will respond by funding a weak SOA science project or by aborting the SOA completely. Companies are stuck in every way that matters because they penalize progressives and reward reactionaries when they should be rewarding progressives and penalizing reactionaries. And that's why a winning SOA needs the chief executive officer's strong support to break the back of this kind of reaction.
The preconditions for SOA success include runtime governance, single source of truth data, canonical service models, a road map with specificity and ROI measures, a skilled staff, and business-driven use cases. Let's assign a zero to each goal. Thus, governance is zero, a road map is zero and so on. All of those zeroes add up to zero unless we put a '1' before that row of zeros. The missing '1' is leadership. Leadership is the edge that gives an SOA project more of a chance of success than a coin toss. The Burton Group observed that a successful SOA was often the result of a new chief information or technology officer followed by a reorganization of functions across service provider and service consumer domains. The chief officer is the vital link and is also often the weakest link. Cigna, the InterContinental Hotels Group, and other companies followed the path of high SOA expectations, initial failure, a new chief, and sustained success.
SOA as deployed technology is highly collaborative. But SOA as a strategic policy must start from the top. Should the board of directors care or even know what an SOA is? No, but they'll ask questions if their company's stock is skidding. They'll inquire as to systemic causes of what is happening and look for answers in their staff, structure, systems, and strategy. Sometimes an enterprise needs a decapitation rather than a band-aid to save itself. The board may bring in new blood on the business side if the partnership between technology and marketing isn't gelling. Only after the corporate door has spun a few times will it sometimes find the right people. Either the board of directors is serious or it isn't serious about transforming its organization. If it isn't serious, it must accept caretakers who nurture broken architecture, operational sluggishness, and eroding stock value. If the board is serious, it may need to replace its CEO and its CTO. They imperil their enterprise by defying the ancient wisdom that "Where there is no vision, the people perish" [REF- 5]. They're in a rut because their rut is the only rut they know. They infect the rest of the enterprise with their sickness of regressive rubbish that poisons any chance for a real SOA. They're not leaders.
Like seeing a flower on a glacier, leadership startles and impresses because it's so rare. But, just as all the other attributes that make us human are sprinkled evenly across the globe, so too is leadership. Leadership is easy to identify but it's hard to define. It's not the image of power. It's not a corner office or walking with a swagger. It's not exuding poise and charm, wearing three-piece suits or power dresses, conforming and never questioning. These are all signs of an enterprise in decline. Leaders are born, not made. They're the misfits, the nonconformists, the renegades, the square peg in the round hole, the devil's advocate, and the rebel to every bureaucrat whose mantra is "but that's the way we've always done things" and its fatalistic tautology "it is what it is."
Success rests with those who can create an inventive culture, tame requirements, and promote best practices. The essence of leadership is to cope with the demands of the moment, to delegate duties and to demand results, to see around the corner and over the horizon, to act firmly and decisively, to make tough calls that effect large groups of people, to shake up and shrink the permanent bureaucracy, to never be satisfied, to articulate a clear vision and a coherent plan, and to provide empowerment and encouragement that ensures success. Weak executives deny the obvious and are bound to friendships that hurt their company. Strong executives take command, not in the military sense of sending men into battle, but in the sense of running the bridge so that the ship is no longer drifting or taking on water. Leaders aren't alpha dogs, cat herders, or control freaks. They're psychologists who understand motivations and are able to motivate with a word or a glance but especially by example. Leaders walk the walk. Their strength comes from their character and that makes them the most dangerous of people. As Lawrence of Arabia said, "Those who dream by night in the dusty recesses of their minds wake in the day to find that it was vanity: but dreamers of the day are dangerous men, for they may act their dream with open eyes, to make it possible."
To make it possible, the chief should understand the firm's social dynamics, noting players who are blocking change, and inviting them to continue their careers elsewhere. This must be done wisely and generously, but it must be done, as it will send a message across the enterprise that change is coming. We must distinguish between dissent that fosters a more robust SOA and dissent that leads to a schism between advocates for an SOA and rejectionists for an SOA. These people might be talented, but out of conviction, they'll undermine the SOA with conscious or subconscious insubordination. To make change the priority is the fundamental problem. The answer is to drain the swamp of toxic attitudes. Without doing this, SOA will be at best like the campfire song where:
The cow kicked Nelly in the belly in the barn,
Didn't do her any good, didn't do her any harm.
One such attitude is the view that we can shepherd a non-intrusive, pain-free, and evolutionary SOA. This masks the body-and-fender work that needs to be done to prepare the way. A leader will make it clear that this effort could be intrusive and painful and, if it succeeds, revolutionary. It wasn't only physical destruction that sparked the industrial resurgence of Japan and Germany after World War II but also the destruction of old ideas and folkways. We cannot build up until we tear down. Leaders must sweep away the cobwebs of entrenched ways to foster entrepreneurial creativity. They must brace themselves for pushback that will come from unexpected quarters, sometimes from their most loyal and trusted allies. The pushback will come in different ways, in action or inaction, in anger or in silence. And, in the absence of leadership, their SOA will die as a triumphant, self-congratulatory, massive but ultimately inconsequential waste of capital.
Merely making SOA a bean-counting exercise or a chance to play with new toys has more to do with leprechauns and unicorns than the real world. Several thousand man-hours multiplied by one or two hundred dollars for each hour plus an ESB isn't an SOA. It's a Never Land where human intentionality is missing. A value-added architecture depends entirely on people, where each person has an inner world filled with doubts, fears, biases, and ambitions. As planning starts, the question shouldn't just be "does it pay?" but also "who's a threat?" It's up to the chief to get rid of those threats. And it's a test of moral courage to make these choices without fear or favor.
The chiefs' direct reports must have unity in the conviction that it's worth making the SOA journey. Like thoroughbred horses stamping their hoofs in a dark stable on a spring day, top-tier managers and experts are ready to explode out of the gate. But it's up to the chiefs to fling wide the gate to give them the maximum freedom to get the maximum results.
Among the rest of the enterprise, there'll be confusion and curiosity. So communication is fundamental. It's also a process. The ground needs to be prepared by all hands meetings, team meetings, one-on-one meetings, and ample training. Employees will swing their support behind new policy well told. The scaffolding of rhetoric must precede that of architecture. The leader should take great care looking for ways to convince the audience in the sincerity of his or her emotions, expressing excitement, enthusiasm, energy, confidence, determination, menace, and hope with oratory. The right words, time, and place will rally a skeptical audience. Persuasion must be more than facts and logic. Since it's true that the intellect of the audience is on the right side of the bell curve, that audience will reject an intellectual argument by itself, as it will neuter every argument with equally cogent counter-arguments. The appeal must be made not only to corporate goals but also to self-interest. This call to act must use the full range of personality and theatre to win the argument and the audience. Friedrich Nietzsche, the 19th century German philosopher, said that "Men believe in the truth of all that is seen to be strongly believed in" and the rank and file will march in a new direction if they see power and purpose. The speeches from the leadership will repeat a few basic themes: "We need an SOA." "I want an SOA." "You have my support." "We'll succeed." Trite? Sure. But it works, and that's all that matters.
Visualizing systems and services is an early step in planning an SOA. Many vendor tools graphically represent system decompositions, such as Visio [REF-6]. While pictures help us understand the system context, they are often vendor based and take time. Box-and-arrow diagrams are also inflexible, preventing drill downs from concepts to code. We're lost in the bowl of spaghetti that SOA is trying to eliminate, as in this example:
Figure 1 – Service-Oriented Spaghetti [REF-7].
It takes effort to fork over this pasta and to try to relate it to something that we should do. What if we had an SOA notation much like logical notation that uses statements rather than images? We process pictures more quickly than symbols as pictures more accurately mirror reality than symbols. Also, most people think visually rather than symbolically. But a notation is terse, mutable, and algorithmic. It allows us to look at the problem in a different way much like the way that we use algebraic geometry. Notation is good, images are better, but an eclectic approach that uses both images and notation to tell our story is best of all.
The volume of SOA design patterns continues to grow to encyclopedic proportions. This article, for example, was based on almost 1,400 pages of text. Our notation has the modest goal of trying to get a handle on these patterns. It's not meant to replace other notations. The idea is to reduce to several lines of what's expressed elsewhere in several pages. It's a short hand for working with multiple patterns on a yellow pad or a whiteboard using only straight lines and simple words. The value of this notion will become more apparent as we try to relate principles and patterns to a use case in the second part of this article.
An alternative, and perhaps a better approach, is for us to understand the SOA design patterns with sufficient depth so that such a notation isn't necessary.
The guiding goals for this notation are clarity and simplicity. Thus, three clear, simple rules are proposed that will let us sketch an SOA using patterns at any level of detail:
- SOA notation consists of sentences that are made of words and connectives with one sentence on each line.
A word consists optionally of a prefix and a name or names. A prefix is an abbreviation of an SOA element. "S" is our abbreviation for 'service'. A name is usually a business entity name. We can also add an object suffix after a period such as ".wsdl" or ".xsd". We want to be crisp but never cryptic.
- We use three kinds of connectives. They show composition, transmission, and connection:
*(See Appendix A for notes on this notation.)
Principles and Patterns
A principle "represents a highly recommended guideline for shaping solution logic in a certain way and with certain goals in mind" [REF-8]. A pattern is a proven solution to a common design problem.
Patterns follow a power law. The larger the system, the more design patterns there are. However, the number of patterns grows sub-linearly. For example, five services might have five different patterns while ten services might have eight patterns and twenty services might have twelve patterns. Scaling optimizes the number of different patterns. This delivers efficiencies in the same way that large cities need fewer gas stations per thousand people than do towns [REF-9]. In the same way also that buying or renting a cloud or an enterprise business suite can save time and money, so too will using patterns save time and money. The only difference is that patterns are logical constructs rather than physical objects.
We knit the patterns together to fulfill enterprise-specific proprietary requirements. Our solution will look like a disk defragmentation were the black areas are seams that need tailoring to complete the solution.
Patterns can also picture the same concept in different but valid ways, akin to how a poet and a scientist try to describe the same sunset. We're not so much pealing an onion as we're turning a prism to reflect different aspects of the same essence. Our model isn't a jigsaw puzzle where we're looking for pieces to complete a picture that is the reality. Rather, it's more like a shattered mirror where each shard imperfectly reflects reality. In the section "rendering a solution", we see an example of this in design patterns numbers four through seven.
Using our notation, we'll see how patterns synergize to the Standardized Service Contract principle (SSCP) and the Service Discoverability principle (SDP). SSCP has 37 design patterns and SDP has four patterns including one pattern that it shares with SSCP.
The Standardized Service Contract Principle
The Standardized Service Contract principle ensures that services "within the same service inventory are in compliance with the same contract design standards." Under this principle, we develop contracts before developing the service logic according to design standards within a service inventory. A service contract is the technical interface of a service. It has published documents that express meta information about a service that establish an interface into the functionality offered by the service's capabilities. Contract design standards shape the structure of WSDL, XML Schema, and WS-Policy definition documents that make up a Web service.
Figure 3 – Contract First Development.
We follow this process in using our notation to describe how we build a service contract:
- State the nouns: Developer, service contract, solution logic.
- State the verbs: Designs, imports, builds.
- Bind the parts to the whole.
- Write the sentences using nouns, verbs, and connectives that express the solution, as this example shows:
Consider a proposed claims service. The firm's claims department has workflows and systems that provide claims processing. Let's build an autonomous service S_Claims that users discover from a service registry.
Figure 4 – XSD-WSDL-XML Contract Design.
The Service Discoverability Principle
To promote discovery of the claim service, we invoke the Service Delivery principle where services "are supplemented with communicative meta data by which they can be effectively discovered and interpreted" [REF-13]. This principle supports discovery by using a service registry as the central repository of service metadata.
Figure 5 – Service Discoverability.
The second article in this two-part series develops the claims scenario using principles and patterns.
[REF-1] Hurwitz, Judith, Bloor, Robin, Kaufman, Marcia, Halper, Fern, Service-Oriented Architectures for Dummies, Wiley, 2009, 251-341.
[REF-2] Wik, Philip, "Architecting Service-Oriented Technologies", Service Technology Magazine, December, 2011. http://www.servicetechmag.com/I57/1211-2
[REF-3] Meehan, Michael, "SOA Adoption Marked By Broad Failure and Wild Success," SearchSOA, 2008. http://searchsoa.techtarget.com/news/1319609/SOA-adoption-marked-by-broad-failure-and-wild-success
[REF-4] Veerman, Erik, Moss, Jessica M, Knight, Brian, Hackney, Jay, Microsoft SQL Server 2008 Integration Services Problem – Design – Solution, Wrox, 2010, 8.
[REF-5] Proverbs 29:18, The Bible (KJV), 1611.
[REF-6] Official SOA Visio stencil. http://www.soapatterns.org/legend.php
[REF-7] W3C Working Group Web Services Architecture, 2004. http://www.w3.org/TR/ws-arch/#gsom
[REF-8] Erl, Thomas, SOA Principles of Service Design, Prentice-Hall, 2008, 28. Principles are illustrated in the following poster: http://www.soapatterns.org/reference_posters.php
[REF-9] Arbesman, Samuel, "The Mathematics of Lego," Wired, January, 2012, http://www.wired.com/wiredscience/2012/01/the-mathematics-of-lego/
[REF-10] Wik, Philip, "Thunder Clouds: Managing SOA-Cloud Risk", Service Technology Magazine, October, 2011. http://www.servicetechmag.com/I55/1011-1
[REF-11] Wik, Philip, "Service-Oriented Architecture and Business Intelligence," Service Technology Magazine, August, 2011, http://www.servicetechmag.com/I53/0811-2
[REF-12] W3C XML Schema Definition Language (XSD) 1.1 Part II: Datatypes, 2011. http://www.w3.org/TR/xmlschema11-2/
[REF-13] Erl, Thomas, SOA Principles of Service Design, Prentice-Hall, 2008, 369.
Appendix A: Symbolic Notational Semantics
- Connectives are the sinews that hold a language together, such as "the", "but", and "is". The connectives < isn't a "less than" operation. Instead, we use it to aggregate parts into a whole. Composition can be logical, as in composing capabilities to solve a problem, or physical, as in building a Web service. The whole is on the left side and its parts are on the right side of the connective. Thus, we don't use the > symbol. We also don't use curved or vertical lines.
- The connective ← or → is in the dative case and is expressed by a verb and the actual or implied preposition "to" or "from". ("The service sent the file to the system", "The service sent the system the file.") This connective is used for messaging, workflows, and file processing where we state what is getting transmitted.
- The connective ___ is in the genitive case and shows a relationship between two nouns indicating ownership, belonging together, or a feature or characteristic ("The color of the book", "The teacher's desk". "The capability of a service."). It's our most flexible connective that means anything that doesn't involve composition or messaging. This symbol, therefore, stands for phrases such as "is realized by", "is equivalent to", "interacts with", "gets requirements from", "derives from", "influences", "operates on", and "requires". It's also a transitive verb in the active voice that shows an action between object and subject ("X___Imports___Y") or a state of being ("X___Exists" or "X___25%"). The inclusion of a verb is optional.
I wish to thank the following individuals who reviewed and critiqued my article: Steve Wisner, Director, IT, Genworth Financial and Errol Ryland, Director, MSS Technologies, Inc.