ServiceTechMag.com > Archive > Issue XXXV: January 2010 > SOA Governance: How Best To Embrace it - Part II: Governance Lifecycle
Farzin Yashar

Farzin Yashar

Biography

Farzin Yashar is a Certified Solution Architect with the SOA Advanced Technologies of IBM's Software Group. In this capacity Farzin helps IBM customers in adoption and implementation of Service Oriented Architecture, creating Enterprise Architectures and SOA Roadmaps.

Farzin has 28 years of experience in the field of Information Technology in multiple industries. Since 1996 Farzin's special focus has been Enterprise Application Integration, Enterprise Information Integration and Process Automation using Service Orientation.

Farzin has a BSc in Chemical Engineering and an MSc in Computer Science. He has lead creation of the architectures for Banking, Insurance, Transportation, Medical Systems and Software Development Tools. He has worldwide experience in people management; he has directed teams of over a 130 individuals. Finally, he has held the title of Technical Director of Alliances building partner relationships and alliances. In this role he has presented and spoken at various Integration Events.

Contributions

rss  subscribe to this author

Bookmarks



SOA Governance: How Best To Embrace it - Part II: Governance Lifecycle

Published: January, 2010 • SOA Magazine Issue XXXV
 

Abstract: In part one of the series, we learned about governance in general and we discussed Enterprise, IT and SOA governance and how they are related. (Part I of this article series can be viewed at http://www.soamag.com/I33/1009-3.php. In part 2, we walk through governance lifecycle and how best we should organize for SOA and SOA governance.


Services Lifecycle and Governance

Services are the heart of the Service Oriented Architecture. Therefore, SOA governance has a special focus on governing services. In this section we define the lifecycle of services and we discuss the fundamental tasks for establishing governance.


The Service Lifecycle Overview

All services go through a consistent set of steps, starting from the creation of an original concept, all the way through analysis, design, development, testing, deployment then eventual retirement, as shown in Figure 1 below:



Figure 1 – The Service Lifecycle State Diagram

This is actually a state-transition diagram in which each valid state is well-defined, and a specific activity has to be performed in order to move from one state to the next.

Like all models, this represents a simplified view of reality: there are no intermediate states like 'under development' for example, since the definitions of such states are subjective and ambiguous.

This development model, however, is an excellent model to define a SOA development approach:

  • Each activity that causes a transition from one state to another can be well defined, and then assigned to a suitably-skilled individual or team to be performed repetitively and to a continuously high standard.
  • Since the stable states can be well-defined, it is practicable to have Quality Assurance (QA) reviews (sometimes known as 'control gates' or 'control points') to determine that each activity has been performed to sufficient quality for migration to the target state to be confirmed reliably
  • Since the activities are repeatable, it is possible to both predict and monitor the development work efforts of an IT solution that requires a significant number of services to be deployed

The main stages in the service lifecycle are:

  • Identification - When the initial need is recognized and the requirements are specified
  • Specification - When the detailed requirements and high-level design are captured
  • Realization - When the service is constructed and deployed
  • Operation - When the service is managed through the rest of its lifetime until it is eventually re-versioned or retired

It is important to notice that not everything can be made into a service, and not all services could, or should be exposed. As part of governing the SOA lifecycle, candidate services need to conform to a service litmus test, whose answers can establish a set of criteria to qualify a service. This litmus test contains questions such as: Is this service reusable and how many consumers will it have? Is this service in line with the goals of the enterprise?, and so on. Once a candidate service is selected, then the next step is to decide if it should be exposed. Exposure of a service means availability for consumption by others in a visible manner, such as through a registry. There are many reasons for not exposing a service. For example, infrastructure services may not be required enterprise-wide and, therefore, not exposed; or, security restrictions and policies may drive the enterprise to expose a service for visibility to only a select set of users.

The service lifecycle is consistent with software/system development methodologies (SDM), like rapid application development (RAD), Rational Unified Process® (RUP®), iterative, spiral and even agile development methods. For example, the Rational Unified Process® (RUP®) consists of the following four incremental phases:

  • Inception
  • Elaboration
  • Construction
  • Transition

Effectively, these phases are performed for each service being created. The phases are not an exact match because the operation stage of the service lifecycle extends somewhat beyond the RUP transition phase.



Figure 2 – RUP Phases

The following diagram depicts the service lifecycle in detail. As indicated, the orange boxes are QA-related steps to ensure effective governance of the SOA development process. The orange boxes could be considered Policy Enforcement Points (PEP). For example, we could have a policy stating that acceptance testing must have 95% success rate prior to exposing the service enterprise-wide. Also, please notice that the bottom part of Figure 3 shows activities that are not part of an SDM, meaning current methodologies should be extended, or augmented to embrace SOA.




View Larger Image
Figure 3 – The Complete Service Lifecycle in Detail (Component Business Model (CBM), Service Level Agreement (SLAs)

Initiating SOA Governance

In order to establish SOA governance, a few fundamental tasks need to be considered.


Task: Define Charter for SOA Center of Excellence (CoE)

Wikipedia (http://www.wikipedia.org/) defines "a Center of Excellence (CoE) or Center of Competence as the formally appointed, and informally accepted, body of knowledge and experience on a subject area. It is a place where the highest standards of achievement are aimed for in a particular sphere of activity"

The SOA CoE is comprised of People, Process, Technology and Services and provides the leadership for successful implementation of services in the enterprise.

Essentially, the SOA CoE's primary mission is change management: Everyone involved in the CoE, from the executive leading it to the professionals that design, develop and test services, needs to be an agent of change.

The SOA CoE should aim to establish services as key enterprise assets by:

  • Providing leadership in SOA vision and execution through a single, cross-organizational, cross-functional authority for SOA related planning and implementation
  • Creating and nourishing expert level SOA skills towards best practices, technology, standards and related SOA methodologies
  • Recommending and helping mobilize organizational and governance models for ongoing SOA adoption
  • Meeting the agreed target maturity level of service-orientation within the lifetime of the CoE
  • Designing infrastructure enhancements for managing the usage of services in areas of security, monitoring, performance, versioning, and shared usage
  • Providing enhancements to IT processes to address funding, sharing, and incentives for sharing and reuse of services as well as for the identification, design and specification of services
  • Helping plan education and training to broaden SOA delivery skills
  • Communicating the strategic intent of the IT department to develop SOA competency into a strategic, core competency for the long term competitive advantage of the organization as a whole

Task: Identify Roles and Responsibilities in Support of SOA

Like any transformation, SOA adoption introduces new challenges. Successful adoption of Service-Orientation requires changes to organizational roles and responsibilities. Organizations must invest time and effort in creating an appropriate organization and support structure to enable a smooth adoption of SOA principles. A first step is establishing a Center of Excellence (CoE) focused on SOA.

We will discuss CoE in detail in a separate section. It is suffice to say the SOA introduces new roles and responsibilities such as Service Registrar or Service Architect.


Task: Define Service Ownership and Development Funding Model

A clear understanding and communication of funding and ownership models is necessary to ensure an optimal adoption rate for SOA. The funding model should be established in so as to encourage sharing, reuse, efficient integration and simplicity. In the absence of clear service ownership and funding model, ownership may default to today's silo application and product lines.

Consider the example of unifying the user experience across multiple lines of business:

In an optimal SOA-based solution, we create a set of shared services across all lines of business in support of a uniform user experience, including services that provide unified access to data.

However, creating such services is not just a technical issue. The lines of business need to be involved to answer the following types of questions:

  • Who owns the data and is there agreement to allow the service access to the data?
  • How can permission to access the data be obtained?
  • Who should fund the shared service? Who owns it, or sponsors its development?
  • Who's responsible to fix it if it breaks?
  • How is the business going to motivate the separate lines of business to reuse enterprise assets and shared business services?
  • Who makes a decision on whether a service can be accessible to other applications? What happens if potential users of the service disagree on its content?

Having a well-established service owning and funding model helps resolve such issues consistently and efficiently. In particular, great care must be given to the data access needs of future, unintended users.


Task: Identify Success Factors, Enablers and Reuse Motivators

Before considering the success factors and motivators for SOA and service reuse, it is important to understand the traditional challenges and constraints with shared service models and reuse.

  • Lack of agreed upon standards, vendor product/platform interoperability
  • Semantics for cross-boundary services, service discovery and visibility into services
  • Challenges with licensing models in shared services model: Given the existence of systems using Commercial Off-the-Shelf (COTS) products that have a pre-established licensing model (for example, per CPU, named-users, seat-based), exposing existing capabilities as services (or operations) and then planning 'future' service consumers has significant cost, legal and organizational issues that deter service sharing. Such obstacles discourage interested service providers.
  • Lack of certification and support: a shared services model historically adds planning overhead in the areas of availability, reliability and security and do not guarantee a high quality of service through a certification.

Task: Design Policies and Enforcement Mechanisms

For the successful adoption of SOA, the SOA Governance Model requires complete, visible support of sponsors and other stakeholders including, but not limited to, the SOA CoE.

Without adequate checks and balances firmly in place, the foundational processes, standards and best practices created by the SOA CoE are less likely to be applied in practice.

Particularly in the early adoption phases of SOA Governance, it is important that regular vitality checks, walkthroughs, and peer reviews ensure that information and approach flows smoothly. Governance enforcement requires empowered entities at multiple levels of enforcement to effectively govern the standards established by the SOA CoE.

The following are some typical and recommended enforcement entities that could have specific responsibilities attached to them

  • SOA Steering Committee - strategic and executive guidance of the SOA journey across the enterprise. This committee ideally should be a subcommittee of an enterprise level governance committee.
  • SOA Control Board - tactical guidance on specific tasks performed for the SOA journey. Again, this should be a part of the enterprise level governance control board.
  • SOA CoE (Center of Excellence) Advisory Group - day to day guidance of the SOA journey.

Service Management and Monitoring

One of the major common mistakes enterprises make is embarking on SOA by writing services/Web Services without thinking about service management and monitoring. This has caused a lot of grief in many organizations. Uncontrollable number of services and many copies of the same service (duplicate functionality) with only minor variations are prime examples of lack of service management and monitoring. For this reason, we are addressing this topic in the service lifecycle. Service management and monitoring should be considered at Design-Time, not as an afterthought associated solely with IT operations.

The key to managing services efficiently is to consider them as another resource type. Services need to be secured, deployed, monitored, versioned, and they should have formal SLAs (service level agreements) associated with them.

Services introduce additional management challenges emanating from the composite nature of the solutions they participate in. Challenges around having to manage the interdependencies among services and maintaining the expected Quality of Service (QoS) becomes as important as managing the resources that comprise the services themselves. As stated in section 2, policy management provides a mechanism to allocate IT resources according to defined policies established by the enterprise. Therefore, in processes, identification of policy decision points (PDPs), where events are defined and decisions are made and policy enforcement points (PEPs), where policies are executed and monitored are essential.

While the underlying technology and standards in support of SOA provide many options to architect and manage the services, it still takes time and effort to actively design the management strategy and execute the strategy consistently.

Service Management in the Context of IT Management and Operations



View Larger Image
Figure 4 – SOA Solution Abstraction Layers

The above graphic depicts a simplified model of atomic and composite services as entities and shows the abstract positioning and relationship amongst systems, components, processes and the consumers.

The intention is to establish the fact that managing services effectively requires management of the service inter-relationships and dependencies, as well as a support model that reflects understanding how services relate to each other and to the IT infrastructure and business process layers.

Flows within the services environment can be controlled through management mediations such as log, filter, route, and transform. Centralized service management policies can define how such mediations are applied.


Service Management and Monitoring Tasks

Complete write-up on Service Management and Monitoring is beyond the scope of this paper. Following is description on two very fundamental tasks that support SOA governance-related goals.


Task: Certification and Publishing of Services

Adding a new 'service' entity to the set of enterprise assets requires a way to announce its status and display details to its potential consumers. Service certification and publishing is a vital process for SOA adoption. One of the biggest advantages of this task is that it stops arbitrary introduction and deployment of services.

While this is not an entirely new step within IT operations procedures, there are aspects such as the loosely coupled nature and new interoperability challenges that are distinctly different from client-server systems and which should be addressed. This is a recognized area where technology standards and tools (registry, repository etc.) are quickly evolving.

Service certification is a critical part of SOA governance. The main objective of certifying services is to actively encourage their reuse by warranting overall service quality and assuring potential new users of the service that it will be fully supported and appropriate to their needs.

Such service assurance may be achieved through requiring additional accountability requirements and by publishing the details of such additional certification for the benefit of potential consumers of the service. Service certification should be carried out under the control of the Quality Assurance department and should include operational tests that approximate the actual deployment architecture associated with its initial intended use. Additional information related to alternative uses, including consumption of the service over a wide area network if that was not the initial deployment architecture can be explicitly mentioned, for example.

The certification step builds on the effort already undertaken during the service identification around reuse and due diligence in identification of potential service consumers.

Various details that were developed and documented during the service design and development phase such as SLA and QoS will contribute to the certification process. Certification formalizes the QA process prior to physically publishing the service as being 'enterprise ready', with an assured quality of service and a full and complete set of support materials.

For example, certification ensures that information such as a version number, ownership, who is accountable for the service, and that classification and availability of the service are published as mandatory service metadata.

Certification and publishing services also serve the following purposes:

  • It helps the IT operations staff capture and document service related metadata. This benefits visibility and system analysis for later projects that may support unintended users
  • During early stages of SOA adoption, it provides a feedback channel from IT operations to design and development teams. This may result in additional design time discipline towards services by prompting reuse questions and considerations associated with the operational aspects of service deployment
  • The process of certification ensures the correctness of service contracts created as part of service specification
  • It establishes the source for communication related to service lifecycle, usage and subscription information
  • It demonstrates that the service has adhered to the required sequence of steps for the service lifecycle status, including intermediate peer reviews. The purpose of defining formal service statuses and allowed status transitions is to clearly provide information related to the stage in which the service is at a given point in time.
  • The certification review results in a pass or fail status for the service in consideration. Pass status indicates that the service is ready to be published. Fail status indicates that further work needs to be done before this can be certified as a service.

NOTE

The certification process also applies to services provided by third parties. Third party arrangements become problematic if the provider changes. Consumer organizations should establish policies that can cope with a chance in the QoS during initial negotiations with the provider.


One way to buffer an enterprise from an uncontrolled or volatile change is to hide the third party service (or services if multiple providers are an option) behind a mediation layer that then manages external / provider volatility without impacting the core business. If a service provider changes, the governance processes might include an introspection of the registry/repository for the mediation "provider" which is under the enterprise's control. Any change in service delivery, interfaces, endpoints, etc. would trigger an announcement to affected consumers.

Some of these requirements might not be established until the service is actually implemented and Service Level Agreements are negotiated with the Service Consumers.


Task: Versioning Services


NOTE

Service versioning is an area that has not yet attained practice maturity. Hence, the context and the guidance below may be considered as leading practices known so far.


Versioning of services requires a mechanism for change tracking and providing visibility into service modifications. Given the nature of SOA drivers (reuse, agility, business alignment, etc) and enablers (technology components, standards), it is natural to expect changes to production services over time. These changes may be a result of a variety of factors such as new requestor, change in compliance/security requirements, and others.

Design decisions need to be made carefully before creating new versions of existing services. Backward compatibility is required in cases where a contract or practical necessity defines the need to support previous versions.

Support for some older service versions may be dropped when the contractual obligation completes or when the need to support the service ends. Some consumers may successfully demand specific versions of services to be created for their sole use.

Unlike the common state in current application scenario, in a services world there are aspects that are native to the programming model that help in versioning. New solutions for running and orchestrating processes (Process Servers), support separation of concerns such as process instance from service implementation.

For example, by specifying a valid "from-date", the Process Server will be able to decide, which of all the deployed versions of a process template to use when creating a given process instance. Importantly, once the instance is created, it will run against that version of the template regardless of what other versions of the process are subsequently deployed.


Organizing for SOA & SOA Governance

SOA is the result of rational and logical evolution of the IT industry, in which careful planning and organization play a huge role. According to Gartner, research indicates that through 2010 a lack of a working SOA governance arrangement will be the most common reason for failure of SOA deployments, with an astounding 80% probability of failure. Nowhere in the annals of SOA best practices is there any notion neither advocating, nor supporting a revolutionary or big bang approach in transitioning to SOA.

The concept of Project (or Program) Management has been around for a long time and most companies with a reasonably sized IT department, or IT activities have a formal Program Management Office (PMO). This is a demonstrated commitment by these companies to apply a disciplined project management methodology to project execution.

The need for an official Program Management Office function has evolved as organizations struggled to make traditional project management adequate to handle multiple interdependent projects having touch points throughout the enterprise. Service orientation demands even higher stakeholder involvement and coordination as the organization tries to create business services, used throughout the enterprise with services, to collaborate and participate in multiple interdependent processes.

As noted in the Gartner statistic above, the risk of failure to SOA projects is high; hence a need for a well organized, strong programmatic approach is required. Organizing for SOA requires an understanding of Program Management, the concept of an SOA Center of Excellence (CoE), and the Roles and Responsibilities that a CoE should consider, and the role of Program Management in the deployment of SOA services.


Program Management

Program Management is usually run out of an enterprise wide Program Management Office (PMO). The Project Management Institute (PMI) defines project/program management as the application of knowledge, skills, tools and techniques to project activities to meet project requirements. PMO ensures that projects within each department or division are managed the same way and are working toward the same goals. This leads to greater efficiency, reduces redundancies and costs and helps the bottom line; very similar to the goals of SOA.

The focus of program management can be summarized in the following 3 categories.

  • Vision and Strategy - Create or refine a business vision and set of enabling strategies. Identify enabling programs, define them in multiple dimensions, quantify their potential value, and justify related costs.
  • Mobilization - Establish a governing structure and roles for the program and its projects. This includes developing /implementing a resourcing plan, ramping up staff, and establishing a Program Management Office (PMO). It also includes defining / implementing the necessary physical and technical infrastructure to support the program staff's work.
  • Planning - Define a strategy to develop and deliver a project plan, calculate overall and detailed sizing, define a work schedule, and establish milestones for major checkpoints. Then, establish program practices for oversight, risk control, quality, and financial management.

Governance plays an important role in Program Management. Usually, in companies with strong Program Management, governance is part of the culture. To this end, implementing SOA can greatly benefit from a strong Program Management discipline. However, having a PMO is not sufficient for SOA because of its unique needs.

SOA Center of Excellence (CoE) is described next. CoE, in fact, complements PMO by having a complete focus on creating and deploying enterprise level services that will be used in the current as well the planned projects. Although, SOA CoE roles and functions could very well be defined within the PMO structure, but it is recommended for SOA CoE to have its own identity at least at the beginning of the SOA journey. This will have the added benefit for companies without a strong PMO; enabling them to establish their SOA CoE and start their SOA journey.


Center of Excellence (CoE)

The SOA Center of Excellence (CoE) is a cross-organization team that guides IT investment, design decisions and implementation towards the strategic shared IT solutions targeted by the SOA Vision and Strategy.

Services must be created as key enterprise assets. The CoE assures this by:

  • Providing leadership in SOA vision and execution through a single, cross-organizational, cross-functional authority for SOA related planning and implementation
  • Creating and nourishing expert level SOA skills towards best practices, technology, SOA standards and related SOA methodologies
  • Recommending and helping mobilize organizational and governance models for ongoing SOA adoption
  • Achieving the agreed target SOA maturity level of service-orientation within the lifetime of the CoE
  • Designing infrastructure enhancements for managing the usage of services in areas of security, monitoring, performance, versioning, and shared usage
  • Providing enhancements to IT processes to address funding, sharing, and incentives for sharing and reuse of services as well as for the identification, design and specification of services
  • Helping plan education and training to broaden SOA delivery skills
  • Communicating the strategic intent of the IT department to develop SOA competency into a strategic, core competency for the long term effectiveness of the organization as a whole

The CoE team will be initially responsible for guiding and augmenting the SOA journey, including selecting which projects adopt a SOA approach, and what services need to be developed. Over time, as SOA becomes 'business as usual', the duties of the SOA CoE may be taken by other bodies such as Architectural Board.


Roles and Responsibilities

There is no standard organizational model for SOA CoE. Leading SOA practices suggest the following responsibilities for SOA CoE.



View Larger Image
Figure 5 – SOA CoE Responsibilities

An appropriate structure to deliver the above responsibilities is shown in Figure 4.2. Note that this structure shows roles rather than individuals. Not all of these roles need to be full-time, and it is perfectly possible for individuals to have multiple roles in the CoE. Five to ten individuals could be a typical size for a CoE when SOA is in its early stages.



View Larger Image
Figure 6 – Example of SOA CoE Roles

Conclusion

In part 2 of the series, we walked through governance lifecycle and how best we should organize for SOA and SOA governance. In the final part of the series, we will cover governance maturity, tooling, vitality and end this article series with governance success patterns.


References

[REF-1] "The Next Revolution in Productivity", Harvard Business Review Article, 1 June 2008, Ric Merrifield, Jack Calhoun, Dennis Stevens, http://harvardbusinessonline.hbsp.harvard.edu/relay.jhtml?name=itemdetail&id=R0806D

[REF-2] IBM EA Academy Study Team, Orlando Workshop, 12th-13th March 2004

[REF-3] See COBIT Framework, Page 19: http://www.isaca.org/AMTemplate.cfm?Section=Downloads&Template=/ContentManagement/ContentDisplay.cfm&ContentID=34172

[REF-4] SOA and the Emergence of Business Technology: How Business Services are Changing the IT Landscape. Farzin Yashar & Robert Laird. http://www.soamag.com/I4/0207-3.asp

[REF-5] Several writings and presentations from Clive Gee, Phd.


Acknowledgements: I would like to thank the following individuals who reviewed and contributed to the original paper: Les Robinson (Boeing), Jay Pollack (Computer Sciences Corp), Bob Riley (Harris Corp), David Almeida (Harris Corp), Mike Moomaw (IBM), Robert Laird (IBM), John Falkl (IBM), Al Secen (Lockheed-Martin), Chris Francis (L3 Communications), Peter Bostrom (Oracle), Kathy Kearns (SITA) and Mansour Rezaei-Mazinani (SITA). My special thanks go to Mr. Robert Laird who generously volunteered his time and expertise providing me with his input on every section of this paper.