ServiceTechMag.com > Archive > Issue XX: July 2008 > An SOA Vendor Evaluation Methodology
Alex Roussekov

Alex Roussekov

Biography

Alexander Roussekov is a Senior SOA Architect with the Firstservis Ltd Pty. In this capacity Alexander helps Firstservis customers in adoption and implementation of service-oriented architecture, creating enterprise architectures and SOA roadmaps.

Alexander has 20 years of experience in the field of Information Technology across multiple industries. His main expertise is in the field of the Service Oriented Architecture, Enterprise Application Integration, and Enterprise Systems Architecture Management.

Alexander is an experienced Consultant/Enterprise Architect in the Private and Government sectors with innovative solutions in respect to SDLC Management, BI Analysis and Implementation, J2EE Architecture and Infrastructure.

Since 1998, Alexander’s special focus has been Enterprise Application Integration, Enterprise Information Integration, and SOA. Alexander has a MS in Computer Science and Software Engineering. He has led and participated in large-scale international software projects in Australia, UK and USA.

Alexander can be reached at: alexzous1@optusnet.com

Contributions

rss  subscribe to this author

Bookmarks



Alex Roussekov An SOA Vendor Evaluation Methodology

Published: July 18, 2008 • SOA Magazine Issue XX
 

Abstract: This article describes a methodology for evaluating SOA vendors that, when applied to analytical sources such as Gartner Group or Forrester reports, can help shortlist appropriate vendors in an objective way. This objectivity is ensured by a ranking and scoring system based on a set of evaluation criteria within the corresponding evaluation models. Toward the end of this article a final short list is revealed based on the correlation between preceding short lists obtained from each model.


Introduction

The following vendor evaluation methodology is intended to be only an initial step of an actual evaluation process that you may want to carry out in order to filter vendors. The approach is based on the analysis of results from studies carried out by the Gartner Group and Forrester Research identifying the "Best-of-Breed" vendors available in the SOA market.

The procedure that we will follow in the evaluation process consists of the following steps:

  1. selection of appropriate sources from Gartner and Forrester
  2. ranking vendors based on the results in the Gartner and Forrester sources
  3. short-listing of vendors
  4. correlation of the short lists
  5. review of the evaluation methodology results

Gartner Group Reports

Three major reports published by the Gartner Group in 2007 are being included in our assessment:


Magic Quadrant for Application Infrastructure for Back-End Application Integration Projects, 2Q07

This report puts the focus on integration with EIS and legacy systems relevant in the current IT landscape. The selection of vendors in relation to appropriate integration frameworks is considered critical. In this context the predominant types of software employed for back-end application integration usage are integration suites and ESBs.


Magic Quadrant for Application Infrastructure for Composite-Application Projects, 2Q07

Composite applications are perceived to be the next technological step on many enterprise SOA roadmaps. It essentially involves building a new style applications based on compositions of services that provide full end-to-end business process automation and reuse of exiting business processes and services. Gartner has identified a composite applications infrastructure market as one that offers integrated suites of application infrastructure components with end-to-end functionalities of user interface management, composition and orchestration, integration and adapters, interoperability and messaging protocols, container technology, etc.


Magic Quadrant for Application Infrastructure for New Service-Oriented Business Application Projects, 2Q07

The topic of this report represents a further step in a typical SOA implementation by addressing the actual construction of service-oriented applications from scratch. The Gartner Group characterizes this market segment by focusing on service orchestration with less of an emphasis on integration. The major product categories expected to constitute the market are application servers, horizontal portals, design, development, modelling and governance tools, SOA infrastructure tools and metadata repositories, business process management (BPM) and orchestration tools, various user-interaction tools and high-end performance enhancement tools.


Gartner Quadrants

The Gartner Group methodology is based on "Magic Quadrants" that position and categorize vendors with respect to their ability to execute and their completeness of vision.

Each of the three major categories represented by the preceding reports is assumed to have an equal weight in the overall evaluation and the same critical impact for the future state of integration and application of services.

Conceptually, this reflects the connections between the past, present and future that corresponds to the evolution path from "Back-End Application Integration projects" to "Composite-Application Projects" and subsequently to "New Service-Oriented Business Application Projects". The importance of these categories is addressed by the Enterprise SOA Maturity Reference Model [REF-1], which also describes the business requirements, key challenges, exit criteria and success factors pertinent to each category.


How We Use Gartner Results

The ranking methodology employed by this study is based on the following rules:

  • Only vendors scoring and represented in each of Gartner's three categories are considered.
  • Only the first six vendors identified in the "Leaders", "Challengers" or "Niche Players" magical quadrants are considered.
  • Vendors are ranked by their position within Gartner's magic quadrants in both the "Ability to Execute" and "Completeness of Vision" dimensions.
  • A ranking score is assigned to each vendor in each dimension as determined by the rank position. The score from the "Challengers" magical quadrant is increased by one, and the score from the "Niche Players" magical quadrant is increased by two.
  • The total score is assigned for each of the three categories calculated as the sum of the scores in the "Ability to Execute" and "Completeness of Vision" dimensions.
  • The final ranking is produced by gradation of grand totals of the totals of each vendor across each investigated category.
  • Vendors not scoring in all categories are dropped from the evaluation.
  • Vendors with lighter scores are considered superior.

You can view graphical representations of the Gartner reports in [REF-1], [REF-2] and [REF-3].


The Scores

Table1 displays the vendor ranking carried out in accordance with the evaluation rules defined previously and based on the results provided in the Gartner reports. Table 2 summarizes the results from Table 1 and shows the total scores for each vendor. (Note that vendor scores are increased by 1 or 2 when the vendors are from the "Challengers" and "Niche Players" Magic Quadrants respectively.)



Table 1
*Vendor is positioned in the "Challengers" Magic Quadrant
**Vendor is positioned in the "Niche Players" Magic Quadrant

The "Combined Ranking" column in Table 1 is produced by adding the ranks of each vendor in respect to a given category. The lower the score, the higher is the rank in the classification ladder. For example, Microsoft is ranked first with respect to "Ability to Execute" criteria and sixth with respect to "Completeness of Vision" criteria based on Gartner's "Back-End Application Integration Projects" category. The overall score for Microsoft is 7, which positions them second in the "Combined" ranking column.



Table 2


Forrester Research Results

Forrester Research Inc. has published a report that analyses the positioning of integration-centric business process management suites (IC-BPMS), a product concept that addresses the link between SOA, legacy integration and BPM. This linkage is reflected in the ranking of the products evaluated. The Forrester Wave Model [REF-5] provides three dimensions used as evaluation criteria:

  • Current Offering – The vendors' capabilities in the areas of connection, BPM, and SOA.
  • Strategy – The vendors' vision and product roadmap, as well as associated solution costs and strategic alliances.
  • Market Presence – The vendors' installed customer base, new customers in the past quarter and year, and the level of global sales and implementation support.

The evaluation of the IC-BPMS market, as defined by Forrester Research, is recommended to be a starting point for evaluation purposes, and vendor analysts are encouraged to adapt the criteria weightings to fit their individual using the Forrester Wave Excel-based vendor comparison tool.


How We Use Forrester Results

This study incorporates the results from Forrester presented in the report [REF-5] and applies the evaluation methodology based on the assumption that all the dimensions (criteria) of the model have equal weightings.

The ranking methodology employed by this study is based on the following rules:

  • Vendors are ranked by their position in the Forrester Wave Model evaluation dimensions: Current Offering, Strategy and Market Presence.
  • Only the top six ranked vendors are considered.
  • The total score is assigned to each vendor as a sum of the three categories rank values.
  • The final ranking is produced by gradation of grand totals of each vendor across each investigated category.

The Scores

Table 3 shows the vendors ranking carried out in accordance with the evaluation rules defined previously.



Table 3

Table 4 summarizes the results from Table 3 and displays the scores of each vendor subject to evaluation.



Table 4


Correlation of the Short Lists from Gartner and Forrester

Now it's finally time to see where all of this research was heading. Table 5 gives a summary of the results of the evaluations carried out in the previous two sections. Only vendors which are represented on both short lists are considered in order to ensure a high level of synergy between the Gartner and Forrester Evaluation Models. Further ranking of the short-listed vendors is achieved by ordering the sums of the scores listed in Table 6.



Table 5


Table 6


Conclusion

The objectivity of this assessment is ensured by a vendor ranking system that determines scores based on a set of common evaluation criteria from two industry-recognized sources (Gartner or Forrester). The final short list is the result of a correlation between the short lists produced by these models and reflects.

Provided in Table 7 is a recap of the criteria from both the Gartner and Forrester assessment models.



Table 7


References

[REF-1] SOA Practitioners' Guide Part 2: SOA Reference Architecture, 2006

[REF-2] Magic Quadrant for Application Infrastructure for Back-End Application Integration Projects, 2Q07

[REF-3] Magic Quadrant for Application Infrastructure for Composite-Application Projects, 2Q07

[REF-4] Magic Quadrant for Application Infrastructure for New Service-Oriented Business Application Projects, 2Q07

[REF-5] The Forrester Wave: Integration-Centric Business Process Management Suites, Q4 2006