Sergey Popov

Sergey Popov

Biography

Sergey Popov is a Certified SOA Architect, Consultant, Analyst, Governance Specialist, Security Specialist, as well as a Certified SOA Trainer. He has broad technical experience with SOA and application integration architecture development, and specializes with integration development based on Oracle applications. During his 15 years of working in the shipping and transportation industry, he helped with establishing communications and message brokering platforms for the world's biggest RO-RO and container shipping companies. In several projects for business critical applications within enterprises Sergey designed and implemented ESB solutions with average transaction volumes of over 100,00 messages per day. Sergey's wide business knowledge in the Telecom industry is further complemented with expertise in the SCM, ERP, Banking and Finance sectors. He was responsible for the implementation of a Pan-European service layer for the second biggest cable and telecommunication provider in Europe. Sergey is also an Oracle Certified professional and Oracle Fusion Middleware architect. His accumulated SOA, Java and Oracle Middleware experience was recently combined in PACKT Publishing book Applied SOA Patterns on the Oracle Platform, August 2014 (http://bit.ly/1uqK9dq

Contributions

rss  subscribe to this author

Bookmarks

Introducing the Agnostic Composition Controller Pattern Published: December 2, 2014 • Service Technology Magazine Issue LXXXVII PDF

Composition Controllers are arguably the ultimate engines, moving contemporary SOA nowadays. Every aspect of complex SOA realisation, from Adaptive Case Management to Cloud Service Balancing using burst-in/burst-out, is based on implementation of this SOA Pattern. We are far from declaring this particular pattern as the main one in the entire SOA patterns catalogue, in the same way we are not placing Composability as the central principle among the other eight. Still, when it comes down to the materialisation of the core SOA promise–gaining/saving money through the reasonably effortless constant reuse of the atomic units of decoupled business logic equipped with standardised and seamlessly discoverable contract—the Composition Controller is the main and probably the only design model which can help. Task-orchestrated services as a nearly static physical implementation of service compositions (mostly for long running compositions) or Atomic Transaction Coordinators (generally for synchronous services) have been pretty much formalized in SOA over the last ten years.

The most fascinating for us in terms of promoting the maximum possible level of reuse are Agnostic Composition Controllers (and sub-Controllers) as the direct realisation of the Composability principle. They are not only fascinating, but also probably the most difficult in practical implementation as we will have to employ almost all other declared SOA patterns in at least six different SOA frameworks. We tried to lay out the practical aspects of this endeavour in the recently published Applied SOA Patterns on the Oracle Platform, Packt Publishing, August 2014 (http://bit.ly/1uqK9dq).

Introduction of Agnostic Composition Controller Pattern

Initially, publication of this subject was planned as three consecutive articles in September – October of 2012. It was intended to logically separate them by the key design aspects of this pattern implementation: what elements of service taxonomy we need for assembling service compositions of fly by Service Broker, what frameworks could host the Service Broker as a controller and, finally, what could be the actual realization of Service Broker in every individual SOA Framework (at least in the main three of them). The first two parts were successfully published here and then presented at the 5th SOA & Cloud Symposium in London in 2012, but the last one proved to be the bite too big to swallow. It took one and half years to complete the third part in a reasonable level of detail suitable for SOA practitioners.

Pattern's Realisation Aspects

Speaking of practical aspects of the presented separation, it is important to mention that it is actually based on the common threefold notion of basic SOA: client-requestor, service-worker, and service registry. Everything is simple in basic SOA. The client could be anything, capable to comprehend and comply with the service-worker contract, the worker doesn't have to be REST or SOAP WS, and the service registry (and its taxonomy) is not compulsory at all. Frustratingly, the large majority of SOA practitioners openly neglect service registry as "redundant and hardly useful." That might be true for simple service activities in basic SOA. Alas, it's not that simple when there are not one but many workers, combined in several transactions with unknown for requestor context, based on business logic which could change depending on a transaction's outcome. More complications are added by the fact that there are many message formats and transport protocols involved, requiring all kinds of transformations and bridging. Generally, we can rely on three whales: WS-Addressing (W3C:WSA), WS-ReliableMessaging (OASIS:WS-RM) and WS-Transaction (OASIS:WS-TX) and we touched on in Chapter 1: SOA Ecosystem – Interconnected Principles, Patterns and Frameworks.

Starting from the third chapter's physical realisation of these core standards in a form of Agnostic Composition Controller has been explained for every framework as it is obvious that each framework will add its own requirements, as follows:

  • For Composition Controller in Enterprise Business Flows Framework (EBF based on BPEL), the classic realisation is based on implementation of agnostic capability would be as creation of a dummy service (called just Service) and using WS-Addressing possibility to substitute the dummy endpoint URL with the actual one. As long as all transactions will be BASE-type, we should equip our composition controller with the ability of compensating transactions in the agnostic way as well. This is something we usually call the Master Composition Controller.
  • Implementation of the Agnostic Composition Controller on the Enterprise Business Services Framework (classic Service Broker in ESB) will also require lookup/substitution of URL on runtime, but as long as all transactions are of ACID-type, Atomic Transaction Coordination (form of WS-TX) is compulsory.
  • Adapter Framework is an EAI approach and not true SOA. Reuse is not really welcome here. Still in Chapter 6: Finding the Compromise we demonstrate how to implement Agnostic Composition Controller in a form of Adapter Factory and reduce the Adapter's footprint in your technical infrastructure in general.

The fact that in every realisation above we will have to perform service metadata lookup and extraction signifies the inevitability of proper Service Repository implementation with Taxonomy, suitable for runtime and design-time SOA governance. This governance is what some call API Management nowadays, and key elements of it (lightweight Service Taxonomy with practical implementation on DB and Service Inventory Endpoint based on ESB) are explained in Chapter 5: Maintaining the Core in great details.

Tackling the Big Problems

With so many moving parts and incorporated standards, it is no wonder that the Agnostic Composition Controller is arguably the most complex SOA pattern for physical realisation. Complexity of the pattern reflects the complexity of the problems it's designed to address and successfully solve. It is clear that using this pattern we can assemble business processes on runtime out of the collection of the service in our inventory or in the Cloud, balance them, throttle them, and even resolve the abnormal situations agnostically. But what about the Real Big Problems (RBP, those which still trigger discussions like "SOA - dead or alive?" or "SOA gets an obituary")? The list of RBPs is quite well known to all SOA practitioners:

  • Big Bang approach: based on belief that from maturity level A an enterprise can push its entire SOA infrastructure to the level B, or with some luck, several levels up in a single transformation. The root of this belief lies in the assumption that SOA cannot coexist with some other traditional architectural approaches.
  • Refrigerator approach: based on disbelief, actually. Initial SOA analysis could take longer than a manager's patience can tolerate. With no tangible initial results, the financing of SOA projects can be ceased for some time, until the necessity of common reusable components becomes apparent again. And again. This "freeze-unfreeze" cycle does not only significantly increase a single project's costs, but also jeopardizes implementation of other dependant SOA projects. It is commonly called "lack of sponsorship from CIO."
  • Focus on pattern implementation, not the problem solving. Based on belief that the Pattern Register is something similar to the modern fashion catalogue - looks trendy and sounds "architecture-ishly" cool.
  • Misbalancing of SOA principles. Based on belief that principles are equal to NFR ("...ilities") or genuine trust that some principles can be completely forgotten in order to ultimately promote others.

You can say that the first two problems are not technical and out of your hands as SOA Architects. First of all, let us not be quick to lay blame on someone. Let's not forget that the "Business and IT alignment" is supposed to be the one of the SOA benefits, and it could be that we didn't walk our half way towards it as we should. Secondly, there could be a good reason for this kind of decision and we can trust that other alternatives might be even worse. What we have to do in this circumstance is to focus on the problem at hand and see how Agnostic Composition Controllers can help.

The "Big Bang approach" could potentially have infinite SOA Adoption Planning and Service-Oriented Analysis projects phases which make top-down approach completely unrealistic. Turning to the meet-in-the-middle (AGILE) approach, we will define only Entity services, leaving Task and Utility mixed and partially undefined. Most probably, the identification of Entity services in a unified way for an entire enterprise is unrealistic either, thus splitting the whole service inventory into manageable domains (geographic or department-based) will be a good idea. With horizontal and vertical layering of services with partially defined metadata and a not clearly defined contract (WSDL will reflects only most obvious operations, i.e. CRUD for Entity, and associated XSD(s) with lots of optional elements of string-type) presence of a universal message container is a necessity. Based on the above, Agnostic Composition Controller(s) in the form of Service Broker will be the main SOA composition engine, handling complex and variable service compositions within single domain and across federated environment. That will buy you some time for proper service layering and modelling, while the first wave of business processes will be automated and provided for rigorous testing.

Problems with "Refrigerator approach" can be solved in a similar way. The reason why management has its doubts is obvious—lack of tangible results for a prolonged time. The classic SOA excuse that "initial service orientation analysis takes a longer time as we have to speculate on reusability" doesn't really work nowadays. SOA as an architectural approach has matured quite well over 15 years, and today we have a lot of ready-to-use essential parts of SOA infrastructure: Service Repository, Recovery Automation, Security/Processing Policies and Policy Store, and we will need them anyway, in almost any of our projects. Thus, at least three tasks can be set for three dedicated (but not exactly separated) task-force teams:

  • Define atomic services (Entity first, as they are limited in numbers and relatively easy to identify);
  • Maintain the Service Repository and register discovered service particulars (just Excel will do initially, no need for a big investment from the start);
  • Implement Service Broker, capable to extract composition invocation list from the Service Repository upon composition initiator's request, and do the invocations sequentially (Keep Agnostic Service Broker as lightweight as possible; if you need parallel processing for some business cases, separate them into individual BPEL, invoke it as atomic service, use FlowN activity inside and return aggregated result back to controller).

This three-tasked parallelism doesn't mean that the overall project's time will be shortened proportionally, but it will allow us deliver tangible results quite quickly and implant the confidence into top management minds. The detailed technical solutions are capable of addressing these two big problems presented in Chapters 3-6 using SCA, OSB, Java and DB (PL/SQL) for ESB, Orchestration and Adapter frameworks. You just have to bear in mind that Agnostic Composition Controllers are really needed for the complex cases with lots of uncertainties and intricate business rules. Do not overcomplicate your design with this pattern where there is no need for that.

The last statement is directly related to the third and last non-technical problem in our short list. Actually, a remedy for it has been provided in every chapter in the form of detail analysis for every identified problem. That is, we should clearly recognise the problem before approaching it with something we believe could solve the problem. Patterns are the sticks for the lame, not otherwise, and do not try to find suitable lame for your beloved stick. How much confusion we can avoid just by renaming the Patterns Catalogue to Problems Catalogue!

The last problem in the list just extends the previous one but has many faces. We started Chapter 1: SOA Ecosystem – Interconnected Principles, Patterns and Frameworks with a real-life case where tactical goals prevailed over the optimal balancing of the SOA principles. The described scenario was so typical and familiar to readers that we had to put in a disclaimer, warning everyone not to follow this avenue. Practically every SOA principle had been broken, and patterns were twisted and misplaced. In every following chapter we carefully assessed and addressed every described problem, relevant to the particular framework. For instance, in Chapter 4: From Traditional Integration to Composition - Enterprise Business Services dedicated to set of pattern, common for ESB we quickly touched on some issues related to balancing of OOP and SOA approaches. Highly interesting subject, but due to space limitations we couldn't delve deep into it, so we will try to add some words to it here.

Balancing OOP and SOA Approaches

Imagine that your team got a pretty standard task—to develop an eCommerce Web Application, a typical web store. A perfect opportunity for Spring Framework with its classic Spring Web MVC (http://docs.spring.io/spring/docs/current/spring-framework-reference/html/mvc.html). You have DispatcerServlet your doGet and doPost, passing them to your business handlers, which deal with you business objects, presented as EJB. The core application can be delivered swiftly, new entities can be provided as you learn new requirements, reflection (dynamic method invocation of new/unknown classes) helps you to plug new modules. All sounds good until you start to realise that first one and half months you and your Client were speaking different languages. What you had in mind was the classic Pet Store Web App while your Client was wondering about set of Web Services as his main focus was on Multichanneling (every Client has a right to set equal sign between Web Applications and Web Services).

Still nothing to worry about. Spring 3 annotations, Spring Integration and some contemplation on involving the lightweight ESB like CXF framework for Proxies and Interceptors will help you to address new requirements. The most natural move in this case could be to expose the methods of existing EJBs in as a REST services, implementation of Client as WS in the following case:

package com.ecommerce.webapp.ws.Client;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.util.Assert;

import com.ecommerce.webapp.model.Client;
import com.ecommerce.webapp.ws.BaseWebServiceImpl;

@Component
public class ClientWSImpl extends BaseWebServiceImpl implements ClientWebService {

  @Override
  public Client getClientProfile(String StoreCode, String LocationCode,
    String ClientCode, String refStoreClientToken, String ClientType) {
    /*....*/
    Assert.hasText(locationId);
    Profile getProfile = .../* actual call to service goes here */
    if (Profile == null)) {
      throw new EntityNotFoundException("no profile found for clients
      code: " + ClientCode);
    }
    return Profile;
  }
}

Listing 1

The preceding example demonstrates the classic OOP approach - in this Client Web Service all WS/REST common functionality is encapsulated in BaseWebServiceImpl superclass. This superclass after initial contemplations should include mandatory:

Query parameters extraction, please see some recommendations from Oracle docs http://docs.oracle.com/cd/E19798-01/821-1841/gipyw/index.html

Generic handling of HttpServletRequest and HttpServletResponse

Common configuration handling for selected eCommerce store - default language, encoding, location properties, and so on. Apache common utils here could be very handy https://commons.apache.org/proper/commons-lang/javadocs/api-2.6/org/apache/commons/lang/LocaleUtils.html

Based on extracted configuration validate common for all WS request path segments (see more http://docs.oracle.com/javaee/6/api/javax/validation/ValidationException.html)

Some of this functionality is presented below (bear in mind that this listing is far from complete):

import java.util.Date;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.util.StringUtils;

import com.google.common.base.Strings;
import com.google.common.primitives.Longs;

public class BaseWebServiceImpl {

  private static final Logger LOGGER =
    LoggerFactory.getLogger(BaseWebServiceImpl.class);

  @Autowired
  private SysConfiguration SysConfig;

  @Autowired
  private CountryConfigStore CountryConfigStore;

  @Context
  private HttpServletResponse httpServletResponse;

  @Context
  private HttpServletRequest httpServletRequest;

  protected final QueryParameters extractQueryParameters(UriInfo uriInfo) {
    QueryParameters query;
    MultivaluedMap<String, String> params =
      uriInfo.getQueryParameters(true);
    if (params != null) {
      query = new QueryParameters(params);
    } else {
      query = new QueryParameters();
    }
    return query;
  }

  protected final void checkCommonPathSegments(String LocationCode,
    String StoreCode, String ClientCode) {
    if (!SysConfig.getLocationCodes().contains(LocationCode)) {
      throw new ValidationException(PATH_PARAM, "LocationCode", INVALID);
    }
    checkStoreCodeForCountry(LocationCode, StoreCode);
    CountryConfig config = CountryConfigStore.lookup(LocationCode);
    if (!config.getClientCodes().contains(ClientCode)) {
      throw new ValidationException(PATH_PARAM, "ClientCode", INVALID);
    }
  }

  protected void checkStoreCodeForCountry(String LocationCode,
    String StoreCode) {
    CountryConfig config = CountryConfigStore.lookup(LocationCode);
    if (!config.getStoreCodes().contains(StoreCode)) {
      throw new ValidationException(PATH_PARAM, "StoreCode", INVALID);
    }
  }

  protected final void checkNotNullRequestBodyField(String fieldPath,
    Object value) {
    if (value == null) {
      throw new ValidationException(REQUEST_BODY, fieldPath, MANDATORY);
    }
  }

  protected final void checkNotBlankRequestBodyField(String fieldPath,
    String value) {
    checkNotNullRequestBodyField(fieldPath, value);
    if (!StringUtils.hasText(value)) {
      throw new ValidationException(REQUEST_BODY, fieldPath, INVALID);
    }
  }

  protected final String getDefaultLocationId(String LocationCode,
    String StoreCode, String ClientCode,
      String extClientId) {
    String cityCode = getDefaultSiteCode(LocationCode, StoreCode);
    return getLocationIdUsingCityCode(LocationCode, StoreCode,
      ClientCode, extClientId,
        cityCode);
  }

  @Context
  public void setHttpServletResponse(HttpServletResponse
    httpServletResponse) {
    this.httpServletResponse = httpServletResponse;
  }

  protected final SysConfiguration getSysConfig() {
    return SysConfig;
  }

  protected final CountryConfig getCountryConfig(String LocationCode) {
    return CountryConfigStore.lookup(LocationCode);
  }
  
  /* ... more functionality is needed */
}

Listing 2

All methods we would like to expose are declared in service contract (as Java interface) ClientWebService.

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;

import com.ecommerce.webapp.model.Client

@Path("/{StoreCode}/{LocationCode}/{ClientCode}/profile")
@Produces("application/json")
@Consumes("application/json")
public interface ClientWebService {
  @GET
  @Path("/{ClientType}")
  Client getClientProfile(@PathParam("StoreCode") String StoreCode,
      @PathParam("LocationCode") String LocationCode,
      @PathParam("ClientCode") String ClientCode,
      @HeaderParam("refStoreClientToken") String refStoreClientToken, @PathParam("ClientType") String ClientType);
}

Listing 3

We skipped the implementation of actual Client EJB for brevity and our realization of BaseWebServiceImpl is exceedingly simple (we will come to that very soon). As a result, WS-orientation (REST-based, of course) will be achieved extremely soon.

Lots of positive things can be said about this approach. Firstly it can be delivered very quickly, so your clients could be stunned by time-to-market you demonstrate. Secondly, Java developers love this approach—it's understandable, transparent, and Spring-based (arguably the most mature framework). Management will have to deal with only one technology, development resources can be acquired with reasonable ease. Reliability will be at its best. Deployment and new versions roll-up are straight forward and simple: solutions can be adapted for a huge variety of application servers, including lightweight and inexpensive, like Tomcat. From a performance standpoint, it will be really swift for some time after initial deployment.

Here we come to the first problem. One business unit (EJB), representing application logic serves two types of requests - WebApp and REST calls from various channels. Although we have maintained maximum level of reuse and delivered our services very fast, we cannot guarantee the linear (i.e. predictable) response for any of our access channels - via service contract and traditional Servlet's dispatcher. Surely this situation shall be anticipated: more requests - higher performance and scalability requirements. We can immediately address this problem by implementing caching for REST sessions (for instance, using http://ehcache.org/). It leads to extending BaseWebServiceImpl with cache functions and session management in general.

Session management is essential as we have to address REST HTTP security, omitted in Listing 2. Obviously, Spring MVC Security model, used in WebApp is not directly applicable for our REST service implementation. We have to handle tokens, credentials and other information, provided in HTTP headers: If-Modified-Since, Last-Modified, Expires, Cache-Control and so on (see standard here http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html).

In addition to caching/scalability methods we have to add to base class functionality for measuring REST call statistic as its highly important for controlling and maintaining our WS SLAs. Now we can extend Listing 2 with functionality, mentioned above (this is partial implementation, simplified for brevity. You will have to elaborate on actual implementation using links to documentation, provided above)

protected final void setCacheHeaders(ClientsCurrentShoppingCart<?> cart,
  boolean setPrivate) {

  setCacheHeaders(cart.getExpires(), setPrivate);
}

protected final void setCacheHeaders(Date expires, boolean setPrivate) {
  if (expires == null) {
    setNoCacheHeader();
  } else {
    httpServletResponse.setDateHeader(HTTP_HEADER_EXPIRES,
      expires.getTime());
    httpServletResponse.setHeader(HTTP_HEADER_CACHE_CONTROL,
   setPrivate ? "private" : "public");
  }
}
protected final void setLastModifiedCacheHeaders(long lastModified,
  boolean setPrivate) {
  httpServletResponse.setDateHeader(HTTP_HEADER_LAST_MODIFIED,
    lastModified);
  String cacheability = setPrivate ? "private" : "public";
  httpServletResponse.setHeader(HTTP_HEADER_CACHE_CONTROL,
    cacheability + ",
  max-age=0");
}

protected final void setNoCacheHeader() {
  httpServletResponse.setHeader(HTTP_HEADER_CACHE_CONTROL, "no-cache,
    no-store,
  must-revalidate");
}

protected final void setResponseAuthHeaders(String username,
  String token) {
  httpServletResponse.setHeader(HeaderNames.ESHOP_USERNAME, username);
  httpServletResponse.setHeader(HeaderNames.ESHOP_TOKEN, token);
}

Listing 4

Well, still looks good, but don't you think that our superclass is getting a bit heavier? Fine tuning with additional sub-classing could help, addressing individual requirements of each REST service, but how far are you ready to go this way? From the very beginning it was clear that refactoring entire application for using REST services only (for channels API and AJAX Views, for instance) would be the best way. Anyway, we managed to maintain WS-orientation quickly and for single (or main) application in application farm of mid-size retail company it will work gracefully.

Here we have come to the main point of this example. What if this application is just eCommerce extension of already existing vast ERP/CRM infrastructure? Just WS-enablement would not be enough; we will need comprehensive SOA-orientation applied here for:

  • Separate core business logic from technical functionality, like caching and session control.
  • Adhere to corporate security requirements in unified way though security perimeter (API gateway, ESB policies, etc.)
  • As a result of the above, include user/session management into corporate SSO under existing brokered authentication with established STS.
  • Maintain service proxies separately on dedicated ESB (we already mentioned CXF, but SeviceMix/Camel would be more appropriate) for throttling, balancing, and SLA control.
  • As a logical outcome from all of the above, all these services will become suitable for the long-running complex compositions (in EBF framework) as Order Management, Order Procurement, Invoice Handling, and Delivery.