ServiceTechMag.com > Archive > Issue LXVIII, November 2012 > Complex Method Design with HTTP and REST
Benjamin Carlyle

Benjamin Carlyle

Biography

Benjamin has been involved with the REST community through his blog and other forums since 2004. He is credited with inspiring the popular Restlet framework for Java, he coined the term "REST Triangle", and has deep understanding of both the theory and practice of REST-style architecture.

As an architect working in the Rail industry he is experienced in bringing together REST architecture, systems architecture, systems integration, and a variety of other topics at an enterprise scale. Benjamin has recently been working with the likes of Thomas Erl on documenting the convergence between SOA and REST.

Contributions

rss  subscribe to this author

Thomas Erl

Thomas Erl

Biography

Thomas Erl is a best-selling IT author and founder of Arcitura™ Education Inc. Thomas has been the world's top-selling service technology author for over seven years and is the series editor of the Prentice Hall Service Technology Series from Thomas Erl (www.servicetechbooks.com ). With more than 300,000 copies in print worldwide, his books have become international bestsellers and have been formally endorsed by senior members of major IT organizations, such as IBM, Microsoft, Oracle, Intel, Accenture, IEEE, HL7, MITRE, SAP, CISCO, HP, and many others.

Several of his books, including Cloud Computing Design Patterns, Cloud Computing: Concepts, Technology & Architecture, SOA Design Patterns, SOA Principles of Service Design, and SOA Governance, were authored in collaboration with the IT community and have contributed to the definition of cloud computing technology mechanisms, the service-oriented architectural model and service-orientation as a distinct paradigm. His more recent title, Service-Oriented Architecture: Analysis & Design for Services and Microservices, formally positions and introduces new patterns for the Microservice architectural model as part of SOA.

As CEO of Arcitura™ Education Inc. and in cooperation with SOA School, Cloud School and Big Data Science School, Thomas has led the development of curricula for the internationally recognized SOA Certified Professional (SOACP), Cloud Certified Professional (CCP) and Big Data Science Certified Professional (BDSCP) accreditation programs, which have established a series of formal, vendor-neutral industry certifications obtained by thousands of IT professionals around the world.

Thomas is the founding member of the SOA Manifesto Working Group and author of the Annotated SOA Manifesto (www.soa-manifesto.com). For 10 years, he was the editor of The Service Technology Magazine, and he further oversees the SOAPatterns.org, CloudPatterns.org and BigDataPatterns.org initiatives, which are dedicated to the on-going development of master pattern catalogs for service-oriented architecture, cloud computing and Big Data.

Thomas has toured more than 20 countries as a speaker and instructor, and regularly participates in international conferences. More than 100 articles and interviews by Thomas have been published in numerous publications, including The Wall Street Journal and CIO Magazine.

Bookmarks



Complex Method Design with HTTP and REST

Published: November 22, 2012 • Service Technology Magazine Issue LXVIII PDF
 

The uniform contract establishes a set of base methods used to perform basic data communication functions. As we've explained, this high-level of functional abstraction is what makes the uniform contract reusable to the extent that we can position it as the sole, over-arching data exchange mechanism for an entire inventory of services. Besides its inherent simplicity, this part of a service inventory architecture automatically results in the baseline standardization of service contract elements and message exchange.

The standardization of HTTP on the World Wide Web results in a protocol specification that describes the things that services and consumers "may", "should", or "must" do to be compliant with the protocol. The resulting level of standardization is intentionally only as high as it needs to be to ensure the basic functioning of the Web. It leaves a number of decisions as to how to respond to different conditions up to the logic within individual services and consumers. This "primitive" level of standardization is important to the Web where we can have numerous foreign service consumers interacting with third-party services at any given time.

A service inventory, however, often represents an environment that is private and controlled within an IT enterprise. This gives us the opportunity to customize this standardization beyond the use of common and primitive methods. This form of customization can be justified when we have requirements for increasing the levels of predictability and quality-of-service beyond what the World Wide Web can provide.

For example, let's say that we would like to introduce a design standard whereby all accounting-related documents (invoices, purchase orders, credit notes, etc.) must be retrieved with logic that, upon encountering a retrieval failure, automatically retries the retrieval a number of times. The logic would further require that subsequent retrieval attempts do not alter the state of the resource representing the business documents (regardless of whether a given attempt is successful).

With this type of design standard, we are essentially introducing a set of rules and requirements as to how the retrieval of a specific type of document needs to be carried out. These are rules and requirements that cannot be expressed or enforced via the base, primitive methods provided by HTTP. Instead, we can apply them in addition to the level of standardization enforced by HTTP by assembling them (together with other possible types of runtime functions) into aggregate interactions. This is the basis of the complex method.

A complex method encapsulates a pre-defined set of interactions between a service and a service consumer. These interactions can include the invocation of standard HTTP methods. To better distinguish these base methods from the complex methods that encapsulate them, we'll refer to base HTTP methods as primitive methods (a term only used when discussing complex method design.)

Complex methods are qualified as "complex" because they:

  • can involve the composition of multiple primitive methods
  • can involve the composition of a primitive method multiple times
  • can introduce additional functionality beyond method invocation
  • can require optional headers or properties to be supported by or included in messages

As previously stated, complex methods are generally customized for and standardized within a given service inventory. For a complex method to be standardized, it needs to be documented as part of the service inventory architecture specification. We can define a number of common complex methods as part of a uniform contract that then become available for implementation by all services within the service inventory.

Complex methods have distinct names. The complex method examples that we cover shortly are called:

  • Fetch - A series of GET requests that can recover from various exceptions.
  • Store - A series of PUT or DELETE requests that can recover from various exceptions.
  • Delta - A series of GET requests that keep a consumer in sync with changing resource state.
  • Async - An initial modified request and subsequent interactions that support asynchronous request message processing.

Services that support a complex method communicate this by showing the method name as part of a separate service capability (Figure 1), alongside the primitive methods that the complex method is built upon. When project teams create consumer programs for certain services, they can determine the required consumer-side logic for a complex method by identifying what complex methods the service supports, as indicated by its published service contract.


img

Figure 1 - An Invoice service contract displaying two service capabilities based on primitive methods and two service capabilities based on complex methods. We can assume that the two complex methods incorporate the use of the two primitive methods, but we can confirm this by studying the design specification that documents the complex methods.


Note

When applying the Service Abstraction principle to REST service composition design, we may exclude entirely describing some of the primitive methods from the service contract. This can be the result of design standards that only allow the use of a complex method in certain situations. Going back to the previous example about the use of a complex method for retrieving accounting-related documents, we may have a design standard that prohibits these documents from being retrieved via the regular GET method (because the GET method does not enforce the additional reliability requirements).

It is important to note that the use of complex methods is by no means required. Outside of controlled environments in which complex methods can be safely defined, standardized, and applied in support of the Increased Intrinsic Interoperability goal, their use is uncommon and generally not recommended. When building a service inventory architecture we can opt to standardize on certain interactions through the use of complex methods or we can choose to limit REST service interaction to the use of primitive methods only. This decision will be based heavily on the distinct nature of the business requirements addressed and automated by the services in the service inventory.

Despite their name, complex methods are intended to add simplicity to service inventory architecture. For example, let's imagine we choose not to use pre-defined complex methods and then realize that there are common rules or policies that we applied to numerous services and their consumers. In this case, we will have built the common interaction logic redundantly across each individual consumer-service pair. Because the logic was not standardized, its redundant implementations will likely exist differently. When we need to change the common rules or policies, we will need to revisit each redundant implementation accordingly. This maintenance burden and the fact that the implementations will continue to remain out of synch make this a convoluted architecture that is unnecessarily complex. This is exactly the problem that the use of complex methods is intended to avoid.

The upcoming sections introduce a set of sample complex methods organized into two sections:

  • Stateless Complex Methods
  • Stateful Complex Methods

Note that these methods are by no means industry standard. Their names and the type of message interactions and primitive method invocations they encompass have been customized to address common types of functionality.

Note

The Case Study Example section at the end of this chapter further explores this subject matter. In this example, in response to specific business requirements, two new complex methods (one stateless, the other stateful) are defined.

Stateless Complex Methods

This first collection of complex methods encapsulate message interactions that are compliant with the Stateless constraint.

Fetch Method

Instead of relying only on a single invocation of the HTTP GET method (and its associated headers and behavior) to retrieve content, we can build a more sophisticated data retrieval method with features such as:

  • automatic retry on timeout or connection failure
  • required support for runtime content negotiation to ensure the service consumer receives data in a form it understands
  • required redirection support to ensure that changes to the service contract can be gracefully accommodated by service consumers
  • required cache control directive support by services to ensure minimum latency, minimum bandwidth usage, and minimum processing for redundant requests

We'll refer to this type of enhanced read-only complex method as a Fetch. Figure 2 shows an example of a pre-defined message interaction of a Fetch method designed to perform content negotiation and automatic retries.


img

Figure 2 - An example of a Fetch complex method comprised of consecutive GET method calls.

Store Method

When using the standard PUT or DELETE methods to add new resources, set the state of existing resources, or remove old resources, service consumers can suffer request timeouts or exception responses. Although the HTTP specification explains what each exception means, it does not impose restrictions as to how they should be handled. For this purpose, we can create a custom Store method to standardize necessary behavior.

The Store method can have a number of the same features as a Fetch, such as requiring automatic retry of requests, content negotiation support, and support for redirection exceptions. Using PUT and DELETE, it can also defeat low bandwidth connections by always sending the most recent state requested by the consumer, rather than needing to complete earlier requests first.

The same way that individual primitive HTTP methods can be idempotent, the Store method can be designed to behave idempotently. By building upon primitive idempotent methods, any repeated, successful request messages will have no further effect after the first request message is successfully executed.

For example, when setting an invoice state from "Unpaid" to "Paid":

  • a "toggle" request would not be idempotent because repeating the request toggles the state back to "Unpaid."
  • the "PUT" request is idempotent when setting the invoice to "Paid" because it has the same effect, no matter how many times the request is repeated

It is important to understand that the Store and its underlying PUT and DELETE requests are requests to service logic, not an action carried out on the service's underlying database. As shown in Figure 3, these types of requests are stated in an idempotent manner in order to efficiently allow for the retrying of requests without the need for sequence numbers to add reliable messaging support.


img

Figure 3 - An example of the interaction carried out by a Store complex method.


Note

Service capabilities that incorporate this type of method are an example of the application of the Idempotent Capability pattern.

Delta Method

It is often necessary for a service consumer to remain synchronized with the state of a changing resource. The Delta method is a synchronization mechanism that facilitates stateless synchronization of the state of a changing resource between the service that owns this state and consumers that need to stay in alignment with the state.

The Delta method follows processing logic based on the following three basic functions:

  1. The service keeps a history of changes to a resource.
  2. The consumer gets a URL referring to the location in the history that represents the last time the consumer queried the state of the resource.
  3. The next time the consumer queries the resource state, the service (using the URL provided by the consumer) returns a list of changes that have occurred since the last time the consumer queried the resource state.

Figure 4 illustrates this using a series of GET invocations.


img

Figure 4 - An example of the message interaction encompassed by the Delta complex method.


The service provides a "main" resource that responds to GET requests by returning the current state of the resource. Next to the main resource it provides a collection of "delta" resources that each return the list of changes from a nominated point in the history buffer.

The consumer of the Delta method activates periodically or when requested by the core consumer logic. If it has a delta resource identifier it sends its request to that location. If it does not have a delta resource identifier, it retrieves the main resource to become synchronized. In the corresponding response the consumer receives a link to the delta for the current point in the history buffer. This link will be found in the Link header (RFC 5988) with relation type Delta.

The requested delta resource can be in any one of the following states:

  1. It can represent a set of one or more changes that have occurred to the main resource since the point in history that the delta resource identifier refers to. In this case, all changes in the history from the nominated point are returned along with a link to the new delta for the current point in the history buffer. This link will be found in the Link header with relation type Next.
  2. It may not have a set of changes because no changes have occurred since its nominated point in the history buffer, in which case it can return the 204 No Content response code to indicate that the service consumer is already up-to-date and can continue using the delta resource for its next retrieval.
  3. Changes may have occurred, but the delta is now expired because the nominated point in history is now so old that the service has elected not to preserve the changes. In this situation, the resource can return a 410 Gone code to indicate that the consumer has lost synchronization and should re-retrieve the main resource.

Delta resources use the same caching strategy as the main resource.

The service controls how many historical deltas it is prepared to accumulate based on how much time it expects consumers will take (on average) to get up-to-date, or in some cases where a full audit trail is maintained for other purposes the number of deltas can be indefinite. The amount of space required to keep this record is constant and predictable regardless of the number of consumers, leaving each individual service consumer to keep track of where it is in the history buffer.

Async Method

This complex method provides pre-defined interactions for the successful and canceled exchange of asynchronous messages. It is useful for when a given request requires more time to execute than what the standard HTTP request timeouts allow.

Normally, if a request takes too long, the consumer message processing logic will time out or an intermediary will return a 504 Gateway Timeout response code to the service consumer. The Async method provides a fallback mechanism for handling requests and returning responses that does not require the service consumer to maintain its HTTP connection open for the total duration of the request interaction.

As shown in Figure 5, the service consumer issues a request, but does so specifying a call-back resource identifier. If the service chooses to use this identifier, it responds with the 202 Accepted response code, and may optionally return a resource identifier in the Location header to help it track the place of the asynchronous request in its processing queue.


img

Figure 5 - An asynchronous request interaction encompassed by the Async complex method.


When the request has been fully processed, its result is delivered by the service, which then issues a request to the call-back address of the service consumer. If the service consumer issues a DELETE request (as shown in Figure 6) while the Async request is still in the processing queue (and before a response is returned), a separate pre-defined interaction is carried out to cancel the asynchronous request. In this case, no response is returned and the service cancels the processing of the request.


img

Figure 6 - An asynchronous cancel interaction encompassed by the Async complex method.


If the consumer cannot listen for call-back requests, it can use the asynchronous request identifier to periodically poll the service. Once the request has been successfully handled, it is possible to retrieve its result using the previously described Fetch method before deleting the asynchronous request state. Services that execute either interaction encompassed by this method must have a means of purging old asynchronous requests if service consumers are unavailable to pick up responses or otherwise "forget" to delete request resources.

Stateful Complex Methods

These next complex methods use REST as the basis of service design but incorporate interactions that intentionally breach the Stateless constraint. Although the scenarios represented by these methods are relatively common in traditional enterprise application designs, this kind of communication is not considered native to the World Wide Web. The use of stateful complex methods can be warranted when we accept the reduction in scalability that comes with this design decision.

Trans Method

The Trans method essentially provides the interactions necessary to carry out a two-phase commit between one service consumer and one or more services (as per the application of the Atomic Transaction pattern). Changes made within the transaction are guaranteed to either successfully propagate across all participating services, or all services are rolled back to their original states.

This type of complex method requires a "prepare" function for each participant before a final commit or rollback is carried out. Functionality of this sort is not natively supported by HTTP. Therefore, we need to introduce a custom PREP-PUT method (a variant of the PUT method), as shown in Figure 7.


img

Figure 7 - An example of a Trans complex method, using a custom primitive method called PREP-PUT.


In this example the PREP-PUT method is the equivalent of PUT, but it does not commit the PUT action. A different method name is used to ensure that if the service does not understand how to participate in the Trans complex method, it then rejects the PREP-PUT method and allows the consumer to abort the transaction.

To carry out the logic behind a typical Trans complex method will usually require the involvement of a transaction controller to ensure that the commit and rollback functions are truly and reliably carried out with atomicity.

Alternative transaction models that have varying degrees of compliance with Stateless are further explored in Chapter 12.

PubSub Method

A variety of publish-subscribe options are available once it is decided to intentionally breach the Stateless constraint. As explained in the Event-Driven Messaging pattern, these types of mechanisms are designed to support real-time interactions where a service consumer must act immediately when some pre-determined event at a given resource occurs. The Event-Driven Messaging pattern is applied as an alternative to the repeated polling of the resource, which can negatively impact performance if the polling frequency is increased to detect changes with minimal delay.

There are various ways that this complex method can be designed. Figure 8 illustrates an approach that treats publish-subscribe messaging as a "cache-invalidation" mechanism.


img

Figure 8 - An example of a PubSub complex method based on cache invalidation. When the service determines that something has changed on one or more resources, it issues cache expiry notifications to its subscribers. Each subscriber can then uses a Fetch complex method (or something equivalent) to bring the subscriber up-to-date with respect to the changes.


This form of publish-subscribe interaction is considered "lightweight" because it does not require services to send out the actual changes to the subscribers. Instead, it informs them that a resource has changed by pushing out the resource identifier, and then reuses an existing, cacheable Fetch method as the service consumers pull the new representations of the changed resource.

The amount of state required to manage these subscriptions is bound to one fixed-sized record for each service consumer. If multiple invalidations queue up for a particular subscribed event, they can be folded together into a single notification. Regardless of whether the consumer receives one or multiple invalidation messages, it will still only need to invoke one Fetch method to bring itself up-to-date with the state of its resources each time it sees one or more new invalidation messages.

The PubSub method can be further adjusted to distribute subscription load and session state storage to different places around the network. This technique can be particularly effective within cloud-based environments that naturally provide multiple, distributed storage resources.

Summary

REST-based service compositions can utilize synchronous or asynchronous message exchanges between composition initiator and composition controller. Idempotency often needs to be guaranteed across multiple service invocations within a given service activity. The advantages and disadvantages of late binding between composition participants need to be understood to design effective REST service composition architectures.

img

This page contains excerps from:

SOA with REST: Principles, Patterns & Constraints for Building Enterprise Solutions with REST
by Raj Balasubramanian, Benjamin Carlyle, Thomas Erl and Cesare Pautasso

Foreword by Stefan Tilkov

(ISBN: 0137012519, Hardcover, 577 pages)

For more information about this book, visit www.servicetechbooks.com.