Gijs in 't Veld is co-founder and CTO at Motion10, specialized in System Integration, Portals and Modern Business Apps. From the beginning of his career in 1986 his main focus has been integration architecture and design using various different platform technologies. Since 2000 he has been involved with hundreds of SOA projects world-wide, solely on the Microsoft Application Platform. Gijs has been awarded Microsoft MVP 7 times in a row during 2006-2013 for his many contributions to the world-wide adoption of Microsoft BizTalk Server / Services and is also an Azure Advisor. He wrote many papers and articles and contributed to a number of books on integration, SOA and cloud computing. Gijs was also a technical reviewer of Thomas Erl's book Cloud Computing - Concepts, Technology and Architecture. Gijs can be followed at Twitter @gintveld.
API Management - Governance of Public Business Services Published: August 1, 2014 • Service Technology Magazine Issue LXXXV PDF
Abstract: If you are involved with cloud computing and mobile (and if you are in IT, hopefully you are), you have probably already bumped into the term API Management. If not, now is the time to start getting to know all about it. It will become a crucial part of your integration platform. It can help you expose your B2B or B2C business services in a more manageable way.
First, it is important to familiarize you with the term API in this context. API, or Application Programming Interface, is an old term. An API is basically a contract for exchanging information between computer programs. In the meantime though, this term does not really cover the real meaning anymore. Because, what is an application these days? Is that still a valid concept? In modern IT environments, an API is basically nothing more than the interface of a web service. These web services are most of the times based on SOAP, but more and more exposed as REST services as well. These services provide a specific piece of functionality (black-box). And they can, if they are developed with the right principles in mind, be used in many ways and thus used in many different contexts.
SaaS and APIs
The trend is that vendors of standard "applications" have made or are making their solutions available as services by means of the software-as-a-service (SaaS) model. And of course there are lots of software solutions available on the internet that have been designed as SaaS solutions from the ground up, like Google, LinkedIn, Bing, Booking.com, etc. Lots of times, the most important functionalities of these SaaS solutions are exposed as APIs. These are the so called public APIs and they are always based on web services technology.
Figure 1 – The rise of public APIs
Marketplaces for services provided by many organizations, exposed as APIs can be found on every cloud platform. To get an idea about what is going on in the so called API Economy, have a look at ProgrammableWeb.com for instance. Among other things, they provide an API directory in which thousands of public APIs have been registered.
APIs and SOA
Most of the times services are RESTful these days, because that's really easy and has a lower overhead. Especially for publicly accessible services, the complexity of "traditional" SOAP services make them a bit harder to use. It's almost like the difference between XML and EDI, where EDI interchanges should adhere to their strict contracts and syntax peculiarities and XML is more readable and easier to parse. We still have to see however, if versioning, error handling and design time governance in a RESTful world won't become a nightmare. But that's another topic that probably deserves some attention as well.
The aforementioned SaaS solutions usually make use of these services internally as well, so that everything can be nicely loosely coupled. The vendors have basically "conspired" to finally shove SOA down your throat whether you like it or not, the only difference now is that they don't call it SOA anymore. But, the same architecture principles that underpin SOA are being followed and that is a great thing! Basically, APIs are used to access (orchestrated) task services, entity services or utility services. And by adhering to the same design principles, re-use, performance, monitor-ability and scalability (among other things) are much easier to accomplish.
In a SOA or architecture based on APIs, in which a set of services can be called in a certain (configurable) order to create real (parts of) business processes, governance is going to be really important. Otherwise we might as well call it JABOWS (just a bunch of web services). In "traditional" SOA environments, implementing run- and design-time SOA governance is now quite the norm. In the latest cloud and mobile solutions based on APIs, this is really not the case yet. APIs are being served and consumed like spaghetti. You'll need a large napkin to cope with that. It is clearly time for a next step in maturity. APIs are more open, readily consumable, mobile friendly and more business oriented, but from an infrastructure, manageability and governance perspective, APIs are more like SOA than API purists would like to admit.
The belief among some end users that API Management has nothing to do with
In figure 1 you can not only see that the amount of public APIs has grown tremendously in the past couple of years, but it is also evident that the vast majority of these APIs is unmanaged. With the growing demand for consumption of content and processes (provided by APIs) in mobile apps, web sites and appliances such as TVs, game consoles and cars basically we're talking about integrating the Internet of Things (IoT). Management and monitoring of all these interconnected consumers and providers will become a bare necessity if we don't want to end up with anarchy on the planet of the APIs.
Application Services Governance
Gartner has placed both SOA governance and API Management in the Application Services Governance category. And quite predictably, they've also introduced a Magic Quadrant for that. There's quite a lot of movement in this Magic Quadrant, not only because existing players are rapidly embracing the technology and improving their offerings, but also because large vendors like Intel, Microsoft and others have made some strategic acquisitions in the past few months. They clearly see the need for API Management and are rapidly integrating these features in their (cloud) platforms. I also predict that vendors in the SOA governance category and vendors in the API Management category will slowly move to a middle ground, providing generic run- and design-time governance solutions and that in the end we will all be talking about Application Services Governance when we discuss design- and runtime governance of APIs and SOAs. Application Services Governance is basically all about service lifecycle management: plan, design, implement, operate, maintain and retire services. API Management currently provides the features to support service lifecycle management specifically for public facing APIs, mostly based on REST. We will however see that lifecycle management for services will become technology agnostic rapidly and that pure-play API Management vendors will incorporate SOA style governance capabilities. At the same time, we will see that pure-play SOA governance vendors will incorporate API Management capabilities and thus focus more on newer service technologies and developer experience.
API Management Defined
To keep the focus on API Management: The nice thing is that you can not only use it to manage new APIs, but you can also use it to expose your existing on-premises business services as APIs towards the cloud and mobile apps and other internal and external consumers. But now you can do it with security, monitor-ability and scalability in mind. Basically we are talking about an intelligent proxy to help expose managed APIs. I'm sure that the large SaaS vendors such as SalesForce.com, Google and Microsoft have put management and monitoring of their public APIs in place, but for smaller organizations wishing to expose APIs to their partners and apps, API Management will also become a crucial part of their integration architectures.
API Management mainly revolves around the following topics:
Figure 2 – Exposing business services as managed APIs
In the following paragraphs, each of these API Management features will be explored.
Service virtualization is about creating an abstraction layer between your "private" services and the outside world. You can expose virtual services with, for example, aggregated functions like "SearchArticlesByName", "GetArticleById", and "PlaceOrder" which in your on-premises back-end systems are handled by completely different, physical applications. It can be thought of as "service mash-up", whereby the API Management layer exposes externally available APIs based on loosely coupled service composition. Using this technique provides the basics for gaining numerous advantages, especially for creating an abstraction layer, but more specifically with regard to security; overcoming DMZ issues is one of the main advantages of service virtualization.
Protocol conversion or mediation makes it possible to convert between transport protocols such as HTTP/s and HTTP or between different queuing mechanisms. For example, SOAP based on-premises functions can, by means of protocol conversion now be exposed as RESTful services towards external consumers. A JSON based data feed is more easily consumable by a mobile app than a SOAP service. We will also see newer transports becoming more and more important in the IoT era, like MQTT and Websockets. These high throughput, low latency protocols will also have to be supported by your API Management solution now or in the near future. At the same time this feature also takes care of applying different security policies and digital encryption certificates to inbound API calls and the bridge to the on-premises SOA architecture. API Management can take care of these conversions and policy enforcement and at the same time provide the loose coupling that we all have learned to love.
You can secure your virtual services in a much more granular way, by only exposing those functions towards certain apps or other consumers that are only important to them and which they can only access with the right certificates. The remaining part of the service operations will not be accessible. This comes in very handy if you have your internal SOA designed in such a way that a high level of re-usability is created, and you want to provide the same set of functionalities to the outside world. However, in most cases authentication and authorization can be handled without too many problems within your own firewalls, but as soon as you expose these services to the public it will become impossible to apply the same security policies. In this case, the combination of service virtualization and applying more granular security will help out. Facilitated by a trust relationship between your on-premises integration middleware and API Management, you can keep on using the same security infrastructure and policies applied to your internal SOA, and at the same time re-use these services to expose business services in a secure way to the outside world.
Scalability and Performance
API Management can also provide caching and load balancing to help improve performance and reduce the risk of overrunning your (back-end) services. Let's face it, in by far the most cases the limitations of your back-end systems in handling large amounts of transactions at the same time proves to be the performance bottleneck. Integration middleware traditionally takes care of this issue by using queueing and throttling mechanisms. The API Management offerings by most vendors have also been equipped with these facilities, including caching mechanisms. This means, for example, that when you expose APIs to the public for providing information about traffic conditions in a certain area, the API Management layer will cache the results and return the information from the cache whenever possible if a traffic request for the same area needs to be processed within the same timeframe.
Runtime Monitoring and Governance
The most relevant (meta) information about runtime calls of services (API calls) will be recorded, so you can exactly monitor how many times services are consumed, when they were invoked, who invoked them and how long it took to process the request end-to-end. Dependencies between APIs and on-premises services are easy to find out. These capabilities provide crucial information for providing performance and scalability. It is also important to make it possible to adhere to service level agreements. And it can be used to make invoicing for the business services provided possible. Monetizing your digital assets will become much easier if you make use of these features of your API Management solution.
Your APIs can be registered in the service catalogue, including all relevant meta data and their descriptions. This will make re-use of services much easier because now your, or 3rd party developers can actually find and assess the re-use potential, instead of re-inventing the wheel for every function they need. It is also vital for creating "products" that consist of mash-ups or compositions of your business services that can be exposed to your customers and partners. The design-time governance features of your API Management solution help you to implement application lifecycle management, including versioning of your APIs. Providing these features through a portal makes it easier to onboard new customers, affiliates and partners that want to make use of your business services, without needing direct access to your on-premises integration middleware.
Apps, APIs and SOA Combined
When there is a need to expose the business services provided through an on-premises full fletched SOA to mobile apps, a layered architecture is needed. This is where, on top of an existing SOA implementation, API Management can play an important role.
A typical architecture that accommodates SOA style integration, together with data services and exposure of business services to mobile apps looks like this:
Figure 3 – An App, API and SOA architecture
In the example architecture depicted in figure 3 you can see that both the integration middleware, operational data store (ODS) and API Management have been placed in the cloud, but you can imagine that an architecture with an ODS and integration middleware hosted on-premises is also capable of providing the same features.
It will be interesting to see how far the Application Services Governance players will go in providing mediation services in their offerings. It will be very tempting for them to also incorporate advanced transformation and content based routing features and maybe throw in a set of legacy adapters and an adapter framework, so you can build your own adapters. But then they are threading into Enterprise Service Bus (ESB) territory. Let's see how that will evolve in the near future; exciting times ahead in the integration space!
By implementing API Management you can now expose B2B and B2C business services provided by your on-premises or other (tailor made) solutions to cloud and apps in a modern, secure, manageable and scalable way, while at the same time leveraging the investments made in your services-based architecture. This will help modernize your solutions and bring new ways for your customers and partners to interact with your business processes, in other words: Opening new sales channels. Bringing down your time to market is crucial to survive in the new, mobile first world. Capitalizing on your digital assets is a must. API Management is here to help you realize that and at the same time make it manageable. Make sure you get to know and embrace it!