Abstract: This article helps understand how a service driven enterprise portal design with SharePoint as the front-end and Microsoft's SOA based integration middleware underneath enables the use of composite services to help build more efficient, scalable and supportable central workspaces that can be used in modern BYOD environments.
Microsoft's SharePoint is a pretty dominant collaboration platform, serving more than 70% of enterprises world-wide. Besides enabling more efficient sharing of documents, improving findability and providing the functionalities to collaborate with team members on projects and documents, SharePoint can also be implemented as an enterprise portal. Basically, by providing document management features, intranet capabilities and integration with other systems you can implement a true central workspace for employees, customers and partners providing them with self-service capabilities.
Of course SharePoint is well known for natively storing and sharing of all kinds of documents and files, as well as allowing people to collaborate with each other on these documents. However, in most SharePoint solutions the information for a great part also comes from external systems. This is why with SharePoint implementation projects integration can often play the greater role.
The integration with other systems and services should be organized according to the SOA principles, thereby providing the means to incorporate external data and processes in the portal, and creating an environment where everything comes together in the right context and with the right security trimming. By implementing a SOA based service layer underneath SharePoint, building such solutions will become much easier, and the end result will be much more scalable and supportable.
The Classic Trap
When leveraging SharePoint as an enterprise portal, it is important to know that it is quite easy to fall into the classic trap that Microsoft tooling is "famous" for: it is quite easy to get to know the product quickly and build your first "hello, world" applications. However, before you know it you will end up with all kinds of functionality that is being used in production environments, and the company is quickly faced with an intranet or portal solution that is not easily manageable, scalable, stable, or even worse an environment that is not supportable.
SharePoint is not a simple product. It can do many things, and inherently its architecture is quite complex. Before designing and building SharePoint solutions, it is therefore crucial that you know the architecture very well, and you know all the ins- and outs of the platform with regard to extensibility. It is very easy to blame the platform on which your badly designed solutions have been built, and before you know it, SharePoint has a bad name within your company, which is just not fair.
In order to prevent such situations, building solutions using SharePoint as the platform have to adhere to a number of simple rules:
- Before you start building "custom solutions" make sure it is not (more-or-less) available out-of-the-box already; discuss your requirements with a SharePoint consultant
- Make sure you build solutions according to the SharePoint architecture; discuss your solution design with a SharePoint architect and have it checked against the Project Start Architecture
- Check your solution design against the SharePoint Governance Plan; this will make sure that the design adheres to the guidelines and rules created for your company's intranet or portal
These rules apply to SharePoint solution design and development in general. Specifically geared towards integration of external content there are two more rules:
- Make sure to apply the principles of service design [REF-1] when exposing data and processes as services in your SOA
- Implement a service layer to streamline the integration of external content and make it manageable and supportable
The remaining part of this article will discuss the integration of external data within SharePoint in more detail.
Business Connectivity Services
Out-of-the-box, SharePoint comes with features to integrate information from other applications. In the latest version this is called "Business Connectivity Services" (BCS). While using this technology, it is possible for "External Content Types" (ECT) to be used just like any other content type in your lists and libraries. But instead of retrieving the information from native data stores, SharePoint will use BCS to retrieve the data. An ECT defines the schema and operations to perform on the data.
Figure 1 – SharePoint Business Connectivity Services (BCS) architecture
As can be seen in figure 1, BCS can retrieve and write data from and to other sources by either using direct SQL queries by invoking .Net components or by consuming Web Services. Through BCS you have full CRUD functionality. There are however several limitations, the lack of functionality to compose services, and the use of data format conversion are the most important ones here. It is not (easily) possible to combine several data sources into one, and display that in a standard SharePoint list or webpart.
Furthermore, it is quite easy to fall into yet another trap; this time it's the "spaghetti integration" trap, where many systems and services communicate with each other in a peer-to-peer fashion and thereby create an unmanageable situation.
When you take a look at the online BCS samples, most of them show you how easy it is to connect to SQL Server tables. Nice indeed, but what if you are not the owner of this SQL database? What if an application update changes column or table names? Are you going to update all ECTs you have defined previously? Even if you can (think about the trouble a table split would cause you), it might incur a lot of rework in all sorts of places. Another example is on WCF (web services) connections: What if your products are now stored in two warehouse locations while they were previously stored in one? You would need to rewrite your BCS connection and change it to a custom .Net Assembly connector.
By implementing a service layer underneath SharePoint BCS, it becomes possible to consume composite services and implement service versioning, real end-to-end tracking and tracing, and monitor features. Also, implementing a service layer gives more flexibility and capabilities with regard to the handling of transactions and providing scalable, failsafe solutions.
And maybe most importantly, BCS in combination with a service layer and the claims based security features provided by SharePoint will create a secure environment, where information will be handled according to the security regulations, and the roles and responsibilities defined in the SharePoint Governance Plan.
Implementing a Service Layer
There are several ways to implement a service layer underneath your SharePoint enterprise portal of which SharePoint Service Applications and BizTalk Server (on-premise) or Azure Service Bus (cloud) are the most commonly used ones in "Microsoft unless" environments. BizTalk Server is used most often because typically implementing a SOA architecture is done through the use of an ESB, and BizTalk Server can be implemented according to the ESB design pattern. If SharePoint runs in the cloud (SharePoint Online), Azure Service Bus (implemented according to the ESB design pattern) is a better choice. For hybrid solutions, where part of the services run on-premise and part in the cloud, the hybrid Federated Service Bus pattern would be the best solution (see my previous article in the April 2012 edition of this magazine [REF-3]).
SharePoint then just leverages the (Federated) ESB for consuming and exposing services. For simple, SharePoint only scenarios though, Service Applications could suffice. The remaining part of this article however focuses on the implementation of an ESB as a service layer.
External Contents Types in SharePoint can be organized and stored in the External Content Type Repository (see figure 2). For example, the ECT "Customer" can be used by a SharePoint power user to display "live" customer data in a webpart on a page that contains all previously exchanged documents for that particular customer. It is also possible to use several different webparts on the same page that through the use of the connected webparts technology can share key data and display information in the right context. For example, showing the "live" customer data, and showing all the relevant RFQ's and Quotes for that customer in other sections on the page is an easy task by just sharing the "customer id" between the webparts.
Figure 2 – SharePoint enterprise portal architecture with a Service Layer
The key is that these ECT's are directly bound to services provided on the underlying ESB. When the Project Site home page for "Project X for customer Y" is displayed, it will retrieve the live Customer, RFQ and Quote data for Customer Y through the ESB, which in its turn gets the data from the underlying back-end systems and services through (orchestrated) web services calls.
It gets more interesting if service composition is needed. For instance, when providing an order interface for your customers in the extranet section of your portal. The "Order" ECT would then consume the Order service on the ESB, which could be implemented as a Scatter Gather pattern like depicted in figure 3. The bus exposes the Order service as a web service consumed by SharePoint through the ECT. The Order service is implemented as a composite service that invokes several other services based on the needs. For example, it could invoke a service provided by a CRM system to check if the customer is a known customer and provide a reference to it. The next step could be a credit check provided by a cloud service. A final step could be submitting the customer's order to the ERP system. The results of all the service calls are aggregated and returned to the SharePoint ECT, which will use it to display the results of the order entry in the portal.
Business processes can be modeled within the ESB, which can combine the various backend systems' functionalities, and is able to expose them as composite services for consumption, by SharePoint. The business process that runs in the ESB is responsible for orchestrating the interaction with all underlying services, and handling this process as a single transaction that can rolled back if necessary.
Figure 3 – Scatter Gather pattern through the ESB
The ESB makes use of its own services to fulfill the requests, such as:
- Itinerary web service: Determine which itinerary (or routing slip) to apply to the request
- Routing service: Routing the request to the different endpoints
- Protocol adaptation service: Make it possible to invoke services that are not based on standard web services
- Transform service: Transformation between requests and canonical data model and canonical data model and service providers
- Process orchestration: For more elaborate orchestration (beyond itinerary actions) of processes, including the use of the business rules engine to make decisions
By implementing a full featured ESB like BizTalk Server or Azure Service Bus underneath SharePoint, the following extra advantages will become available:
- Service location and version transparency
- Transport protocol conversion
- Error handling and repair
- Data format transformation
- Message interactions
- End to end tracking and tracing
- (Business) activity monitoring
The last one is a particularly interesting one. Because all the (composite) services handled by the ESB are monitored already, you can easily extend the information being monitored to the level that Business Activity Monitoring becomes possible. By tracking milestones in your business processes or even tracking certain business content retrieved from the transactions processed by the ESB, you can feed your Business Intelligence solution (for example through SQL Server and SharePoint BI capabilities) and use the information to fine-tune your business processes. A true Business Process Management (BPM) implementation is not farfetched anymore then.
Performance, Latency and Mobile Access
Performance and latency is always an issue with any integration solution or pattern. Most of the times, the performance of the underlying services is the bottleneck. But in order to make the ESB itself perform as quickly as possible and to be able to handle large volumes of service calls and transactions, a number of things can be done. First, it is important to notice that BCS does not support caching (apart from metadata caching). Second, it is important to know that in an ESB that provides guaranteed delivery, transaction handling and restart ability, it is evident that data and milestones are persisted to a database. Therefore, two things are absolutely crucial to implement in order to have a great performing ESB underneath your SharePoint:
- A dedicated, high performance and scalable database
- A forward cache
The 2nd point is a typical solution to provide for high performance, low latency service call replies in environments where lots of the same data is queried over and over again, such as customer data, employee data, etc. This pattern is described in Enterprise Integration Patterns [REF-2].
By instrumenting your ESB correctly by means of performance counters, correct milestones and other metering information in your activity monitoring layer, it is always possible to at any given time determine where bottlenecks are in your integrated application landscape and adjust accordingly.
On a last note, SharePoint portals can be implemented and extended as such that touch-friendly mobile access is possible as well. There are several technologies available for developing such solutions, such as jQuery Mobile. A lot of times, SharePoint is implemented in such a way that you do not even know SharePoint is the platform underneath all the user interaction. For example, lots of SharePoint implementations use the Office Applications as the end-user interface, such as Outlook. A human workflow started from within a process orchestration in the ESB could easily create a task in Outlook through a SharePoint workflow. Mobile access to your Outlook or other Office applications then automatically takes care of the rest.
While implementing a SharePoint enterprise portal with the above guidelines in mind, a true any time, any place, any device environment can be created that will deliver the right information in the right context at the right time to employees, customers and partners.
Leveraging SharePoint as an enterprise portal requires extensive knowledge of the SharePoint platform architecture and service-oriented architecture. By building and exposing services according to the principles of service design and leveraging the services provided by an ESB (either on-premise or in the cloud, or in a hybrid fashion using the Federated Service Bus design), any SharePoint enterprise portal becomes a much more scalable, supportable and sustainable integrated solution.
Implementing SharePoint on top of a SOA based service bus makes it possible to design true BPM solutions that can help companies quickly adapt their business processes to the employee's and customer's needs.
[REF-1] Erl, Thomas, "SOA Principles of Service Design", Prentice Hall, 2007. http://servicetechbooks.com/psd
[REF-2] Hohpe, Gregor and Woolf, Bobby, "Enterprise Integration Patterns", Addison-Wesley Professional, 2003.
[REF-3] Veld, Gijs, "Guidance for Integration Architecture on the Microsoft Business Platform", Service Technology Magazine, April 2012. http://www.servicetechmag.com/I61/0412-4