Benedikt is a Consultant working in Integration and CRM projects for large enterprises. He has experience with different technologies, governance frameworks and roles, from developing and designing to leading and budgeting. Benedikt is an active contributor to Bitcoin meetups, and he follows discussions and congresses about cryptography and enterprise security. One of his main interests is tackling well-known enterprise software challenges with cryptography and blockchain technology.
Integration – The Blockchain 'Killer Usecase' – Part I Published: December 31, 2015 • Service Technology Magazine Issue XCIII PDF
Abstract: There is an interesting feature of the Bitcoin protocol that doesn't receive the attention we should attribute to it. Bitcoin is able to integrate and have endpoints (in Bitcoin terminology 'wallets' and 'miners') seamlessly talk to each other in a large and dynamic network. Anyone can easily setup a wallet and within minutes will be able to interact with other members of the network.
That cannot be taken for granted. Take a typical Enterprise for example at a Telecommunications company. Within their enterprise network you will find a large amounts of disparate sometimes small, often large Application like CRM, Billing which need to talk to each other, for example to exchange customer sales orders.
Making these application talk to each other is a huge problem: Check the top X lists why large IT projects fail, you will find Application Integration issues. Search for Enterprise Service Bus or Middleware and you will see names of the largest software vendors on the. Consider Internet of Things and its growing demand to integrate protocols of different devices and you will get a sense of the magnitude of the problem and the size of the opportunity.
This paper will try to show, where Bitcoin and the underlying Blockchain and Consenus Technology offer a different approach to integrating members of a network. We will show how the Bitcoin Blockchain manages to integrates members of a large dynamic network via enforcing not only one set of common data but also one language only to communicate. Trying to adapt to the situation in enterprises with multiple disparate Applications, we will suggest a version of Bitcoins 'proof of work' consensus mechanism to translate messages from different Applications into each other without reverting to a trusted middle man. In the last part, we will discuss, how one can describe the dominant approach to Enterprise Application Integration as another form of the centralization approach banks take to facilitating monetary exchange. Our conclusion here is no different than contemporary Service Oriented Approaches like Microservices but we come to the conclusion from a different set of arguments and suggest different solutions, which are inspired from Bitcoin and Blockchain design patterns.
The most important motivation for this article is therefore to try to arrange a marriage between two ecosystems of current Information Technology. Blockchain is a superior and elegant technology, but it has very specialized financial and legal use cases. Enterprise Integration represents a major business case, but orbits around service orientation and message exchange patterns. Brought together, these two citizens of IT society would form a marriage in heaven.
Integration – Some Major Difficulties
Enterprise Integration Problems start where languages differ.
Consider the case of a CRM and Billing System and a Provisioning System in a Telecommunications enterprise. Client order are taken in with the CRM System, the order is send to the Provisioning System, which sets some switches in the network to give clients product Internet or TV channels. The Provisioning System might kick off a Shipping System and initiate sending over a modem and other hardware. The Billing System will be responsible to generate a bill to the client and will do so with recurring cycles after the Provisioning System reports the order has been fulfilled. All three Systems (in fact there are many more) know the concept of a Client and of Products, but they have different ways of talking about them and their internal Data Models differ. Often, this is the case simply because such systems grew historically and were produced from different vendors. Sometimes this is the case because different systems are interested in different parts of concepts, for example the billing system doesn't have to know about technical details of a product while the provisioning system doesn't have to know about prices. There are no structural reasons that languages about common concepts differ, in fact you will find in most many enterprises and industries efforts to standardize ways of describing concepts, because having one language to talk about the same things in different Systems would greatly reduce effort to integrate such Systems. Albeit such efforts, Systems remain heterogeneous partially because they historically grew as they are sometimes because some software vendors don't have an interest to integrate with anything other than their own components, mostly though coordinating concepts between Systems and agreeing on one nomenclature is a task not be underestimated and even smaller enterprises fail to have sufficient governance in place for this.
The most common approach to overcome that 'language barrier' these days is to place a so called Enterprise Service Bus (ESB) in between these Systems. An ESB has several tasks, among other things he provides the technical infrastructure to exchange mass amounts of messages. As part of this, this piece of software is connecting all participants in the communication and sends messages between the systems to update their local data stores. This way, a client in CRM, Billing and Provisioning should always have the same address.
Often, these layers offer a very generic language for common concepts like clients or products. Here you will find aforementioned Industry or Enterprise Standard, which that Organization is trying to enforce. All connected Systems need to map their application specific concepts to this generic layer. One of the promises connected is that this way, replacing one of the systems e.g. the CRM can happen without an affect to another system, e.g. the billing system. All necessary is that the new CRM system needs to translate its concepts to that generic language and all other systems will never have to know about that change. Of course in practice things are not quite that simply.
Even if this generic language approach would work, there is still a major problem when you replace your old with a new CRM system: you have to get the data from one old system to the new one. Experience tells that this is a lengthy and sensitive task for enterprise clients, even though after all two different CRM Systems should have more or less the same understanding of among other things Clients and Products. This is because computers are much more rigid in concepts than our human language concepts, they think in tables and fields and even subtle differences can cause major difficulties migrating data reliable.
The Bitcoin Blockchain turns these paradigm around: The transactions, the Blockchain is permanent and primary. Endpoints like wallets and miners plug in to the Blockchain, derive their status (e.g. balance of your Bitcoin account, the public key) from it and write back any updates (e.g. sending Bitcoins to another public key) to the Blockchain, where it is accessible to anyone with the right public / private key combination. We will discuss this in the next section.
Before we continue, here is an approach to solve these integration problems. So called 'Master Data Management' (MDM) approaches are all about storing Enterprise Data, which is relevant for several applications, in one central database. This could be a customer or a product database, these master data management databases then feed applications in need of this data like a CRM or Billing System and hence represent one single of source truth. Whereas such an approach can succeed in storing data in one central place (provided am enterprise has rigid governance in place to enforce this) this approach does per definition not succeed in establishing one common language for common concepts: CRM and Billing might still have different data models for client data, even though they would be fed by the same data source. Therefore, a Master Data Management System will have just as many difficulties integrating with CRM and Billing as CRM and Billing will have integrating between each other as long as there is not one unified language in place. Using the EBS's generic language to mediate between the MDM and the slave Systems will also cause just the advantages and disadvantages as using this generic layer to communicate between CRM and Billing.
Our point here is not that the described problems cant be solved conceptually, that indeed long happened. One creates generic message exchange languages, one builds up Master Data Management Systems and back these up by organizations trying to enforce language standards and proper usage. Reality bites, the size of the market speaks a clear language, it is as big as it is because the problem in practice is not solved and consequences for clients and IT as a whole are severe because a large part of financial and talent resources get used up in this endless 'integration battle'. Any IT Implementation knows some external systems, where someone has to take the effort to translate data from one to another format be it towards an outdated printing street with an unknown data format or towards the 'latest & greatest' update of a Billing System, where the vendor decided to change dataformats and API's and is only half hearted about enterprise and industry standards. Most IT Implementation Projects have to deal with some kind of data migration, where most if the difficulties arise from transferring data from one format to another without achieving anything other of business value than doing just this translation. When it comes to communication – and gladly not only there Computers are very different from human beings, for different teams with different nomenclature it is easy to communicate about the same thing ('the client', 'the product') even though they look at the concepts from different functional context like CRM and Billing.
Did You ever hear of any such problems in 'Bitcoinland'? Not me, but Bitcoin is among other things also integrating different wallets and miners. So, what is Bitcoin doing better?
Using Blockchains in an Enterprise
Some Bitcoin and Blockchain Background
Bitcoin was invented in 2009 and combines principles of distributed Systems and cryptography such that it is able to create a currency out of an algorithm with no back up of a government or central bank and is able to establish a payment system with a bank and its infrastructure. An indispensable feature for this is that it is able to create trust within those participating in Bitcoin transactions. Technically, Bitcoin and other crypto currencies do this via implementing a distributed System, where all nodes hold a copy of a Transaction Ledger recording all Transactions between owners of Bitcoin, which are identified by public cryptographic keys and are in principal anonymous. Owners of such transactions sign transactions by private keys, which are only known to them and allow the Bitcoin network protocol to verify ownership. The underlying data structure, the distributed Blockchain has the most astounding feature, that any transactions that happened a certain period of time ago are by any practically means immutable. Bitcoin achieves that by using hashing algorithms in a smart way. Transactions are assembled in blocks and block headers always depend on previous blocks, hence trying to change transactions in a block, which was added to the Blockchain a while ago, requires to always recalculate all ensuing hashs. The community debates one risk of fraud under the label '51% attacks' for the hypothetical case where a node manages to assume control over a large part of the network. Such an attacker might want to double spend its Bitcoin, that is pay a merchant, who sends you some goods and then try to pay very same Bitcoin to an alter ego public address of yourself.
Even if we assume an attacker would be able changing the Blockchain in a malicious way, difficulties start before that: Computing power within a distributed network can be huge and is difficult to control and coordinate for any such attacker trying to assume power. With the popularity of Bitcoin, there is also a significant social control: if an evil attacker would try to assume power this would immediately go through the news and there would be social and legal sanctions against such a player. In that respect the Bitcoin eco system isn't different from ant other system, that enforces its rules by social control. Bitcoin however offers algorithms as part of its protocol that make it mathematically quite difficult to abuse the System.
One function of Bitcoins is that they serve as an incentive for network members, so called 'miners', to find hashes necessary to build Blocks. Miners compete to find solutions to artificial riddles for essentially finding numbers with certain features. Whoever solves the riddle first, will get rewarded and can mine some Bitcoin via placing a respective transaction in its Block. Blocks represent a set of transactions. Transaction typically send an amount of Bitcoin from one public key to another one and are broadcasted to all members of a peer 2 peer. Mining is a metaphor reminiscent of the fact, that Bitcoin money supply is designed to be limited. Like in a mine in the early days of exploitation it will generally be easier to find a natural resource, it used to be very easy to mine Bitcoin but with time it will become more 8 and more difficult. Miners will have to fallback towards benefiting from fees users of the Bitcoin system can pay to incent executing transactions fast.
Anyone who solved such a 'riddle' has also submitted a so called 'proof of work' to the protocol, that is a proof that a miner used a certain amount of computing power to solve a cryptographic riddle. These riddles are just designed the way that from a statistically point of view they always require a certain amount of (guess work to be solved. 'Proof of work' is a key to ensuring security, because one attacker would need to bring many 'proof of works' to change the Blockchain and hence have to assemble so much of computing power, that cheating is not only difficult but has high economic costs. The better strategy to maximize profit is to 'join the crowd' and mine 'honestly'. In a non centralized system, where all participants will seek to maximize their benefit, the most promising strategy is to 'go with the rules', for which the System was designed.
The Bitcoin network shows that a currency working based on algorithms can create true financial value for miners powering the Blockchain and can be used by ordinary people to execute transactions in a network that has no notion of banks or any other trusted central middle man.
A more recent attempt to use Blockchain Technology for a wider context of usecase is the Ethereum protocol. Ethereum attempts to achieve this by offering a turing complete language on top of their proprietary Blockchain. Bitcoin offers a turing incomplete scripting language for executing simple verifications on transactions. Bitcoin restricts itself deliberately, because turning complete languages are exposed to the so called halting problem. Its relevant form for Bitcoin and Ethereum is that a program which uses loops, can never for certain be pre determined to ever stop. As a result on attacker could use this for a Denial of Service attack. Ethereum introduces the notion of 'gas', which essentially is a mechanism attaching costs to executing a program in this Turing complete language. If there is no more credit the script will simply be stopped and denial of service attacks are therefore only possible with infinite credit. Ethereum introduces its own currency, called ether. Core use cases Ethereum support are any type of crypto currency and among other things so called 'smart contract'. These are contracts, which execute and enforce themselves automatically.
Think about a 'kill switch' in any electronically empowered product you would buy. You buy a car on credit but fail to make your crypto currency repayments. If you signed a smart, electronically scripted contract your car simply locks its doors and in the not so far away future potentially even drives itself back to the showroom. Of course, the dealer might as well call the local police, but having such a smart contract might be much more efficient and safe for the dealer, since he doesn't need any state authority as a trusted and potentially costly middle man. Even more, imagine you trade with strangers on the other side of world. Smart contracts could enable people to trade and do business with strangers, usually over the internet, without the need for a large centralized authority site to act as a middleman.
Ethereum uses independent software agents to codify such smart contracts on top of their Blockchain platform in a Turing complete language. 9 Ethereum chooses a different Consensus Mechanism than Bitcoin. Like with other crypto currency, there is dissatisfaction with the costs of the computing power (electricity, hardware) the Bitcoin protocol implies. 'Proof of work' is essentially expensive CPU work with no other goal than achieving consensus on the state of the Blockchain in a distributed system. Proof of stake asks users to prove ownership of a certain amount of currency (their "stake" in the currency) when deciding n the correct state of a system, the general idea being that stake holders will have less interest to fraud, because they endanger the acceptance of the eco system and the value of the currency they themselves own. Proof of stakes save energy costs but are inherently more complex than proof of work mechanisms and there is an ongoing discussion if they can achieve the level of security the proof of work consensus achieves.
We will come back to this discussion later, while suggesting a mechanism to use proof of work to translate messages of Systems into each other.
Private Blockchains – Confidentiality in an Enterprise
Here is one important aspect of using Bitcoin and Blockchain inspired technologies in an Enterprise Context, that will need to be tackled to make Blockchains fit for the enterprise market. 'Trust' within an Enterprise Network must per se be managed differently from 'Trust' within the internet. Within an Enterprise Network, there are numerous security layers within the network, the operating Systems and Application all with the goal to ensuring that only those who should access have access. But Bitcoin 'lives' within the Internet, anyone can download the Bitcoin Blockchain. Even though messages are encrypted, in many cases a smart researcher can understand patterns and connect the dots and find out who is behind a seemingly anonymous private key and what transactions he undertakes.
For most enterprises, exposing its transaction in a distributed network to any network client is not acceptable, even if the data cannot be changed and even if there might be advanced cryptographic algorithms claiming to maintain privacy and confidentiality.
But then, within a closed and secured enterprise network, one might not see the benefit of an immutable Blockchain with hashing, cryptography and consensus mechanisms. One also doesn't need a currency to incent participants: one buys servers, software licenses and pays employees their salary. A properly setup transaction log might be all what is left of the Blockchain in an Enterprise and to solve the described Enterprise Application Problems, this might just be enough. Storing Transactions alone, if immutable or not, would enable traceability and allow enterprises to run analysis and build intelligence on their data with a granularity unseen so far. That would be a desirable side effect (if not more) of using a Blockchain inspired Transaction Log with an Enterprise. All that, integrating your applications and enabling Intelligence on top of Your Blockchain can be reaped without the Blockchain immutability features.
Nevertheless, also within an enterprise context there are many situations where immutability of transactions is desired
Using Blockchain Technology knows valuable use cases also within secured networks. However giving up an open distributed System implies a gradual shift towards less Decentralizations and that in itself implies risk of loosing many desirable features of the Blockchain. Think about how much easier it could be for an evil player within an Enterprise network – if he assumed power – to overpower other miners, just because there will not be so many of them and their computing power much lesser to overcome. In practice, for most real life use cases one can be optimistic that there will be a compromise between confidentiality (closed networks and participants in the mining and consensus algorithm and immutability (decentralization with many participants).
One can search for such 'consortium' or 'fully private' Blockchains, some further reading is also listed below. We will not investigate this part here further but investigate Blockchain Integration Patterns, which would be relevant for any Blockchain Implementation, be it public or private.
Blockchain Integration – the Master Data Language Management Approach
Bitcoin turns the relation between Endpoints and Integration Layer upside down and that is the reason why wallets or miners can so seamlessly integrate with each other. In a typical Application landscape, applications hold the values of endpoints, they define which products a Client has. The Integration layer sends over orders (combination of clients with instances of products) from CRM to Billing to update another application of the status changes. The application end points are primary and permanent. The integration layer is a message oriented system and messages are derived from values on the endpoints and are temporary.
Figure 1 – Blockchain vs. Enterprise Service Bus Integration
In some aspects the ESB communication pattern looks like a pattern humans have: There are different functional teams like CRM and Billing and they send each other emails to update each other on statuses. This is a good way to communicate for humans, who are able to translate and communicate to each other about common concepts like clients, even if they represent them differently. For IT Systems, with their inflexible and rigid way of representing concepts, these subtle differences can cause major difficulties and lead to the problems we described above.
The 'trick' of the Blockchain here is to turn around the relationship between permanent endpoints and non permanent message layer. Blockchain takes the transaction between addresses as permanent and primary, wallets hook on to the Blockchain and derive the status of the public keys for which they hold the private keys. The fact that the Blockchain is a distributed System and works with cryptography and a consensus system can for the Integration question be regarded as optional. Bitcoins way to enable communication in large distributed Systems is to have the transaction layer as the primary and permanent single source of truth.
Talk of 'Single source of truth' sounds an awful lot like terminology known from Master Data Management, where data relevant for several application is stored in one database and is meant to feed other systems. A Blockchain is a large 'single source of truth' and in that sense representing a Master Data Base.
However, one important difference to a traditional Master Data Management System is that the Bitcoin Blockchain is existing as several copies of this Master Data Base on several nodes. Hence, logically we have one large Master Data Base but from an Implementation point of view the System is distributed and all full nodes hold a copy. That has important consequences on politically and economically desirable attributes of decentralization, which make a significant part of the attractiveness of the Bitcoin Blockchain. No one central agency is required to build trust in the network, the network equipped with decentralized copies of the Master Data Base and the Consensus System is able to generate security and build trust among network participants all by itself, therefore eliminating the need for a middle man like a bank.
Another important difference between a Master Data Management and Blockchain, is that the Blockchain is not only Master of the Data, but that it also controls the language used to communicate between endpoints. There is no other language than Blockchains transactions to talk about transactions in Bitcoin. It is not as if any wallet could introduce its own Data Model and the Bitcoin Blockchain would then mediate and translate between different wallets. The Bitcoin Blockchain is hence not only a Master Data Management (MDM) but also a Master Language Management System (MLM), which for the sake of brevity will refer to as an MLDM (Master Language & Data Management) System.
Here are some aspects in which the Blockchain differs from a traditional Master Data Management System.
Point (2) is illustrated by the Ethereum Blockchain. The Bitcocin Blockchain is designed for financial transactions but even the Bitcocin Blockchain is 'abused' by some for storing information permanent und immutable. The Ethereum Blockchain tries to generalize the concept of a Blockchain and use it for storing any kind of information so called 'contracts' deem valuable. Contracts can be seen as agents running on top of the Ethereum Blockchain with a Turing complete language executing tasks typically attributed to smart contracts, e.g. pay out a bonus to an employee if his performance rating is such and such (internal data feed) and stock values (several external data feeds) exceed values xyz. Blockchain technology can address a broader set of questions than a simple Master Data Management System can.
The last point is worthwhile considering for a moment. Master Data Management Systems as currently known iterate the Integration Problem: Just as CRM and Billing have different nomenclatures for the same thing ('the client'). A Master Data Management System will have its own understanding of 'the client' and hence have the same difficulties to integrate with CRM and Billing that CRM and Billing have between each other. MDM as known does achieve having one set of data, that represents a single source of truth. It does not, per design, solve the issue of having different languages for the same thing, which need to be translated into each other: 'Unified Data: yes. Unified Language: No'. Blockchain Technology enforces not only one common set of data, it also enforces the nomenclature, one language for all concepts. Wallets plug in to the Blockchain and either speak 'Blockchainish' or they cant connect. There is no choice and no room for application specific dialects. The Bitcoin Blockchain facilitates communication in large distributed and dynamic networks via this concept of Master Data and Language Management. Cryptography, Consensus Mechanisms and Distribution are not necessary to achieve this.
One way of leveraging Bitcoin Blockchain Technology within an Enterprise would be to try transfer this approach to an enterprise Applicaton landscape. We would have Applications write transaction data, that is relevant for other Applications into a Blockchain. Such Applications would derive the status of important concepts like the Client from this Enterprise Blockchain. They would have the choice to have local databases for Data that is relevant only for them, e.g. notes on calls in a CRM System are not relevant to Billing. One important question would be.
Advantage of such a System would be that just like wallets, connecting new Systems to the Blockchain would require no Data Migration effort. All relevant data would be stored within in the Blockchain and status can be derived. Such an approach could significantly enhance agility, think of a client facing web front end which enterprises might want to replace based n marketing and campaign reasons.
Another set of advantages become apparent, when an enterprise is interested in financial or legally relevant use cases as listed before. Such use cases could be implemented by a Bitcoin style Blockchain, where the Application logic would stay outside the Blockchain or an Ethereum style Blockchain, where one would build agents enforcing smart contract rules on top of a Blockchain. We will not further investigate this approach here, even though we do claim that there is significant potential in these approaches.