Once upon a time HTTP and HTML were trucking along nicely. This meant that we had a truly open, meaningfully interoperable framework to allow for rich human based web interactions. The next obvious step was to look into machine to machine web interactions. A bunch of work went into this area. XML, WS-*, POX, JSON, RSS, OData, etc. With the exception of RSS none of it came to much of anything, especially in the personal space. In this article I look at why I think the previous efforts to create an open data web failed.
Ads
When the Internet was first getting broad public attention there was a major question of how it would make any money. The value of what the Internet in general and the web in particular provided was obvious, but who would pay the bills? The answer turned out to be ads. But if you make money via ads then you absolutely must control the entire end to end user experience so you can make sure that your users will see your ads. This means that any kind of data level interoperability was anathema since it meant people used your infrastructure without seeing your ads.
Feudalism
Because of the math of network effects if you can build a big enough network then displacing you is very hard and keeping off competitors is pretty easy. This makes Internet services a winner takes all proposition. So everyone is out to create their own feudal kingdom and rake in the bucks. Well the last thing you do when building a castle is open a bunch of holes in the walls so interoperability in an open data web is maximally uninteresting.
Bonus Points: why the combination of ads and Feudalism killed privacy for everyone
The amount of money that advertisers will pay to have their ads shown/clicked is directly related to how much information the service showing the ad has about the person who will see the ad so the advertiser’s algorithms can decide if the user is interesting to them. In addition since many ads are paid for ’by the click’ as a service showing ads you want to maximize the probability that you will show ads the user will click on. The way you do that is by collecting as much information as possible about the user so you can model their likely behavior and pick the right ads to show them. So the ad business drives a situation where those showing ads wants to collect as much information about users as possible, even information that has nothing to do with the functionality the service provides. We have essentially incentivized all Internet services to want to turn themselves into Big Brother.
But it turns out that the network effects that led to feudalism also work in ads. If you manage to collect the largest, most accurate, collection of information about users and if you can keep it locked up in your service then you become the most compelling place to show ads and so can charge more. This becomes a zero sum game as advertisers send their dollars to you and thus starve your competitors. So not only do you want to know everything about everyone but you don’t want to ever share that data.
So now what?
If we want an open data web then we need to change the dynamics. I must admit that I’m not quite sure what the solution is for an ad based economy. Micropayments seem like a fairly obvious choice but I just don’t know. However I have a suspicion that much like how the Internet and the web developed, if we build a compelling enough experience with an open data web then this will motivate a bunch of folks to figure out the economics aspect. So I’m going to focus on actually building the open data web which means tackling feudalism. The details of how to do that will be in a future post.
Disclaimer
It’s worth noting that this work is being done on my own time with my own resources and is completely independent of my employer. So whatever you think of this article don’t blame my employer, it’s not their fault.