Open Data isn’t free data

An early mashup using open data from Chicago’s police force

I support the principle of Open Data; and I’ve been creating commercial value from it since at least 2007, when as part of IBM’s Emerging Technologies team I developed scenarios to show how our customers could exploit it using early implementations of “Mashup” technology.

Here’s an example of what we were up to in those days, using alpha code for IBM Mashup Centre to integrate open data from Chicago’s public services with business data from insurance applications running in CICS. CICS is a transaction engine that’s now 43 years old and is used by 90% of Fortune 500 companies. When you take money out of a cashpoint, book an airline seat or renew your home insurance, there’s a decent chance CICS is involved somewhere. So there was (and is) vast economic potential in what we were doing.

But it’s not always straightforward to realise that value. It’s no accident we based our demonstration scenario in Chicago, which has long been at the leading edge of cities promoting Open Data. (It’s well worth catching up with how Chicago’s new CTO John Tolva is driving this agenda forwards, by the way). At the time, many other cities published similar data; but it wasn’t usable in the same way that Chicago’s was. It had been published in the form that was possible, cheaply, rather than in a form that was useful.

My point is: Open Data won’t deliver the value we all want it to unless we answer some hard questions. Such as:

Who will use Open Data, and why?

There are too many Open Data sites that don’t attract users and activity; so the investment in operating them doesn’t result in the creation of new value. That’s a shame; and we should try to understand why it happens. Often, I think it’s because they focus on making as much data as possible available in whatever form it’s in.

The Knight Commission report “The Information Needs of Communities” emphasised instead the need to consult with communities to find out what they need, rather than to publish data in anticipation of innovation. They are now publishing further guidance on implementing their ideas to promote open government.

Obviously, the problem with the extreme of this position is that if we restrict our Open Data efforts to providing only that data which is proven to be required through extensive consultation, we will limit the opportunity for spontaneous innovation. So a balance needs to be found.

How much does open data cost?

My experience building Open Data scenarios for our early Mashup technology taught me that high quality open data in a useable form was very rare. That’s because it’s expensive.

If producing highly usable information from the applications that manage the world’s information was easy or cheap, a good part of the IT industry would disappear overnight (whether you think that would be good or bad: it hasn’t happened). If we want usable data, we’re going to have to find ways and reasons to pay for it.

The cost to public sector organisations of processing Freedom of Information requests will sometimes provide the business case for spending money to open up data, but not always. Recent Government initiatives to make Open Data a criteria of future procurements will bake the cost of it into vendor proposals; but that won’t address the cost of opening up data from existing systems.

Finally, there will be many cases where clear value can be derived from open data; but not by the organisation that bears to cost of creating or distributing it. In order to balance the need for open innovation with the need to flow cost and revenue between organisations in a reasonable way, commercial models such as “freemium” will need to be explored. The “Dublinked” Open Data portal is doing that, for example.

How do we access and use Open Data?

As William Perrin argued recently, we need to think about how Open Data will be used beyond the community of technologists. I’ve blogged before about the need for technology and information to be accessible; and the need for our education system to provide us with the skills to use technology to manipulate and understand information. Those are both big challenges that we won’t overcome any time soon.

Where do we go next?

The potential value of Open Data is too great for us to afford to be negative, cynical or apathetic. Software automation and information technology are changing the way that value is created in the economy (see work on this from Imperial College and McKinsey), and the concept of Open Data is crucial to providing access to that potential across all sectors of society. But we will only realise that value if we find ways to addressing the cost of providing usable information; and to invest in making it accessible.

Acknowledgement: I’d like to thank Simon Whitehouse for discussions leading to this post, and for the link to William Perrin’s article.