Read this first: Why it matters that the Oireachtas just killed KildareStreet.com. I was going to comment on Simon McGarr’s blog post but the comment grew long enough to merit a post on my own very occasional blog.
I worked in the Oireachtas as a Parliamentary Assistant between 2005 and 2007. At this time there seemed to be a bit of a push behind “eDemocracy” and I was aware of efforts to ensure the provision of acts, bills and debates in structured formats. The grand vision seemed to be that the authoring, amendment and publication of all data related to the Oireachtas would ultimately be compatible with its exposure to the public as structured data. This was very much a work-in-progress at the time, and legacy systems were still centre-stage, in particular the little-loved Lotus Notes.
My understanding of the workflow then and now is that Oireachtas debate records are inputted into Lotus Notes initially and this is the format in which they are first made available to members and staff. Prior to this summer’s changes, they were then parsed into XML files from which the Oireachtas Debates web site was generated. The raw XML files were also publicly available, from which KildareStreet.com was generated. Theoretically anyone could use these XML files to provide web interfaces to debates but as far as I’m aware KildareStreet.com is the only site to actually have done so.
The process of converting the raw Lotus Notes records into XML appears to have been outsourced, either to Cahill Printers or Propylon. Propylon certainly created the XML schema and authoring system and their web site claims they “manage and publish the Verbatim Debate records of both houses in multiple formats”. However this useful site by Leo Bollins, who is the Principal of the eDemocracy Unit of the Oireachtas, suggests that Cahill Printers does the actual conversion.
Mark Mulqueen, Head of Communications for the Oireachtas, confirmed to me on Twitter that the recent changes to the site were designed to achieve efficiencies by ending the outsourcing of “a large amount of work involved in debates. That’s where a saving arises.” I asked him if this meant that Propylon were no longer managing the debate records and he replied, “Yes, I can confirm that to be the case. Using existing resources we will provide access to debates more quickly”.
— mark mulqueen (@MarkMulqueen) September 19, 2012
So it appears that the Oireachtas has decided to save time and money by eliminating entirely the stage in their workflow that parsed raw debates records into XML. This stage has been replaced with a (presumably automated) process that generates web pages from Lotus Notes. It’s easy to see how somebody with little appreciation of the value of providing open public data in a structured format could have viewed this stage as a costly luxury, and its elimination as a simple and obvious “efficiency”. It’s particularly disappointing, however, that nobody in the decision-making process seemed to be aware of how much of a backward step this “efficiency” would represent. As John Handelaar of KildareStreet.com told The Irish Times, “We are replacing 2012 with 1995 overnight”.
The great promise of open data is that it breaks the one-to-one relationship between those who produce data and those who provide services on top of that data. If members of the public want to search or manipulate the Oireachtas records in particular ways, they don’t have to wait for the Oireachtas itself to produce the necessary tools, they can use third-party tools where available, or even build their own. The Houses of the Oireachtas, in their search for efficiencies, have reinstated this one-to-one link, and the public are now entirely dependent on their staff to produce services on top of the debates records. At present this amounts to a keyword search facility that currently produces an “internal server error”. This is not good for the public, and it’s not good for the management of the Oireachtas either.
The standard response of the Oireachtas authorities to the large number of TDs and Senators who have raised the issue is that they want to meet with John Handelaar to try and resolve the issue. That’s the sort of reply that will satisfy most members for the time being, but it reduces the issue to a question of the interface between the Oireachtas and one individual web site (albeit the only web site that has done anything useful with the Oireachtas data). A properly-briefed TD or Senator should be pursuing the issue as a generic open data question: why has the Oireachtas apparently taken such a huge step backwards in its provision of public data in a modern, clean, structured format?
A further pedantic point is that it’s important that queries on this issue are directed to those responsible, i.e. the Houses of the Oireachtas Commision. This Commission, made up of the Ceann Comhairle, Clerk of the Dáíl and a number of TDs and Senators, manages the business of the Oireachtas independent of the government. Its chair is the Ceann Comhairle, Sean Barrett, and his office seems to be the most obvious first port of call.