Francis Maude has said that he would like to make the Freedom of Information Act, which allows citizens to request data and documents from the government, redundant by publishing government data sets online.

"I’d like to make Freedom of Information redundant, by pushing out so much [open] data that people won’t have to ask for it," he said last night.

However, Maude also revealed the reluctance of some public sector bodies to meet their open data obligations, which include making data available in a machine readable format. "One local authority was painstakingly converting their parking meter data from a machine readable format into PDFs."

He spoke about the need for consistent data that could allow meaningful comparisons between departments and over time. "What we particularly lack are comparable data across government and data sets that don’t change [format] over time," he said.

"When [TopShop boss] Philip Green came in to do a review of government efficiency for us, his views on the quality of government data were unrepeatable."

"Comparability is the handmaiden of accountability, which is one of the reasons why it’s resisted," he added. "Some parts of government are unwilling or unable to provide data in common definitions."

"Transparency can be uncomfortable for politicians, which is why it always popular with the opposition, and for the first 12 months of government when we’re talking about [the former government’s performance]," he said.

"We’ve stuck with it, and we’ve taken it further," he claimed. (The Cabinet Office itself has not always met its own open data commitments, however. In 2010, it committed to publishing "regular" data on the performance on major IT projects, but has not updated the information on data.gov.uk since February 2011).

Big data report

Maude was speaking at the launch of a report into the use of ‘big data’ in government, sponsored by storage vendor EMC and conducted by Policy Exchange, the think tank he co-founded.

He took the opportunity to criticise laws that prevent government departments from sharing citizen’s personal data, and made specific reference to attempts to fight fraud.

"The UK loses £36 billion to fraud every year, in large part due to [laws that prevent data sharing]," he said.

"Our attempts to cut down on debt fraud are undermined by seemingly contradictory legal constraints. There are 86 legal gateways that limit data sharing between debt agencies. We want a single legislative framework that allows sharing of data on debtors."

Attendees at the launch issued concern that the use of big data – defined by Policy Exchange as "data sets that are too awkward to work with using traditional, hands-on database management tools" – by government could threaten citizen privacy.

"We have to be vigilant on retaining trust," said Maude. "We are determined to strengthen protection of privacy, and have appointed a privacy expert on our Transparency Board.

"In this hugely powerful big data movement, if one set of anonymised data allows personal data to be revealed, it will set the whole practice back," he said. "At the same time, we cannot let privacy be used as an excuse not to release data."

Privacy Exchange’s report recommended that the government adopts of a "Code for Responsible Analytics", in order to ensure that big data analytics does not lead to unethical use of data. This should include an agreement only to use big data analytics when it serves a specific policy objective; to respect privacy; and to develop big data systems in experimental conditions – i.e. not using live citizen data.

It also recommended that the Cabinet Office create an "elite data team", staffed with "data scientists" to identify big data opportunities. The budget for this team could "in the low millions of pounds", but could achieve savings for central government "worth at least £1 billion".

Maude said the report had given the Cabinet Office "much to think about". James Petter, EMC’s UK and Ireland managing director, said the company "fully endorses the proposal of a specialist data force in the Cabinet Office".

He added that EMC would "offer the use of" its Greenplum Analytics Workbench, a hosted service for testing and developing Hadoop-based big data analytics systems. EMC has confirmed that use of the Workbench service will be free of charge for the government.

Last month, the Government Procurement Service announced a forthcoming software procurement framework that could be worth up to £1 billion over two years. It included a lot to provide ‘big data’ software and services, which covers "the implementation, operation and support of big data solutions to efficiently process large quantities of data", including massively parallel processing (MPP) databases, data mining grids, and distributed file systems".