It is impossible to analyse central government performance on the basis of the data it publishes online, a new study by the Institute for Government has found.
This means members of public cannot hold Whitehall to account, and departmental managers themselves are not able to assess their own performance effectively.
The think tank looked at over 500 data sets published by central government departments for its first annual Whitehall Monitor report. In doing so, it found serious shortcomings in the way that data is published and organised.
"Whitehall has increased the amount of data in the public domain through its transparency agenda, which is commendable," the Institute said yesterday. "However, while we can get insights on parts of the picture, it is impossible to comprehensively assess Whitehall’s effectiveness."
Some issues concerned the accessibility of data. Much of the information that the Institute analysed was contained in PDFs, making it laborious to extract and analyse.
Other issues related to the comparability of the data. The same information is often presented in different formats or with different labels between or even within departments, making it difficult to make meaningful comparisons.
In particular, it is very difficult to analyse resources (i.e. its budget), how it uses those resources (i.e. policies and legislation), and what the outcomes are (i.e. societal benefits).
"There are insufficient links between the resources that departmental leaders receive, how they are deployed and the real-world outcomes that the government is pursuing."
Addressing these issues would not only help interested members of the public ("armchair auditors", as they often called) analyse Whitehall departments' performance, it could also help departmental managers and even MPs.
"We hope that would lead to better scrutiny from the public as well as from parliament," researcher Petr Bouchal told Information Age this morning.
Tips for presenting data
The Institute makes a number of recommendations for improving the accessibility and comparability of Whitehall data, that could equally apply to businesses. They are as follows:
Access and availability
- Provide an index of key data and an explanation of what it covers
- Label and name data files in an informative way
- Make it clear whether you are looking at the latest version of the information
- Explain the update cycle
- Enable users to find all of the data in a given series easily
- Collate data that is published in frequent releases by multiple organisations
- Signposting other relevant data is helpful
- Enable different users to access data in a form that is most useful for their needs
- Provide key data in a machine-readable format
- Structure data to make it easy to analyse
- Any time data is presented in an 'advanced' way, add a straightforward way too
- Try to make the source data stable where possible
Data purpose and documentation
- Provide clarity as to what and what is not in scope of the data
- Be clear what the data’s purpose is, who it is for and how it should be treated
- Document the degree of confidence there should be in the data and the level of quality assurance
- Devise a consistent set of labels for organisations and ensure these remain stable over time
- Ensure all variables and specialist terms are explained
- Mark missing data consistently to distinguish it from zeros and omissions
- Provide contact details for someone who understands and manages the data if users have further queries or want to report errors
Comparability
- Report on consistent boundaries
- Make groupings explicit
- Provide consistent sets of information across departments
- Document changes between sources to allow reconciliation
- Explain changes over time
- Publish restated series where basis for reporting has changed
The Institute for government's complaints about the comparability of the data echo remarks by data visualisation expert Stephen Few after the launch last month of the UK government's new spending comparison portal, the Government Interrogating Spending Tool.
The site is "either an attempt to obscure the data under the guise of transparency or the work of people who have no knowledge of data visualisation", Few told Information Age. "The charts in every case are either inappropriate for the data or appropriate but ineptly designed."
He criticised the poor comparability of departmental spending data over time. "Data sensemaking consists primarily of making comparisons, but few comparisons are enabled by this system," Few said. "Items that you would want to compare must be viewed separately, such as spending and budget.
"This is an embarrassing mess."
