One part of ridding Tower Hamlets of a corrupt mayor in 2014 was the proof that council grants were being made on the basis of who you voted for, not what you needed. This was done using ‘Open Data’ published on the council’s website.
It is ironic that over a year later, with a democratically elected Labour administration in power publicly dedicated to ‘openness and transparency’ that the Open Data published now is less accurate than under the previous corrupt administration.
No malpractice or criminal intent by any individual is implied but something is fundamentally wrong somewhere.
LW knows a reasonable amount about how P2S data should be published as our Open Data Analysts spent a year digging through P2S data files before striking gold.
Here are the problems we found with the data published by Tower Hamlets:
- Inaccurately named files (January 2015 containing January 2014 data)
- Inconsistent order of data in fields (sometimes the data might be the first column, other times the sixth)
- Payment data for two months in one file
- Data not formatted at all (mainly dates of transactions)
- Data not formatted consistently
- Internal codes in payment detail fields which are incomprehensible to anyone outside the Council – 83168/5618 means what exactly?
- Up until the ‘Payment for March 2015’ file the data is presented in the form of a tabbed Excel file, one tab for each Council directorate then one for all Payment Card spends.
- Ordinary payments and card payments have very different data fields. But beginning with the ‘April 2015’ file the card payments have been included in the single tabbed Excel file. They are also identified as ‘payment card’ by an entry in the ‘Transaction Number’ field. Which is, at best, bad practise. Two key data fields specific to payment cards are also missing.
This makes the payment card data far less informative than it should be.
And less informative than it was under Lutfur Rahman.
In practise this means that because of the issues listed above it is that much harder for ordinary residents to check the Payments to Suppliers over £500 data.
LW is used to this sort of work and it is real a pain for us. We download, clean and format all the Council data then put it into our own database so we can check for anything interesting. It adds to our workload and means we have to spend more time doing what the IT Department of the Council should be doing and less time analysing data for interesting stories for residents to read.
Poorly presented Open Data may lead to inaccurate analysis and incorrect conclusions.
Presenting the data properly is really basic stuff. Really basic.
LW politely suggests that the Council gets this sorted because without getting the fundamentals correct the internal procedures of the Council will remain opaque.