Sorry...we cannot share the data with you !

The truth is in the verb

One of the most tragico-amusing facts that most researchers, students, consultants etc get mis-led by when going data hunting (especially for digital data) in government offices, is that when the concerned officials say, "We cannot share that data with you !" (which they almost always do) - they are telling the truth....as in literally !

For sure, they may not "want" to share it with you, but in most cases they simply "cannot".

In order to share any data - they must first have it in their possession. Well, it's a bit hard to part with what you don't have isn't it ? 

So, let's not be overly harsh on the officials of refusal, and focus instead on what are the deeper structural reasons which create this absurd situation, where the government itself is not in effective possession of its own data. And just as Thomas Piketty often says, that the structural economic problems such as wealth inequality can be better understood by resorting to the novels of Austen and Balzac than the works of economists, I would like to approach our present puzzle through the medium of personal experiences and anecdotes...and believe me, I have tonnes of them. 

Looking for data...and finding the cave of Ali Baba

Couple of years ago when I started working as a consultant for the Housing & Urban Development Department in Odisha, one of my tasks was to collect whatever data was available in the various offices of the department on the slums of Odisha. 

All through my years as a student and researcher of city planning, I was accustomed to all kinds of data collection warfare that one has to conduct in government offices - combined arms, protracted, geurrilla, psy-ops, corridor-corner-mugging...you name it - to get even the most obvious and useless bits of information. So, going data-digging with the seal of the Principal Secretary was like a walk in the park and I was quite enjoying the breezy feel of it. Even so, I didn't really expect to find anything useful at all. 

After visiting desks and cubicles, where I would be kindly directed to other desks and cublicles (I love strolling the halls and corridors of bureaucracy...it's not half as masochistic as it sounds and it has its benefits), I ended up in a corner desk in the office of the Bhubaneswar Municipal Corporation (BMC), where I struck a conversation with another consultant who was nearing the end of his contract. He seemed to remember that there was some data stored in a desktop, which was barely ever switched on. As he searched for the folder, he reminisced more, "This data was collected during the RAY scheme...then the scheme was abandoned, and no one used it anymore."

As many would remember, RAY was a promising scheme of the central government to create slum free cities which met a pre-mature end in 2014 when the Congress led UPA-II government was defeated by the BJP led NDA. While the scheme was abandoned before any serious implementation could be done, it seemed that it did generate some data before dying.

When he finally opened the folder, it turned out to be a treasure cove containing detailed shapefiles of each of the 436 slums of Bhubaneswar, organised neatly in 67 sub-folders corresponding to each ward of the city. The slum-level data consisted of separate map layers for the slum boundary, land ownership, open spaces, infrastructure and dwelling units along with the full household survey data of over 90000 households. Then there were city level map layers showing the city agglomeration area, the muncipal area, municipal wards, location of slums and the road network. From the pdf files containing the layout maps prepared for each slum, one could even find the details of the private consultancy that had prepared the maps (that's always a lucky find, as GIS databases in India almost never contain any metadata files....more on the horror of GIGO - Garbage In Garbage Out - of digital data in later blogs).

The RAY data that we discovered in that abandoned, dust-coated desktop, had been produced sometime around 2013-14. Judging by the time-stamp in the pdf files it must have been handed over to the Government in 2014.  

Data absence through Data obsolescence

Whether, a scheme gets abandoned or not, there is nothing stopping the government from making full use of such a comprehensive and detailed database and also update it in the process of using it. However, a general lack of familiarity with GIS  (although everyone keeps talking about it all the time...agreed, mostly the talk is on buzzwords and eye-candies and not on serious cartography or computing), the dependence on expensive and proprietary software (which ensures that the GIS consultants produce and process the data...and the government has eyes on it only through the consultants and for just as long as the contract period lasts) and lack of awareness regarding free and open-source alternatives creates a situation where the government just does not have the systems and capacities in place to "receive" the data even if it is "handed-over".

Well, what inevitably happens to something which you never use ??....you gradually forget that it even exists. And as the consultants (for what is the government nowadays but an army of consultants embedded in various departments and offices doing everything from preparing reports, drafting standard operating procedures, analysing data, preparing powerpoint presentations...you name it) involved with a certain project leave at the end of their contract period, the last traces of the organisational memory regarding the data disappear with them. 

It was clear from the RAY survey guidelines and the database that I discovered that the survey involved an elaborate process involving the collection of cartosat-1 (2.5 m resolution) and, the now decomissioned, cartosat-2 (< 1 m resolution) satellite imagery; differential global position system (DGPS) surveys to mark the edges of each structure inside the slum; extensive digitization to prepare the maps; detailed household surveys of over 90000 slum families etc. For sure, it must have been an expensive process too. 

And here we were, 3 years later, where the best case scenario for the data turned out to be that it was forgotten in a desktop in the corner of BMC to be accidentally discovered by me on a random data safari. If this happens in one case, then, for sure, it happens in other cases too, especially considering that the underlying causes remain mostly unchanged.

It so happened, that a few years later, when I showed that data to the then Commissioner of Bhubaneswar in a presentation, he made a sincere request that I share the data with his office as they didn't possess any GIS data on the slums of Bhubaneswar. 

So here I was in a surreal situation where BMC was asking me to share BMC's data because BMC did not possess it. 

I had half a mind to say "Sorry, I cannot share the data with you." 

Well, the trouble was that I..."could" ;))