Sorry...we cannot share the data with you !

The truth is in the verb

One of the most tragico-amusing facts that most researchers, students, consultants etc get mis-led by when going data hunting (especially for digital data) in government offices, is that when the concerned officials say, "We cannot share that data with you !" (which they almost always do) - they are telling the truth....as in literally !

For sure, they may not "want" to share it with you, but in most cases they simply "cannot".

In order to share any data - they must first have it in their possession. Well, it's a bit hard to part with what you don't have isn't it ? 

So, let's not be overly harsh on the officials of refusal, and focus instead on what are the deeper structural reasons which create this absurd situation, where the government itself is not in effective possession of its own data. And just as Thomas Piketty often says, that the structural economic problems such as wealth inequality can be better understood by resorting to the novels of Austen and Balzac than the works of economists, I would like to approach our present puzzle through the medium of personal experiences and anecdotes...and believe me, I have tonnes of them. 

Looking for data...and finding the cave of Ali Baba

Couple of years ago when I started working as a consultant for the Housing & Urban Development Department in Odisha, one of my tasks was to collect whatever data was available in the various offices of the department on the slums of Odisha. 

All through my years as a student and researcher of city planning, I was accustomed to all kinds of data collection warfare that one has to conduct in government offices - combined arms, protracted, geurrilla, psy-ops, corridor-corner-mugging...you name it - to get even the most obvious and useless bits of information. So, going data-digging with the seal of the Principal Secretary was like a walk in the park and I was quite enjoying the breezy feel of it. Even so, I didn't really expect to find anything useful at all. 

After visiting desks and cubicles, where I would be kindly directed to other desks and cublicles (I love strolling the halls and corridors of bureaucracy...it's not half as masochistic as it sounds and it has its benefits), I ended up in a corner desk in the office of the Bhubaneswar Municipal Corporation (BMC), where I struck a conversation with another consultant who was nearing the end of his contract. He seemed to remember that there was some data stored in a desktop, which was barely ever switched on. As he searched for the folder, he reminisced more, "This data was collected during the RAY scheme...then the scheme was abandoned, and no one used it anymore."

As many would remember, RAY was a promising scheme of the central government to create slum free cities which met a pre-mature end in 2014 when the Congress led UPA-II government was defeated by the BJP led NDA. While the scheme was abandoned before any serious implementation could be done, it seemed that it did generate some data before dying.

When he finally opened the folder, it turned out to be a treasure cove containing detailed shapefiles of each of the 436 slums of Bhubaneswar, organised neatly in 67 sub-folders corresponding to each ward of the city. The slum-level data consisted of separate map layers for the slum boundary, land ownership, open spaces, infrastructure and dwelling units along with the full household survey data of over 90000 households. Then there were city level map layers showing the city agglomeration area, the muncipal area, municipal wards, location of slums and the road network. From the pdf files containing the layout maps prepared for each slum, one could even find the details of the private consultancy that had prepared the maps (that's always a lucky find, as GIS databases in India almost never contain any metadata files....more on the horror of GIGO - Garbage In Garbage Out - of digital data in later blogs).

The RAY data that we discovered in that abandoned, dust-coated desktop, had been produced sometime around 2013-14. Judging by the time-stamp in the pdf files it must have been handed over to the Government in 2014.  

Data absence through Data obsolescence

Whether, a scheme gets abandoned or not, there is nothing stopping the government from making full use of such a comprehensive and detailed database and also update it in the process of using it. However, a general lack of familiarity with GIS  (although everyone keeps talking about it all the time...agreed, mostly the talk is on buzzwords and eye-candies and not on serious cartography or computing), the dependence on expensive and proprietary software (which ensures that the GIS consultants produce and process the data...and the government has eyes on it only through the consultants and for just as long as the contract period lasts) and lack of awareness regarding free and open-source alternatives creates a situation where the government just does not have the systems and capacities in place to "receive" the data even if it is "handed-over".

Well, what inevitably happens to something which you never use ??....you gradually forget that it even exists. And as the consultants (for what is the government nowadays but an army of consultants embedded in various departments and offices doing everything from preparing reports, drafting standard operating procedures, analysing data, preparing powerpoint presentations...you name it) involved with a certain project leave at the end of their contract period, the last traces of the organisational memory regarding the data disappear with them. 

It was clear from the RAY survey guidelines and the database that I discovered that the survey involved an elaborate process involving the collection of cartosat-1 (2.5 m resolution) and, the now decomissioned, cartosat-2 (< 1 m resolution) satellite imagery; differential global position system (DGPS) surveys to mark the edges of each structure inside the slum; extensive digitization to prepare the maps; detailed household surveys of over 90000 slum families etc. For sure, it must have been an expensive process too. 

And here we were, 3 years later, where the best case scenario for the data turned out to be that it was forgotten in a desktop in the corner of BMC to be accidentally discovered by me on a random data safari. If this happens in one case, then, for sure, it happens in other cases too, especially considering that the underlying causes remain mostly unchanged.

It so happened, that a few years later, when I showed that data to the then Commissioner of Bhubaneswar in a presentation, he made a sincere request that I share the data with his office as they didn't possess any GIS data on the slums of Bhubaneswar. 

So here I was in a surreal situation where BMC was asking me to share BMC's data because BMC did not possess it. 

I had half a mind to say "Sorry, I cannot share the data with you." 

Well, the trouble was that I..."could" ;))

Reviving a dormant blog...And a few words on Lefebvre, Cartesian geometry and Open-source GIS

The Lefebvrian Paradox

In his famous and influential book ‘The Production of Space’, Henri Lefebvre had written that ‘social space, and especially urban space, emerged in all its diversity – and with a structure far more reminiscent of flaky mille-feuille pastry than of the homogenous and isotropic space of classical (Euclidian/Cartesian) mathematics.’ 

Ironically, in the decades following the publication of Lefebvre’s book, it is precisely the dramatic developments in the application of ‘Eucliain/Cartesian’ mathematics to geographical analysis – namely through the development of various Geographic Information Systems (GIS) software – that helped unravel the innumerable layers of the urban mille-feuille. However, such technologies were proprietary, expensive and largely out of the reach of community groups. For the power of computerised geographical analysis to address the complexities of social space, it had to be accessible by the community and become the technological arm of 21st century participatory development. That became possible with the emergence of the Linux operating system and the free and open-source software movement as a powerful alternative to proprietary software.

Altruism through technical necessity - the revolutionary potential of the open-source movement

The democratic and community oriented nature of the open-source movement is not just a function of altruism (which definitely motivates many members of the movement) but of technical necessity. It is only through the collaborative efforts of millions of developers around the world that the open-source movement is able to create its software. The fact that the opensource geo-spatial foundation describes itself as a ‘not-for-profit organization...devoted to an open philophy and participatory community driven development’ (https://www.osgeo.org/about/) has a lot to say about the movement’s potential role in the field of participatory urban development.

A quick glance at the remarkably informative maps prepared by the citizens of Kibera slum in Nairobi demonstrates the power of this combination. My personal favourite is a thematic map on the status of safety and security in the slum, which identifies dangerous places through a clustering of “bottles” (signifying alcohol vending spots) and “bulbs” (signifying operational or non-operational street lights). A dense clustering of bottles along with black bulbs (non-operational) suggests dangerous locations. The map is prepared using the opensource QGIS software and uses openstreetmap as its base layer (https://mapkibera.org/download/maps/Security%20map%20final.pdf).

The process effectively combines the depth of qualitative information, which can only be aquired through detailed community mapping exercises, with the accuracy, speed and processing power of modern computing. This is where Cartesian/Euclidian mathematics (what are the “bottles” and “bulbs” if not coordinates on an x-y cartesian plane ?) meets the local knowledge of the community. 

The major distinction between proprietary and opensource software is not that one is generally expensive and the other is mostly free. While proprietary software treats users primarily as consumers of technology (causing stready technological dependence, while simultaneously creating an illusion of being tech-savvy), opensource software encourages users to transform into collaborative developers (encouraging community participation and causing technological empowerment).

Pro-Poor Development goes Big Business - Government in hypersonic mode

Although community organisations around the world have effectively used opensource software to map their settlements and exert their agency, the increasing size, scale and speed of implementation of urban poverty alleviation programmes in the global south creates its own challenges.

In this new context it is no longer enough for community organisations to create detailed maps of their own settlements and prepare their own databases. Now, instead of having to deal with a lethargic, inefficient and apathetic bureacracy, they increasingly have to deal with a highly energetic, quick and effective state machinery, which is flush with funds and operates in partnership with large private sector consultants and technology firms to implement gigantic urban poverty alleviation programs simultaneously across thousands of slum settlements in hundreds of cities. Instead of being a reluctant antagonist, the state has turned into an extremely vigorous ally of the urban poor – in both cases community participation may suffer, but the community mobilisation methods of the past are simply not adequate in this drastically altered reality. Many NGOs and community groups have simply not grasped this altered reality well enough.

The challenge of technological development

Moreover, different slum improvement projects would require different types of maps. A Kibera like self-assessment need not map every single dwelling in the slum, but a land titling project like Jaga Mission of Odisha would require an exact mapping of the dwellings (as land rights are issued based on the area of actual occupation). Discussions on participatory mapping are incomplete if not seen in the context of the overall objective of the task and the technical requirements for accomplishing it.

In Jaga Mission, high resolution images of slums were captured using drones and GIS databases of slums were prepared by private sector technical consultants by digitizing these drone images and linking them to the household survey data. The fact that the mapping of 1725 slums spread across 109 small and medium towns of the state (spread across the 155707 sq.km area of Odisha) and the survey of about 170000 slum households was completed within a period of 7 months gives some idea of the speed, scale and intensity of the exercise. It averages out to the preparation of detailed and accurate GIS databases of about 62 slums every week for 7 months straight. 

Jaga Mission achieved this through a creative combination of high technology activities (such as drone mapping and GIS) with field-based community participation activities. About 24 NGOs were engaged to undertake community mobilisation exercises and create Slum Dwellers Associations (SDAs) in each slum. Both the drone mapping by the technololgy consultants and the household surveys by the NGOs were done in coordination with the SDAs. 

The combination was achieved through standardised operating procedures (SOPs) which, in turn, were based on the parameters of granting land rights as detailed out in “The Odisha Land Rights to Slum Dwellers Act, 2017”. While the slum dwellers did not prepare their own maps, as in the case of Kibera, they played a crucial role in ensuring that the mapping of the slums was done correctly. 

Infantile optimism or realistic pessimism ?

Jaga shows a possible model in which community groups can become partners of large and technology heavy processes and hopefully, in the future, take over of the databases of their slums from the private technology consultants and the government. They can then initiate a Kibera like process in which the maps of Jaga are constinuously enriched with qualitative information with the help of open-source software and technology volunteers. 

This may not appear as romantic as the vision of slum dwellers preparing their own maps using open-source software, but given the present reality of large-scale and high-speed projects, such approaches may be the only ones operationally possible to allow a successful egagement of communities with the developmental blitz. Over time they may even overcome the blitz and achieve the romantic vision at a massive scale – not unlike the way Linux overcame Microsoft.