The importance of Operational Parameters...and the pointlessness of Data-hunting-gathering

The tragedy of data-hunting-gathering 

A defining feature of urban poverty alleviation programs in India in the present times is their ever increasing size, scale and speed of implementation. The Government plays a leading role in the planning and execution of these programs in collaboration with large (often multi-national) private sector consultants, think-tanks and non-profits with financial support from international donors and philanthro-capitalist foundations.

Increasingly, the implementation of these projects involves the creation of vast amounts of digital data, which fuels an ever growing data-hunger among all kinds of development organisations and professionals. 

Everyone is perenially looking for data these days – students, researchers, the organisation that just partnered with the government, the organisation that wants to partner with the government, the organisation that will never partner with the government, the odd travelling scholar in search of a good case study...the list goes on. 

Most of the time, the DHG (data hunter-gatherer) community is not even aware of what kind of data they need, the reason they need it for or the use they intend to put it to once they acquire it. The belief seems to be ("belief" is the right word, for there seems to be very little of science in such an approach), that once these mythical Himalayan data-sets would be acquired all the other questions would magically get answered too.

Rather than searching for data in such a hopeless manner, it could be far more useful to understand the technical aspects of the development programs that generate this data and the operational parameters they need to adhere to. Exploring in detail "how" a program is executed, also throws light on "who" actually implements it and with what tools and techniques. One can then also understand "what" data gets generated in such programs and what uses it could have.

The usefulness of studying operational parameters

Compared to the data, which may be extremely hard to obtain, the goals, objectives and  standard operating procedures guiding the implementation of these projects are more accessible. Even if they are not available in the public domain, the government is generally far more comfortable sharing these documents than the actual data. 

Having even a general understanding of the operational parameters of a project can help one deduce what kind of data may or may not have been produced as part of the project. 

For example, in the case of Jaga Mission, the operational parameters were clearly laid out in the legislation - "The Odisha Land Rights to Slum Dwellers Act, 2017" that guided its implementation.

Let’s look a bit deeper into the operational parameters of the mission in order to understand what kind of data was produced and why it was produced. 

The operational parameters of Jaga Mission

According to the Act

• A slum is defined as a compact settlement of minimum 20 households; 
• Land rights shall be granted based on the area of actual occupation; 
• The maximum ceiling for granting land rights is 45 sqm for slums located in Municipal Councils and 60 sqm for slums located in Notified Area Councils (NACs); 
• Land rights shall be granted free of charge for up to 30 sqm for families belonging to the Economically Weaker Section (EWS) category; 
• Non-EWS households shall pay a certain percentage (later fixed at 50 percent) of the benchmark value of land for securing the land rights and EWS households shall pay a lesser percentage (25 percent) for the amount of land occupied above 30 sqm.

From the above parameters, we can get a pretty clear idea of what kind of data we require in order to execute the program. It is clear that the implementation of the Mission would not only require the boundaries of slum settlements to be clearly delineated (according to the definition) but also an exact mapping of every single dwelling inside the slums in order to calculate the area under actual occupation. In its first phase, Jaga Mission covered 168000 households in 1725 slums in 109 cities, which are spread across the length and breadth of the state of Odisha (155707 sq.km), turning the mapping and survey process of the Mission into a huge logistical challenge. 

It was this massive geographical scale and the need to complete the survey within a reasonably short time which made the Government consider the use of drones. Teams of 3 to 4 professional surveyors would travel by road and reach the cities allotted to them. They would then team up with the local municipal staff and NGOs to visit the slum settlements. GIS companies and NGOs were hired and simultaneously deployed to cover all the cities and slums of Jaga. On reaching the slum, the survey team would prepare the drone flight plan (based on inputs from the municipal staff, NGO representatives and slum dwellers), set up the ground control points, fly the drone and then move on to the next slum. For an average sized slum (1.2 hectares), this process would take about 1.5 to 2 hours. After covering all the slums in a city, the survey team would pack up and drive on to the next city. The raw drone captures for a batch of cities would then be sent for processing and turned into extremely high-resolution (2 cm) ortho-images (i.e. geographically corrected images which allow true distances to be measured). While the image processing was being done, the survey teams would continue their surveys in other cities, thus ensuring a simultaneity of data capturing and data processing activities. The high resolution allowed the digitization of even the tiniest dwellings to be done directly on the ortho-image, without the need for additional DGPS surveys to be conducted in the slums (as was done in the case of the RAY project a few years earlier). 

The following three map layers (vector data) were the most crucial and where prepared for each slum -

i) Slum boundary layer – showing the exact extent of the slum on the date of the survey.

ii) Slum dwellings layer – polygons showing each dwelling unit inside the slum (along with the household survey data).

iii) Revenue parcels layer – showing the ownership details and formal records of the land parcels on which the slum is located.

Together with the ortho-image (raster) layer and an additional land-use (vector) layer, this translated into a huge dataset of 8625 map layers for the 1725 slums of Jaga.

But, once the above map layers were ready, all sorts of geographical and mathematical operations related to mission implementation could be done easily. Let's consider the following cases in a slum located in a Municipal Council (with a maximum ceiling of 45 sqm for being eligible). Let’s assume the benchmark value of land to be INR 80,00,000/acre (i.e. INR 1977/sqm).

1. Beneficiary A - EWS family occupying 27 sqm.
2. Beneficiary B - EWS family occupying 33 sqm.
3. Beneficiary C - Non-EWS family occupying 35 sqm
4. Beneficiary D - EWS or Non-EWS family occupying 50 sqm

Solution of land rights eligibility criteria for each case would be -

• Beneficiary A - Family would get land right free of cost for 27 sqm.
• Beneficiary B - Family would get land right for 33 sqm on payment of INR 1482.75/- (0.25 x 1977 x3)
• Beneficiary C - Family would get land right for 35 sqm on payment of INR 34597.5/- (0.5 x 1977 x 35)
• Beneficiary D - Ineligible until family agrees to surrender the extra 5 sqm.

We can see clearly from the above example that the operational parameters of the project as defined in the Act pretty much dictated what kind of data needed to be generated and how that data was to be used to calculate the eligibility criteria for settling the land. 

But it also becomes clear from the above, what kind of data was not generated as part of the Mission. 

The truth is in the grind

The bottom-line is that the data that governments produce for the implementation of their schemes and projects may or may not be of any use to researchers, scholars, practitioners and organisations which are not directly involved with the project. The operational parameters help us understand what kind of databases may or may not have been generated.

It is far better to have ones own objectives and questions clearly articulated and produce the datasets accordingly. In this digital age it is easy to imagine that the grind of being a solid development researcher can be avoided by just scooping up all the data that exists in government offices and working the magic of the computer on it - but that does not work.

No amount of aimless data-hunting-gathering can replace the power of scientific knowledge, well articulated research questions and a robust methodology.

The truth is in the grind.

Sorry...we cannot share the data with you !

The truth is in the verb

One of the most tragico-amusing facts that most researchers, students, consultants etc get mis-led by when going data hunting (especially for digital data) in government offices, is that when the concerned officials say, "We cannot share that data with you !" (which they almost always do) - they are telling the truth....as in literally !

For sure, they may not "want" to share it with you, but in most cases they simply "cannot".

In order to share any data - they must first have it in their possession. Well, it's a bit hard to part with what you don't have isn't it ? 

So, let's not be overly harsh on the officials of refusal, and focus instead on what are the deeper structural reasons which create this absurd situation, where the government itself is not in effective possession of its own data. And just as Thomas Piketty often says, that the structural economic problems such as wealth inequality can be better understood by resorting to the novels of Austen and Balzac than the works of economists, I would like to approach our present puzzle through the medium of personal experiences and anecdotes...and believe me, I have tonnes of them. 

Looking for data...and finding the cave of Ali Baba

Couple of years ago when I started working as a consultant for the Housing & Urban Development Department in Odisha, one of my tasks was to collect whatever data was available in the various offices of the department on the slums of Odisha. 

All through my years as a student and researcher of city planning, I was accustomed to all kinds of data collection warfare that one has to conduct in government offices - combined arms, protracted, geurrilla, psy-ops, corridor-corner-mugging...you name it - to get even the most obvious and useless bits of information. So, going data-digging with the seal of the Principal Secretary was like a walk in the park and I was quite enjoying the breezy feel of it. Even so, I didn't really expect to find anything useful at all. 

After visiting desks and cubicles, where I would be kindly directed to other desks and cublicles (I love strolling the halls and corridors of bureaucracy...it's not half as masochistic as it sounds and it has its benefits), I ended up in a corner desk in the office of the Bhubaneswar Municipal Corporation (BMC), where I struck a conversation with another consultant who was nearing the end of his contract. He seemed to remember that there was some data stored in a desktop, which was barely ever switched on. As he searched for the folder, he reminisced more, "This data was collected during the RAY scheme...then the scheme was abandoned, and no one used it anymore."

As many would remember, RAY was a promising scheme of the central government to create slum free cities which met a pre-mature end in 2014 when the Congress led UPA-II government was defeated by the BJP led NDA. While the scheme was abandoned before any serious implementation could be done, it seemed that it did generate some data before dying.

When he finally opened the folder, it turned out to be a treasure cove containing detailed shapefiles of each of the 436 slums of Bhubaneswar, organised neatly in 67 sub-folders corresponding to each ward of the city. The slum-level data consisted of separate map layers for the slum boundary, land ownership, open spaces, infrastructure and dwelling units along with the full household survey data of over 90000 households. Then there were city level map layers showing the city agglomeration area, the muncipal area, municipal wards, location of slums and the road network. From the pdf files containing the layout maps prepared for each slum, one could even find the details of the private consultancy that had prepared the maps (that's always a lucky find, as GIS databases in India almost never contain any metadata files....more on the horror of GIGO - Garbage In Garbage Out - of digital data in later blogs).

The RAY data that we discovered in that abandoned, dust-coated desktop, had been produced sometime around 2013-14. Judging by the time-stamp in the pdf files it must have been handed over to the Government in 2014.  

Data absence through Data obsolescence

Whether, a scheme gets abandoned or not, there is nothing stopping the government from making full use of such a comprehensive and detailed database and also update it in the process of using it. However, a general lack of familiarity with GIS  (although everyone keeps talking about it all the time...agreed, mostly the talk is on buzzwords and eye-candies and not on serious cartography or computing), the dependence on expensive and proprietary software (which ensures that the GIS consultants produce and process the data...and the government has eyes on it only through the consultants and for just as long as the contract period lasts) and lack of awareness regarding free and open-source alternatives creates a situation where the government just does not have the systems and capacities in place to "receive" the data even if it is "handed-over".

Well, what inevitably happens to something which you never use ??....you gradually forget that it even exists. And as the consultants (for what is the government nowadays but an army of consultants embedded in various departments and offices doing everything from preparing reports, drafting standard operating procedures, analysing data, preparing powerpoint presentations...you name it) involved with a certain project leave at the end of their contract period, the last traces of the organisational memory regarding the data disappear with them. 

It was clear from the RAY survey guidelines and the database that I discovered that the survey involved an elaborate process involving the collection of cartosat-1 (2.5 m resolution) and, the now decomissioned, cartosat-2 (< 1 m resolution) satellite imagery; differential global position system (DGPS) surveys to mark the edges of each structure inside the slum; extensive digitization to prepare the maps; detailed household surveys of over 90000 slum families etc. For sure, it must have been an expensive process too. 

And here we were, 3 years later, where the best case scenario for the data turned out to be that it was forgotten in a desktop in the corner of BMC to be accidentally discovered by me on a random data safari. If this happens in one case, then, for sure, it happens in other cases too, especially considering that the underlying causes remain mostly unchanged.

It so happened, that a few years later, when I showed that data to the then Commissioner of Bhubaneswar in a presentation, he made a sincere request that I share the data with his office as they didn't possess any GIS data on the slums of Bhubaneswar. 

So here I was in a surreal situation where BMC was asking me to share BMC's data because BMC did not possess it. 

I had half a mind to say "Sorry, I cannot share the data with you." 

Well, the trouble was that I..."could" ;))

Reviving a dormant blog...And a few words on Lefebvre, Cartesian geometry and Open-source GIS

The Lefebvrian Paradox

In his famous and influential book ‘The Production of Space’, Henri Lefebvre had written that ‘social space, and especially urban space, emerged in all its diversity – and with a structure far more reminiscent of flaky mille-feuille pastry than of the homogenous and isotropic space of classical (Euclidian/Cartesian) mathematics.’ 

Ironically, in the decades following the publication of Lefebvre’s book, it is precisely the dramatic developments in the application of ‘Eucliain/Cartesian’ mathematics to geographical analysis – namely through the development of various Geographic Information Systems (GIS) software – that helped unravel the innumerable layers of the urban mille-feuille. However, such technologies were proprietary, expensive and largely out of the reach of community groups. For the power of computerised geographical analysis to address the complexities of social space, it had to be accessible by the community and become the technological arm of 21st century participatory development. That became possible with the emergence of the Linux operating system and the free and open-source software movement as a powerful alternative to proprietary software.

Altruism through technical necessity - the revolutionary potential of the open-source movement

The democratic and community oriented nature of the open-source movement is not just a function of altruism (which definitely motivates many members of the movement) but of technical necessity. It is only through the collaborative efforts of millions of developers around the world that the open-source movement is able to create its software. The fact that the opensource geo-spatial foundation describes itself as a ‘not-for-profit organization...devoted to an open philophy and participatory community driven development’ (https://www.osgeo.org/about/) has a lot to say about the movement’s potential role in the field of participatory urban development.

A quick glance at the remarkably informative maps prepared by the citizens of Kibera slum in Nairobi demonstrates the power of this combination. My personal favourite is a thematic map on the status of safety and security in the slum, which identifies dangerous places through a clustering of “bottles” (signifying alcohol vending spots) and “bulbs” (signifying operational or non-operational street lights). A dense clustering of bottles along with black bulbs (non-operational) suggests dangerous locations. The map is prepared using the opensource QGIS software and uses openstreetmap as its base layer (https://mapkibera.org/download/maps/Security%20map%20final.pdf).

The process effectively combines the depth of qualitative information, which can only be aquired through detailed community mapping exercises, with the accuracy, speed and processing power of modern computing. This is where Cartesian/Euclidian mathematics (what are the “bottles” and “bulbs” if not coordinates on an x-y cartesian plane ?) meets the local knowledge of the community. 

The major distinction between proprietary and opensource software is not that one is generally expensive and the other is mostly free. While proprietary software treats users primarily as consumers of technology (causing stready technological dependence, while simultaneously creating an illusion of being tech-savvy), opensource software encourages users to transform into collaborative developers (encouraging community participation and causing technological empowerment).

Pro-Poor Development goes Big Business - Government in hypersonic mode

Although community organisations around the world have effectively used opensource software to map their settlements and exert their agency, the increasing size, scale and speed of implementation of urban poverty alleviation programmes in the global south creates its own challenges.

In this new context it is no longer enough for community organisations to create detailed maps of their own settlements and prepare their own databases. Now, instead of having to deal with a lethargic, inefficient and apathetic bureacracy, they increasingly have to deal with a highly energetic, quick and effective state machinery, which is flush with funds and operates in partnership with large private sector consultants and technology firms to implement gigantic urban poverty alleviation programs simultaneously across thousands of slum settlements in hundreds of cities. Instead of being a reluctant antagonist, the state has turned into an extremely vigorous ally of the urban poor – in both cases community participation may suffer, but the community mobilisation methods of the past are simply not adequate in this drastically altered reality. Many NGOs and community groups have simply not grasped this altered reality well enough.

The challenge of technological development

Moreover, different slum improvement projects would require different types of maps. A Kibera like self-assessment need not map every single dwelling in the slum, but a land titling project like Jaga Mission of Odisha would require an exact mapping of the dwellings (as land rights are issued based on the area of actual occupation). Discussions on participatory mapping are incomplete if not seen in the context of the overall objective of the task and the technical requirements for accomplishing it.

In Jaga Mission, high resolution images of slums were captured using drones and GIS databases of slums were prepared by private sector technical consultants by digitizing these drone images and linking them to the household survey data. The fact that the mapping of 1725 slums spread across 109 small and medium towns of the state (spread across the 155707 sq.km area of Odisha) and the survey of about 170000 slum households was completed within a period of 7 months gives some idea of the speed, scale and intensity of the exercise. It averages out to the preparation of detailed and accurate GIS databases of about 62 slums every week for 7 months straight. 

Jaga Mission achieved this through a creative combination of high technology activities (such as drone mapping and GIS) with field-based community participation activities. About 24 NGOs were engaged to undertake community mobilisation exercises and create Slum Dwellers Associations (SDAs) in each slum. Both the drone mapping by the technololgy consultants and the household surveys by the NGOs were done in coordination with the SDAs. 

The combination was achieved through standardised operating procedures (SOPs) which, in turn, were based on the parameters of granting land rights as detailed out in “The Odisha Land Rights to Slum Dwellers Act, 2017”. While the slum dwellers did not prepare their own maps, as in the case of Kibera, they played a crucial role in ensuring that the mapping of the slums was done correctly. 

Infantile optimism or realistic pessimism ?

Jaga shows a possible model in which community groups can become partners of large and technology heavy processes and hopefully, in the future, take over of the databases of their slums from the private technology consultants and the government. They can then initiate a Kibera like process in which the maps of Jaga are constinuously enriched with qualitative information with the help of open-source software and technology volunteers. 

This may not appear as romantic as the vision of slum dwellers preparing their own maps using open-source software, but given the present reality of large-scale and high-speed projects, such approaches may be the only ones operationally possible to allow a successful egagement of communities with the developmental blitz. Over time they may even overcome the blitz and achieve the romantic vision at a massive scale – not unlike the way Linux overcame Microsoft.