The Data exists...right under our Mouses !

The capital irony

It is perhaps a capital irony of our times that precisely at the time when computers are more powerful and affordable than ever before and the access to powerful and previously expensive software provided by the  Linux + FOSS (Free and Open Source Software) movement, the general ability to use computers effectively to address the various problems faced by our cities is at an all time low.

I myself come from a background of primarily qualitative and participatory techniques in urban planning. I continue to have a natural fondness for such techniques, but have increasingly also discovered the power that effective use of computers bring to my work.

Contrary to the myth that the quantitative and qualitative worlds are poles apart (which leads to the further myth that professionals dealing with qualitative techniques cannot use computers for serious quantitative analysis), the two are in fact friends and allies of each other and help each other continuously.

Without waxing complex, think of a rather simple example. I would like to undertake participatory exercises in various slums in my city and I use all kinds of creative ideas to undertake the same inside those communities.

But alongside that, I could also prepare a GIS database of the slums in the city that gives me spatial and quantitative information on slums - such as their location, distance from each other, distance from other city facilities, size and density of the settlements, the population and occupational characteristics of the settlements etc.

This quantitative database can actually help me increase the effectiveness of my qualitative techniques by helping me to schedule meetings, use different techniques in slums of different sizes and shapes, check the probability of consensus-building (fewer meetings could build consensus faster in a smaller slum than in a larger and denser slum) etc.

Rather than focusing too much on whether to deploy quantitative or qualitative methods, it is better to focus on the problem that needs to be solved and deploy whatever methods that may be necessary.

Why computers ?

As long as I want to do participatory activities in a handful of slums, I may not need any support of computers at all. However, if I would like to undertake such activities in tens or hundreds or thousands of slums, then I begin to feel the need of the processing power of the computer.

It is as simple as that.

As more and more resources are made available to various urban development programs and schemes in India, their sizes, duration and scale of operation are all increasing. It is not difficult to understand that in a country the size of India, urban development projects would need to be undertaken at a scale where one can at least hope to make a meaningful difference. 

But of course, computers need instructions to follow - and they need data to work on.

The data exists...right under our mouses

Quite often, the impossibility of obtaining data is cited as one of the main barriers to effective use of computers in solving urban problems in India. I have written on this topic on multiple occasions. And I have stressed on earlier blogs that mere accumulation of digital data is not of much use if one does know how to use computers effectively to process it.

However, another capital irony of our times is that much of the data whose absence we so lament - does indeed exist...and sometimes right under our noses (or mouses).

Let me demonstrate.

This particular link will take you to the dashboard of the "GIS based Master Plan" sub-scheme of AMRUT. 

The very first component of this sub-scheme was geo-database creation and in the following screen-shot of the dashboard we can see its status -

 

If we look at the first three steps of the component, we can see that satellite data had been acquired and processed for about 450 cities. In the pie chart on administrative works is not self-explanatory, but it could mean that the National Remote Sensing Centre (NRSC) of the Indian Space Research Organisation (ISRO) may have handled the satellite data acquisition and processing for 240 cities and private companies may have done it for another 220 cities.

In any case, according to the official dashboard itself, we can conclude that processed satellite data exist for about 450 cities. As per the status chart, final GIS maps also seem to exist for 351 cities.

From if it exists...to where it exists

Finding evidence and clear arguments for the claim that something exists, is the first step in finding something. If I know for sure that something exists, then I need not succumb to the fallacy that it doesn't even exist. 

The task after that is to discover, where it exists, rather than wonder if it exists.

The same method can be applied to understand exactly what all data has been collected and processed under the myriad central and state government schemes that are going on in the country and have already been executed in the past.

Believe me, we will have more data than we would need for getting most of our tasks done.

The catch here is this...a person who does not have the imagination to discover the data, most likely would not have the imagination to use that data either.

But let's keep that blast for a later post ;)

Solving the "land" variable in land-titling equations (and some more on tackling complex urban problems)

My previous blogs have stressed the importance of operational parameters of large and complex urban development projects and how their implementation can become very difficult - if not impossible - without a creative combination of modern computing and community feedback. Perhaps nowhere does this reality hit home harder than in the implementation of mega-sized slum land-titling and upgrading initiatives such as Jaga Mission.

The Odisha Land Rights to Slum Dwellers Act, 2017, which guides the implementation of Jaga Mission, states in section 3, sub-section 1, that 'every landless person occupying land in a slum in any urban area by such date as may be notified by the State Government, shall be entitled for settlement of land and certificate of land right shall be issued in accordance with the provisions of the Act.'

All very well-intentioned and clear so far, but a whole operational quagmire opens up when one begins to implement the provisions of the Act. The art of land administration in India as it exists today, is an elaborate one that pre-dates the Mughal era. Words like 'kissam' (a derivative of the Farsi 'Qism', referring to land-use type) co-exist with the English 'Record of Right' in land revenue records, reflecting the deep and layered history of the subject. The typical planning student of professional is often not even familiar with these terms used by the revenue department, let alone being able to intervene in the operational matters of land administration.

The slums in any city may be located on hundreds of separate parcels of land, which may belong to diverse 'kissam' types, which, further, could be a mix of non-reserved (on which land rights can be given) and reserved (on which land rights cannot be given without initiating a process of re-classification of land in consultation with the revenue department) categories. Furthermore, parcels belonging to different kissams could be owned by an array of government departments or private entities, depending on which it may or may not be possible to grant land rights. It is these attributes that must be cross-referenced with each other and with the location and household data of the slum dwellers in order to satisfy all the conditions necessary to settle land rights.

Quantifying Complexity

Let's take the example of a single city of Balasore in Odisha. The 3128 families living in 41 slums of this city are located on 735 land parcels with belong to 33 different 'kissams'.

Let's further consider a particular 'kissam' called 'gharbari', which refers to homestead. There are 238 parcels corresponding to this kissam out of which 121 parcels are owned by private enitities; 71 parcels are owned by the railways; 23 parcels are owned by temple trusts and 23 parcels are owned by various departments and offices of the state government. Only the slum houses located on the last 23 parcels in the above list, can be settled without getting into special arrangements and negotiations with other government departments and private entities.

This is just one kissam out of 33. Now, also consider the location of the 41 slums in the city and how the 3128 slum houses intersect with the land parcels of various kissam and ownership types. 

What has been described above is the case of just one city out of 115; just 41 slums out of 2919; and a mere 3128 slum houses out of a total of above 400,000.

This is complexity quantified.

And this is why we need the processing power of modern computers (and not for making powerpoint presentations on the achievements of the Mission).

Political will is a necessary condition...but not a sufficient one

This is also the reason why in the first phase of Jaga Mission, which covered about 170,000 slum households in 109 small and medium towns of Odisha, the government reached as impasse after granting about 70,000 land rights certificates. The remaining 100,000 households fell on land parcels belonging to various reserved kissam categories, restricted central government lands, private entities, temple trusts or environmentally hazardous lands.

Instead of basking in glory for having distributed 70,000 land rights certificates in less than two years (no mean feat !) the Department of Housing and Urban Development, the nodal agency overseeing the implementation of the Mission, went out of its way to initiate inter-departmental negotiations with the revenue department, forest and environment department, private entities such as royal families, temple trusts etc to find workable solutions to grant land rights to the remaining slum households. Special standard operating procedures were also developed to address the challenge of slums located on highly restricted central government lands such as those belonging to railways and defence, which may require relocation.

The special measures initiated by the Government were a clear indicator of the strong political will that backed Jaga Mission. It was also a clear indicator that even when a strong political will exists (which is rare in itself), the implementation of any large pro-poor intervention may face serious challenges due to technical and operational reasons.

And it is extremely important for people not directly involved with the implementation of such projects to have a thorough understanding of these operational reasons if they wish to engage effectively and critique accurately.

Consider this news clip (in Odia) from 2019, by a regional news channel which was critical of the Mission. It showed the residents of a slum called 'Godhi Basha', who had not received land rights certificates. However, the news anchor could not give any reason for the state of affairs apart from the usual one that the government was failing to keep its promise to slum dwellers. At the 33 second mark, the clip showed a beneficiary called Ms. Anjana Das holding the card showing her Jaga Mission house number.

However, as we have already seen, lack of political will is definitely not a problem with Jaga Mission. What then is the mystery of the Godhi Basha slum ?

Had the journalist investigated just a little more, he would have discovered that the whole slum was situated on a parcel of land that belonged to "South Eastern Railways" - one of the 72 land parcels in the city which are owned by the Indian Railways. As a matter of fact, the journalist also got the name of the beneficiary wrong.

It was possible to identify all these issues in time less than the duration of the news report, thanks to the digital data collected as part of the Mission, relevant open-source software and quick data analysis on the command line, which we started discussing in the last two blogs. 

Unfortunately, as I have shown through case examples in another blog, the government itself fails to utilise available computing technologies effectively by continuing to rely on archaic bureaucratic methods and pointless application of manual labour (relying on paper maps and field visits despite possessing high-resolution imagery and GIS databases).

It definitely retains operational overview but it cannot solve the complex problems (where complexity is merely a function of processing power available i.e. with respect to the computer, they are simple problems) which not only prevent it from overcoming its operational impasses but also cause community level confusion as shown in the case of the news clip.

Community Empowerment at Slum level...Community dis-empowerment at Mission Level

Just as the Government faces its own difficulties for not using available technologies effectively, so does the community.

As I have pointed out earlier, given the size and complexity of projects such as Jaga Mission, the typical methods of community participation, engagement and critiquing are simply not adequate. 

It is not enough to understand why something is not working in one's own slum - it is only one out of 3000 slums ! 

That means 0.0003 % of the Mission in terms of the number of slums.

How can the community get effective overview of the implementation of the Mission while covering (at the level of each slum) a microscopic 0.0003 % of the Mission, when the Government and its consultants have overview of 100 % ?

Just have a look at the maps above. The one on the left, shows the location of Godhi Basha among all the slums in Balasore city; and the one on the right shows the location of Balasore city among all the cities covered by Jaga Mission. 

I hope this gives some sense of the scale....in other words, what 0.0003 % looks and feels like.

Unfamiliar methods...or merely abandoned ?

This is why it is neither enough for governments to continue using their familiar bureaucratic methods, nor is it enough for community organisations to continue using their familiar participatory development methods. 

Probably it is time to embrace the unfamiliar methods.

The size and complexity of urban challenges in the present times demands the use of methods which were created specifically for tackling large, complex and dynamic systems -- more extensive use of open-source geo-spatial software and data analysis; further refinement of mathematical models for urban and regional planning; application on the principles of cybernetics to understand the functioning of urban systems; techniques of operations research etc. 

It is an irony that huge progress was made in the development and application of many of the above techniques in solving pressing social problems precisely when computing power was very low, and abandoned in favour of feel-good but ineffectual qualitative approaches (not to mention the pseudo-scientific farce that passes for 'tech' in contemporary urban discourses) when computing power is at its peak.

More on that in forthcoming blogs...

The importance of Operational Parameters...and the pointlessness of Data-hunting-gathering

The tragedy of data-hunting-gathering 

A defining feature of urban poverty alleviation programs in India in the present times is their ever increasing size, scale and speed of implementation. The Government plays a leading role in the planning and execution of these programs in collaboration with large (often multi-national) private sector consultants, think-tanks and non-profits with financial support from international donors and philanthro-capitalist foundations.

Increasingly, the implementation of these projects involves the creation of vast amounts of digital data, which fuels an ever growing data-hunger among all kinds of development organisations and professionals. 

Everyone is perenially looking for data these days – students, researchers, the organisation that just partnered with the government, the organisation that wants to partner with the government, the organisation that will never partner with the government, the odd travelling scholar in search of a good case study...the list goes on. 

Most of the time, the DHG (data hunter-gatherer) community is not even aware of what kind of data they need, the reason they need it for or the use they intend to put it to once they acquire it. The belief seems to be ("belief" is the right word, for there seems to be very little of science in such an approach), that once these mythical Himalayan data-sets would be acquired all the other questions would magically get answered too.

Rather than searching for data in such a hopeless manner, it could be far more useful to understand the technical aspects of the development programs that generate this data and the operational parameters they need to adhere to. Exploring in detail "how" a program is executed, also throws light on "who" actually implements it and with what tools and techniques. One can then also understand "what" data gets generated in such programs and what uses it could have.

The usefulness of studying operational parameters

Compared to the data, which may be extremely hard to obtain, the goals, objectives and  standard operating procedures guiding the implementation of these projects are more accessible. Even if they are not available in the public domain, the government is generally far more comfortable sharing these documents than the actual data. 

Having even a general understanding of the operational parameters of a project can help one deduce what kind of data may or may not have been produced as part of the project. 

For example, in the case of Jaga Mission, the operational parameters were clearly laid out in the legislation - "The Odisha Land Rights to Slum Dwellers Act, 2017" that guided its implementation.

Let’s look a bit deeper into the operational parameters of the mission in order to understand what kind of data was produced and why it was produced. 

The operational parameters of Jaga Mission

According to the Act

• A slum is defined as a compact settlement of minimum 20 households; 
• Land rights shall be granted based on the area of actual occupation; 
• The maximum ceiling for granting land rights is 45 sqm for slums located in Municipal Councils and 60 sqm for slums located in Notified Area Councils (NACs); 
• Land rights shall be granted free of charge for up to 30 sqm for families belonging to the Economically Weaker Section (EWS) category; 
• Non-EWS households shall pay a certain percentage (later fixed at 50 percent) of the benchmark value of land for securing the land rights and EWS households shall pay a lesser percentage (25 percent) for the amount of land occupied above 30 sqm.

From the above parameters, we can get a pretty clear idea of what kind of data we require in order to execute the program. It is clear that the implementation of the Mission would not only require the boundaries of slum settlements to be clearly delineated (according to the definition) but also an exact mapping of every single dwelling inside the slums in order to calculate the area under actual occupation. In its first phase, Jaga Mission covered 168000 households in 1725 slums in 109 cities, which are spread across the length and breadth of the state of Odisha (155707 sq.km), turning the mapping and survey process of the Mission into a huge logistical challenge. 

It was this massive geographical scale and the need to complete the survey within a reasonably short time which made the Government consider the use of drones. Teams of 3 to 4 professional surveyors would travel by road and reach the cities allotted to them. They would then team up with the local municipal staff and NGOs to visit the slum settlements. GIS companies and NGOs were hired and simultaneously deployed to cover all the cities and slums of Jaga. On reaching the slum, the survey team would prepare the drone flight plan (based on inputs from the municipal staff, NGO representatives and slum dwellers), set up the ground control points, fly the drone and then move on to the next slum. For an average sized slum (1.2 hectares), this process would take about 1.5 to 2 hours. After covering all the slums in a city, the survey team would pack up and drive on to the next city. The raw drone captures for a batch of cities would then be sent for processing and turned into extremely high-resolution (2 cm) ortho-images (i.e. geographically corrected images which allow true distances to be measured). While the image processing was being done, the survey teams would continue their surveys in other cities, thus ensuring a simultaneity of data capturing and data processing activities. The high resolution allowed the digitization of even the tiniest dwellings to be done directly on the ortho-image, without the need for additional DGPS surveys to be conducted in the slums (as was done in the case of the RAY project a few years earlier). 

The following three map layers (vector data) were the most crucial and where prepared for each slum -

i) Slum boundary layer – showing the exact extent of the slum on the date of the survey.

ii) Slum dwellings layer – polygons showing each dwelling unit inside the slum (along with the household survey data).

iii) Revenue parcels layer – showing the ownership details and formal records of the land parcels on which the slum is located.

Together with the ortho-image (raster) layer and an additional land-use (vector) layer, this translated into a huge dataset of 8625 map layers for the 1725 slums of Jaga.

But, once the above map layers were ready, all sorts of geographical and mathematical operations related to mission implementation could be done easily. Let's consider the following cases in a slum located in a Municipal Council (with a maximum ceiling of 45 sqm for being eligible). Let’s assume the benchmark value of land to be INR 80,00,000/acre (i.e. INR 1977/sqm).

1. Beneficiary A - EWS family occupying 27 sqm.
2. Beneficiary B - EWS family occupying 33 sqm.
3. Beneficiary C - Non-EWS family occupying 35 sqm
4. Beneficiary D - EWS or Non-EWS family occupying 50 sqm

Solution of land rights eligibility criteria for each case would be -

• Beneficiary A - Family would get land right free of cost for 27 sqm.
• Beneficiary B - Family would get land right for 33 sqm on payment of INR 1482.75/- (0.25 x 1977 x3)
• Beneficiary C - Family would get land right for 35 sqm on payment of INR 34597.5/- (0.5 x 1977 x 35)
• Beneficiary D - Ineligible until family agrees to surrender the extra 5 sqm.

We can see clearly from the above example that the operational parameters of the project as defined in the Act pretty much dictated what kind of data needed to be generated and how that data was to be used to calculate the eligibility criteria for settling the land. 

But it also becomes clear from the above, what kind of data was not generated as part of the Mission. 

The truth is in the grind

The bottom-line is that the data that governments produce for the implementation of their schemes and projects may or may not be of any use to researchers, scholars, practitioners and organisations which are not directly involved with the project. The operational parameters help us understand what kind of databases may or may not have been generated.

It is far better to have ones own objectives and questions clearly articulated and produce the datasets accordingly. In this digital age it is easy to imagine that the grind of being a solid development researcher can be avoided by just scooping up all the data that exists in government offices and working the magic of the computer on it - but that does not work.

No amount of aimless data-hunting-gathering can replace the power of scientific knowledge, well articulated research questions and a robust methodology.

The truth is in the grind.

Reviving a dormant blog...And a few words on Lefebvre, Cartesian geometry and Open-source GIS

The Lefebvrian Paradox

In his famous and influential book ‘The Production of Space’, Henri Lefebvre had written that ‘social space, and especially urban space, emerged in all its diversity – and with a structure far more reminiscent of flaky mille-feuille pastry than of the homogenous and isotropic space of classical (Euclidian/Cartesian) mathematics.’ 

Ironically, in the decades following the publication of Lefebvre’s book, it is precisely the dramatic developments in the application of ‘Eucliain/Cartesian’ mathematics to geographical analysis – namely through the development of various Geographic Information Systems (GIS) software – that helped unravel the innumerable layers of the urban mille-feuille. However, such technologies were proprietary, expensive and largely out of the reach of community groups. For the power of computerised geographical analysis to address the complexities of social space, it had to be accessible by the community and become the technological arm of 21st century participatory development. That became possible with the emergence of the Linux operating system and the free and open-source software movement as a powerful alternative to proprietary software.

Altruism through technical necessity - the revolutionary potential of the open-source movement

The democratic and community oriented nature of the open-source movement is not just a function of altruism (which definitely motivates many members of the movement) but of technical necessity. It is only through the collaborative efforts of millions of developers around the world that the open-source movement is able to create its software. The fact that the opensource geo-spatial foundation describes itself as a ‘not-for-profit organization...devoted to an open philophy and participatory community driven development’ (https://www.osgeo.org/about/) has a lot to say about the movement’s potential role in the field of participatory urban development.

A quick glance at the remarkably informative maps prepared by the citizens of Kibera slum in Nairobi demonstrates the power of this combination. My personal favourite is a thematic map on the status of safety and security in the slum, which identifies dangerous places through a clustering of “bottles” (signifying alcohol vending spots) and “bulbs” (signifying operational or non-operational street lights). A dense clustering of bottles along with black bulbs (non-operational) suggests dangerous locations. The map is prepared using the opensource QGIS software and uses openstreetmap as its base layer (https://mapkibera.org/download/maps/Security%20map%20final.pdf).

The process effectively combines the depth of qualitative information, which can only be aquired through detailed community mapping exercises, with the accuracy, speed and processing power of modern computing. This is where Cartesian/Euclidian mathematics (what are the “bottles” and “bulbs” if not coordinates on an x-y cartesian plane ?) meets the local knowledge of the community. 

The major distinction between proprietary and opensource software is not that one is generally expensive and the other is mostly free. While proprietary software treats users primarily as consumers of technology (causing stready technological dependence, while simultaneously creating an illusion of being tech-savvy), opensource software encourages users to transform into collaborative developers (encouraging community participation and causing technological empowerment).

Pro-Poor Development goes Big Business - Government in hypersonic mode

Although community organisations around the world have effectively used opensource software to map their settlements and exert their agency, the increasing size, scale and speed of implementation of urban poverty alleviation programmes in the global south creates its own challenges.

In this new context it is no longer enough for community organisations to create detailed maps of their own settlements and prepare their own databases. Now, instead of having to deal with a lethargic, inefficient and apathetic bureacracy, they increasingly have to deal with a highly energetic, quick and effective state machinery, which is flush with funds and operates in partnership with large private sector consultants and technology firms to implement gigantic urban poverty alleviation programs simultaneously across thousands of slum settlements in hundreds of cities. Instead of being a reluctant antagonist, the state has turned into an extremely vigorous ally of the urban poor – in both cases community participation may suffer, but the community mobilisation methods of the past are simply not adequate in this drastically altered reality. Many NGOs and community groups have simply not grasped this altered reality well enough.

The challenge of technological development

Moreover, different slum improvement projects would require different types of maps. A Kibera like self-assessment need not map every single dwelling in the slum, but a land titling project like Jaga Mission of Odisha would require an exact mapping of the dwellings (as land rights are issued based on the area of actual occupation). Discussions on participatory mapping are incomplete if not seen in the context of the overall objective of the task and the technical requirements for accomplishing it.

In Jaga Mission, high resolution images of slums were captured using drones and GIS databases of slums were prepared by private sector technical consultants by digitizing these drone images and linking them to the household survey data. The fact that the mapping of 1725 slums spread across 109 small and medium towns of the state (spread across the 155707 sq.km area of Odisha) and the survey of about 170000 slum households was completed within a period of 7 months gives some idea of the speed, scale and intensity of the exercise. It averages out to the preparation of detailed and accurate GIS databases of about 62 slums every week for 7 months straight. 

Jaga Mission achieved this through a creative combination of high technology activities (such as drone mapping and GIS) with field-based community participation activities. About 24 NGOs were engaged to undertake community mobilisation exercises and create Slum Dwellers Associations (SDAs) in each slum. Both the drone mapping by the technololgy consultants and the household surveys by the NGOs were done in coordination with the SDAs. 

The combination was achieved through standardised operating procedures (SOPs) which, in turn, were based on the parameters of granting land rights as detailed out in “The Odisha Land Rights to Slum Dwellers Act, 2017”. While the slum dwellers did not prepare their own maps, as in the case of Kibera, they played a crucial role in ensuring that the mapping of the slums was done correctly. 

Infantile optimism or realistic pessimism ?

Jaga shows a possible model in which community groups can become partners of large and technology heavy processes and hopefully, in the future, take over of the databases of their slums from the private technology consultants and the government. They can then initiate a Kibera like process in which the maps of Jaga are constinuously enriched with qualitative information with the help of open-source software and technology volunteers. 

This may not appear as romantic as the vision of slum dwellers preparing their own maps using open-source software, but given the present reality of large-scale and high-speed projects, such approaches may be the only ones operationally possible to allow a successful egagement of communities with the developmental blitz. Over time they may even overcome the blitz and achieve the romantic vision at a massive scale – not unlike the way Linux overcame Microsoft.