You cannot "find" data, if you cannot recognise it

I have written in earlier blogs about the trouble with GIGO (Garbage-In-Garbage-Out), which basically means that if the data that goes into your software is garbage, then your output would be garbage too. However, sometimes the problem may be what Andrei Martyanov describes as "double-GIGO" (I can't remember which video it was in but feel free to check out this one anyway), where both the data used and the formulation of the problem may be garbage. Needless to say, that it is one hell of a tragedy when that happens. Unfortunately, it is not a rare phenomenon at all.

However, quit often, the problem is simply not knowing what data to look for and where to look. For example, last year the CITIIS program of the National Institute of Urban Affairs (NIUA) organised a training workshop for the Department of Housing and Urban Development in Odisha for mapping of water bodies in urban areas of the state. The program started with the mapping of 19 water bodies in the pilot phase.

Having engaged with the implementation of such programs from the inside, I know for a fact that "there is many a slip betwixt the cup and the lip", when it comes to their objectives and the data that they use for achieving it. I have written extensively on the state of data in government projects in multiple blogs.

Even without going into any detail of this specific program, isn't it a bit peculiar to do such a gala workshop on such a theme, when the following already exists ?? 

This is the dashboard of the Water Bodies Information System (WBIS) of the Indian Space Research Organisation (ISRO). Here is the link to it. 

The site clearly mentions -

"The water spread area information is extracted using images of 6 m to 56 m resolution for delineating water bodies of sizes as small as 0.025 ha and 50 ha respectively. This information is made available as Water Bodies Information System (WBIS) for visualisation and download."

One would expect that the organisers of the workshop to be atleast aware of the existence of such an information system prepared by such a respectable organisation - Alas, nothing of the sort !

When the knowledge and technical abilities of urban development professionals increasingly gets limited to preparing powerpoint presentations, having endless online meetings and organising various kinds of "capacity-building" events, then the probability of being able to identify relevant data (even when it stares at you point blank) is bound to decrease...and at a steep slope.

One can safely argue that it is hovering somewhere very close to zero if not already there

The ability to find data is very much a function of being able to recognise relevant data and that in turn is dependant of ones knowledge of the subject, which needs constant updating.

But is there any time left for that after the relentless "labour" of back-to-back online meetings ? 

Planning and the Complexity Conundrum

Implementing at the cost of Planning

A characteristic feature of large urban development schemes in India is that they are heavily implementation oriented. In a way, they seem to have overcome the "the-plan-was-good-but-the-implementation-was-bad" impasse in Indian urban planning.

To be sure, this lacuna of planning has been overcomeby abandoning planning itself and opting for large, sectoral schemes implemented by specific line departments of the government.  For what else are these large urban development schemes in India if not projects of specific infrastructure verticals undertaken in mega-scale?

It is not difficult to resolve the difficulties of planning, if urban planning itself -- the science and art of forecasting future development scenarios in a city or region and preparing for it by integrating the activities of multiple components of the urban system i.e. land, housing, economy, infrastructure -- is abandoned.

Urban systems are complex by their very nature. The discipline of urban planning, therefore, by its very nature, is tasked with anticipating the behaviour of this highly complex and dynamic system in a scientific manner and preparing for it. 

Planning and Complexity

Rather than talking in vague and general terms (as is increasingly common these days), it is perhaps better to use the concept of "variety", which is used in the field of cybernetics as a measure of complexity.

W. Ross Ashby, one of the pioneers in the field of cybernetics, described "variety" as the number of possible states that a system can take. As systems become larger in size, the amount of "variety", and therefore complexity, increases exponentially.

Looking at urban systems, examples of this could be found everywhere. For a very small town a single, small commercial area with a small cluster of shops of various kinds may suffice. But a larger city would inevitable give rise to a whole range of commercial areas of various sizes and types, having different areas of coverage and located in various parts of the city.

I sometimes gave the example of a footpath to my planning students, which, in a sweet little Scandinavian town would be just a footpath, but in a city of even moderate size in India could turn into a space for shopping, hawking, begging, sleeping, living, storing, parking...and, if possible, a space for pedestrians for walking. 

In the former case the footpath would have a variety of 1 and in the latter a variety of n...and counting !

In his book "Designing Freedom", Stafford Beer showed the calculation to estimate variety -

If there are n people in a system, and each of them has variety x (each can adopt x number of possible states), then the variety of the total system thus defined will be xⁿ.

So if there are only 40 people (n=40), each of whom has only two possible states (x=2), there are still 2⁴⁰ possible states of the system.

2⁴⁰ =  1,099,511,627,776

      ('Designing Freedom', Stafford Beer, p - 11)                 

 

This is complexity quantified.

Better to not even attempt to calculate the total possible states that our second footpath can take ! 

Now that we have developed a healthy respect for the mind-numbing salvo of variety (therefore complexity) that urban systems can hurl at the planning profession, it is perhaps possible to at least understand (if not entirely forgive) why the profession often fails to successfully execute the task that it has taken on.

The planning paradox and the world of probabilities

Any serious discussion on a topic as complex as the planning of urban systems in the 21st century, has to begin by acknowledging that it is a near impossible task. And yet, it has to be done.

Therefore, the discipline has to be approached like any complex system has to be approached - not with the demands of certainty...but with the estimation of probability.

As the geopolitical expert Andrei Martyanov said in a recent talk -

"The world of prognostication and serious analysis is the world of probabilities."

In the context of planning, this was discussed wonderfully by Poulicos Prastacos in one of his papers on the Projective Optimization Land Use Information System (POLIS) land-use transportation model. He wrote that one of the problems of the first generation of land use-transportation models, developed by planners in the US during 1960-75, was that their goals were too ambitious. 

Prastacos wrote the following lines way back in 1985, which I find extremely relevant given the planning challenges we face today -

"Critics of urban modelling were correct in pin-pointing the limitations of the early models, but failed to notice that most of these arose from either the overambitious expectations about the role of models in planning or the general lack of knowledge about the state of the art and the capability to implement successfully complex mathematical equations. They did not provide an alternative methodology that could address some of the more modest goals and potential applications of large-scale models (consistent set of forecasts, evaluation of alternative transportation improvements)."

Instead of an abandonment of the models, the empirical criticism should have instead allowed for the calibration of the goals and ambitions - applying them to more modest problems and building them up based on the results.  

If 'certainty' is impossible, then it is pointless to keep it as the only measure of success. There is no methodology that exists which can forecast the future population of a city with certainty. However, there are many splendid methods by which the future population can be estimated, depending on the assumptions used. 

Abandoning probability based scientific methods (just because they "fail" to offer the certainty demanded by decision-makers) relegates planning to a position where its only hopes are various kinds of purely qualitative discursive practices; the tribal knowledge of long established planning offices and the individual genius of this or that planning officer, engineer or administrator etc.

That's no way to handle a system, let alone a complex and dynamic one such as the urban system. 

This dooms the profession to be eternally engaged with last minute fire-fighting with ad hocism as its primary tool.

Compared to this visible form of acting and responding to various urban challenges, the voluminous master plans (where they exist) with all their guidelines, development controls,  land-use plans seem unrealistic and farcical like the detailed diet-chart of a person who is gobbling junk food everyday because he never gets enough time to cook and eat healthy food.

This is also what strengthen the arguments in favour of ditching planning altogether, or effectively bypassing it by directly implementing the separate infrastructure components without requiring an overall plan to guide the process. 

Thus a mission like the Atal Mission for Rejuvenation and Transformation  (AMRUT) would cover a range of infrastructure veticals such as water supply, sewerage and septage management, storm water drainage to reduce flooding, green spaces and parks and non-motorised transport. Similarly, Swachh Bharat Mission would focus on construction of toilets, solid waste management etc. 

As the implementation of these verticals can be measured in the form of easily quantifiable metrics - number of toilets constructed; kilometers of drains laid; number of parks made etc - they, naturally, become favourites of politicians and bureaucrats alike.

But this can lead to serious problems.

Duplication dilemma

A distinct benefit of planning, even when it totally fails to predict or influence the course of urban development, is its ability to get some sense - however limited - of how the different components of the urban system interact with each other. Being obliged to operate over a specific geographical territory it can at least figure out how the various sectoral components are located with respect to each other. Operating purely within the sectoral domains eliminates this advantage. In fact, this is not very different from the arguments offered in favour of economic planning - the ability to monitor the activities of individual firms and attempt to coordinate them for the fulfillment of overall plan targets - as opposed to a purely market driven approach where each firm strives to maximise its profits irrespective of the consequences that may have for the overall economy and the environment.  

The eagerness to maximise the implementation of individual sectoral schemes leads to a tendency to overlook how different sectors interact with each other in an urban system.

Just take a look at the image below. It shows the location of a slum in the northern part of the city of Bhubaneswar. Right next to it, one can see the affordable housing units being constructed to house the residents of the slum shown in the image and also the residents of other neighbouring slums. 

The affordable housing units were being constructed under a public private partnership model and overseen by the Bhubaneswar Development Authority and the Bhubaneswar Municipal Corporation (a process which had been in the works almost since 2017). In the meantime, Jaga Mission - the flagship slum land-titling and upgrading programme of the Government of Odisha - was launched and was overseen by the Department of Housing and Urban Development.

Jaga Mission selected the same slum in its pilot phase of slum upgrading in 2020-2021 and did a very fine job of upgrading it in consultation with the residents of the slum.

However, a successful of upgrading of this slum means that there really is no need for these families to move into the nearby affordable housing site. Not only would the families have no need of moving there, the units of the housing site may now need to be filled by families of slums which are located further away - hence increasing the probability of reluctance of the residents of even those slums to move in here. 

Each scheme aimed at maximising their individual benefits, without considering that they may just end up duplicating the benefits churned out by another scheme.

A more planned and coordinated approach would probably have been to choose other slums for the pilot upgrading phase and let the present slum be catered to by the affordable units - since they were already under construction. 

This is just one example. In a situation where such lack of coordination and a continuous maximisation of implementation of individual sector verticals is the norm, such examples are innumerable and constantly proliferating. 

Yet, there are ways to turn this situation around by making intelligent use of the tools and techniques that are available to us - that we either tend to forget or abandon.

As Prastacos pointed out - we don't need to throw away our tools...we may only need to make the goals more modest and realistic.

More on that in future blogs...

 

 

Regarding farcical flirting...and GIS based master plans

Earlier today, I wrote the following on my linkedin status -

GIS based Master Plans...it's a bit like saying pen based novels...or camera based photographs.
India's farcical flirting with technical terms has to stop.
It shows an unreflected acceptance of meaningless sentences - in other words, it helps accelerate collective stupidity.

I felt that this should be elaborated upon. 

To be sure, I am not against flirting - it is a creative art. Anyone who wishes to study it seriously could turn to Act 5, Scene 2 of William Shakespeare's play "Henry V". 

Here is a youtube link to the scene in the classic film adaptation by Sir Lawrence Oliver -

 

Of course, one is free to point out that strictly speaking Henry was wooing Lady Katherine and not flirting with her. 

But then I am equally free to reply, that while Henry was indeed wooing Katherine...he was also flirting - not with her - but with the just altered geo-political situation in Europe following the battle of Agincourt where Henry achieved decisive victory over the French and was in a position to dictate terms to her. 

Yes, flirting is art indeed - of geo-political scale and significance.

So much for my admiration for genuine (geo-political or not) flirting. 

But the recent tendency (increasing at an exponential rate) in India to flirt with "tech" terminology (including the ridiculous sounding word "tech") without any regard for what they really mean, can be considered farcical indeed.

Sentences beginning with the following should immediately put one's BS-filtering systems on high alert -

• "We are developing digital tools for..."
• "According to our AI based tools..."
• "As per our machine learning algorithms..."
• "We use high-resolution satellite imagery for..."
• "We have adopted a data-driven approach for...."

The trouble is not with terms like digital, AI, machine learning, high-resolution satellite imagery etc., but with the things that generally appear in the second part of the sentence.

Consider the following statement - "We use high-resolution satellite imagery to track land-use violations in Bhubaneswar smart-city on a monthly basis."

If this video clip (with an animation showing a satellite "diving" from its orbit every time it tries to get a better look at Bhubaneswar and other such wonderful things) is not enough to turn you numb, consider the following questions -

• How exactly does a high-resolution satellite imagery help me understand what use the buildings it shows are put to - commercial, residential etc. ?
• How does it help me understand violations in bye-laws such as height restrictions ?
• How exactly does it help me understand the prescribed use of the land on which the building is situated ?

 

 AND...

• What about the fact that last I checked the master plan of Bhubaneswar was under revision the new one is not even out in the public yet ??

High-resolution satellite imagery is indeed extremely useful for city planning purposes, but let's just say -

there is many a slip 'twixt the cup and the lip.

 

In other words, there is a whole range of activities that have to be done, and done properly, before high-resolution satellite imagery can perform the task of tracking and preventing land-use violations.

By beginning the sentence with the 'cup' and ending it conveniently with the 'lip', without ever clearly mentioning what all lies in between betrays either complete ignorance of the processes involved or a sly ploy to shock-n-awe anyone not familiar with such technology for the sake of furthering ones agenda.

As regarding all that lies in between the cup (the technology) and the lip (its successful application), we can turn to Shakespeare again and say,

"Ay, there is the rub !"

It is precisely in all the activities that need to be done to make a technology effective, that the real professional grind lies - and that which most would like to steer clear of.

It is easy to write a software...but notoriously hard to organise and clean the data that would be used by the software; and fix the organisations that would use the software.

 

And now let's turn to the star of the show...one that really cracks me up every time I hear it - 

GIS based Master Planning.

I wonder who came up with that one, and most importantly - why ?

Do we ever say ludicrous things such as a pen-based novel; or a camera-based photograph; or a type-writer based article  etc ?

What exactly is the point of defining a planning process using one of the many tools, which may be deployed to aid its preparation, apart from either or both of the following -

    a) Zero understanding of planning.

    b) Zero understanding of the role of GIS in planning.

Well, there is a set of more realistic causes which are far more sinister than the above two - but let's go with these for the moment.

Applying the GIGO (garbage-in-garbage-out) model - which suggests that if the input (data) is garbage, then the output (solution) would be garbage too - to our present topic, we can argue that if the input (the planning approach) is meaningless, then the output (plans produced), would be meaningless too.

Thankfully, we have an elaborate dash-board available in the public domain to support our argument. 

As expected, the dash-board is a cool and elaborate one containing all kinds of information, except the most important one - the plans themselves. 

The first alarm bell rings when we scroll to the middle of the page and look at the master plan formulation status. We learn, that after 8 years of implementation, only 135 out of the total 500 cities covered by the scheme have reached the final step of "Final Master Plan". A total of 257 have reached the level of "Draft Master Plan".

But where are these amazing 135 GIS based Master Plans ?

For that one has to scroll right to the bottom of the page and click on the relevant states (or cities) on a map to access the plans. One has to repeat a few more rounds of needless clicking until one finally reaches the page from which two separate files can be downloaded - Master Plan Report and Land Use Map.

As Gujarat was proudly colored green (all tasks completed for all cities), I decided to click on a random city called Botad.

I didn't expect to hit jackpot on the first city I clicked on, but this is the Master Plan Report that popped up -

 

 You can download the report directly from this link.

It was a pdf copy of a 30 page long report in Gujarati language. And here is a screenshot and direct link to the land use map that accompanied it -

The map too is in pdf format; a jumble of different layers overlaid on top of each other; with no clear distinction between existing and proposed land-use. 

The map is a cartographic nightmare but that's the least of our worries.

Forget about using available technology to develop a planning approach based on decades of theoretical and practical advances and reflections in the field of planning, we are offered a national level scheme which purports to use GIS but provides us with pdf files that tell us nothing and which we can use for nothing.

The reports and plans of different cities seem to be prepared by different private consultants, each following their own cartographic rules; structure of report; and occupying their own unique positions on the scale of being slapdash.

The only thing that we can be certain of regarding the meaning of the term "GIS based Master Planning" is that all consultants have made some use of GIS software in preparing the maps.

Pardon my French, but what the F*** is the use of that ?

And yet, why is it that we don't burst out laughing and brush aside the moment we are presented with these ludicrous "tech"-loaded planning terms, be it GIS based planning or the galaxy of terms following that other ubiquitous and incomprehensible adjective - Smart ?

What makes us take such tragic farces seriously....discuss them, develop projects around them, organise conferences and webinars on them ?

May be because we are distracted from the real tasks that face us...and alienated from the scienctific knowledge that we need to tackle the urban challenges facing us.

This nonsense seems to be everywhere...and relentless.

But it seems to contain the petrified brittleness of all things insecure...one tiny push and it may crumble to dust.

Then why not give it a resounding whack ?

 

 

   

 

 

 

 

 

Solving the "land" variable in land-titling equations (and some more on tackling complex urban problems)

My previous blogs have stressed the importance of operational parameters of large and complex urban development projects and how their implementation can become very difficult - if not impossible - without a creative combination of modern computing and community feedback. Perhaps nowhere does this reality hit home harder than in the implementation of mega-sized slum land-titling and upgrading initiatives such as Jaga Mission.

The Odisha Land Rights to Slum Dwellers Act, 2017, which guides the implementation of Jaga Mission, states in section 3, sub-section 1, that 'every landless person occupying land in a slum in any urban area by such date as may be notified by the State Government, shall be entitled for settlement of land and certificate of land right shall be issued in accordance with the provisions of the Act.'

All very well-intentioned and clear so far, but a whole operational quagmire opens up when one begins to implement the provisions of the Act. The art of land administration in India as it exists today, is an elaborate one that pre-dates the Mughal era. Words like 'kissam' (a derivative of the Farsi 'Qism', referring to land-use type) co-exist with the English 'Record of Right' in land revenue records, reflecting the deep and layered history of the subject. The typical planning student of professional is often not even familiar with these terms used by the revenue department, let alone being able to intervene in the operational matters of land administration.

The slums in any city may be located on hundreds of separate parcels of land, which may belong to diverse 'kissam' types, which, further, could be a mix of non-reserved (on which land rights can be given) and reserved (on which land rights cannot be given without initiating a process of re-classification of land in consultation with the revenue department) categories. Furthermore, parcels belonging to different kissams could be owned by an array of government departments or private entities, depending on which it may or may not be possible to grant land rights. It is these attributes that must be cross-referenced with each other and with the location and household data of the slum dwellers in order to satisfy all the conditions necessary to settle land rights.

Quantifying Complexity

Let's take the example of a single city of Balasore in Odisha. The 3128 families living in 41 slums of this city are located on 735 land parcels with belong to 33 different 'kissams'.

Let's further consider a particular 'kissam' called 'gharbari', which refers to homestead. There are 238 parcels corresponding to this kissam out of which 121 parcels are owned by private enitities; 71 parcels are owned by the railways; 23 parcels are owned by temple trusts and 23 parcels are owned by various departments and offices of the state government. Only the slum houses located on the last 23 parcels in the above list, can be settled without getting into special arrangements and negotiations with other government departments and private entities.

This is just one kissam out of 33. Now, also consider the location of the 41 slums in the city and how the 3128 slum houses intersect with the land parcels of various kissam and ownership types. 

What has been described above is the case of just one city out of 115; just 41 slums out of 2919; and a mere 3128 slum houses out of a total of above 400,000.

This is complexity quantified.

And this is why we need the processing power of modern computers (and not for making powerpoint presentations on the achievements of the Mission).

Political will is a necessary condition...but not a sufficient one

This is also the reason why in the first phase of Jaga Mission, which covered about 170,000 slum households in 109 small and medium towns of Odisha, the government reached as impasse after granting about 70,000 land rights certificates. The remaining 100,000 households fell on land parcels belonging to various reserved kissam categories, restricted central government lands, private entities, temple trusts or environmentally hazardous lands.

Instead of basking in glory for having distributed 70,000 land rights certificates in less than two years (no mean feat !) the Department of Housing and Urban Development, the nodal agency overseeing the implementation of the Mission, went out of its way to initiate inter-departmental negotiations with the revenue department, forest and environment department, private entities such as royal families, temple trusts etc to find workable solutions to grant land rights to the remaining slum households. Special standard operating procedures were also developed to address the challenge of slums located on highly restricted central government lands such as those belonging to railways and defence, which may require relocation.

The special measures initiated by the Government were a clear indicator of the strong political will that backed Jaga Mission. It was also a clear indicator that even when a strong political will exists (which is rare in itself), the implementation of any large pro-poor intervention may face serious challenges due to technical and operational reasons.

And it is extremely important for people not directly involved with the implementation of such projects to have a thorough understanding of these operational reasons if they wish to engage effectively and critique accurately.

Consider this news clip (in Odia) from 2019, by a regional news channel which was critical of the Mission. It showed the residents of a slum called 'Godhi Basha', who had not received land rights certificates. However, the news anchor could not give any reason for the state of affairs apart from the usual one that the government was failing to keep its promise to slum dwellers. At the 33 second mark, the clip showed a beneficiary called Ms. Anjana Das holding the card showing her Jaga Mission house number.

However, as we have already seen, lack of political will is definitely not a problem with Jaga Mission. What then is the mystery of the Godhi Basha slum ?

Had the journalist investigated just a little more, he would have discovered that the whole slum was situated on a parcel of land that belonged to "South Eastern Railways" - one of the 72 land parcels in the city which are owned by the Indian Railways. As a matter of fact, the journalist also got the name of the beneficiary wrong.

It was possible to identify all these issues in time less than the duration of the news report, thanks to the digital data collected as part of the Mission, relevant open-source software and quick data analysis on the command line, which we started discussing in the last two blogs. 

Unfortunately, as I have shown through case examples in another blog, the government itself fails to utilise available computing technologies effectively by continuing to rely on archaic bureaucratic methods and pointless application of manual labour (relying on paper maps and field visits despite possessing high-resolution imagery and GIS databases).

It definitely retains operational overview but it cannot solve the complex problems (where complexity is merely a function of processing power available i.e. with respect to the computer, they are simple problems) which not only prevent it from overcoming its operational impasses but also cause community level confusion as shown in the case of the news clip.

Community Empowerment at Slum level...Community dis-empowerment at Mission Level

Just as the Government faces its own difficulties for not using available technologies effectively, so does the community.

As I have pointed out earlier, given the size and complexity of projects such as Jaga Mission, the typical methods of community participation, engagement and critiquing are simply not adequate. 

It is not enough to understand why something is not working in one's own slum - it is only one out of 3000 slums ! 

That means 0.0003 % of the Mission in terms of the number of slums.

How can the community get effective overview of the implementation of the Mission while covering (at the level of each slum) a microscopic 0.0003 % of the Mission, when the Government and its consultants have overview of 100 % ?

Just have a look at the maps above. The one on the left, shows the location of Godhi Basha among all the slums in Balasore city; and the one on the right shows the location of Balasore city among all the cities covered by Jaga Mission. 

I hope this gives some sense of the scale....in other words, what 0.0003 % looks and feels like.

Unfamiliar methods...or merely abandoned ?

This is why it is neither enough for governments to continue using their familiar bureaucratic methods, nor is it enough for community organisations to continue using their familiar participatory development methods. 

Probably it is time to embrace the unfamiliar methods.

The size and complexity of urban challenges in the present times demands the use of methods which were created specifically for tackling large, complex and dynamic systems -- more extensive use of open-source geo-spatial software and data analysis; further refinement of mathematical models for urban and regional planning; application on the principles of cybernetics to understand the functioning of urban systems; techniques of operations research etc. 

It is an irony that huge progress was made in the development and application of many of the above techniques in solving pressing social problems precisely when computing power was very low, and abandoned in favour of feel-good but ineffectual qualitative approaches (not to mention the pseudo-scientific farce that passes for 'tech' in contemporary urban discourses) when computing power is at its peak.

More on that in forthcoming blogs...

(Smart when you create...dumb when you consume) <-- How to recognise unnecessary technology and some Jaga Mission Stories

The challenge of humanity, since the industrial revolution, has not been one of scarcity, but one of excess (and of the exploitation and inequality that naturally appear when that excess - which can provide for the whole world - ends up being controlled by the few). 

The trouble with technology is the same. How much is enough ? Where does one draw the line between the need and the want...the useful and the useless ?

Perhaps, there is a simple way to distinguish the technology that one needs from the technology that is unnecessary, superfluous and, in all probability, harmful.

Any technology which makes you smarter while you use it and keeps making you smarter and more creative the more you use it, is a healthy and useful technology for you. The technology that makes you dumb and dependent when you use it, is neither very useful nor very healthy in the long run, irrespective of the convenience that it brings.

Most of the time, we use (rather consume) technology that makes the creator of that technology - and those who control the creators - smarter and more powerful, while making us dependent and constantly distracted (which cannot but cause a steady dumbing down over time).

Technology ...Consumer and Creator

A casual glance at the way people use their computers - one of the most powerful tools of modern technology - can confirm the above statements.

Google-maps may make map-reading, navigation and orientation very easy, but it can also decrease our ability to use our sense of direction, powers of observation and of memory to remember and locate landmarks, judge distances etc.

Typically, when we use google-maps our eyes stay glued to the smart-phone screen (the effect is the same even if we are looking at the road while driving...the app tells us everything), but if we were to go old-style with paper maps we would have to constantly look up from the map to scan the surroundings and ensure that we are at the right spot.

Of course, I am absolutely not suggesting dumping google-maps and returning to paper maps. After all the whole purpose of technology is to make life more convenient for humans so that they don’t have to engage in ceaseless manual labour and mundane tasks and engage in more meaningful pursuits instead.

But if the consequence of such “convenience” is a steady process of dumbing down and engaging in such "meaningful pursuits" like spending hours on social media and the "struggles" of becoming an influencer, then it might actually be healthier and more meaningful to return to a life of heavy manual labour (if one can, that is).

The idea is not to turn ones back on technology (it simply cannot be done), but to be aware of the manner in which the technology is owned, controlled and provided to us so that we can be conscious of its effects on us.

GPS and the Kargil Lesson

Talking about the conveniences of positioning systems, that is exactly the lesson that the Indian army learnt the hard way during the Kargil war of 1999. India was denied access to the Global Positioning System (GPS) by USA exactly when she needed it most in the context of high-altitude mountain warfare. The Kargil war was also a trigger event that led to the development of NavIC (’Navigation with Indian Constellation’...the word ‘navik’ means sailor in Hindi), India’s alternative to the GPS. 

Therefore, the simple learning that the above case provides, is that it is alright to be a user of technology, as long as you also play a role in developing it (or at least understanding how it operates), but it can be downright dangerous if you forever remain merely a consumer of technology.

Still, no matter how dependent google-map may make you, it is still a very useful tool. What arguments could one possibly offer to justify the helpless addictions that are caused by the largely useless social-media platforms ?

I am sure the people who develop these platforms continue to sharpen and develop their skills in programming and problem solving, whereas the users continuously lose the ability to use the computer for the main task it was created to perform – to compute. On top of that, the more they use...the more they generate data for these very same companies.

One cannot wait for society to change to protect oneself from such devastating trends...one simply has to jump off this crazy train oneself.

Linux and Synaptic Connections

For me that jump was in April 2016, when I made the switch from Windows to Linux...and never looked back.

I realised that the users of open-source operating systems and software, inevitably, start transforming into developers with time.

Or as my dear friend Titusz Bugya, who introduced me to Linux and taught me pretty much everything that I know about the proper use of computers, once put it jokingly-

"Linux IS user-friendly !! It is a friend of the User...not of the idiot !"

As the Linux beginner starts overcoming the hesitation and fear of the terminal window and has the first conversations with the computer using the command line, s/he begins to hone that most essential and fundamental skill required for solving a problem, no matter how complex -- the ability to formulate a question.

The clearly formulated question leads to the precisely formulated command and that leads to the desired result.

Here, the Unix philosophy, which is also used in Linux, of using programs that do only one thing and do it very well becomes a great tool. It encourages you to break down complex tasks into component parts - which by themselves may not be as overwhelming - and then deploy the appropriate programs to tackle them one at a time. 

It is not just about approaching and successfully completing a task, but about developing a certain way of thinking and approaching a problem - or as the geo-political expert Andrei Martyanov put it in his brilliant book - "to develop complex synaptic connections which are applicable for everyday life."

Some more hands-on stuff...(OR) how Linux helped Jaga Mission in Odisha

In the previous post I had started discussing about the Linux command line and the incredible flexibility and power it provides to the user. 

Using the command line, and progressing (which happens quite naturally) towards scripting and programming, also halts and reverses the "Smart when you create - Dumb when you consume" process.

The simple fact is that we can't depend on an external IT specialist or a ready-made software for most of the problems that we face regularly in our work.

Only we know the specific problems that we face in our particular work environments - and they may pop up anytime. It is impossible to out-source all such problem situations to an external software consultant.

Similarly, there may be many tasks at work, which could be solved and/or automated through the command line or scripts (a series of commands written down in a file for execution). I have already showed some examples in the previous post. 

In this post let me show another example of a slightly higher order of complexity than the ones I showed earlier.

The implementation of Jaga Mission, the flagship slum improvement project of the Government of Odisha, where I worked as a consultant urban planner, involved the creation of a pretty huge geo-spatial database.

In the first phase of the project, about 2000 slums located in 109 cities and towns of the state were mapped using quadcopter drones. The very high resolution (2.5 cm) imagery was geo-referenced and digitized to create the necessary layers of geo-spatial data layers. 

The following were the major data layers that were prepared for each slum settlement -

a) the high-resolution drone image

b) layer showing the individual slum houses

c) layer showing the slum boundary

d) layer showing cadastral (land ownership/tenancy) data corresponding to the extent of the slum settlement.

e) layer showing the existing land-use of the slum settlement

This led to the creation of a pretty substantial geo-spatial database of about 10,000 map layers. In an earlier blog on operational parameters, I have explained how this database was crucial to fulfilling the goal of Jaga Mission of granting in-situ land rights to slum dwellers. 

The geo-spatial data was particularly useful when encountered with complex situations, such as slums located on certain specific categories of land, where granting in-situ land rights may not be possible. 

When this data was handed over to the Jaga Mission office by the technology consultants, the data-sets were organised in a manner which made quick retrieval and analysis difficult.

The individual layers were stored in a series of folders and sub-folders in a manner as shown in the diagram below -

 

In order to retrieve any layer of a particular type (say, the slum household map) of any slum, one would have to first open the folder of the respective district; then the folder of the respective ULB (Urban Local Body i.e. the city) ; then the folder of the respective slum and then the necessary layer(s).

The file names of the individual layers just mentioned the type (e.g. "hhinf" for the household layer; "rplot" for the revenue plot/cadastral layer etc), without giving any further information suggesting the name of the slum or city.

While this is absolutely fine for manually retrieving the separate layer files and operating on them on a Geographic Information System (GIS) software, this method of data management is incompatible with any attempt at programming, automating or quick retieval.

And when we are dealing with 30 districts; 109 ULBs; 2000 slums; and 10000 data layer, then quick and precise retrieval is essential. Any kind of programming or process automation could also be extremely useful. 

For example, it was decided by the Government that slums located on land belonging to the Railways may need to be re-located to alternate sites. The process could be done by filtering the cadastral layers based on land ownership by the Railways and then selecting the houses which intersect with those parcels from the slum-households layer. 

However, given the manner in which the files were named and organised, this process would have to be done manually on a slum by slum basis. In the absence of an army of GIS technicians (something that the Jaga Mission did not possess), the process was bound to revert to an even more laborious process of municipal staff and revenue officials physically visiting the slums and checking if they were located on railway land.

It was almost as if the elaborate digital database had never been created.

Titusz and I wished to rename and re-organise the data-files in a manner which would enable near instantaneous retrieval and processing. But, of course, even to rename the files (in order to enable scripting), we would need - you guessed it - scripting !....or else how to rename 10000 files stored in separate folders and sub-folders ??

So, we wrote a script which would loop over each of the 30 district folders and recursively go down each sub-folder until it reached the bottom-most level where the data files where stored. 

Every time the script would move down a folder level, it would store the name of the folder as a variable. Once it reached the level of the individual file it would rename it by adding the relevant stored variables as prefixes to the original name of the file. The resultant file name would therefore contain the name of the city, the name of the slum and the type of the data layer (there was no need to add the name of the district to the file name).

The following diagram shows the concept behind the script -

 

Once this process was completed, there was no need to store the files in separate folders and sub-folders. They could be kept in a single folder and files of any combination of city name, slum name and type could be retrieved instantly.

Not only did we have fun trying to create a script that would solve our problem by making use of the names of the very folders in which they were stored (which was precisely the problem that we were trying to solve !), but we also ended up creating a fresh system which drastically reduced the time taken for analysis and decision-making regarding all future tasks.

Effectively, we used the problem to solve the problem.

As a direct consequence, it reduced the burden of manual labour which would have fallen on the shoulders of municipal workers and also reduced the problems faced by slum dwellers due to incorrect decision-making in a process as challenging as relocation.

More on those stories in the forthcoming blogs...

 
 

 

 

With great power comes great...Idiocy !

One of the tragedies of computers in the present times is that they are almost always used for things which they are not supposed to be used for. 

As the name suggests, the primary task of the computer is to compute - to take over the boring, repetitive and labour consuming computing tasks of human beings so that the species can focus on more creative and meaningful pursuits (i.e. human pursuits). 

However, if I were to observe the use of the computer by development sector managers and professionals, it would appear sans doute that the electronic computer was created for the sake of making unnecessarily heavy and pointless graphics loaded power-point presentations. 

Indeed, if one counts the amount of hours (sometimes, all the hours) that young professionals spend adjusting images, animations and texts on their power-point presentations one wonders whether computers have increased office-based manual-labour by orders of magnitude instead of reducing it. 

Of course, a major cause of this, which I have already written about in a previous blog , is the continued dependence on proprietary software despite the steady growth of open-source software and operating systems. 

While proprietary software and operating systems treat computers users primarily as consumers of technology (distracting them with ever flashy and "user-friendly" software products, which are designed to make the consumer feel tech-savvy while simultaneously building a technological dependence akin to substance addiction), open-source software and operating systems encourage users to gradually transform into a free community of developers...liberating them from the addiction of any specific product created by a company (for its own profits of course) and enabling them to create their own products which are suited to their needs.

The super-computers in our pockets

In his brilliant talk titled "You should learn to Program", computer scientist Christian Genco said something, which was quite eye-opening and embarrassing at the same time. 

He showed a photo of the Apollo Guidance computer that was created by NASA engineers to do the complex calculations necessary for the Apollo-11 moon-landing mission. He then flipped out a smart phone from his pocket and said, 

"Your cell phone in your pocket right now, has the computing power to do the calculations of a million Apollo-11 missions simultaneoulsy ! NASA scientists in 1961 would have fallen to their knees and worshipped you for having this kind of technology !"

The Apollo Guidance Computer 

Christian then wrapped up the irony by showing a clip of a little video game and said,

"And what are you using it to do ??"

Here we are, faced with a zillion challenges that are hammering our clueless city governments from all sides....the challenge of affordable land; of urban poverty; of the vulnerability of the informal sector; of climate; of traffic; of housing...(the list goes on and on)...and we use one of the most powerful tools ever created by human beings to play video games, spend hours on social media, make colorful powerpoint presentations etc.

Enter the Command Line...may the Force be with You

Many years ago, when I was in high school, it was screens like the one shown below that made me run away from computers.

 

Little did I know that many years later this grim, dark screen - known as the terminal window - would turn into my biggest ally. I shall tell the story of my introduction to Linux in another blog. But for now, let us talk a bit about this dark screen. The terminal window is the one in which you can type in commands to your computer...when the computer responds, i.e. executes your command, you begin to sense both the true power of the machine and of yourself.

Of course, in the beginning the advantage is not very evident and I wondered why I should not go click-click with the mouse on the graphical user interface (GUI). After all, that is the most familiar and convenient way of interacting with computers running Windows on them. 

But very soon the things you can do with the command line starts overtaking all that you can do with your mouse...and then, suddenly, it goes totally beyond the reach of the mouse as if a spaceship from "Star Trek" just jumped into warp speed and disappeared from sight.

Let's say, I want to create a folder called "data" using the command line. I would type the following at the command prompt and press enter -

mkdir data

On the terminal it would look like this -

 

 

But I can easily do the same by right clicking on the windows file manager and then selecting the option for creating a new folder right ? 

But let us now try to make five folders with names data-1; data-2; data-3 etc. Now it begins to get slightly inconvenient to use the mouse.

But on the command line you would just have to type the following line and press enter -

 

And it's done.

But now....let's make 500 folders !

On the command all that you have to type is the tiny command-

That's it ! When you open your file manager, you see this waiting for you -

You have activated warp drive and left the mouse somewhere far behind in the universe. 

Now, imagine that these folders have been filling in over a long time with all kinds of files (word files; excel files; image files; pdf documents etc) concerning your work.

You would create a new folder called data-pdf and copy just the pdf files into it. It may be that all the 500 folders could contain pdf files or only some of them...you cannot be sure. What would be a mouse based way to do this ? Open each folder, select the pdf files and then copy them into the new folder. 

Or you could type the following two commands -

 

The first command -- mkdir data-pdf -- makes a new folder with the name data-pdf.

The second command -- cp  -r */*.pdf  data-pdf -- uses a command called cp (copy) to recursively (i.e. goes inside each and every folder) check for files ending with a pdf extension and then copy just those files into the newly created folder called data-pdf.

 

Well, well ! Now that is something isn't it ? We asked the computer to perform quite a complex search-n-retrieval task and it did it...in much much shorter time than the blink of an eye. 

And we did this with just two lines of ultra-simple commands.

Imagine what all we could do with a series of such commands.

A series of commands tied together....well, that's a program !!

If you could do so much with these mighty single-line commands (and they are mighty as we shall see)....what all can you do with a bunch of them tied together !!

That's what we will explore here.

(to be continued...)

 

 

 

 

The importance of Operational Parameters...and the pointlessness of Data-hunting-gathering

The tragedy of data-hunting-gathering 

A defining feature of urban poverty alleviation programs in India in the present times is their ever increasing size, scale and speed of implementation. The Government plays a leading role in the planning and execution of these programs in collaboration with large (often multi-national) private sector consultants, think-tanks and non-profits with financial support from international donors and philanthro-capitalist foundations.

Increasingly, the implementation of these projects involves the creation of vast amounts of digital data, which fuels an ever growing data-hunger among all kinds of development organisations and professionals. 

Everyone is perenially looking for data these days – students, researchers, the organisation that just partnered with the government, the organisation that wants to partner with the government, the organisation that will never partner with the government, the odd travelling scholar in search of a good case study...the list goes on. 

Most of the time, the DHG (data hunter-gatherer) community is not even aware of what kind of data they need, the reason they need it for or the use they intend to put it to once they acquire it. The belief seems to be ("belief" is the right word, for there seems to be very little of science in such an approach), that once these mythical Himalayan data-sets would be acquired all the other questions would magically get answered too.

Rather than searching for data in such a hopeless manner, it could be far more useful to understand the technical aspects of the development programs that generate this data and the operational parameters they need to adhere to. Exploring in detail "how" a program is executed, also throws light on "who" actually implements it and with what tools and techniques. One can then also understand "what" data gets generated in such programs and what uses it could have.

The usefulness of studying operational parameters

Compared to the data, which may be extremely hard to obtain, the goals, objectives and  standard operating procedures guiding the implementation of these projects are more accessible. Even if they are not available in the public domain, the government is generally far more comfortable sharing these documents than the actual data. 

Having even a general understanding of the operational parameters of a project can help one deduce what kind of data may or may not have been produced as part of the project. 

For example, in the case of Jaga Mission, the operational parameters were clearly laid out in the legislation - "The Odisha Land Rights to Slum Dwellers Act, 2017" that guided its implementation.

Let’s look a bit deeper into the operational parameters of the mission in order to understand what kind of data was produced and why it was produced. 

The operational parameters of Jaga Mission

According to the Act

• A slum is defined as a compact settlement of minimum 20 households; 
• Land rights shall be granted based on the area of actual occupation; 
• The maximum ceiling for granting land rights is 45 sqm for slums located in Municipal Councils and 60 sqm for slums located in Notified Area Councils (NACs); 
• Land rights shall be granted free of charge for up to 30 sqm for families belonging to the Economically Weaker Section (EWS) category; 
• Non-EWS households shall pay a certain percentage (later fixed at 50 percent) of the benchmark value of land for securing the land rights and EWS households shall pay a lesser percentage (25 percent) for the amount of land occupied above 30 sqm.

From the above parameters, we can get a pretty clear idea of what kind of data we require in order to execute the program. It is clear that the implementation of the Mission would not only require the boundaries of slum settlements to be clearly delineated (according to the definition) but also an exact mapping of every single dwelling inside the slums in order to calculate the area under actual occupation. In its first phase, Jaga Mission covered 168000 households in 1725 slums in 109 cities, which are spread across the length and breadth of the state of Odisha (155707 sq.km), turning the mapping and survey process of the Mission into a huge logistical challenge. 

It was this massive geographical scale and the need to complete the survey within a reasonably short time which made the Government consider the use of drones. Teams of 3 to 4 professional surveyors would travel by road and reach the cities allotted to them. They would then team up with the local municipal staff and NGOs to visit the slum settlements. GIS companies and NGOs were hired and simultaneously deployed to cover all the cities and slums of Jaga. On reaching the slum, the survey team would prepare the drone flight plan (based on inputs from the municipal staff, NGO representatives and slum dwellers), set up the ground control points, fly the drone and then move on to the next slum. For an average sized slum (1.2 hectares), this process would take about 1.5 to 2 hours. After covering all the slums in a city, the survey team would pack up and drive on to the next city. The raw drone captures for a batch of cities would then be sent for processing and turned into extremely high-resolution (2 cm) ortho-images (i.e. geographically corrected images which allow true distances to be measured). While the image processing was being done, the survey teams would continue their surveys in other cities, thus ensuring a simultaneity of data capturing and data processing activities. The high resolution allowed the digitization of even the tiniest dwellings to be done directly on the ortho-image, without the need for additional DGPS surveys to be conducted in the slums (as was done in the case of the RAY project a few years earlier). 

The following three map layers (vector data) were the most crucial and where prepared for each slum -

i) Slum boundary layer – showing the exact extent of the slum on the date of the survey.

ii) Slum dwellings layer – polygons showing each dwelling unit inside the slum (along with the household survey data).

iii) Revenue parcels layer – showing the ownership details and formal records of the land parcels on which the slum is located.

Together with the ortho-image (raster) layer and an additional land-use (vector) layer, this translated into a huge dataset of 8625 map layers for the 1725 slums of Jaga.

But, once the above map layers were ready, all sorts of geographical and mathematical operations related to mission implementation could be done easily. Let's consider the following cases in a slum located in a Municipal Council (with a maximum ceiling of 45 sqm for being eligible). Let’s assume the benchmark value of land to be INR 80,00,000/acre (i.e. INR 1977/sqm).

1. Beneficiary A - EWS family occupying 27 sqm.
2. Beneficiary B - EWS family occupying 33 sqm.
3. Beneficiary C - Non-EWS family occupying 35 sqm
4. Beneficiary D - EWS or Non-EWS family occupying 50 sqm

Solution of land rights eligibility criteria for each case would be -

• Beneficiary A - Family would get land right free of cost for 27 sqm.
• Beneficiary B - Family would get land right for 33 sqm on payment of INR 1482.75/- (0.25 x 1977 x3)
• Beneficiary C - Family would get land right for 35 sqm on payment of INR 34597.5/- (0.5 x 1977 x 35)
• Beneficiary D - Ineligible until family agrees to surrender the extra 5 sqm.

We can see clearly from the above example that the operational parameters of the project as defined in the Act pretty much dictated what kind of data needed to be generated and how that data was to be used to calculate the eligibility criteria for settling the land. 

But it also becomes clear from the above, what kind of data was not generated as part of the Mission. 

The truth is in the grind

The bottom-line is that the data that governments produce for the implementation of their schemes and projects may or may not be of any use to researchers, scholars, practitioners and organisations which are not directly involved with the project. The operational parameters help us understand what kind of databases may or may not have been generated.

It is far better to have ones own objectives and questions clearly articulated and produce the datasets accordingly. In this digital age it is easy to imagine that the grind of being a solid development researcher can be avoided by just scooping up all the data that exists in government offices and working the magic of the computer on it - but that does not work.

No amount of aimless data-hunting-gathering can replace the power of scientific knowledge, well articulated research questions and a robust methodology.

The truth is in the grind.