Automating Planning Tasks 2 - (Programming with GRASS + Linux)

In the first part of this blog we had discussed how the various steps of a particular planning problem (in this case the slum-proofing problem of Jaga Mission) can be articulated, algorithmised and then automated. In the second part we shall see how the specific commands in the computer program work.

The technical steps necessary for achieving the goal (given the capabilities and constraints of government organisations executing the mission) were identified and are listed below -

(a) Identify the location of existing slums 

(b) identify vacant government land parcels near the existing slums 

(c) check them for suitability 

(d) generate map outputs for further visual analysis and verification. 

It is clear that the problem solution involves spatial analysis tasks to be performed on a Geographic Information System (GIS) software. 

While QGIS is the more familiar and user-friendly option, my preferred software for tasks like this is GRASS, which stands for Geographic Resources Analysis Support System.

GRASS is an extremely powerful and versatile open-source software, which was originally developed by the US Army - Construction Engineering Research Laboratory (CERL) and then by OSGEO - The Open Source Geospatial Foundation. 

A very useful feature of GRASS is that when you undertake any operation on it (e.g. clipping features in one map layer using features in another map layer), the command line version of the operation is also shown.

This is because GRASS is not just a point-and-click software (software which rely on a Graphical User Interface - GUI - for performing the operations by clicking with a mouse), but it also contains a command line option.

Most computer users are unfamiliar with the Command Line Interface (CLI) - even intimidated by it (I certainly used to be not so long ago). They, therefore continue to work on the user-friendly environment of the GUI and doing all their tasks with the mouse. However, by doing that they fail to harness even a fraction of the power of these wonderful machines - there is a reason why users who work comfortably with the CLI are called Power-Users. I have written about the advantages of the CLI in earlier blogs.

When GRASS is launched, it opens three windows simultaneously - the Graphic console (for performing operations using the mouse) ; a display window (for viewing the maps and results of operations) ; and a command line window (for performing tasks by typing in commands).

An interesting thing happens when you do any GIS operation on GRASS. Let's say you wish to execute the module called v.extract , which creates a new vector map by selecting features from an existing map (quite like exporting selected features to a new map layer in QGIS). The series of steps is similar to that in QGIS, where you specify the input file name, the output file name (of the new file), any queries that specify which features are to be selected etc.

You will notice, that while you are specifying these details by clicking with the mouse and entering file names, the following line is getting typed at the bottom of the module window -

This line is basically the command line version of the v.extract operation. This is how it reads --

< v.extract input=bbsr_slum_houses@training where="ward_id = 16" output=ward_16_slum_houses --overwrite>

The command begins with the name of the module, then specifies name of input file, name of input file and the sql query (in this operation the query selects all slum houses that are located in ward 16). The optional --overwrite can be added if you run the operation multiple times and would want the previous output files to be overwritten automatically (useful during scripting/programming).

When you click the "copy" button in the module window, this command line gets copied to memory. You can then paste it in a text file to make it a part of your program.

A good practice is to run all the operations manually once using the mouse, just so that you are clear about all the operations and all the command line versions of the operations. With practice you would not need to copy the commands. You will simply be able to type them out from memory or my referring to the manual page of the respective module.

Turning individual commands to a script 

Commands are very powerful because instead of clicking with a mouse many times, you can just type a short line and get the job done. But the main power of the commands is that you can write a bunch of them down in a sequence in a text file and execute it as a program to automate all your computing tasks. 

Consider the following 4 commands written down in a sequence -->

(1) v.buffer input=centroids output=buffer type=point distance=$buffer

(2) v.clip input=orsac_tenable clip=buffer output=vacant

(3) v.out.ogr --overwrite input=vacant output=$LAYER_OUTPUT/$ulb\_vacant_$buffer\_m_buffer.gpkg format=GPKG

(4) db.out.ogr --overwrite input=vacant output=$LAYER_OUTPUT/$ulb\_vacant_$buffer\_m_buffer.csv format=CSV

Of course, it looks a bit overwhelming ! But nothing to worry, cos everything useful feels a bit overwhelming at the start. Here is what is going on -

In step 1, the v.buffer module draws a buffer of user defined distance around the centroids of slums and saves it in a file called buffer. In step 2, the v.clip module clips features from the land parcels layer using the buffer layer file. In step 3, the v.out.ogr module exports the clipped land parcels as a gpkg vector file and stores it in your chosen folder. In step 4, the db.out.ogr exports the attribute table of the clipped file (containing details of the land parcels) as a comma separated value (csv) spreadsheet file and stores it in your chosen folder.

When you run these commands as a script then all these tasks get performed automatically in this sequence. 

It is clear from the above, that you can tie together any number of commands in an appropriate sequence to handle analytical tasks of varying levels of volume and complexity. 

GRASS + BASH

Bash - an acronym for Bourne Again Shell - is basically a "shell", a program in Unix like operating systems such as Linux, which is used to communicate with the computer i.e. it takes keyboard commands and then passes them onto the operating system for execution. 

The unique thing about Bash is that it is not just a powerful method for passing commands to the operating system but also an effective programming language.

The advantage of combining GRASS with Bash is that you can take the GIS commands of GRASS and use them as it is in a Bash script without any change in syntax by adding a flag called --exec at the start of the command.

By doing that you can not only execute the GIS tasks but combine those tasks with all the other programs and the full power of your Linux computer.

For example, in the same script, you can select GIS files of a few specific cities, undertake all the operations that your need to undertake, then save them in a separate folder, extract the attribute data as csv files, undertake data analysis tasks on them, put them in some other folder, turn both the spreadsheets and the output maps into pdf files etc etc. 

I guess this is already somewhat of an overload...and I feel tired of typing too ! I will show the combination with Bash in the next blog.

The request and the reward

Using the command line and scripting with GRASS and Bash are extremely powerful, flexible and fun processes, but they do request a readiness to learn and an openness to be creative.

The rewards...well they are immeasurable.

Data and its Non-Use

Tackling the dynamic with the static

One of the primary challenges facing the management and governance of our cities is the fact that they are extremely complex and dynamic systems. At any point of time, the decision-maker has to juggle multiple unknown and possibly unknowable variables. Dealing with our cities, therefore, must be seen as a fascinating challenge of having to deal with uncertainty.

But how does one achieve it ? How does one master the uncertain and the unknown? There are many ways of doing it, which, unfortunately remain unused and abandoned by our city planners and decision-makers. Our planners, obsessed with the preparation of voluminous master plans, often ignore something very fundamental - you cannot tackle something extremely dynamic with something extremely static.

As the renowned architect-planner Otto Koennisberger had already observed six decades ago when he was preparing the master plan of Karachi, that by the time such master plans are ready they are already out of date. In such a situation not only do the decisions of the planners hit the ground late, but the feedback is hopelessly delayed too.

The importance of feedback...and of cognitive dissonance

Anyone dealing with complex systems would be familiar with the crucial value of feedback. A system that continuously fails to correct itself in time is a doomed system.

Yet, are things like handling vast amounts of data, taking decisions in real-time, continuous course-correction based on feedback through a network of sensors really such insurmountable problems in the present times ?

On the contrary, a characteristic feature of the present times is not only a complete technological mastery over these challenges but also the increasing affordability and accessibility of such technology. Does Google maps wait for a monthly data analysis report to figure out how many people had difficulty reaching a selected destination and modify its algorithm accordingly? It shows an alternate route instantly when it senses that the user has mistakenly taken a wrong turn.

We take this feature of Google maps as much for granted as we take a 20 year perspective plan of a city for granted - such is the collective cognitive dissonance of out times.

Tackling uncertainty by shortening the data collection cycle

Let us consider a problem more serious than reaching the shopping mall successfully using google navigation. We are all aware of the havoc that flash floods cause in our cities. They are hard to anticipate because they can occur within minutes due to extremely high rainfall intensities. As a consequence of climate change we can only expect such events to turn more erratic and intense over time. While the rain happens for a short duration, the available rain gauges still measure the average rainfall over a 24 hour cycle. Therefore, despite collecting vast amounts of rainfall data, we may still not be able to use it for predicting the occurrence of flash floods.

However, in the last few years, the Indian Space Research Organisation (ISRO) and the Indian Meteorological Department (IMD) have installed Automatic Weather Stations (AWS) at various locations across the country which record rainfall data at intervals of less than an hour. Based on the data from the AWS we not only get better data sets for analysis but can respond in near real-time when the event happens.

 

In this case, we tackled the uncertainty of the rainfall event by reducing the data-collection cycle from 24 hours to less than an hour. We didn't eliminate uncertainty but we definitely limited it.

While tackling complex and uncertain situations, our main allies are not needlessly vast quantities of data but the ability to clearly articulate the problems faced and then attempting to algorithmise the problem - the more articulate the problem statement, the more effective the algorithm.

 

Algorithmising Jaga Mission tasks

Let's take another example of Jaga Mission - Government of Odisha's flagship slum-empowerment program. Jaga Mission has arguably created the most comprehensive geo-spatial database of slums in the world. Its database consists of ultra-high resolution (2 cm) drone imagery of each and every slum in the state.

But the vastness of the data by itself achieves nothing - except increasing the headache of custodians of the data who do not possess necessary data handling skills.

It is only when the variables contained in the data are identified and linked with each other, that one can feel its true power.

The database essentially consists of four main components for each of the 2919 slums of the state -

    (a) The ultra-high resolution drone imagery  

    (b) The slum boundary map layer
    (c) The slum houses map layer (containing household information)
    (d) The cadastral map layer (showing land-parcels and ownership)

These four components can be combined in myriad ways to tackle a whole range of complex problems encountered during implementation.

Does a city wish to know whether some slums lie on land belonging to the forest department in order to avoid problems during upgrading ?

No problem! Just filter the forest parcels in the cadastral layer (d) and find out exactly which households are affected by creating an intersection with the slum houses layer (c). Similarly, total area lying on forest land can be found by creating an intersection with the slum boundary layer (b).

 

Does the forest department wish to know the condition of vegetation in such slums before it allows the upgrading process? Easy. Turn on the ultra-high resolution drone image (a) and count every leaf if you wish.

Do revenue officials in cities located hundreds of kilometers from each other need the details of a specific kind of protected land parcel (let's say "gochar kissam" - grazing land) that the slums may be located on ?

Well, just create a computer program that loops through the layers of the slums of different cities and undertakes the intended operation. I showed one such program in the last blog.

The revenue inspectors would not have to run around in each and every slum of each and every city to manually verify something that is easily done using the digital data-sets.

Such a well articulated system design can not only solve its own internal problems but can also offer solutions to other related systems - for example environmental sustainability; disaster adaptation etc.

However, the mainstream approach in Jaga Mission ultimately involved turning the geo-spatial database into thousands of PDF files and paper print-outs and then proceeding to tackle these complex tasks with the brute force of manual labour.

What a data-loss !

Automating Planning Tasks - Part 1 --> (100 mb Powerpoint file Vs 3 kb text file)

What computers were not meant to do

"But in running our institutions we disregard our tools because we do not recognise what they really are. So, we use computers to process data, as if data had a right to be processed, and as if processed data were necessarily digestible and nutritious to the institution, and carry on with the incantations like so many latter-day alchemists."

- Stafford Beer, 'Designing Freedom'

The cyberneticist Stafford Beer wrote these lines in his typical humorous style way back in the 1970s.

Despite Beer's best efforts, in the years since the publication of his essays and with the tremendous increase in the processing power, storage capacity and affordability of modern computers, this obsession with data has also kept on increasing till it reached the ludicrous levels that we see today when the act of collecting of vast amounts of data itself justifies the purpose for collecting vast amounts of data.

A particularly tragic situation is one which is quite typical in the offices of the urban development sector (the field which I am most familiar with) and involves highly educated professionals spending tens of person-hours preparing graphic-heavy power-point presentations. Nothing against power-point at all ! It is a great software. The problem lies in this undue importance that professionals in the development sector feel obligated to attach to visual presentations and the time and effort they end up dedicating to the task.

Instead of making a clear presentation of the activities being undertaken by the organisation (the main purpose of a software like power-point), the making of the presentation itself becomes a big chunk of the activities being performed by the organisation. 

And these files are heavy ! Tens of mega-bytes just for making the whole thing cluttered with images, data visualisation charts, animations  etc. 

The same philosophy extends to online dash-boards and cluttered charts that urban planning graduate students in India increasingly make for their project presentations. 

Whether by design or not, the only effect such presentations have is to visually overwhelm and confuse the viewer, and not bring clarity to the topic being discussed.

We have all seen those bloated power-point files...no need to share examples of those eye-sores here.

Now let's see instead the power of a simple text file containing a script, and with a size of only 3 kilobytes.

The 3 kb text file

The following screen-shot is of a program I wrote for automating the technical steps of the slum-proofing vertical of Jaga Mission - the landmark slum land titling and upgrading initiative of the Government of Odisha.

I will explain the slum-proofing vertical in detail in another blog. In this one I will just outline the structure of the program.

The process involved certain very concrete technical steps - (a) Identify the location of existing slums (b) identify vacant government land parcels near the existing slums (c) check them for suitability (d) generate map outputs for further visual analysis and verification.

The program automates that planning process by performing the following steps -

1) selects a user-designated city from the list of total cities; 

2) draws buffers of user-designated radius length around the centroid of each slum; 

3) clips suitable land parcels (i.e. filters out categories such as waterbodies, ponds, tanks, forests etc) that fall within the buffer from the cadastral map layer containing vacant land parcels owned by the government; 

4) calculates the total vacant land available and approximate number of households that could be accommodated; 

5) outputs a report stating the total vacant land available and total residential plots that could be created on that land assuming a plot size of 30 sqm and 60 percent land coverage by residential plots.

6) outputs vector maps of the vacant land parcels for further visual scrutiny and human analysis

7) outputs maps in pdf format for a quick look by team members unfamiliar with GIS and for printing out.

It took a few seconds for this process to be completed for a city that contained about 40 slums.

If the user would like to change the city or alter the buffer distance (for example, if suitable land is not available within the buffer of the designated radius length), it can be easily done by just typing the desired inputs in the prompt asking for the city code and the buffer radius.

Considering the fact the the Mission involves 115 cities and 2919 slums, this program shortens the analytical process by orders of magnitude and allows time to be devoted to study the outputs, have discussions, refine the overall strategy and assess the probability of effective implementation.

And most importantly, writing such programs is an extremely interesting, fun and creative process. 

Have fun doing creative work and automate the rest...what could be more delightful than that ??

The size of the text file that contains this program and undertakes all these tasks in a matter of seconds is 3 kilobytes.

Is it so hard to see which one is really our friend and ally ??

To part 2...

Smartness...without the Stutter - what India's Smart Cities Mission could be about

An obsession with "implementation" can create problems not just for planning (which is still struggling to open its parachute after being jettisoned), but also for urban missions of the government whose performance should not and probably cannot be measured quantitatively.

For example, one should not attempt to either shape or evaluate the Smart Cities Mission using metrics that one could use for AMRUT, PMAY or the Swachh Bharat Mission.

If one attempts to measure "intelligence" using metrics like the number of integrated command and control centres made, or the ridiculously high figure of lakhs or crores spent on the 5858 smart city projects or that other wholly irrelevant variable - numbers of place-making projects implemented - then one may totally lose ones way and end up with a complete hotch-potch of things where anything and everything could be described as a smart city.

Perhaps, this is exactly what made the present minister, former seasoned diplomat and articulate speaker, appear rather at a loss for words while describing the achievements of the Smart Cities Mission at the National Urban Planning Conclave in July this year. You can listen to him here.

The fact that while talking about the command and control centres he could only mention their role as war rooms during covid says a lot about the lack of clarity regarding one of the most significant components of the Smart Cities Mission. Yet, it is precisely the points that he mentioned and the manner in which he mentioned that reveal what is fundamentally problematic in the conceptualisation of the Smart Cities Mission.

Not a separate mission...but the brain linking all missions

To see the Smart Cities Mission as a distinct mission itself seems to be an erroneous approach. Instead of trying to create a green-field "smart city" or develop parts within the city as area based development zones or initiate a random traffic signaling project here or a place making project there, the smart cities mission should be about being the "brain" that links, coordinates and charts out the path for all the various urban missions and activities being undertaken in a particular city. It should be the "intelligence" that guides the actions of all the on-going and future missions. 

Seen in this way, there would be no need to justify the achievements of the mission by quoting figures for numbers of smart parks developed or the expenditure incurred in the construction a super-duper integrated command and control centre building.

Unfortunately, the work of being the brain is precisely what the smart cities mission has not been able to do. In order to do that one needs the freedom to adopt a way of thinking that focuses on how things are being implemented rather than explicitly focus on what is being implemented -- on effectiveness and functionality rather than appearing cool and spewing buzzwords.

One has to somehow initiate a "capacity building" programme for the very politicians, government officials and private sector consultants who are engaged with the smart cities mission....for, I hate to say it, but they really don't seem to have a clue when it comes to the science of creating intelligent systems capable of tackling complexity and uncertainty.

The fact that the mission is often under vigorous criticism from a bunch of even more clueless social and political scientists, journalists and activists is another reason for its inability to correct its course. This crowd is even more difficult to handle because its members are not only comfortably ignorant of the fact that they are completely ignorant but also pretty smug in their confidence that their critique is spot on.

What Smart systems are like

In my early days of exploring the smart cities mission I often came across this trivia that the concept of smart cities has its roots in some products and systems developed by the IBM company. 

Well, if that is so, then how do we explain the following lines written in 1965 ?

"Planning is not centrally concerned with the design of the artefacts, but with a continuing process that begins with the identification of social goals and the attempt to realise these through the guidance of change in the environment. At all times the system will be monitoring to show the effects of recent decisions and how these relate to the course being steered. This process may be compared to that encountered in the control mechanisms of living organisms, part of the subject matter of cybernetics."

- J.B. Mcloughlin

 

The highlighted line in the passage is precisely the task that the integrated command and control centres should be performing. And as Mcloughlin righly points out, the roots of such a continuous and dynamic style of planning (which strongly resembles a Smart City approach) lie in the fields of cybernetics and operations research.

How can a system develop a kind of "intelligence" where it can take decisions and optimise its course by continuously receiving feedback from a variety of sensors. The principle of automatic control is built into such an approach. At its most basic it could be something as simple and effective as a water tank with a floating valve stopper that stops the flow of water into to the tank when the water reaches a certain level and then resumes it again when it falls below that level. 

And at a large and advanced level it could be something like this -

This is a diagram of an interconnected power generating system. The components comprising the system and shown in the diagram were described by the author A.A. Voronov as follows -

"A few hydroelectric stations (A) and the thermal stations (B) tied into a ring network operating on a common constant load (C) under direct digital control from a control room (D) common for all the utilities."

- A.A. Voronov, "Specific Features Involved in the Development of Large Automatic Control Systems"

The component (D) highlighted by me in the line above and shown as "Supervising Computer" in the diagram IS an Integrated Command and Control Centre.

It is not about what kind of building it is located in, but the function that it performs that is important.

And what is that function ? Voronov explains it as follows -

"Depending on weather conditions the amount of water collected and stored in the station reservoirs can vary significantly. At a low water level the hydroelectric stations should cut their daily discharge from the reservoirs so as to conserve the accumulated water. To compensate the reduced capacity of the water stations the network has to increase the amount of fuel burnt in the thermal units, which implies increased requirements for railway transportation delivering this fuel. Conversely, with high water levels in the reservoirs, the requirements for railway transportation are reduced accordingly.

The algorithm computing these optimal powers is implemented on the supervising computer of the system. From the general incoming information visualised on the display the operator may adjust the algorithm to take into account the changing external conditions of the system such as fuel characteristics, flooding terms, and so on."

 

The network described above coordinates varying hydrological conditions; power generation and freight transportation through necessary algorithms working through the integrated command and control centre.

Does anything more need to said to explain what a smart and intelligent system should be doing ? 

And by the way, this example is from a book published 40 years ago !

When one starts to understand what such systems are really about and for how long they have been around then one can start going a little deeper than be preoccupied with scientifically profound concerns such as terminators enslaving humanity. 

Consider the following observation by the remarkable Soviet mathematician Elena S. Wentzel -

"Even with totally automatic systems of control which seem to make decisions with no human interference, the judgement of a human is always present in the form of the algorithm employed by the system. 

The functions of the human are not taken up by a machine, rather, they shift from a basic level to a more intelligent level. To add more weight to the argument, some automatic control systems are developed so that the human may actively interact to aid the process of control."

- Elena S. Wentzel, "Operations Research: A Methodological Approach"

And by the way, the above lines are from a book published 43 years ago !

As I have discussed earlier, the tremendous increase in the processing power of computers, the growth of GNU/Linux and the free and open-source software (FOSS) movement and the availability of open data allow us to develop such systems on our own laptops and desktops. All it requires is the courage and the willingness to learn things that are important for our work but are, as yet, unfamiliar to us. 

It is always more liberating to learn and do, than having to pretend and defend.

In the forthcoming blogs we will continue to look at how such systems can be developed to tackle complex real world problems, using the resources that are readily available to us.

Split City - have our urban missions caused multiple personality disorder in our cities ?

I had written in an earlier blog how the urban development process in India has overcome the "plan-was-good-but-implementation-was-poor" impasse by abandoning planning itself. And by planning I am referring not to the oft encountered terms such as "governance", "resilience", "sustainability" etc. etc. which end up meaning pretty much everything (and therefore - nothing), but the art and science of making sense of the future in order to better prepare for it. 

As Brian G. Field and Bryan D. MacGregor wrote:

"Planning is a process of analysis and action which is necessarily about the future. It involves intervention to manipulate procedures or activities in order to achieve goals. Forecasting is crucial to such a process." 

- 'Forecasting Techniques for Urban and Regional Planning'

Our political leaders and government officials love to mention in public speeches how the length of roads and pipes built in the country could well nigh be measuring tapes for the solar system and the number of dwelling units built under our housing programs could house the population of certain developed countries six times over (that is, provided they agree to stay in homes of the size of 25 to 30 square meters. I have been involved in building a few of these...I have some idea).

But the mere reaching of astronomical numbers and sizes is not a measure of the functionality of the amenities created.

I wrote in the blog how new affordable housing units ended up getting constructed under an earlier central government scheme right next to a slum which was being upgraded under a current state government scheme.

Presumably, the affordable housing scheme (intended to house the residents of the slum), and the slum upgrading scheme (intended to keep the slum residents in their existing settlement) were not exactly on talking terms. The result was a creation of new housing units for households that didn't need them anymore.

The phenomenon is commonly described by concerned folk as the challenge of "working in silos". The solution prescribed is often a healthy dose of the magic medicine called "convergence".

But the problem is a bit more complex than remaining and operating in silos. It seems to be one of full blown Dissociative Identity Disorder (or split personality disorder).

Split City Syndrome

Working in silos would suggest that a particular unit dealing with a project or its part is not communicating adequately with other units dealing with related projects or other parts of the same project.

However, a multiple personality syndrome would be when the same unit acts like totally different entities when dealing with separate parts of the same project or with different but inter-related projects. For example, when working with Swachh Bharat Mission the same planning office may be oblivious of the fact that Pradhaan PMAY houses contain toilets and then while working with PMAY, it may forget that toilets for the targeted households may already have been constructed under SBM. This transition may happen seamlessly over the hours of the same working day.

This is not mere speculation. This is exactly what's going on in the field, in the cities and towns of India (comfortably far away from the large halls of Vigyan Bhawan where the National Urban Planning Conclave was recently held).

While attempting to saturate each and every slum settlement of a state with individual household level toilets, the alarm suddenly rings that the households that have applied for PMAY funding for benficiary led house construction (BLC) should be factored in. 

That is certainly a wise thing to do, except that "factoring it in" effectively translates into counting the houses that have applied for PMAY BLC funding as "toilets that have already been constructed."

When thinking about toilets, the SBM personality dominates. Like a hammer seeing everything as a nail, that personality trait sees everything as a toilet. It eliminates all other rooms in a BLC house and sees only the toilet. It cares not if the house has been built or not. Then when the time comes to look at what is going on with the PMAY BLC house construction, all hell breaks loose, because it is realised that the construction has not even started as funds from the centre have not yet been released. When the PMAY personality trait dominates, then the SBM trait dissolves -- only the ghosts of toilets in yet-to-be-built houses populate the spreadsheets of SBM. 

Convergence is desirable...but not at the top 

The attempt to use the concept of "convergence" to link and coordinate the various mission verticals is a good idea. Overcoming the analysis-paralysis of long drawn planning exercises was necessary. If planning lags behind reality by an ever increasing margin then such planning is useless. As Otto Koenisberger had remarked already in the 1960s, after his experience with plan making in Karachi, that by the time the plan was prepared it was already out of date.

However, an implementation frenzy of the kind we are seeing in our cities, where the various mission verticals try to maximise their own outputs without paying any heed to how other missions are intertwined with it, is not desirable either. 

Advances in technology and computing make it possible to have dynamic, real-time systems that can coordinate the functioning of inter-connected mission verticals. The so called "convergence" of these missions is not supposed to happen at a higher level of decision-making - for example, an over-arching department of planning and convergence or in the office of a senior IAS officer who is in-charge of multiple mission streams.

It is supposed to happen throughout the network of the mission streams and through the rank-and-file of the organisational system that is in-charge of the implementation. 

More on that in forthcoming blogs...

Open Data...or Open Disdain ? Cognitive dissonance and Urban India's Open (GIGO) Portals

Have you even visited your own "Open Data" portals ?

In Satyajit Ray's classic film "Jana Aranya", the character played by Utpal Dutt asks a banana seller if he had ever himself tasted the bananas that he sells. 

Better hear the voice of the great Utpal Dutt yourself, and keep the tone in your mind -

We can then ask the officials and consultants of India's various urban missions in the same tone, if they have ever visited their own "open data" portals. 

A few months ago, at a conference on the smart cities mission, a senior official of the mission said that he was very glad that so much data was now available freely to the public through open data platforms such as the Open Government Data (OGD) platform , the Indian Urban Data Exchange (IUDX) platform etc. 

One wonders how he could say something like that at a public forum and how none of the die-hard supporters and critics of the mission in the audience had no questions regarding such a statement.

Open data and my neighbour's laundry list

I have made multiple visits to both the above platforms and downloaded various data files. Never have I ever found anything with any more usefulness or relevance to my work than ... let's say....my neighbour's laundry list. It seems as if the professionals tasked with uploading data to these portals found every scrap of excel spreadsheet lying around in their respective offices and dumped them in these digital bins. May be they are rewarded for the sheer number of files that they upload rather than what those files contain.

It is also amusing to discover that individuals who talk passionately about these platforms have never visited them or have downloaded any data from them. A large part of the problem is also the unfamiliarity with the basic standards of data storing and a lack of clarity regarding the tasks that the data should be used for.

The fact that this useless data is available in a range of file types such as csv, json, ods etc further compounds the irony of the situation.

And of course, let's not forget that entering the website and accessing the data are not always the same thing. 

Often you will encounter this at some point -

Or this -

Strangely, when I had checked the portal sometime back, many of the "private" buttons were "open" and coloured a welcoming green. Of course in the name of smart traffic signal data they often contained something as amazing as column containing names of certain squares (all the rest is left to the Sherlock Holmsian powers of imagination and deduction on the part of the website visitor).

Consider the following csv (comma separated value) file available on the OGD portal -

This is all the information that this downloaded csv file contains. The file contains no metadata (which means there is no data on the data itself) such as - when was it uploaded, who uploaded it, which period is represented in the data, what do the fields mean (does "Nos. of IHHL" mean number of individual household toilets under construction or already constructed or targeted ?), does the data correspond to the Swachh Bharat Mission (SBM) or some other project...and, how on earth does a person not dealing daily with Indian development lingo know what "IHHL" stands for in the first place ??...etc. etc. etc.

The incompleteness of the data further renders it useless. Even if we were to assume that the data shows how many toilets have been constructed, what is the use of that if not compared against the total toilets that were supposed to be constructed ? Even if that data were available in another file on the portal, they would not be comparable due to the lack of metadata.

The hard fact regarding any data management process is that any data that does not contain meta-data is garbage data. And considering the inevitable thing that happens when garbage data is fed into any analytical or decision-making system (Garbage In - Garbage Out....aka GIGO)....none of this data should be used by anyone actually trying to do something useful.

Genuine open data portals

It is normal for the "defenders" of these portals to wax apologetic when confronted with these issues with predictable statements such as - "Yes...but it also contains things which are useful...with time it will improve...it takes time to build something like this" etc etc. 

There is no need for all that. Just a quick visit to any of the following would give one a clear idea on what serious open data portals should be like.

Bhuvan portal of the Indian Space Research Organisation (ISRO)+National Remote Sensing Centre (NRSC)

Automatic Weather Station portal of the Indian Meteorological Department. 

USGS EarthExplorer of the United States Geological Survey.

AND not to forget the wonderful....

Census of India.

At the end it all boils down to this - if you have something serious to do and you know what you are talking about, your data won't have to be a pile of BS...open or not.

Disaggregation Dilemma - Part 2...(from static land-use color codes to something like OSM features indexing)

Information density at lowest levels of disagregation

Let us continue the discussion on information loss and data aggregation by comparing the maps of different planning levels that I showed in the first part of the blog, with their corresponding scales on OpenStreetMap and google satellite imagery. 

Instead of starting from the top (the city level), let's start from the bottom (the layout level) this time, and remember the words of Prastacos again- 

that computerisation basically allows us to maintain data at the lowest level of disaggregation and then readily aggregate it as the need arises.

There is of course no restriction on further zooming into the osm (OpenStreetMap) or the satellite imagery to study the area on greater detail. Similary, one can zoom out and reduce the scale to any extent to study larger areas. One does not have to stay restricted to certain categories of pre-defined map-scales (and needless to say, we also get our freedom from the scanned copies of water-soaked blue-prints that the government generously shares as "open data" and get a feel of the power of the REAL open data).

 

What is information loss due to aggregation ?

If we zoom into the area of the layout plan in the City level land use plan, then this is all the detail that we could possibly get -

 

Now compare the above level of detail (left) with what we saw in the case of the osm image (right) -

 

The above comparison is a simple visual representation of the amount of information loss that happens when spatial data is aggregated to higher levels using non-computerised cartographic methods.

There was simply no other way - in the absence of computers - than to prepare maps of different scales; covering different geographical extents in order to show different planning levels.

However, none of those limitations remain if one is working with digital spatial data and computers - there is no information loss at higher levels of aggregation of the same map. 

The trouble is that we continue to operate with the same methodology even when we have computers and geo-spatial software at out disposal.

Coding spatial information - learning from OSM features

When one understands the fundamental manner in the way in which computerisation allows aggregation and disaggregation of data, then one can also understand that the manner in which land and building uses were coded in earlier non-computerised map-making systems are no longer adequate or relevant.

Incidentally, a very powerful and effective alternative to older methods of land-use coding has already started appearing in the form of the "Map Features" of OpenStreetMap. 

Here is a description of the system from osm's wiki page -

OpenStreetMap represents physical features on the ground (e.g., roads or buildings) using tags attached to its basic data structures (its nodes, ways, and relations). Each tag describes a geographic attribute of the feature being shown by that specific node, way or relation.

Most features can be described using only a small number of tags, such as a path with a classification tag such as highway=footway, and perhaps also a name using name=*. But, since this is a worldwide, inclusive map, there can be many different feature types in OpenStreetMap, almost all of them described by tags.

The osm feature indexing system is extremely thorough and exhaustive and designed for use by the computer. Have a look at the difference between a typical color coded land use system and the osm map features system below.

This is how land uses are color coded in a typical land-use plan -

And this is how the osm map features indexing system looks like -

Just a casual glance is enough to see the power of this feature indexing system. It lists the various types of uses as key-value pairs, states what osm map elements they belong to, provide a clear description of each feature, the rendering and also photographs of typical examples.

The wealth of information that gets collected and maintained using such an indexing system is truly mind-boggling.

The analytical opportunities such systems open up can help us go toe-toe with the most complex urban problems that we face - and win.

 

Disaggregation dilemma - Part 1...(Of GIS based PDFs and Water Soaked Blue-prints)

What is wrong with our land-use plans ?

Well...nothing. Except, perhaps, the fact that they belong to an earlier epoch of technological development - a period when one necessarily had to prepare maps at different spatial scales in order to show greater or lesser detail; and use specific colours to aggregate the primary land uses at different scales - for example, yellow for residential use; red for commercial use (depending on prevalent cartographic rules).

One can also say that the technology of land-use maps, as they continue to be used to urban planning in India, corresponds to the period of map making prior to the advent of computerised cartography and geo-spatial analysis.

Using present technology, we do not need to switch between different maps prepared at different scales to study different degrees of spatial detail. Instead, we can simply zoom in and out within the same map. 

In most aspects of our lives we take this for granted - when we are booking an uber; or checking directions to a destination on google maps; or checking how far the swiggy delivery partner is at a particular point of time. 

In all such businesses, computerised geo-spatial analysis and decision-making is not just one of the components to be considered -  it is the most fundamental science and technology on which the business operations play out.

However, in a vital and complex activity such as urban planning, whose social and economic significance far exceeds that of profit maximisation in the gig economy, such technology is still a sort of a novelty which is far from having been internalised by the rank and file of the profession.  

In fact, the inadequacy of technical knowledge becomes amply clear precisely when one takes a look at the manner in which the planning profession attempts to internalise geo-spatial technologies. I discussed this in an earlier blog.

It is perhaps too difficult for our planning professionals and educators - too busy flaunting tech-terms and buzzwords - to come to terms with the simple fact that if your planning maps are made using GIS software then you do not need separate sets of maps at the levels of the city - i.e. the city level, the zone level and the layout level -- they are all part of the same geo-spatial database ! 

I am not even getting into the travesty of making such "GIS" maps available online in PDF format and then providing the attribute data in separate spreadsheet files and THEN announcing this pointless hotch-potch as Open-data ! A tighter slap on the face of the open-data movement was never landed. This is not open-data...this is an open disdain of the citizen.

 

From GIS based PDFs to Water soaked Blueprints

Let's have a look at such maps as they are available from the website of the Delhi Development Authority -

a) Here is the "big honcho" - the proposed land-use map of all Delhi. The highest level of the plan and the one with the smallest geographical scale and level of detail. Most of the time lay-persons attempting an analysis of the Delhi Master Plan remain pre-occupied with this level. Of course, it shows nothing more than the most general and most aggregated land-use distribution at the level of the city.

b) The next level of planning detail comes in the form of Zone level land-use maps. Shown below is the map of Zone-F in South Delhi. As per the zonal plan report, already in 2001, this zone had an area of 11958 hectares (i.e. 119.5 square kilometres) and a population of 12,78,000. That basically means that while it is just a part of the city of Delhi, it is still larger than many smaller sized cities of India (it is, in fact, larger than the smart city of Bhubaneswar in terms of population). 

The Zone too, therefore, is at a substantially high level of aggregation and can be compared to city level land use plans of one-million plus cities in India.

(NOTE - pay attention to the key-map in the attachment below and marvel at the cartographic genius of whoever prepared this "GIS based" pdf output)

c) And something peculiar happens when we go down to the level of the layout that contains the maximum geographical detail - the layout plans; which are more like a plan for a cluster of neighbourhood blocks. 

Here is what the plan of one of the layouts constituting Zone-F looks like...if you can make anything out that is. The keen observer would realise that this is actually a well drafted layout map (at least the key map is correct !), but we have suddenly descended from the world of GIS based PDF map outputs, to the world of water-soaked and worn-out archives of crumpled gateway sheets and blueprints. 

This is what gets uploaded as digital layout maps on the website of the premier urban planning agency of the capital of the country. 

There is therefore a complete dissonance between what digital and geo-spatial technologies truly are and how they are being utilised. 

In this matter the critics and activists of the civil-society and consultants of the private sector are often more technically incompetent than government planners. The government officials may not be familiar with the modern software but they know their cartography well enough (as illustrated by the water-soaked map), while civil society critics and private sector consultants (who often actually prepare the "GIS" outputs) are often poor in both technology and cartography.

 

In the next part we will see how computerised geo-spatial methods eliminate the problems of aggregation by allowing data to be maintained at as disaggregated a level as allowed by its granularity and aggregating the base-data as per requirement to whatever level necessary processing power of the computer.

In the words of planning expert and theorist Poulicos Prastacos -

"Data should be maintained at the lowest level of disaggregation and then readily aggregated as the need arises."

(Source - 'Integrating GIS technology in urban transportation planning and modeling' - P. Prastacos)

To be continued...

The Data exists...right under our Mouses !

The capital irony

It is perhaps a capital irony of our times that precisely at the time when computers are more powerful and affordable than ever before and the access to powerful and previously expensive software provided by the  Linux + FOSS (Free and Open Source Software) movement, the general ability to use computers effectively to address the various problems faced by our cities is at an all time low.

I myself come from a background of primarily qualitative and participatory techniques in urban planning. I continue to have a natural fondness for such techniques, but have increasingly also discovered the power that effective use of computers bring to my work.

Contrary to the myth that the quantitative and qualitative worlds are poles apart (which leads to the further myth that professionals dealing with qualitative techniques cannot use computers for serious quantitative analysis), the two are in fact friends and allies of each other and help each other continuously.

Without waxing complex, think of a rather simple example. I would like to undertake participatory exercises in various slums in my city and I use all kinds of creative ideas to undertake the same inside those communities.

But alongside that, I could also prepare a GIS database of the slums in the city that gives me spatial and quantitative information on slums - such as their location, distance from each other, distance from other city facilities, size and density of the settlements, the population and occupational characteristics of the settlements etc.

This quantitative database can actually help me increase the effectiveness of my qualitative techniques by helping me to schedule meetings, use different techniques in slums of different sizes and shapes, check the probability of consensus-building (fewer meetings could build consensus faster in a smaller slum than in a larger and denser slum) etc.

Rather than focusing too much on whether to deploy quantitative or qualitative methods, it is better to focus on the problem that needs to be solved and deploy whatever methods that may be necessary.

Why computers ?

As long as I want to do participatory activities in a handful of slums, I may not need any support of computers at all. However, if I would like to undertake such activities in tens or hundreds or thousands of slums, then I begin to feel the need of the processing power of the computer.

It is as simple as that.

As more and more resources are made available to various urban development programs and schemes in India, their sizes, duration and scale of operation are all increasing. It is not difficult to understand that in a country the size of India, urban development projects would need to be undertaken at a scale where one can at least hope to make a meaningful difference. 

But of course, computers need instructions to follow - and they need data to work on.

The data exists...right under our mouses

Quite often, the impossibility of obtaining data is cited as one of the main barriers to effective use of computers in solving urban problems in India. I have written on this topic on multiple occasions. And I have stressed on earlier blogs that mere accumulation of digital data is not of much use if one does know how to use computers effectively to process it.

However, another capital irony of our times is that much of the data whose absence we so lament - does indeed exist...and sometimes right under our noses (or mouses).

Let me demonstrate.

This particular link will take you to the dashboard of the "GIS based Master Plan" sub-scheme of AMRUT. 

The very first component of this sub-scheme was geo-database creation and in the following screen-shot of the dashboard we can see its status -

 

If we look at the first three steps of the component, we can see that satellite data had been acquired and processed for about 450 cities. In the pie chart on administrative works is not self-explanatory, but it could mean that the National Remote Sensing Centre (NRSC) of the Indian Space Research Organisation (ISRO) may have handled the satellite data acquisition and processing for 240 cities and private companies may have done it for another 220 cities.

In any case, according to the official dashboard itself, we can conclude that processed satellite data exist for about 450 cities. As per the status chart, final GIS maps also seem to exist for 351 cities.

From if it exists...to where it exists

Finding evidence and clear arguments for the claim that something exists, is the first step in finding something. If I know for sure that something exists, then I need not succumb to the fallacy that it doesn't even exist. 

The task after that is to discover, where it exists, rather than wonder if it exists.

The same method can be applied to understand exactly what all data has been collected and processed under the myriad central and state government schemes that are going on in the country and have already been executed in the past.

Believe me, we will have more data than we would need for getting most of our tasks done.

The catch here is this...a person who does not have the imagination to discover the data, most likely would not have the imagination to use that data either.

But let's keep that blast for a later post ;)

Indian Space Assets and Urban Planning

Smart in Space...Clueless on Land

As India's capabilities in the field of space technologies increase continuously, the gap between the data generated by our space based assets and the utilisation of the same for solving pressing social and economic problems is felt palpably...and painfully.

After all, the vision of our space program has always been to -

"Harness, sustain and augment space technology for national development, while pursuing space science research and planetary exploration."

And this is the level that we have already reached in this domain -

 

When it comes to space, we are not just good - we are among the best in the world and sometimes better.

The Indian satellite cartosat-3, launched in November 2019, is one of the most advanced high-resolution earth observation satellites in the world. 

With a resolution of 25 cm, Cartosat-3 surpasses the American World View-3 satellite owned by Maxar Technologies, which has a resolution of 31 cm.

And guess what is written in the Mission document of Cartosat-3 as the primary application of this third generation satellite -

You can access the document on this link.

One would expect that such high resolution products would be developed for the defense sector alone, but we have reached a level of technological development, where the mission document lists purely civilian sectors as the target users of Cartosat-3.

I seriously wonder whether the scores of professionals of India's urban development sector are aware that one of the most advanced products of one of the most advanced fields of human technology has been produced by their country for them.

Is it really so hard to imagine, what all becomes possible when you have high resolution satellite data available for the whole city and the region ?

From data collection to serious data analysis

With the development of advanced earth observation satellites we can finally take a pause from the gigantic spatial survey exercises that take up all the energy and creativity that could and should be dedicated to data analysis, forecasting, modelling etc. In any case, the fragmented and project specific data generated by these large urban development projects is used poorly and then abandoned and forgotten the moment the projects come to an end (refer to previous blogs for more details).

Consider the fact that Jaga Mission - which created one of the largest high-resolution geo-spatial database of urban slums in the world, had to deploy three separate drone survey companies, multiple quadcopter drones, teams of surveyors and the resources of over a hundred city governments to complete that survey in less than a year.

Given the logistics of covering almost 2000 slums in 109 small and medium cities spread across the length and breadth of the state of Odisha, which has an area of 155,000 square kilometres, it was a daunting task. Typically, in large urban development projects in India (and they are all large nowadays), so much energy and resources are devoted to conduct the surveys and create the datasets that serious analysis never gets a chance to take off. 

Such a daunting logistics would also imply, that while the urban reality is extremely dynamic, the probability of repeating such a spatial survey exercise at regular intervals would be very low.

From the point of view of temporal change, the ultra-high resolution data collected under Jaga Mission in 2018 may already be out of date. And there exist no plans of updating that data set.

Satellite data has no such problem with temporal resolution (the interval of time after which the same area of the surface of the earth would be captured again by the satellite in orbit).

With Cartosat-3 data, one can not only get a detailed picture of slum settlements (not just in Odisha, but through the country), but one can also analyse the spatial changes over time and also relate the location of slums to other features in the city (none of these are possible with Jaga Mission drone data, which only captured images of individual slum settlements.

Data without scientific knowledge is useless

There is a never-ending clamour for data among urban development professionals and researchers in India...a tendency I described as "data hunting-gathering" in a previous blog. However, it was not access to unlimited amounts of data that made the Indian space program what it is today - but scientific knowledge and the intelligent application of that knowledge.

Data is crucial - but only when one possesses the necessary scientific knowledge to effectively use that data. 

The trouble is that many (definitely not all) professionals demanding access to all kinds of digital data - i) would not be able to recognise that data if it were staring at them from their computer screens; ii) would not know what to do with the data even if by some miracle they figured out that it was indeed the data that they were looking for.

The usual excuse given by scholars and practitioners alike is that urban challenges are too complex. Of course, they are complex - but complexity of a problem is as much a function of the knowledge of the problem-solver as it is an intrinsic characteristic of the problem itself.

Finding an address too can be a very complex task - if someone doesn't know how to read a map. 

It is always nice to check whether a problem is really complex or if I am too dumb ! It may be very nice to discover that it is the latter, because then I know that the problem is solvable and I also get an opportunity to study and learn something new and useful.

Unless that knowledge gap is bridged, the brilliant developments in the field of Indian space technologies shall be unable to solve the relatively mundane challenges of urban planning and development.

And that would be a real shame.

You cannot "find" data, if you cannot recognise it

I have written in earlier blogs about the trouble with GIGO (Garbage-In-Garbage-Out), which basically means that if the data that goes into your software is garbage, then your output would be garbage too. However, sometimes the problem may be what Andrei Martyanov describes as "double-GIGO" (I can't remember which video it was in but feel free to check out this one anyway), where both the data used and the formulation of the problem may be garbage. Needless to say, that it is one hell of a tragedy when that happens. Unfortunately, it is not a rare phenomenon at all.

However, quit often, the problem is simply not knowing what data to look for and where to look. For example, last year the CITIIS program of the National Institute of Urban Affairs (NIUA) organised a training workshop for the Department of Housing and Urban Development in Odisha for mapping of water bodies in urban areas of the state. The program started with the mapping of 19 water bodies in the pilot phase.

Having engaged with the implementation of such programs from the inside, I know for a fact that "there is many a slip betwixt the cup and the lip", when it comes to their objectives and the data that they use for achieving it. I have written extensively on the state of data in government projects in multiple blogs.

Even without going into any detail of this specific program, isn't it a bit peculiar to do such a gala workshop on such a theme, when the following already exists ?? 

This is the dashboard of the Water Bodies Information System (WBIS) of the Indian Space Research Organisation (ISRO). Here is the link to it. 

The site clearly mentions -

"The water spread area information is extracted using images of 6 m to 56 m resolution for delineating water bodies of sizes as small as 0.025 ha and 50 ha respectively. This information is made available as Water Bodies Information System (WBIS) for visualisation and download."

One would expect that the organisers of the workshop to be atleast aware of the existence of such an information system prepared by such a respectable organisation - Alas, nothing of the sort !

When the knowledge and technical abilities of urban development professionals increasingly gets limited to preparing powerpoint presentations, having endless online meetings and organising various kinds of "capacity-building" events, then the probability of being able to identify relevant data (even when it stares at you point blank) is bound to decrease...and at a steep slope.

One can safely argue that it is hovering somewhere very close to zero if not already there

The ability to find data is very much a function of being able to recognise relevant data and that in turn is dependant of ones knowledge of the subject, which needs constant updating.

But is there any time left for that after the relentless "labour" of back-to-back online meetings ?