Disaggregation Dilemma - Part 2...(from static land-use color codes to something like OSM features indexing)

Information density at lowest levels of disagregation

Let us continue the discussion on information loss and data aggregation by comparing the maps of different planning levels that I showed in the first part of the blog, with their corresponding scales on OpenStreetMap and google satellite imagery. 

Instead of starting from the top (the city level), let's start from the bottom (the layout level) this time, and remember the words of Prastacos again- 

that computerisation basically allows us to maintain data at the lowest level of disaggregation and then readily aggregate it as the need arises.

There is of course no restriction on further zooming into the osm (OpenStreetMap) or the satellite imagery to study the area on greater detail. Similary, one can zoom out and reduce the scale to any extent to study larger areas. One does not have to stay restricted to certain categories of pre-defined map-scales (and needless to say, we also get our freedom from the scanned copies of water-soaked blue-prints that the government generously shares as "open data" and get a feel of the power of the REAL open data).

 

What is information loss due to aggregation ?

If we zoom into the area of the layout plan in the City level land use plan, then this is all the detail that we could possibly get -

 

Now compare the above level of detail (left) with what we saw in the case of the osm image (right) -

 

The above comparison is a simple visual representation of the amount of information loss that happens when spatial data is aggregated to higher levels using non-computerised cartographic methods.

There was simply no other way - in the absence of computers - than to prepare maps of different scales; covering different geographical extents in order to show different planning levels.

However, none of those limitations remain if one is working with digital spatial data and computers - there is no information loss at higher levels of aggregation of the same map. 

The trouble is that we continue to operate with the same methodology even when we have computers and geo-spatial software at out disposal.

Coding spatial information - learning from OSM features

When one understands the fundamental manner in the way in which computerisation allows aggregation and disaggregation of data, then one can also understand that the manner in which land and building uses were coded in earlier non-computerised map-making systems are no longer adequate or relevant.

Incidentally, a very powerful and effective alternative to older methods of land-use coding has already started appearing in the form of the "Map Features" of OpenStreetMap. 

Here is a description of the system from osm's wiki page -

OpenStreetMap represents physical features on the ground (e.g., roads or buildings) using tags attached to its basic data structures (its nodes, ways, and relations). Each tag describes a geographic attribute of the feature being shown by that specific node, way or relation.

Most features can be described using only a small number of tags, such as a path with a classification tag such as highway=footway, and perhaps also a name using name=*. But, since this is a worldwide, inclusive map, there can be many different feature types in OpenStreetMap, almost all of them described by tags.

The osm feature indexing system is extremely thorough and exhaustive and designed for use by the computer. Have a look at the difference between a typical color coded land use system and the osm map features system below.

This is how land uses are color coded in a typical land-use plan -

And this is how the osm map features indexing system looks like -

Just a casual glance is enough to see the power of this feature indexing system. It lists the various types of uses as key-value pairs, states what osm map elements they belong to, provide a clear description of each feature, the rendering and also photographs of typical examples.

The wealth of information that gets collected and maintained using such an indexing system is truly mind-boggling.

The analytical opportunities such systems open up can help us go toe-toe with the most complex urban problems that we face - and win.

 

Disaggregation dilemma - Part 1...(Of GIS based PDFs and Water Soaked Blue-prints)

What is wrong with our land-use plans ?

Well...nothing. Except, perhaps, the fact that they belong to an earlier epoch of technological development - a period when one necessarily had to prepare maps at different spatial scales in order to show greater or lesser detail; and use specific colours to aggregate the primary land uses at different scales - for example, yellow for residential use; red for commercial use (depending on prevalent cartographic rules).

One can also say that the technology of land-use maps, as they continue to be used to urban planning in India, corresponds to the period of map making prior to the advent of computerised cartography and geo-spatial analysis.

Using present technology, we do not need to switch between different maps prepared at different scales to study different degrees of spatial detail. Instead, we can simply zoom in and out within the same map. 

In most aspects of our lives we take this for granted - when we are booking an uber; or checking directions to a destination on google maps; or checking how far the swiggy delivery partner is at a particular point of time. 

In all such businesses, computerised geo-spatial analysis and decision-making is not just one of the components to be considered -  it is the most fundamental science and technology on which the business operations play out.

However, in a vital and complex activity such as urban planning, whose social and economic significance far exceeds that of profit maximisation in the gig economy, such technology is still a sort of a novelty which is far from having been internalised by the rank and file of the profession.  

In fact, the inadequacy of technical knowledge becomes amply clear precisely when one takes a look at the manner in which the planning profession attempts to internalise geo-spatial technologies. I discussed this in an earlier blog.

It is perhaps too difficult for our planning professionals and educators - too busy flaunting tech-terms and buzzwords - to come to terms with the simple fact that if your planning maps are made using GIS software then you do not need separate sets of maps at the levels of the city - i.e. the city level, the zone level and the layout level -- they are all part of the same geo-spatial database ! 

I am not even getting into the travesty of making such "GIS" maps available online in PDF format and then providing the attribute data in separate spreadsheet files and THEN announcing this pointless hotch-potch as Open-data ! A tighter slap on the face of the open-data movement was never landed. This is not open-data...this is an open disdain of the citizen.

 

From GIS based PDFs to Water soaked Blueprints

Let's have a look at such maps as they are available from the website of the Delhi Development Authority -

a) Here is the "big honcho" - the proposed land-use map of all Delhi. The highest level of the plan and the one with the smallest geographical scale and level of detail. Most of the time lay-persons attempting an analysis of the Delhi Master Plan remain pre-occupied with this level. Of course, it shows nothing more than the most general and most aggregated land-use distribution at the level of the city.

b) The next level of planning detail comes in the form of Zone level land-use maps. Shown below is the map of Zone-F in South Delhi. As per the zonal plan report, already in 2001, this zone had an area of 11958 hectares (i.e. 119.5 square kilometres) and a population of 12,78,000. That basically means that while it is just a part of the city of Delhi, it is still larger than many smaller sized cities of India (it is, in fact, larger than the smart city of Bhubaneswar in terms of population). 

The Zone too, therefore, is at a substantially high level of aggregation and can be compared to city level land use plans of one-million plus cities in India.

(NOTE - pay attention to the key-map in the attachment below and marvel at the cartographic genius of whoever prepared this "GIS based" pdf output)

c) And something peculiar happens when we go down to the level of the layout that contains the maximum geographical detail - the layout plans; which are more like a plan for a cluster of neighbourhood blocks. 

Here is what the plan of one of the layouts constituting Zone-F looks like...if you can make anything out that is. The keen observer would realise that this is actually a well drafted layout map (at least the key map is correct !), but we have suddenly descended from the world of GIS based PDF map outputs, to the world of water-soaked and worn-out archives of crumpled gateway sheets and blueprints. 

This is what gets uploaded as digital layout maps on the website of the premier urban planning agency of the capital of the country. 

There is therefore a complete dissonance between what digital and geo-spatial technologies truly are and how they are being utilised. 

In this matter the critics and activists of the civil-society and consultants of the private sector are often more technically incompetent than government planners. The government officials may not be familiar with the modern software but they know their cartography well enough (as illustrated by the water-soaked map), while civil society critics and private sector consultants (who often actually prepare the "GIS" outputs) are often poor in both technology and cartography.

 

In the next part we will see how computerised geo-spatial methods eliminate the problems of aggregation by allowing data to be maintained at as disaggregated a level as allowed by its granularity and aggregating the base-data as per requirement to whatever level necessary processing power of the computer.

In the words of planning expert and theorist Poulicos Prastacos -

"Data should be maintained at the lowest level of disaggregation and then readily aggregated as the need arises."

(Source - 'Integrating GIS technology in urban transportation planning and modeling' - P. Prastacos)

To be continued...

The Data exists...right under our Mouses !

The capital irony

It is perhaps a capital irony of our times that precisely at the time when computers are more powerful and affordable than ever before and the access to powerful and previously expensive software provided by the  Linux + FOSS (Free and Open Source Software) movement, the general ability to use computers effectively to address the various problems faced by our cities is at an all time low.

I myself come from a background of primarily qualitative and participatory techniques in urban planning. I continue to have a natural fondness for such techniques, but have increasingly also discovered the power that effective use of computers bring to my work.

Contrary to the myth that the quantitative and qualitative worlds are poles apart (which leads to the further myth that professionals dealing with qualitative techniques cannot use computers for serious quantitative analysis), the two are in fact friends and allies of each other and help each other continuously.

Without waxing complex, think of a rather simple example. I would like to undertake participatory exercises in various slums in my city and I use all kinds of creative ideas to undertake the same inside those communities.

But alongside that, I could also prepare a GIS database of the slums in the city that gives me spatial and quantitative information on slums - such as their location, distance from each other, distance from other city facilities, size and density of the settlements, the population and occupational characteristics of the settlements etc.

This quantitative database can actually help me increase the effectiveness of my qualitative techniques by helping me to schedule meetings, use different techniques in slums of different sizes and shapes, check the probability of consensus-building (fewer meetings could build consensus faster in a smaller slum than in a larger and denser slum) etc.

Rather than focusing too much on whether to deploy quantitative or qualitative methods, it is better to focus on the problem that needs to be solved and deploy whatever methods that may be necessary.

Why computers ?

As long as I want to do participatory activities in a handful of slums, I may not need any support of computers at all. However, if I would like to undertake such activities in tens or hundreds or thousands of slums, then I begin to feel the need of the processing power of the computer.

It is as simple as that.

As more and more resources are made available to various urban development programs and schemes in India, their sizes, duration and scale of operation are all increasing. It is not difficult to understand that in a country the size of India, urban development projects would need to be undertaken at a scale where one can at least hope to make a meaningful difference. 

But of course, computers need instructions to follow - and they need data to work on.

The data exists...right under our mouses

Quite often, the impossibility of obtaining data is cited as one of the main barriers to effective use of computers in solving urban problems in India. I have written on this topic on multiple occasions. And I have stressed on earlier blogs that mere accumulation of digital data is not of much use if one does know how to use computers effectively to process it.

However, another capital irony of our times is that much of the data whose absence we so lament - does indeed exist...and sometimes right under our noses (or mouses).

Let me demonstrate.

This particular link will take you to the dashboard of the "GIS based Master Plan" sub-scheme of AMRUT. 

The very first component of this sub-scheme was geo-database creation and in the following screen-shot of the dashboard we can see its status -

 

If we look at the first three steps of the component, we can see that satellite data had been acquired and processed for about 450 cities. In the pie chart on administrative works is not self-explanatory, but it could mean that the National Remote Sensing Centre (NRSC) of the Indian Space Research Organisation (ISRO) may have handled the satellite data acquisition and processing for 240 cities and private companies may have done it for another 220 cities.

In any case, according to the official dashboard itself, we can conclude that processed satellite data exist for about 450 cities. As per the status chart, final GIS maps also seem to exist for 351 cities.

From if it exists...to where it exists

Finding evidence and clear arguments for the claim that something exists, is the first step in finding something. If I know for sure that something exists, then I need not succumb to the fallacy that it doesn't even exist. 

The task after that is to discover, where it exists, rather than wonder if it exists.

The same method can be applied to understand exactly what all data has been collected and processed under the myriad central and state government schemes that are going on in the country and have already been executed in the past.

Believe me, we will have more data than we would need for getting most of our tasks done.

The catch here is this...a person who does not have the imagination to discover the data, most likely would not have the imagination to use that data either.

But let's keep that blast for a later post ;)

Indian Space Assets and Urban Planning

Smart in Space...Clueless on Land

As India's capabilities in the field of space technologies increase continuously, the gap between the data generated by our space based assets and the utilisation of the same for solving pressing social and economic problems is felt palpably...and painfully.

After all, the vision of our space program has always been to -

"Harness, sustain and augment space technology for national development, while pursuing space science research and planetary exploration."

And this is the level that we have already reached in this domain -

 

When it comes to space, we are not just good - we are among the best in the world and sometimes better.

The Indian satellite cartosat-3, launched in November 2019, is one of the most advanced high-resolution earth observation satellites in the world. 

With a resolution of 25 cm, Cartosat-3 surpasses the American World View-3 satellite owned by Maxar Technologies, which has a resolution of 31 cm.

And guess what is written in the Mission document of Cartosat-3 as the primary application of this third generation satellite -

You can access the document on this link.

One would expect that such high resolution products would be developed for the defense sector alone, but we have reached a level of technological development, where the mission document lists purely civilian sectors as the target users of Cartosat-3.

I seriously wonder whether the scores of professionals of India's urban development sector are aware that one of the most advanced products of one of the most advanced fields of human technology has been produced by their country for them.

Is it really so hard to imagine, what all becomes possible when you have high resolution satellite data available for the whole city and the region ?

From data collection to serious data analysis

With the development of advanced earth observation satellites we can finally take a pause from the gigantic spatial survey exercises that take up all the energy and creativity that could and should be dedicated to data analysis, forecasting, modelling etc. In any case, the fragmented and project specific data generated by these large urban development projects is used poorly and then abandoned and forgotten the moment the projects come to an end (refer to previous blogs for more details).

Consider the fact that Jaga Mission - which created one of the largest high-resolution geo-spatial database of urban slums in the world, had to deploy three separate drone survey companies, multiple quadcopter drones, teams of surveyors and the resources of over a hundred city governments to complete that survey in less than a year.

Given the logistics of covering almost 2000 slums in 109 small and medium cities spread across the length and breadth of the state of Odisha, which has an area of 155,000 square kilometres, it was a daunting task. Typically, in large urban development projects in India (and they are all large nowadays), so much energy and resources are devoted to conduct the surveys and create the datasets that serious analysis never gets a chance to take off. 

Such a daunting logistics would also imply, that while the urban reality is extremely dynamic, the probability of repeating such a spatial survey exercise at regular intervals would be very low.

From the point of view of temporal change, the ultra-high resolution data collected under Jaga Mission in 2018 may already be out of date. And there exist no plans of updating that data set.

Satellite data has no such problem with temporal resolution (the interval of time after which the same area of the surface of the earth would be captured again by the satellite in orbit).

With Cartosat-3 data, one can not only get a detailed picture of slum settlements (not just in Odisha, but through the country), but one can also analyse the spatial changes over time and also relate the location of slums to other features in the city (none of these are possible with Jaga Mission drone data, which only captured images of individual slum settlements.

Data without scientific knowledge is useless

There is a never-ending clamour for data among urban development professionals and researchers in India...a tendency I described as "data hunting-gathering" in a previous blog. However, it was not access to unlimited amounts of data that made the Indian space program what it is today - but scientific knowledge and the intelligent application of that knowledge.

Data is crucial - but only when one possesses the necessary scientific knowledge to effectively use that data. 

The trouble is that many (definitely not all) professionals demanding access to all kinds of digital data - i) would not be able to recognise that data if it were staring at them from their computer screens; ii) would not know what to do with the data even if by some miracle they figured out that it was indeed the data that they were looking for.

The usual excuse given by scholars and practitioners alike is that urban challenges are too complex. Of course, they are complex - but complexity of a problem is as much a function of the knowledge of the problem-solver as it is an intrinsic characteristic of the problem itself.

Finding an address too can be a very complex task - if someone doesn't know how to read a map. 

It is always nice to check whether a problem is really complex or if I am too dumb ! It may be very nice to discover that it is the latter, because then I know that the problem is solvable and I also get an opportunity to study and learn something new and useful.

Unless that knowledge gap is bridged, the brilliant developments in the field of Indian space technologies shall be unable to solve the relatively mundane challenges of urban planning and development.

And that would be a real shame.