Why physical usability really matters

We teach computer science, but usually relegate the human-computer interaction and usability aspects of it to upper year classes, where its relevance may be “too little, too late”. Software is a means to an end, whether it makes a washing machine function, control a self-driving car, or run a social media app. Software should possess seven qualities: user experience, availability, performance, scalability, adaptability, security, and economy. But, if you get the user experience wrong, then none of the rest matter. Unfortunately software exhibiting bad user experiences abounds, from terrible websites, to lacklustre apps, and awful appliance interfaces. Part of the problem is that software designers rely on advice from people who *aren’t the end-users* – sometimes they have focus-groups, but it doesn’t seem like often enough.

A case in point is Metrolinx Preto readers on various transit systems. I don’t know who chooses the device interfaces, but I imagine it is the transit systems themselves. On some systems, the readers are designed in such a manner that once tapped, the user is provided with confirmation via a sound, and a brief verification of the balance on the card. This is useful, because otherwise the user is forced to login to the Presto website to find the balance on the card. This is done on a two-line screen. The TTC on the other hand provides a huge screen area, and the only feedback they provide is success or failure – no information on the current balance. There are supposedly 10,000+ readers out there, and I would imagine transparency would be a key factor, making all the readers function in the same way from a user experience point-of-view. Consider the TTC Presto reader found in the buses.

The area associated with tapping the Presto card represents roughly 12% of the readers, front surface area. The feedback screen on the other hand takes up 30% of the real estate. Now many people using the reader for the first time will wrongly try and tap the screen, because it is the first thing the eyes are drawn too – not the small area below it. There are two problems, one is that the actual human-machine interface is very small, and the second is that the large screen basically mimics the visual instructions  already on the tap area. A better interface would have concentrated more on the interface area, and less on a huge feedback area (which serves no other purpose really). These other Presto card readers do the job *way* better from the perspective of a person interacting with a reader. Feedback is provided by means of the 2-line LED display, and its almost impossible to get the card-tapping wrong.

Considering the market, you would also think that Presto would have a mobile app, not force users into using a mobile web site. It just makes sense.

 

Software – a step back in time

It has been 60 years since the we crawled out of the primordial muck of assembly coding and started to program with the aid of Fortran. Fortran was our friend in the early days of coding, as much as Cobol, and many other languages that were to evolve. However times were different, programs were small, and usually geared towards performing a task faster than the human mind could. Ironically, our friends Fortran and Cobol are still with us, often to such an extent that it is impossible now to extricate ourselves from them.  The problem is we are still doing things the same way as we were 30 years ago, and software has ballooned to gargantuan proportions. It’s not only size, it’s also complexity. Writing a program 1000 lines long to perform some mathematical calculation, is vastly different to designing a piece of software a 50 million lines long to run a car. The more complex the software, the more chances that it conceals hidden bugs, and the greater the probability that no one really understands how it works. It’s not the languages that are the problem, it is the methodology we use to design software.

Learning to program is not fundamentally difficult. Note that I emphasize the word program, which encompasses taking a problem, solving the problem, designing an algorithm, *coding* the algorithm in a language, and testing the program to make sure it works. This is of course true if the problem can be solved. Can I write a program for cars to detect STOP signs using a built-in camera? Probably. Can it be tested? Probably? Can you create autonomous vehicles? Yes. Can you guarantee they will operate effectively in all environments? Unlikely. What happens to a self-driving car in a torrential downpour? Or a snowstorm? Autonomous trains/subways work well because they run on tracks in a controlled space. Aerospace software works well because avionics may be taken more seriously. Cars? Well who knows. A case in point – a Boeing 787 has 6.5 million lines behind its avionics and online support systems, the software in a modern vehicle? – 100 million LOC. Something not quite right there…

Fundamentally I don’t think we really know how to build systems 100 million LOC in size. We definitely don’t know how to test them properly. We don’t teach anyone to build these huge systems. We teach methods of software development like waterfall and agile, which have benefits and limitations, but maybe weren’t designed to cope with these complex pieces of software. What we need is something more immersive, some hybrid model of software development that takes into account that software is a means to an end, relies heavily on usability, and may be complex.

Until we figure out how to create useful, reliable software, we should likely put the likes of things like AI back on the shelf. Better not to let the genie out of the bottle until we know how to properly handle it.

I highly recommend reading this article: A small group of programmers wants to change how we code—before catastrophe strikes.

Coding? Why kids don’t need it.

There has been a big hoopla in the past few years about learning to code early. The number of articles on the subject is immense. In fact some people even think two year olds should start to code. I think it is all likely getting a bit out of hand. Yeah, I guess coding is somewhat important, but I think two year olds should be doing other things – like playing for instance. People seem to forget that kids should be kids. Not coders. Some places can’t even get a math curriculum right. For years Ontario has used an  inquiry-based learning system for math in elementary schools. This goes by a number of different names : discovery learning, or constructivism, which focuses on things open-ended problem solving. Problem is it doesn’t really work. Kids can’t even remember the times tables. So if you can’t get the basics of math right, then likely you won’t get coding right either.

Why do we code? To solve problems right? Coding, however, is just a tool, like a hammer. We use a hammer to build things, but not until we have some sort of idea *what* we are building. To understand how to build something, we first have to understand the context in which it will be built, the environment, the materials, the design. You can’t teach coding in isolation. That’s almost like giving someone a hammer, a bucket of nails and a skid of lumber and saying “build something”. The benefits of teaching kids to code are supposedly numerous. from increased exploration and creativity, mastery of new skills, and new ways of thinking. Now while it may be fun to build small robots and write small programs to have them do things, it is far from the reality of real computer science. The most code written every year is still in Cobol, to maintain the legacy code base that underpins the world’s financial system (and other systems alike). A world far removed from learning to code in Alice.

We have been here once before, in the 1980s there was an abundance of literature exploring the realm of teaching children to program (using visual languages such as Logo). It didn’t draw masses of students into programming then, and adding coding classes to the curriculum in elementary school now won’t do it either. Some kids will enjoy coding, many likely won’t.  Steve Jobs apparently once said “Everyone should learn how to program a computer, because it teaches you how to think.” But it doesn’t. You can’t write programs, if you don’t know how to solve the problems the programs are meant to deal with. Nearly anyone can learn to write basic programs like Python and Julia, but if you don’t have a purpose, then what’s the point? It’s nice to have coding as a skill, but not at the expense of learning about the world around you. Far too many people are disassociated from the world around them. Sitting in front of a machine learning to code, may not be the best approach to broaden the minds of our youth. We could simply start by having kids in school do more problem solving tasks, from the wider world – build things with Lego, make paper airplanes, find ways of building a bridge across a creek, cooking. The tasks are practical, and foster a means of problem solving. With a bag of problem solving skills, they will be able to deal with lots of things better in life.

Learning to code is not some form of magic, and sometimes kids should just be allowed to be kids.

 

When things aren’t quite what they seem

Some algorithms I don’t get. I just read about an algorithm “Treepedia“, which measures the canopy cover in cities, using Google imagery – or actually it measures the “Green View Index (GVI). As their website describes, they don’t count the number of individual trees, but have created a “scaleable and universally applicable method by analyzing the amount of green perceived while walking down the street”. This gives Toronto a “Green View Index” of 19.5%. Nice. But look a little closer and they also state “The visualization maps street-level perception only, so your favorite parks aren’t included!” So, it focuses on street trees, which is interesting, but doesn’t represent the true picture. The green cover in an urban environment cannot be estimated from a simple series of street-view photographs.

Now that’s a problem. The algorithm clearly does *not* calculate the amount of land covered by trees.  That’s a problem because Toronto has a bunch of ravines, and numerous parks filled with trees. Official estimates put the actual tree canopy cover at somewhere between 26.6% and 28%. Vancouver is the opposite – Treepedia gives it a rating of 25.9%, whereas official documents show the amount closer to 18% (2013). That’s made up of canopy on private property (62%), parks (27%), and streets (11%).

So why didn’t they include parks, and forests? Because it’s not a trivial thing to do. Green spaces from aerial images in urban areas also include grassed areas, and gardens, shrubs etc. But the algorithm is not entirely to blame though, maybe the logic behind it? Look, representing canopy cover is important, but that’s not what this algorithm does. It calculates some measure of “greenness” calculated at street level. It doesn’t take into account the 70ft silver maple in my back yard. It might be useful tool for filling in the street canopy with new plantings, but a metric which in the words of the website  “by which to evaluate and compare canopy cover” it is clearly not. The problem is compounded when the media, run articles and misinterpret the data. Even cities publish the findings, as shown on the City of Toronto website, proclaiming “Toronto was named one of the greenest cities in the world”. They fail to mention that only 25 cities are shown, and mislead readers with statements like “Each city has a percentage score that represents the percentage of canopy coverage”, which is clearly not the case. Besides you can hardly calculate the GVI of only 25 cities and call it a “world ranking”.

P.S. For those interested, there is an abundance of literature relating to actual canopy cover estimation using airborne LiDAR, aerial imagery, and satellite imagery.

 

 

Fixing photographs (e.g. travel snaps) (ii)

3. FIXING LIGHT with B&W

There are some images which contain shafts of light. Sometimes this light helps highlight certain objects in the photograph, be it as hard light or soft light. Consider the following photo of a viking carving from the Viking Ship Museum in Oslo. There are some nice shadows caused by the light streaming in from the right side of the scene.

One way to reduce the effects of light is to convert the photograph to black-and-white.

By suppressing the role colour plays in the image, the eyes become more fixated on the fine details, and less on the light and shadows.

4. IMproving on sharpness

Sometimes it is impossible to take a photograph with enough sharpness. Tweaking the sharpness just slightly can help bring an extra crispness to an image. This is especially true in macro photographs, or photographs with fine detail. If the image is blurry, there is every likelihood that it can not be salvaged. There is only so much magic that can be performed by image processing. Here is a close-up of some water droplets on a leaf.

If we filter the image using some unsharp masking to sharpen the image, we get:

5. saturating colour

Photographs of scenes containing vivid colour may sometimes appear quite dull, or maybe you want to boost the colour in the scene. By adjusting the colour balance, or manipulating the colour histogram, it is possible to boost the colours in a photograph, although they may end up “unrealistic” colours in the processed image. Here is a street scene of some colourful houses in Bergen, Norway.

Here the image has been processed with a simple contrast adjustment, although the blue parts of the sky have all but disappeared.

 

 

Fixing photographs (e.g. travel snaps) (i)

When travelling, it is not always possible to get a perfect photograph. You can’t control the weather – sometimes it is too sunny, and other times there is not enough light. So the option of course is to modify the photographs in some way, fixing what is considered “unaesthetic”. The problem lies in the fact that cameras, as good as they are, don’t always capture a scene the way human eyes do. Your eyes, and brain correct for many things that aren’t possible with a camera. Besides which we are all tempted to make photographs look brighter – a legacy of the filters in apps like Instagram. Should we fix photographs? It’s one of the reasons the RAW file format exists, so we can easily modify an images characteristics. At the end of the day, we fix photographs to make them more aesthetically pleasing. I don’t own a copy of Photoshop, so I don’t spend copious hours editing my photographs, it’s usually a matter of adjusting the contrast, or performing some sharpening.

There is of course the adage that photographs shouldn’t be modified too much. I think performing hundreds of tweaks on a photograph results in an over-processed image that may not really represent what the scene actually looked like. A couple of fixes to improve the aesthetic appeal?

So what sort of fixes can be done?

1. Fixing for contrast issues

Sometimes its not possible to take a photograph with the right amount of contrast. In an ideal world, the histogram of a “good” photograph should be uniformly distributed. Sometimes, there are things like the sky being overcast that get in the way. Consider the following photo, which I took from a moving train using shutter-priority with an overcast sky.

The photograph seems quite nice right? Does it truly reflect the scene I encountered? Likely not quite. If we investigate the histogram (the intensity histogram), we notice that there is one large peak towards the low end of the spectrum. There is also a small spike near the higher intensity regions, most likely related to the light regions such as the sky.

So now if we stretch the histogram, the contrast in the image will improve, and the photograph becomes more aesthetically pleasing, with much brighter tones.

2. Fixing for straight lines

In the real world, the lines of buildings are most often straight. The problem with lenses is that they are curved, and sometimes this impacts the form of photograph being acquired. The wider the lens, the more straight lines converse to the centre of the image. The worse case scenario are fish-eye lenses, which can have a field of view of up to 180°, and result in a barrel distortion. Take a photograph of a building, and the building will appear distorted. Human eyes compensate for this with the knowledge that it is a building, and its sides should be parallel – they do not consciously notice converging vertical lines. However when you view a photograph, things are perceived differently – it often appears as though a building is leaning backwards. Here is an photograph of a building in Bergen, Norway.

 

Performing a perspective correction creates an image where the vertical lines of the building are truly vertical. The downside is of course that the lower portion of the image has been compressed, so if the plan is to remove distortion in this manner, make sure to allow enough foreground in the image.

Obviously it would be better to avoid these problems when photographing buildings.

 

 

Making a simple panorama

Sometimes you want to take a photograph of something, like close-up, but the whole scene won’t fit into one photo, and you don’t have a fisheye lens on you. So what to do? Enter the panorama. Now many cameras provide some level of built-in panorama generation. Some will guide you through the process of taking a sequence of photographs that can be stitched into a panorama, off-camera, and others provide panoramic stitching in-situ (I would avoid doing this as it eats battery life). Or you can can take a bunch of photographs of a scene and use a image stitching application such as AutoStitch, or Hugin. For simplicities sake, let’s generate a simple panorama using AutoStitch.

In Oslo, I took a three pictures of a building because obtaining a single photo was not possible.

This is a very simple panorama, with feature points easy to find because of all the features on the buildings. Here is the result:

It’s not perfect, from the perspective of having some barrel distortion, but this could be removed. In fact the AutoStitch does an exceptional job, without having to set 1001 parameters. There are no visible seams, and the photograph seems like it was taken with a fisheye lens. Here is a second example, composed of three photographs taken on the hillside next to Voss, Norway. This panorama has been cropped.

This scene is more problematic, largely because of the fluid nature of some of the objects. There are some things that just aren’t possible to fix in software. The most problematic object is the tree in the centre of the picture. Because tree branches move with the slightest breeze, it is hard to register the leaves between two consecutive shots. In the enlarged segment below, you can see the ghosting effect of the leaves, which almost gives that region in the resulting panorama a blurry effect.

Ghosting of leaves.

So panorama’s containing natural objects that move are more challenging.

Image sharpening – image content and filter types

Using a sharpening filter is really contingent upon the content of an image. Increasing the size of a filter may have some impact, but it may also have no perceptible impact – what-so-ever. Consider the following photograph.

The image (which is 1500×2000 pixels – down sampled from a 12MP image) contains a lot of fine details, from the stores signage, to small objects in the window, text throughout the image, and even the lines on the pavement. So sharpening would have an impact on the visual acuity of this image. Here is the image sharpened using the “Unsharp Mask” filter in ImageJ ((radius=10, MW=0.3). You can see the image has been sharpened, as much by the increase in contrast than anything else.

Here is a close-up of two regions:

Pre-filtering (left) vs. post-sharpening (right)

Now consider this image of a landscape:

The impact of sharpening will be reduced in most of the image, and will really only manifest itself in the very thin linear structures, such as the trees. Sharpening tends to work best on features of interest with existing contrast between the feature and its surrounding area. Features that are too thin can sometimes become distorted. Indeed sometimes large photographs do not need any sharpening, because the human eye has the ability to interpret the details in the photograph, and increasing sharpness may just distort that. Again this is one of the reasons image processing relies heavily on aesthetic appeal.

Here is the image sharpened using the same parameters as the previous example:

There is a small change in contrast, most noticeable in the linear structures, such as the birch trees.  Again the filter uses contrast to improve acuity (Note that if the filter were small, say with a radius of 3 pixels, the result would be minimal). Here is a close-up of two regions.

Pre-filtering (left) vs. post-sharpening (right)

Note that the type of filter also impacts the quality of the sharpening. Compare the above results with those of the ImageJ “Sharpen” filter, which uses a kernel of the form:

ImageJ “Sharpen” filter

Notice that the “Sharpen” filter produces more detail, but at the expense of possibly overshooting some regions in the image, and making it appear grainy. There is such as thing as too much sharpening.

Original vs. ImageJ “Unsharp Masking” filter vs. ImageJ “Sharpen” filter

So in conclusion, the aesthetic appeal of an image which has been sharpened is a combination of the type of filter used, the strength/size of the filter, and the content of the image.

Some of the issues with taking holiday pictures

As I have mentioned before, some of the problems associated with digital photographs manifest themselves at the acquisition stage of the process, i.e. the human part. It is impossible to avoid, primarily because the human visual system perceives things quite differently from a camera. They are not the same, nor will they ever be. It’s impossible. This is partially because the human visual system compensates for things the camera can not. One good example is shadows. It is challenging for a photo with a hard shadow in it to appear aesthetically pleasing, because a digital camera will likely make the shadow appear quite harsh. Try and compensate for it, and the lighter areas appear washed-out. Your eyes will generally compensate for the shadows. Here are some of the issues with taking holiday snaps.

• It is not easy to take photographs of scenes with hard shadows.
• The AUTO setting does not guarantee a good photograph, and neither does M (manual). Ideally shooting in P (program) mode probably gives the most sense of flexibility.
• Shooting photographs from a moving object, e.g. a train requires the use of S (shutter priority). You may not get good results from a mobile device, because they are not designed for that.
• Using a flash for landscapes is useless.
• Photographing landscapes is not a trivial task.
• Mobile devices are not that great for holiday pictures (even iPhones).
• Huge lenses are a waste of money (and heavy).
• A super sunny day may not always provide for the best photographs.

If you get a travel photograph wrong, it’s very hard to fix it, short of going back to where you took the shot.