The art of tagging (ii) – describing cereal boxes

If we search for the word alpine in a bibliographic database, we are likely to tag all articles “of or relating to high mountains”. This is a task of simple string matching. Take a page, scan it in, perform OCR, and obtain a natural language description of what is written. It isn’t as simple to reduce an image to a set of descriptors. For example it may be difficult to ascertain the contents of a cereal box from visual images on the carton, but automatically identifying the words “SUGAR STARS” and “Kellogg’s” gives a computer a good indication of it’s contents.

We can use a list of words or tags to describe the cereal packages:

However to obtain a contextual understanding of the images of cereal, we have to produce a description, of the form:

The food manufacturer General Mills produced a cereal called “Twinkles”, in the 1960s, which was a star-shaped cereal made of toasted oats and corn. It was advertised using a red  elephant named Twinkles, who wears a bow-tie. On this particular package the elephant is sliding down the side of an iceberg into the ocean with a ship on the horizon with steam coming out of its stack. 

Even though the description doesn’t do justice to the artwork and colours shown in the image, it does portray the overall content of the image. If we look at the description, we find that all the keyword tags are contained within.

The food manufacturer General Mills produced a cereal called “Twinkles”, in the 1960s, which was a star-shaped cereal made of toasted oats and corn. It was advertised using a red  elephant named Twinkles, who wears a bowtie. On this particular package the elephant is sliding down the side of an iceberg into the ocean with a ship on the horizon with steam coming out of its stack.

Consequently, the tags could be thought of as a précis of the description. It would be hard for a computer vision based algorithm to produce a similar result.

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s