Tag Clouds

Visualizing term frequency

Web 2.0 has made the tag cloud an ubiquitous form of text visualization. To create a tag cloud for a text, you first remove stop words, then take a couple dozen of the most frequently occurring terms, alphabetize them, and render each so that its font size is proportional to the number of times that it occurs in the text. The result typically looks something like the following:

Chirag Mehta's US Presidential Speeches Tag Cloud

This example is taken from Chirag Mehta's US Presidential Speeches Tag Cloud website. The site shows tag clouds for various historic presidential speeches, beginning in 1776 and running through George W. Bush's State of the Union addresses. You move through speeches by adjusting a horizontal time-line slider along the top. The display responds by providing a tag cloud for each speech. As you move back and forth through time, you can see different words become more or less prominent. Mehta also uses color to indicate the relative newness of a given term, fading from white to brown over time. Take a few moments to familiarize yourself with Mehta's website, to get some idea of the potential of this kind of visualization. We're going to start by visualizing the term frequencies of a single text; later you'll learn how to extend the method to synchronous or diachronous collections of texts.

Mapping one range onto another

You already did most of the work that you'll need to make a tag cloud visualization when you learned how to compute word frequencies. We start by deciding how many terms we want to have in our cloud. Usually this will be somewhere on the order of 30 to 100. You want to show enough terms to capture the most interesting aspects of your text, but not so many that the distinctive features are obscured by noise. Once the size of the tag cloud is determined, you take that many elements from the top of the sorted dictionary of word-frequency pairs. Suppose we're working with Dollard's biography and we've decided that we want the size of our tag cloud to be 100 elements. Then we take the top 100 items from our dictionary. The most frequent term, 'dollard', occurs 91 times. The least frequent term occurs 6 times. Subtracting one from the other, we have a frequency range of 85.

Now we need to map this range onto a range of possible font sizes. We've chosen 24 pixels as our smallest font and 54 as our largest, giving us a range of 30 font sizes. So we need to map 85 different frequencies down to 30 different fonts. We could work with these exact values, but it makes more sense to come up with a general formula.

Let's start with the ends of our ranges. We want a frequency of 6 to be mapped to a font size of 24, or, more generally, we want minfreq to be mapped to minfont. Likewise, we want maxfreq to be mapped to maxfont.

Now consider the second-most frequent term, 'iroquois', which occurs 64 times in Dollard's biography. We first need to determine what proportion of the way it is between minfreq and maxfreq. In this case, it is (64-6)/85 = 58/85 = 0.68235. In other words, 'iroquois' occurs about 68% of the way between the least frequent term and 'dollard'. More generally, let's define a frequency scalingfactor for term k as:

(We have to use the float function to tell Python that we're working with floating point numbers and it shouldn't convert everything to integers.)

That deals with the range on the frequency side of our mapping. On the font side, we will use a similar logic:

The floor function in the math module rounds a floating point number down to the nearest integer.

A little bit of CSS

In the early days of the web, HTML tags specified both what something was (e.g., a title or a paragraph) and how something should look (e.g., italics or boldface). This made it difficult to write web pages that would look good and be usable on a variety of different computers. A very large font might make a good headline on a wide screen, but not fit on a narrow one. A page that looks good in color might be illegible when printed on a laser printer. To get a sense of the problem, compare the Google home page with the mobile version of the page that is designed to be used on devices like cellphones. The obvious solution was to separate the content of a page from its form. HTML tags are now generally used to specify what something is, and Cascading Style Sheets (CSS) are used to specify how it should look under various conditions.

You've already seen one example of CSS, when we used the style property of an HTML a tag to indicate that we didn't want the browser to underline hyperlinks. Putting CSS into HTML tags like this is known as inline CSS. It sort of defeats the original purpose of separating form and content, but it will make it easier for us to do so later on.

We can bundle this into a function that returns the HTML a tag as a string, given a url and link name. Add the following function to the dh.py module.

Given the undecoratedHyperlink function, we can rewrite our keywordListToGoogleSearchLink function, too. Replace the old version in dh.py with the following version:

In order to apply a style to a small region of your web page, you usually use a span tag. For example, if you wanted to print a word in red in HTML, you could use either of the following expressions. You can test these two HTML expressions online with the W3 Schools TryIt editor.

The first example uses a predefined color name. The second uses an RGB function that defines a color in terms of how much red, green and blue it contains, each on a scale ranging from 0 to 255. It is also possible to change font sizes using inline CSS. Try copying the following example into the HTML editor:

To use a particular style for a large region of your page, you use the HTML div tag. Think of div like a generalization of a paragraph. In the program that we're developing, we are going to use span tags for each term in the tag cloud, but the whole cloud will be sitting inside of a div. Try the following code in the HTML editor. The CSS properties set the width of the div to 560 pixels, set the background color to a very light grey, put a solid one-pixel grey border around it, and center any text within it. As long as you end each property-value pair with a semi-colon, you can include as many as you want in the style.

Functions to write HTML divs and spans

We now have enough background information to write some Python functions that will automatically create HTML div and span tags. Copy the following function and add it to dh.py.

If you copy this function to a Python shell and execute it, you can see that it allows you to add additional properties to a default div.

We will also bundle up our code to create a scaled font. Copy the following function into the dh.py module.

Other dimensions for visualization

Given our scaledFontSizeSpan function, we now have the ability to map any scaling factor between 0 and 1 onto a range of font sizes that we've chosen. Font size is not the only variable that we could use for visualization, of course. Any range of potentially

interesting differences in our data can be mapped onto any other range of perceptually salient properties. It is quite easy, for example, to create a function that adjusts the lightness or darkness of a greyscale font as well as the font size. Study the following and add it to dh.py.

When rgb is given three equal parameters, it returns a value between black (0, 0, 0) and white (255, 255, 255). In this case, we start with a relatively light grey (200, 200, 200) and subtract larger and larger values from it as the scaling factor increases. When the scaling factor is 1, our font color is black. Recall that the %d formatting character allows us to interpolate an integer into a string.

It is also possible to adjust each of the red, green and blue color components independently. The following function creates a heat map, shading from a cool blue when the scaling factor is 0, to a hot red when it is 1. Make sure that you understand how the code works, then add it to the dh.py module.

Putting it all together

We can now write a program that makes use of these functions (and ones that we've written previously) to create a tag cloud for Dollard's biography. First add the following to dh.py:

Now copy the following code to Komodo Edit, save it as html-to-tag-cloud.py and execute it.

If you substitute dh.scaledFontShadeSpan or dh.scaledFontHeatmapSpan for dh.scaledFontSizeSpan you can create any of three different tag cloud visualizations like the ones show below. You can also modify the code to change the range of font sizes, the typeface, the background color of the div, the width of the border, and anything else you can think of.

Three Tag Cloud Visualizations

Combining the tag cloud with KWIC

We're going to make one final refinement to this program, combining it with our ability to generate keyword in context (KWIC) displays. We want to create a web page with a tag cloud at the top and a number of KWIC displays in alphabetical order. When you click on a particular term, you're taken down the page to the KWIC display for that term. There is a link you can click to return to the tag cloud at the top. The code for this program is listed below. Much of it will be familiar, but it makes use of a few new techniques. Copy it to Komodo Edit, save as html-to-tag-cloud-kwic.py and execute it.

The first section of the code creates a sorted dictionary of word-frequency pairs, as you've done before. In the second section, we need to add a list containing (n//2) padding characters (hash marks) to the beginning and end of our full list of words before creating a dictionary of n-grams. We use two new commands to do this. The first replaces the empty slice before the beginning of the list with the list of padding characters. The second extends the word list by adding the list of padding characters to the end of it. To experiment with this, open a Python shell and try the following:

Note that the extend function changes the list, but doesn't return a value (hence the 'None'). This is called changing a list in place.

In the third section we create a tag cloud. Each term in the cloud has a hyperlink that looks like this:

Putting a hash mark in front of a string creates a relative link, a link to a place within the current web page. (Don't confuse this with our use of the hash mark as a padding character above. The two are completely unrelated.) Each of these links has a corresponding named anchor farther down the page. The anchor looks like this:

We also automatically create a number of links which take you back to the top of the page. Each of them looks like this:

At this point it is probably a good idea to spend a few minutes studying the HTML file that this program created automatically. It is named html-to-tag-cloud-kwic.html.