Doing Digital History with Python II: creating custom Word Clouds

by Monika Barget

In the second edition of Doing digital history with Python, I would like to address word clouds as a visual method of finding patterns in texts (see critical reflection in Basic Text Mining: Word Clouds, their Limitations, and Moving Beyond Them). Word clouds display the frequency or importance of individual keywords in individual texts or entire corpora. There are many ready-made tools in multiple languages that help you create word clouds in different designs, such as the in-built word cloud generator in Voyant Tools or browser-based tools such as Wortwolken.com. However, not all of them may be suitable for your specific use case.

When it comes to changing the number of words shown, colour patterns or text orientation, most standard word clouds are fairly limited, so installing the WordCloud package for Python is highly recommended. Like any other non-standard Python package, WordCloud needs to be initiated via import WordCloud in your code. The actual plotting of the graph is performed with Python’s matplotlib package.

If the text you want to visualise only contains words that should be part of the actual wordcloud, the process is fairly simple. The code becomes complex, however, if your text file contains articles, pronouns, auxiliary verbs and punctuation which need to be excluded. In this case, you may want to use Python’s Natural Language Processing kit NLTK to pre-process the input file.

I wrote the following code to visualise words from a tri-lingual early modern text file, containing words in German, English and French:

# Generate word cloud from text in English, French and German

from wordcloud import WordCloud # this is the package for generating the actual word cloud
import matplotlib.pyplot as plt # this package is needed for plotting the graph
import string # this package may be used for pre-processing
import nltk # this is Python's NLP library and is only needed if the text needs pre-processing
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
nltk.download('punkt')

infile=("C:\\Users\\yourname\\yourpath\\yourfile.txt")  

# define stopwords and convert them to string
# in-built NLTK stopwords are monolingual (e.g. 'english')
# list used below is my own tri-lingual stopword list stored in the nltk-data folder

stopwords_multiling=stopwords.words('en_fr_de')
data4=set(stopwords_multiling)
print(type(data4)) # wordcloud only accepts stopwords if class is 'set'

# read and pre-process data
  
with open(infile, 'r', encoding='utf-8') as f: # open file
    data1=f.read() # read content of file as string
    data2=data1.translate(str.maketrans('', '', string.punctuation)).lower() # remove punctuation
    data3=" ".join(data2.split()) # remove additional whitespace from text

# define word cloud layout
  
wordcloud = WordCloud(width = 1000, height = 1000, 
                background_color ='white', 
                max_words=300,      
                stopwords=data4, 
                min_font_size = 12).generate(data3) 
  
# plot the WordCloud image   

plt.figure(figsize = (10, 10), facecolor = None) 
plt.imshow(wordcloud) 
plt.axis("off") 
plt.tight_layout(pad = 10) 
  
plt.show() 

Wordcloud-Python.py [github]

The multilingual stopword list I use is, unfortunately, not delivered with NLTK but a list I had to assemble and ingest myself. How this is done will be discussed in another post in this on-going Python series.

To call the in-built German stopword list in NLTK, simply change
stopwords_multiling=stopwords.words('en_fr_de')
to
stopwords_multiling=stopwords.words('german').

Processing multiple files can also be done quickly as seen in this modified script.

Be creative and plot your first word cloud in Python 3! Be sure to consult the official documentation of WordCloud for more features.

A detailed tutorial in English, explaining advanced layout options such as colouring, is available on the “Datacamp” blog. “Python Lernen” (in German) offers a basic step-by-step tutorial for simple word clouds that do not require imported stopword lists. This tutorial explains the installation of the WordCloud package via “pip”, but we recommend the “anaconda” environment and installation via “conda”.

In order to find out about the functionalities of Voyant Tools, you may want to go back to the presentation that my colleague Demival Vasques-Filho delivered at our Dariah-DE Workshop in September 2019: Tools for quantitative text analysis [github].

---

---

Featured Image: Coluber argus, Print, between 1700 and 1880, 26,96 cm x 21,21 cm, Special Collections of the University of Amsterdam, Wikimedia Commons.

---

---