Uncovering censorship in the 16th century with Transkribus and Python. Episode V: How did the censors actually change the text?

by Markus Müller

We have come a long way since episode I of this miniseries: After digitizing the texts, normalizing the orthographic variants, and resolving the abbreviations, we used an interactive web app to find and correct remaining transcription errors. Now that the texts are free of mistakes we can finally use them for comparisons. In this episode, we will compare an original text with an expurged reprint to find censorship. Since the censors sometimes manipulated only one or two characters in a word, thereby changing the meaning of the whole sentence, we will compare the texts word by word using the Python module difflib.

---

This miniseries describes a digital workflow that starts with raw images of 16th century Latin prints, extracts the text, normalizes it, corrects transcription errors, and finally compares two texts to uncover differences, i.e. censorship. These are the episodes:

1. OCR with 16th century prints.
2. Let Python speak to Transkribus.
3. Normalizing 16th century raw text.
4. Detecting OCR transcription errors.
   
5. How did the censors actually change the text?
   
6. Finding text re-use.
   

---

In the 16th century, the Mainz cathedral preacher Johannes Wild (Ioannes Ferus) wrote a series of commentaries on various books of the Bible. The Archbishop of Mainz considered them to be perfectly Catholic and encouraged their printing. In other Catholic territories, however, especially in Spain and Italy, Wild’s books were considered “heretically contaminated,” whereas the author himself considered Catholic. Therefore, the Spanish Inquisition as well as the Roman Congregation of the Index decided to “expurgate” Wild’s books, meaning that they deleted or modified passages that were considered “heretical” and then reprinted the book. Interestingly, the expurgated versions differ considerably.

Take, for example, a short passage from the commentary on Matthew, chapter 16, verse 13, where Wild promotes unity in the faith by quoting the apostle Paul:

If Paulus would be here and see all the sects in every single city, what would he say other than what he said to the Corinthians: “I thank my God always for you because of the grace of God that was given you. I appeal to you, brothers, by the name of the Lord Jesus, that all of you speak the same thing. For it has been reported to me that there are contentions among you. One of you says, ‘I follow Paul’, another, ‘I follow Apollo’ [1 Cor 1:4,10–12]. For while there is this kind of strife among you, are you not of the flesh? Who then is Apollo, who is Paul? Ministers by whom you believed [1 Cor 3:4f.].”

Manually comparing the original Latin passage with the Spanish and Roman editions uncovers considerable modifications in the Roman text. In the Spanish reprint, however, this passage remained unchanged. The Roman censors replaced the names “Paul” and “Apollo” with the names of the leading Protestant theologians Luther and Calvin, so that the apostle Paul himself seemed to condemn them as “servants of the Devil”.

Translated to English, the expurgation of Wild’s text would look like this:

If Paulus would be here and see all the sects in every single German cities, what would he say other than what he said to the Corinthians: “I thank my God always for you because of the grace of God that was given you. I appeal to you, brothers, by the name of the Lord Jesus, that all of you speak the same thing. For it has been reported to me that there are contentions among you. One of you says, ‘I follow Paul Luther’, another, ‘I follow Apollo Calvin’ [1 Cor 1:4,10–12]. For while there is this kind of strife among you, are you not of the flesh? Who then is Apollo Luther, who is Paul Calvin? Ministers of Satan by whom you believed were seduced [1 Cor 3:4f.].”

Automate text comparisons with Python’s standard library

Comparing two texts manually is a tedious but quite simple procedure: First, we need to find the a starting point. Using our fingers as pointers, we then move forward one word in each text and check whether the two words differ or not. If they are different, one of the finger moves on until we find the next corresponding word. The words skipped in this process could be insertions, deletions, or substitutions by the censor.

Although this procedure is quite simple for humans, it would be quite complex to translate it into efficient Python code, because every little detail of the algorithm would affect the performance. Fortunately, the problem of text comparison is well known in the world of programmers: Developers often need to compare different versions of the same code to track of changes, especially if they work in a team. Of course, they don’t compare the different versions manually, but use so-called diff algorithms specialized in efficiently finding differences between two strings.

Python’s standard library comes with the difflib module, which uses one of these algorithms (called Gestalt Pattern Matching, developed by Ratcliff and Obershelp in the 1980s). The difflib module is easy to use and produces a human-readable output:

import difflib

list_1 = ["Finding", "diferences", "is", "complicated", "."]
list_2 = ["Finding", "differences", "is", "easy", "."]

d = difflib.Differ()
delta = d.compare(list_1, list_2)  # returns a generator
delta = [*delta]  # converts the generator into a list

for line in delta:
    print(line)

Listing 1: simple_diff.py compares two lists of strings (GitHub).

The output looks like this:

  Finding
- diferences
+ differences
?   +
  is
- complicated
+ easy
  . 

Each line of the diff output starts with a two-letter code: ~ ~ precedes lines common to both lists. Lines that occur only in list_1 start with -, and those unique to list_2 start with +. A question mark indicates intra-line differences. Although this format is human-readable, it is not very intuitive for our purpose. It would be better to have a markup similar to the “track changes” feature of modern word processors, where deletions appear strikethrough and additions are underlined.

To achieve this kind of markup, we evaluate the first character of each line of diff output and add a corresponding markup to the rest of the line before printing it to the screen, e.g. lines starting with - get a “strikethrough” markup, and lines starting with + are underlined. Lines starting with ? can be ignored. For a quick and easy output on the console, we can implement the markup using combining unicode characters: print(' \u0336a\u0336b\u0336c') results in “abc”, using \u0332 instead yields “abc”.

import difflib

list_1 = ["Finding", "diferences", "is", "complicated", "yet", "it", "could", "be", "so", "easy", "."]
list_2 = ["Finding", "differences", "is", "easy", "."]

d = difflib.Differ()
delta = d.compare(list_1, list_2)  # returns a generator
delta = [*delta]  # converts the generator into a list

# Functions providing markup using combining unicode characters
def strikethrough(text):
    return ''.join([u'\u0336{}'.format(c) for c in text])

def underline(text):
    return ''.join([u'\u0332{}'.format(c) for c in text])

# Process the differences found with difflib:
output = []
for line in delta:
    # Extract the code and the data from each line:
    code = line[:1]
    data = line[2:]

    # Add markup:
    if code == " ":
	output.append(data)
    elif code == "-":
	output.append(strikethrough(data))
    elif code == "+":
	output.append(underline(data))

print(" ".join(output))

Listing 2: simple_diff_2.py renders the diff output in a more readable way (GitHub).

Finding diferences differences is complicated yet it could be so easy.

Getting Python and Transkribus back into conversation

Instead of feeding this script with hard-coded input (list_1, list_2), we can connect it directly to Transkribus via the Transkribus client, the normalization pipeline and the simple command line interface used in episode III. The only thing missing is a way to select a range of pages from two different documents on the Transkribus server. Both stacks of pages are then run through the normalization pipeline before the normalized text gets extracted into two lists of words. Finally, these lists are fed into the script above.

That means we need to add some functionality to the simple command line interface from episode III. You find the code in the GitHub repository for this article. Take a look at the select_range_pipeline() (GitHub) and compare_pipeline() (GitHub) functions to get an idea of how this can be done. In the compare_word_lists() (GitHub) function, you will find the script from above, except that it gets its data directly from the normalization pipeline and Transkribus. Three additional lines of code render page breaks and line numbers of the original text. This meta data is crucial if we want to cite our findings in an academic publication.

Running normalize_and_compare.py (GitHub), the script asks whether you want to just print a normalized page or compare pages. If you select the “compare pages” option, the script will let you select two ranges of pages from two of your documents on the Transkribus server. The output will look something like this:

Using CTS to track passages in a text

In the first block, the script repeats the selected page ranges in a CTS-like format. CTS stands for Canonical Text Service and is a very simple protocol developed at Leipzig University for addressing text passages. A Canonical Text Service URN conists of four parts: namespace:work:passage@subreference. I have called the namespace tr for “Transkribus”. The work is referenced by the collection and document IDs separated by a dot, and I use the page number as passage. TextRegion and line numbers could be added as subreference, e.g. @r2l10 for “region 2, line 10”. In the example above, tr:37299.291391:430 means page 430 of document number 291391 in collection number 37299 on the Transkribus server.

The second block of the output shows the differences found between the two page ranges. Every line starts with meta data about the text region (r) and the line number (l) in the original text, so that you can easily find the passage in the original book. Since the page breaks usually differ quite a lot, the first line of the output does not make much sense and should be ignored. The interesting part begins in line 5 (r0l5) where the output shows the differences we already found by hand in the example above.

Adding a “Compare” feature to the web app

As you can see, the script has some limitations: it is black and white, ignores punctuation, and is case sensitive. The output is fine for testing purposes, but the command line interface does not make much sense if you want to use the code in your daily research.

Therefore, I integrated the logic into episode IV’s web interface: Here you first have to “export” the pages you want to compare. The graphical user interface in the “Export” tab, and the JavaScript code behind it, make it easy to do that. (See the corresponding code in the app.)

The exported page ranges are cached as csv files and can be reviewed in the “Download” tab.

The “Compare” tab lists all the exported csv files. First select the original text (it will be tagged as “original”) and then the expurged reprint. You can either download the raw diff output or open it in the “Compare & Analyze” editor.

The visualization in this editor makes it easy to review the differences found by diff and to decide whether they are actually censorship. (See the corresponding html and js files in the app.)

If so, you can highlight the passage with the cursor and press R to annotate it and register it as censored.

All the passages registered as censored are listed in the “Censorship” tab. The CTS address in the first column (it follows a slightly different pattern than in the command line interface) helps you to find the text and context of the registered passage.

These four tabs and the corresponding workflow may seem complicated at first glance. However, the advantage of this approach is its flexibility. You can download the exported data for testing and development purposes. You can easly implement additional export formats. The same is true for the “Compare“ functionality. The “Censorship” tab helps to compare different censorships and identify patterns in the censorship practises. Again, the Python and JavaScript code behind this user interface is far from being perfect, but it works and it speeds up the search for censorship considerably.

In the next episode we will try to use a similar approach to tackle an even more time consuming task: Searching for verbatim quotes with no reference to the source.

---

Featured Image: Let Python Talk to Transkribus, 2021, by Markus Müller, CC BY-SA 4.0,
build upon Wolpertinger by Rainer Zenz, CC BY-SA 3.0; Python natalensis, in: Andrew Smith: Illustrations of the zoology of South Africa, Reptilia. Smith, Elder, and Co., London 1840, PD; Markus Müller: Sammelband mit Schriften Johann Wilds, Wissenschaftliche Stadtbibliothek Mainz, Signatur 559 q 1; The ‘haunted table’ in the bar of the Black Horse Inn public house, on Nuthurst Street in Nuthurst, West Sussex, England, by Acabashi, CC BY-SA 4.0.

---

---