Uncovering censorship in the 16th century with Transkribus and Python. Episode IV: Detecting OCR transcription errors

by Markus Müller

In the last episode, we built a pipeline to convert a diplomatic transcription into normalized Latin text. The code works fine as long as the diplomatic transcription is correct. But what happens if the transcription contains errors or, even worse, if the printer in the 16th century misspelled a word? – Right now, nothing would happen at the moment because our pipeline cannot detect these errors. This is a problem because as soon as we start comparing two editions of the same text to check for censorship (and that’s where we are going!), the slightest difference between the two texts may be interpreted as censorship. Can we solve this problem? – Yes, we can!


This miniseries describes a digital workflow that starts with raw images of 16th century Latin prints, extracts the text, normalizes it, corrects transcription errors, and finally compares two texts to uncover differences, i.e. censorship. These are the episodes:

  1. OCR with 16th century prints.
  2. Let Python speak to Transkribus.
  3. Normalizing 16th century raw text.
  4. Detecting OCR transcription errors.
  5. How did the censors actually change the text? (coming soon)
  6. Finding text re-use. (coming soon)

Two printer’s errors in 16th century Latin texts.

The ‘analogue’ solution is to manually decide whether a difference between the two texts is an actual censorship or just a misprint or our own transcription error. If it is a transcription error, we should correct it in Transkribus, since text recognition models trained with an erroneous transcription will reproduce these errors (cf. episode I). If, on the other hand, it is a printing error, the transcription does not need to be modified, although the normalization pipeline should definitely correct the misprint from 400 years ago in order to prevent false alarms when searching for censorship.

It would be complicated or even impossible to automate this decision. But we could at least try to identify the problematic words in the text automatically and make the manual review as easy and fast as possible.

We need a Latin spell checker!

A simple way to find problematic words is to take the normalized text and check each word with a Latin dictionary. If the word is not in the dictionary, it could be a misprint or a transcription error. There is yet another possibilty: the suspicious word could be correct, but the dictionary does not know it – think of strange and rare names like “Thesbite“ or “Thisbite” (cf. 1 Kings 17:1), alternative spellings (like “Abraam” instead of “Abraham”), or neo-Latin words that did not exist in Antiquity and therefore are not included in today’s standard Latin dictionaries (like “philautia” = self-love, selfishness, a Greek word (φιλαυτία) introduced into Latin by humanists in the 16th century and used by theologians like Erasmus of Rotterdam).

The dictionary.py module from the last episode could do the job. Though it would be nice to have a fully interactive user interface that highlights the suspicious words, or even better, to get a list of possible corrections. The cyhunspell module we used in dictionary.py can generate such a list, but the quality is not very good and the algorithm is surpisingly slow: generating lists of possible corrections for fifty erroneous words with cyhunspell takes 3.8 seconds (on my seven-year-old laptop).

Building a super-fast Latin dictionary

Wolf Garbe’s SymSpell algorithm takes only 7 milliseconds to generate the same lists of possible corrections for fifty words. Fortunately, there is a nice Python implementation called SymSpellPy. The only problem is that SymSpell requires a frequency list for the language in question, i.e., statistics on how often a word occurs in a given corpus. The bigger the corpus, the better.

Compared to the many high-quality frequency lists for modern languages (see the links on Wolf Garbe’s GitHub page), which are based on corpora with billions of words, the Latin lists are quite small and focussed on Antiquity. The late-medieval Latin in our texts contains loads of biblical names and theological/ecclesiastical vocabulary, so that the available Latin word lists are far too small. We have to build our own corpus and generate our own frequency list.

Luckily, in recent decades many Latin texts have been digitized and are freely available. The Corpus corporum, for example, reached 160 million words in March 2019, of which 95 million words are theological texts from the “Patrologia Latina” (one of the largest editorial projects of the 19th century). This is exactly the type of text we need. Instead of downloading hundereds of individual files from “Corpus corporum”, you can get most of them as txt files by downloading the Latin Text Library which is part of Python’s CLTK (Classical Language Tool Kit). This corpus contains 2,141 Latin text files from Antiquity up to the modern era. Unfortunately, the files are quite messy and sometimes suffer from very strange spelling and/or numerous transcription errors. Therefore I tried to clean the texts with a Python script before building the word list. The script normalizes non-standard characters (like “æ”, “œ” etc.) and typical spelling variants (“j” → “i”, “v” → “u”), then filters out all punctuation, numbers, and Greek characters, and finally it checks every word against our Latin Hunspell dictionary from episode III. The result is not bad, but the Hunspell dictionary is very picky when it comes to proper names (case-sensitive!) and medieval spelling variants. You can preserve at least some of them by applying some tricks on words that are not recognized at first glance (e.g. by replacing “ci“ → “ti”, “e“ → “ae”, capitalizing the word, etc.). See my Jupyter notebook build_SymSpell_dictionary.ipynb on GitHub for details.

Using this method, we can extract 12,829,421 words from the 2,141 text files in the CLTK corpus. After counting them, we end up with 408,396 unique words (the most frequent ones are “et” (423,445), “in” (261,237), and “est” (159,846). Hunspell considers 125,141 of them are incorrect, but after applying the tricks described above, we can “save” around 30,000 words from being deleted. In the end, our frequency list contains 313,881 unique words, which is a fairly good starting point for SymSpell.

Finally, we load the frequency list into a SymSpell object (which takes some seconds), et voilà: our super-fast Latin dictionary is ready. Caveat: rarely used inflected forms as well as some spelling variants and proper names are still missing. Therefore, the spell checker we are going to build will use Hunspell as a fallback option if SymSpell does not know a word. If Hunspell also reports an error, we should check the word manually and either correct it or add it to the SymSpell dictionary.

To implement this fallback procedure we add a function to cleaner.py from episode III (cf. GitHub for details). Now that the dictionary works on a technical level, let’s put some thought into the user interface.

Building a spell checker GUI with HTML, CSS and JavaScript

Theoretically, it would be possible to highlight erroneous words in the rudimentary command line interface we built in episode III, but the command line approach does not allow for a fully interactive user experience. Therefore, a graphical user interface (GUI) is the way to go, either as a desktop application (cf. this review of 10 GUI frameworks for Python) or as a web application, meaning the GUI would be a web page written in HTML. As you can see below, with a combination of HTML and Cascaded Style Sheets (CSS) we can display the normalized text below the diplomatic transcription, highlight misspelled words and provide alternative spellings in a drop-down menu:

<!DOCTYPE html>
<head>
  <meta charset="utf-8">
  <link rel="stylesheet" href="editor.css">
</head>
<body>
  <div>
    <h3>Latin Spell Checker</h3>
    <p>Document: Wild1559-Mt,Paris</p>
    <p>Page: 273</p>
    <p>Status: GT</p>
  </div>
  <div>
    <div>
      <div>tamen laudem appetimus, vilipendi ægrè ferimus, & cæt.</div>
      <div>tamen laudem appetimus , uilipendi aegre ferimus , etc</div>
      <div>deinde proni ſumus ad iudicandum. Hæc tria Chriſtuus po</div>
      <div>deinde proni sumus ad iudicandum . Haec tria 
        <div class="word wrong">Christuus
          <div class="dropdown">
	    <div>Christus</div>
	    <div>Christum</div>
	  </div>
        </div>
        potissimum 
      </div>
      <div>tiſsimùm in ſermone, quem in monte habuit, impugnauit:</div>
      <div>in sermone , quem in monte habuit , impugnauit :</div>
    </div>
  </div>
</body>
</html>

Listing 1: Simplified snippet of an HTML page called editor.html (on GitHub) visualizing the raw transcription together with the normalized Latin text. Words missing in the dictionary are underlined in red by adding the word wrong class. A drop-down menu shows possible corrections.

.word {
  position: relative;
  display: inline-block;
}

.word.wrong {
  position: relative;
  display: inline-block;
  text-decoration: underline; 
  text-decoration-style: wavy; 
  text-decoration-color: red;
}

.dropdown {
  display: none;   /* make the dropdown menu invisible */
  position: absolute;
  background-color: #f1f1f1;
  box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.2);
}

.word:hover .dropdown {
  display: block;    /* make the dropdown menu visible */
  color: black;
  text-shadow: none;
}

Listing 2: Cascaded Stylesheet (editor.css, on GitHub) describing the format of the HTML elements (like the red wavy line below mispelled words) and implementing the drop-down functionality.

This is how editor.html looks in the browser.

In this prototype, the text content is hard-coded. For a truly interactive user experience, we need a way to dynamically change the content, like on a news website where a computer program on the server responds with the latest news as soon as you request the web page by typing the corresponding URL in your browser.

That means, we have to build our own “spell checker” server and make it send the latest transcription to the browser along with the normalized and spell-checked Latin text. We could even give the user to send commands back to the server by including some input elements of HTML5 (like buttons, text input fields etc.) in the web page.

To implement this truly interactive scenario, we need to step into web development and learn HTML, CSS and some JavaScript. But don’t be afraid, there are very good tutorials to learn all these things. The good news is: programming a server can be done with Python using a so-called “web framework” like “Flask” (see below). Let’s focus on programming the server.


TIPS FOR LEARNING HTML, CSS, JavaScript AND Flask

A good starting point for web development is David J. Malans “CS 50” course at Harvard, which includes a very nice session about HTML, CSS and JavaScript, as well as an introductory lecture on Flask. Since Flask hides a lot of functionality behind easy-to-use objects and functions, you may also benefit from Malan’s general introduction into databases and the Structured Query Language (SQL), which helps to understand what’s going on behind the scenes. An alternative would be to work through online courses, either for free (like w3schools or the tutorials of Mozilla’s Developer Network) or paid (I very much liked the (partly free) JavaScript course of The Net Ninja). To dive deeper, you should check out the documentation of Flask and bookmark technical references of HTML, CSS and JavaScript, like the MDN Web Docs. Be warned: You will need to invest quite a bit of time.


A spell checking web app with Python and Flask

The task of our “spell checker” server will be to send data to the client (= the browser) and handle the requests it receives from the client (e.g. when the user fills in a text input field and presses a button). A web framework like Flask provides ready-to-use modules to do all these things.1

Let’s have a look at a simplified example that demonstrates the basic idea of Flask and shows how to integrate the building blocks from episode II and III into a web application. By convention, Flask expects specific declarations and functions in your code and a certain folder structure. In your project folder, app.py is the main Python script that creates the web app and provides the core logic for managing requests and responses. HTML files must be located in a templates/ subfolder. CSS and JavaScript belong in the static/ subfolder. This is how the complete folder structure looks like:

app.py
static/
  editor.css
  editor.js
templates/
  editor.html

The following code sets up a rudimentary web app that serves the HTML prototype we saw above when the user visits the /editor route in the browser:

from flask import Flask, render_template

app = Flask(__name__)      # <-- create app object

@app.route("/editor")      # <-- define a route
def editor():              # <-- function which sends
                           #     data to the client
    return render_template("editor.html")   # <┐
                            #     template sent to client

Listing 3: A very simple web app with Flask (on GitHub).

The cool thing about Flask is that you can run a development server locally without having to install any additional software. After installing Flask (pip install Flask, I strongly recommend to use a virtual environment, either conda or venv) and downloading and unzipping the code from GitHub just open a terminal or command line, go to the project folder and say flask run. The development server will start and as soon as you open your browser and visit http://127.0.0.1:5000/editor, you should see the spell checker prototype from above.

Unfortunately, the web site still shows hard-coded content. But thanks to Flask we can easily change that. Compare the rudimentary Flask app from above with this one:

from flask import Flask, render_template

app = Flask(__name__)

@app.route("/editor")
def editor():

    page = {'lines': [
    # raw transcription from Transkribus:
	{'raw_data': 'deinde proni ſumus ad iudicandum. Hæc tria Chriſtuus po',
	'words': [
	    {"data-type": "word", # <word | punctuation | unreadable>
	    "data": "deinde",     # <the actual word as a string>
	    "spelling": "ok",     # <ok | probably ok | wrong>
	    },
	    {"data-type": "word", 
	    "data": "proni", 
	    "spelling": "ok", 
	    },
	    # etc. ... etc. ...
	    {"data-type": "punctuation", 
	    "data": ".", 
	    "spelling": "ok", 
	    },
	    # etc. ... etc. ...
	    {"data-type": "word", 
	    "data": "Christuus", 
	    "spelling": "wrong", 
	    "suggestions": [{"term": "Christus", # <suggested word>
	        "count": 1432}]      # <frequency in the corpus>
            },
	    ]}
	]}
                                          ┌> send the page variable
    return render_template("editor.html", page=page)

Listing 4: Sending data and an HTML template with Flask (on GitHub).

This time, the content is no longer hard-coded in the HTML file itself. Instead, there is a page variable that contains all the data and is sent to the browser along with editor.html in the return render_template() statement. The page variable is a Python dictionary containing the raw_data taken from Transkribus and a list of words. Each word is a Python dictionary that stores the word itself (in the data attribute), its type (data-type), and information about whether the word is misspelled or correct.

<div class="page">
   {% for line in page.lines %}
   <div class="rawText">{{ line.raw_data }}</div>
   <!-- etc. … … … etc. -->
   {% endfor %}
</div>

Listing 5: A snippet of the corresponding editor.html (on GitHub). When rendering the page, the Jinja templating engine will convert the placeholders with curly braces into actual HTML markup.

If you take a look at the new editor.html, you will notice a strange syntax with curly braces. Similar to a form on paper, this syntax marks gaps which Flask should fill in when it renders the template. For example, {% for line in page.lines %} means: “Iterate over the lines of the page”, and {{ line.raw_data }} means: “Fill in the raw transcription of this line here”, and so on. This syntax and the templating functionality are part of the Jinja templating engine, which is included into Flask. The resulting HTML code is identical to our hard-coded prototype, but this time, the server generated it dynamically based on the data provided by the page variable. If we change the page variable, the text content of the resulting HTML would change as well.

This is where the building blocks from episodes II and III come into play: Imagine we used Python to change the page variable automatically! Transkribus_Web, for example, could download the raw transcription from Transkribus and store it in the page variable. normalize.py would do its job and generate the list of normalized words. Last but not least, dictionary.py would check the spelling of every word and generate the list of suggestions if necessary. All the data would be stored in the page variable. The result would be a page variable just like the one we saw in the example above. This time, however, the page variable would be generated dynamically. That means that we can normalize and render the data from Transkribus flawlessly and display the result as a web page!

Let’s go even one step further and make the content of the web page, especially the raw transcription, editable so that the user could react when a word is highlighted as “wrong” and possibly correct the transcription error. Then, by pressing a button on the web page, the user could ask the server to re-run the spell check or save the modified transcription to the Transkribus server. All of these features can be implemented with Flask and HTML. If you want even more interactivity on the web page, you should outsource parts of the logic from the Flask server to the browser by adding JavaScript to the HTML template so that the HTML document can be manipulated from within the browser.

I can’t explain all the details here because the blog article is already too long. The code of the app I built (see my GitHub repository) has grown over the last three years and is still work in progress. The parts of the code I wrote at the beginning are often quite clumsy, and the details of the architecture of the software are sometimes a crude mixture of different approaches. I am a historian, not a professional programmer. The app can only be used by small teams (two or three people at most) because it lacks a proper user rights management and the data management is far too inefficient for larger numbers of users. Writing this web app, I learned a lot, but I also recognized how complex programming is and how much expertise and experience is needed to write really good and efficient code.


HINTS FOR BEGINNERS WHO ALREADY HAVE SOME EXPERIENCE WITH HTML, CSS, JavaScript AND Flask

  • Making your HTML content look right with CSS takes a lot of time. There are CSS frameworks like Bootstrap or W3.CSS that provide ready-to-use CSS designs that look professional and reduce the lines of CSS lines you have to write.
  • An efficient graphical user interface typically uses icons. Using a CSS icon library is an easy way to integrate professionally designed icon sets into your web application.
  • Writing pure JavaScript code (aka “Vanilla JavaScript”) can sometimes be tedious. Using a JavaScript framework like jQuery or Vue.js can make life easier. The downside is that you have to learn yet another set of commands.
  • To transfer data from the Flask server to your JavaScript, you could implement your own REST-API in Flask and use JavaScript to retrieve the data. Alternatively, render_template could send your data along with the HTML template. In the HTML template, you should then include a line of JavaScript in the <head> tag with a syntax like var my_data = {{ my_data|safe }}, meaning that Jinja should use the safe filter to render the variable as JSON. This prevents crashes caused by escaped characters.

On the way to a fully-featured Transkribus spell checker web app

To conclude, I briefly explain the structure of the web app. The user interface consists of three main parts, the document manager, the editor and the dictionary. In the document manager you can browser your collections, documents and pages on the Transkribus server. The other tabs will be explained in the next episodes.

By clicking on a page, you can open it in the editor, which is an extended version of the prototype we saw above. It offers much more interactivity. As soon as you select a line of text for editing, the photo of the original page jumps to the corresponding passage and highlights the line in red, so you can easily decide whether you deal with a transcription error or a misprint. If a highlighted word turns out to be correct, you can add it to the dictionary or to a list of printer’s errors with just one click in the dropdown menu (⨁ dictionary).

The buttons on the left and right margin of the screen give you access to a virtual keyboard for special characters and a comments side panel (you can attach comments to words).

Clicking the books icon in the editor opens the user dictionary, where you can review entries and add or delete words. The “Reload abbreviations table from Google Docs” button at the top of the page refers to the Cleaner’s replacement table: the table is stored on Google Drive and can be edited by all members of the team. Clicking the button synchronizes the web app with the Google spread sheet.

Things like the user dictionary would not work without a custom database in the background. Flask provides efficient ways (Flask SQLAlchemy) to set up and communicate with a database, or in other words, to “manipulate the database model”. The overall architecture of the app follows the so-called MVC pattern, where “M” stands for the model, i.e. the database(s), “V” is the view, i.e. the HTML templates like editor.html, and “C” stands for the controller, i.e. the normalization pipeline and all other parts of the Python code explained above.

The schema shows how the Flask server talks to the Transkribus REST-API to download the raw transcriptions. It processes them and stores some data in the database and/or in the in-memory cache, from where the views get their data before they are sent to the client via the render_template() command. In the browser, the JavaScript code generates an interactive user interface and allows the user to send commands back to the controller, which then saves the modified transcription on the Transkribus server, loads new data or modifies the user dictionary in the database etc. On GitHub you find a schema of the database model.

Conclusion: Programming a well-designed user interface is quite complicated. The solution presented here is only one of many possible approaches. It is still work in progress, the code is far from beeing professional, some bugs are not fixed yet, others still need to be found. Nevertheless, together with a student assistant I used the app for more than a year, and the interactive user interface has made the whole procedure of transcribing, training a Transkribus model and correcting automated transcriptions much faster and easier. The road is rocky, but worth it.


Footnotes

1 An even more powerful web framework for Python would be Django.

Cite this article as: Markus Müller, "Uncovering censorship in the 16th century with Transkribus and Python. Episode IV: Detecting OCR transcription errors," in Digital Humanities Lab, 24/09/2021, https://dhlab.hypotheses.org/2191.

Featured Image: Let Python Talk to Transkribus, 2021, by Markus Müller, CC BY-SA 4.0,
build upon Wolpertinger by Rainer Zenz, CC BY-SA 3.0; Python natalensis, in: Andrew Smith: Illustrations of the zoology of South Africa, Reptilia. Smith, Elder, and Co., London 1840, PD; Markus Müller: Sammelband mit Schriften Johann Wilds, Wissenschaftliche Stadtbibliothek Mainz, Signatur 559 q 1; The ‘haunted table’ in the bar of the Black Horse Inn public house, on Nuthurst Street in Nuthurst, West Sussex, England, by Acabashi, CC BY-SA 4.0.


Autor: Markus Müller

Studied Catholic theology and education science in Tübingen (2000–2006) and Salamanca (2002/2003). 2012: PhD in Catholic theology (ecclesiastical history) in Tübingen. Research assistant in Tübingen, assistant lecturer in Frankfurt and Mainz. Since 2018, postdoctoral researcher at the Institute of European History in Mainz. – Research interests: history of religious education, censorship in the 16th century, Digital Humanities.

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.