Uncovering censorship in the 16th century with Transkribus and Python. Episode II: Let Python speak to Transkribus

by Markus Müller

In the first episode of this miniseries, I explained how to use OCR with Transkribus to create a diplomatic transcription from images of 16th century Latin prints. However, the resulting text is full of abbreviations and may contain transcription errors. In both cases, Python scripts could help to normalize the text and to detect possible errors.

An indispensable prerequisite for pimping the Transkribus workflow with Python is flawless access to the data stored on the Transkribus server. It would not make much sense to manually download the data, then run some Python scripts on the command line, and finally upload the data again manually, especially when we talk about hundreds of pages to be processed. This episode will show how to download and upload transcriptions from and to the Transkribus server using Python. (Basic to intermediate knowledge of Python is required.)


This miniseries describes a digital workflow that starts with raw images of 16th century Latin prints, extracts the text, normalizes it, corrects transcription errors, and finally compares two texts to uncover differences, i.e. censorship. These are the episodes:

  1. OCR with 16th century prints.
  2. Let Python speak to Transkribus.
  3. Normalizing 16th century raw text.
  4. Detecting OCR transcription errors. (coming soon)
  5. How did the censors actually change the text? (coming soon)
  6. Finding text re-use. (coming soon)

Fortunately, Transkribus offers an Application Programming Interface (API) that allows users to download and upload data programmatically via the HTTP protocoll. The Transkribus API is a REST-API, i.e. it follows a standard convention called “REpresentational State Transfer” to organize the communication between the Transkribus server and the client (in this case the Python script). REST-APIs typically provide CRUD functionality, i.e. they expose so called “endpoints” for Creating, Reading, Updating and Deleting data on the server. An endpoint is basically a URL, e.g. https://api.zippopotam.us/DE/55116. As soon as you request this URL, the server responds with a dataset containing the requested information, in this case geodata for the German (“DE”) postal code “55116” in the JSON format (try it out by opening the URL in your browser). To learn more about REST-APIs in general, this page offers a very good introduction.

For all the steps in the communication between server and client there are specialized packages in Python. The “requests” package sends HTTP requests and receives the response from the server. The response usually contains data in a JSON or XML format. The “json” package transforms JSON into a Python dictionary and vice versa. The “lxml” package is a very powerful toolbox for reading, manipulating and writing XML. With these ingredients we can build a script that communicates with a REST-API.


JSON and XML

A simple way to store structured data on a computer (without using special software like a database) is to enrich normal text with special characters that a computer can “understand”. For example, the information “My name is Markus and I live in Mainz” could be encoded as {'name': 'Markus', 'address': 'Mainz'} or <myself><name>Markus</name><address>Mainz</address></myself>. Although the information is structured and thus machine-readable, it is still just a string, i.e. a series of characters that can easily be stored on disk in a plain text file or sent to another computer on the internet using the HTTP protocol. The first method to encode the information is called “JavaScript Object Notation” (JSON) because it is the standard way of describing “objects” in the JavaScript programming language. The second example uses the “markup” technique of the “eXtensible Markup Language” (XML): The words enclosed in angle brackets are called “tags”. Every piece of information is framed by an opening (<name>) and a closing tag (</name>). Tags can be nested (<myself><name>XYZ</name></myself>) to represent a tree-like structure. XML is widely used to enrich pure text with additional information, e.g. layout and formatting of the text (as in HTML pages) or semantic metadata (as in digital editions using the TEI XML standard). Cf. Monika’s XML & JSON tutorial for another use case.


To start with, a simple API will do the job: Let’s send a GET request to the Open Notify API (which has only two endpoints) to see who is on the International Space Station at the moment:

""" List the astronauts that are currently on the ISS
    using the Open Notify API: http://api.open-notify.org/ """

import requests
import json

# Build the URL using the "astros.json" endpoint:
api_base_url = "http://api.open-notify.org/"
endpoint = "astros.json"
url = api_base_url + endpoint

# Make a GET request and store the response:
response = requests.get(url)

# Evaluate the response:
if response:
    try:
        json_response = response.json()
        print("The raw JSON response sent by the server:")
        print(json.dumps(json_response))   
        print("\nPeople on the ISS:")
        for astronaut in json_response['people']:
            if astronaut['craft'] == "ISS":
                print("- " + astronaut['name'])
    except:
        print(f"ERROR: Something went wrong. {response.content}")
else:
    print(f"ERROR: Something went wrong. HTTP status code: {response.status_code}")

Listing 1: astronauts.py on Github

The very same approach can be used to make Python speak to Transkribus, although there are two major differences between the Transkribus API and the Open Notify API: First, the Transkribus API has more endpoints. A description of the most important endpoints can be found in the documentation along with a comprehensive list of all available endpoints. Secondly, in contrast to the Open Notify API, we have to log in on the Transkribus server before we can access its API. There is, of course, an endpoint for that: …/auth/login. Other than in listing 1, the login endpoint expects a special type of request called POST request (GET and POST being the two most important HTTP request types). A POST request can send an additional „payload“ to the server. In this case, the payload is a JSON object containing your user name and password: {'user': 'your_user_name', 'pw': 'your_password'}. In Python, the corresponding request command would be response = requests.post(url, data={'user': 'your_user_name', 'pw': 'your_password'}).

Having sent the request, the Transkribus server responds with XML data containing the user’s name, the user’s ID and a unique session ID. The “objectify” class of the “lxml” package allows us to deal with this XML data elegantly in a ‘pythonic’ way. The most important piece of data in this response is the so-called “session ID”, which we need to add to our further requests as a cookie. The session ID will allow the server to identify us so that we gain access to all the other endpoints of the API. Sending a request to the …/auth/logout endpoint invalidates the session ID, just like clicking the “log out” button in a web app. The following code snippet will log in, print the list of our collections stored on the Transkribus server, and log out again. (Make sure to replace “YOUR_USER_NAME” and “YOUR_PASSWORD” with the actual credentials of your Transkribus account before running the script. If you do not have a Transkribus account yet, you can sign up on https://readcoop.eu/transkribus/.

""" Template for using the Transkribus API. (Python 3.6+) """

import requests
import sys
from lxml import objectify

# Define building blocks for the endpoints:
api_base_url = "https://transkribus.eu/TrpServer/rest/"
login_endpoint = "auth/login"
collections_endpoint = "collections/list"
logout_endpoint = "auth/logout"

# Log in:
## Prepare the payload for the POST request:
credentials = {'user': 'YOUR_USER_NAME',
           'pw': 'YOUR_PASSWORD'}
## Create the POST request (the requests package converts the credentials into JSON automatically):
response = requests.post(api_base_url + login_endpoint, data=credentials)

## Evaluate the response:
if response:
    r = objectify.fromstring(response.content)
    print(f"TRANSKRIBUS: User {r.firstname} {r.lastname} ({r.userId}) logged in successfully.")
    session_id = str(r.sessionId)
else:
    sys.exit("TRANSKRIBUS: Login failed. Check your credentials!")

# Get the list of collections (using a GET request):
cookies = dict(JSESSIONID=session_id)
response = requests.get(api_base_url + collections_endpoint, cookies=cookies)
if response:
    collections = response.json()
    print("TRANSKRIBUS: Your collections:")
    for collection in collections:
        print(f"- {collection['colName']}, ({collection['nrOfDocuments']} document(s))")
else:
    sys.exit("TRANSKRIBUS: Could not retrieve collections.")

# Log out (using a POST request):
response = requests.post(api_base_url + logout_endpoint, cookies=cookies)
if response:
    print("TRANSKRIBUS: Logged out successfully!")
else:
    sys.exit("TRANSKRIBUS: Logout failed.")

Listing 2: transkribus_api_template.py on Github


How to learn Python?

Since Python is a very popular language nowadays, you can find many very good online courses, tutorials and books on the internet. Just pick one that you like and that fits your needs. My own journey started with CS50, a series of freely available Computer Science lectures by David J. Malan at Harvard University. CS50 offers a general introduction into programming as well as more advanced courses like Web Programming with Python and JavaScript. If you prefer a book, “Automate the Boring Stuff” by Al Sweigart could be a starting point. A specialized text editor, like Microsoft’s Visual Studio Code, will help you to avoid errors in your code (the infamous bugs). As soon as you start your first ‚real‘ project, I would recommend using a so called “environment manager” like Anaconda which includes the very useful Jupyter Notebook application mentioned below. The following paragraphs require intermediate Python skills.


Following the code pattern from above, you can access all the endpoints available in Transkribus. You may have noticed that the code is very repetitive: For example, in every step we use the same response = requests.get command. Therefore, it is more convenient to put the repetitive code into reusable functions and wrap these functions in a class. Inspired by the TranskribusPyClient (a huge class that covers all endpoints of the Transkribus API), I wrote my own Transkribus_Web class that covers selected endpoints and includes some additional functionality that fits the needs of my project. You can check out my code in this Jupyter notebook. Using a class like this, the conversation between Python and Transkribus is a piece of cake. We can achieve the same result as above with only five lines of code:

from transkribus_web import Transkribus_Web

client = Transkribus_Web()
client.login("YOUR_USER_NAME", "YOUR_PASSWORD")
print(client.get_collections())
client.logout()

Listing 3: Cf. chapter A custom Transkribus REST-API Client class in the transkribus_client.ipynb notebook on Github.

Apart from getting a list of your collections, the Transkribus_Web class can also list your documents and the pages within a document. As you can see in the output of the code above (yes, it’s a JSON object!), every collection in Transkribus has its unique ID (“colId”). The same is true for documents and pages. Thus you need the colId, docId and pageNumber to address a specific page.

If you open the above mentioned Jupyter notebook, you can use the get_page_xml function of the Transkribus_Web class to download the content of a specific page. The page is delivered as XML by the Transkribus Server. The structure of the raw XML seems to be quite complicated at first glance. However, the objectify API of the lxml package makes life much easier, since you do not have to deal with the XML markup directly. Instead, the XML data is converted into a Python object, let’s call it “X”, with two attributes: X.Metadata and X.Page. X.Page is empty if there are no transcripts yet, otherwise it contains a series of nested attributes that represent the structure of a page in Transkribus (containing TextRegions, TextLines, BaseLines and the actual transcription). The following overview can help you to get started:

You can modify the objectify object like any other Python object, e.g. you could correct some errors in the transcription, i.e. modify .TextEquiv.Unicode as shown in the last paragraphs of the Jupyter notebook. If you convert the objectify object back to a XML string (i.e. “serialize” the object) using the etree API of the lxml package, you can upload the serialized data back to Transkribus. The upload_page_xml function of the Transkribus_Web class can do this for you.

At this point things become interesting, because now we have a complete toolset to communicate with Transkribus: We can browse our collections, documents and pages. We can download the content of a page, modify it, and send it back to Transkribus. At the same time, things have become quite complicated. I struggled a lot to wrap my head around the REST-API stuff, functions and classes in Python as well as object oriented programming in general. But it is definitely worth to learn these basics, because automating the communication with Transkribus will save a lot of time as soon as we start to normalize the transcription and search for transcription errors in the following episodes of this miniseries. Apart from Transkribus, there are other interesting use cases of APIs relevant for historians, and you could even build a REST-API yourself to provide your own data.

Cite this article as: Markus Müller, "Uncovering censorship in the 16th century with Transkribus and Python. Episode II: Let Python speak to Transkribus," in Digital Humanities Lab, 23/07/2021, https://dhlab.hypotheses.org/2114.

Featured Image: Let Python Talk to Transkribus, 2021, by Markus Müller, CC BY-SA 4.0,
build upon Wolpertinger by Rainer Zenz, CC BY-SA 3.0; Python natalensis, in: Andrew Smith: Illustrations of the zoology of South Africa, Reptilia. Smith, Elder, and Co., London 1840, PD; Markus Müller: Sammelband mit Schriften Johann Wilds, Wissenschaftliche Stadtbibliothek Mainz, Signatur 559 q 1; The ‘haunted table’ in the bar of the Black Horse Inn public house, on Nuthurst Street in Nuthurst, West Sussex, England, by Acabashi, CC BY-SA 4.0.


Autor: Markus Müller

Studied Catholic theology and education science in Tübingen (2000–2006) and Salamanca (2002/2003). 2012: PhD in Catholic theology (ecclesiastical history) in Tübingen. Research assistant in Tübingen, assistant lecturer in Frankfurt and Mainz. Since 2018, postdoctoral researcher at the Institute of European History in Mainz. – Research interests: history of religious education, censorship in the 16th century, Digital Humanities.

Ein Gedanke zu „Uncovering censorship in the 16th century with Transkribus and Python. Episode II: Let Python speak to Transkribus“

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.