Uncovering censorship in the 16th century with Transkribus and Python. Episode VI: Finding text re-use

by Markus Müller

In 1550, the Catholic cathedral preacher Johann Wild (Ioannes Ferus, 1495–1554) admitted in the preface to his commentary on the Gospel of John that he reused texts of Protestant authors such as Johannes Brenz (1499–1570) and Johannes Oekolampad (1482–1531) in his book, but that he only borrowed thoughts that were compatible with Catholic teaching.¹ Unfortunately, in his text, there are no footnotes or other references in his text to the authors he presumably cited. Since 1950, however, various historians have found verbatim parallels or at least significant similarities between Wild’s commentaries and Protestant authors.² But these findings were more or less accidental. It is still unclear to what extend Wild actually quoted Protestant authors, which authors he used in particular, and so on. The main reason for that is that a manual search for verbatim parallels is very time-consuming, even more time-consuming than searching for censorship. So, is there a way to hack the search for literal quotes in 16th century books?

---

This miniseries describes a digital workflow that starts with raw images of 16th century Latin prints, extracts the text, normalizes it, corrects transcription errors, and finally compares two texts to uncover differences, i.e. censorship. These are the episodes:

1. OCR with 16th century prints.
2. Let Python speak to Transkribus.
3. Normalizing 16th century raw text.
4. Detecting OCR transcription errors.
5. How did the censors actually change the text?
6. Finding text re-use.

---

Text re-use can be difficult to find

As explained in the last episode, the algorithm for searching differences between two texts is quite simple and Python’s difflib is easy to use. So, isn’t searching for similarities more or less the same as searching for differences? – Unfortunately, detecting the re-use of text is a more complex problem:

• For example, it may happen that the author quotes a certain sentence twice or reverses the order of the quoted sentences. In this case, we would have to search the entire source text again and again to make sure that we don’t miss any of these quotes.
• Sometimes the author may even modify the sentences themselves, e.g. by changing the word order, etc. In this case, the comparison between the original and the quoted sentence would not yield an exact match and we would need some sort of fuzzy comparison, i.e. an algorithm that can handle a change in word order.
• It gets even more complicated if the author puts the sentence into passive voice or changes the tense, because then he manipulates the actual words, and we need an algorithm for approximate string matching. There are various approaches to implementing this kind of fuzzy search in Python, but things get very complicated very soon. For example, if the texts undergo several types of modifications at the same time (e.g. changed sentence order plus passive voice), we would have to combine different types of search strategies.

TRACER: 700 algorithms to uncover text re-use

Fortunately, we do not have to go down this rabbit hole ourselves, thanks to TRACER, “a powerful and flexible suite of some 700 algorithms for the automatic detection of (historical) text reuse”, according to the manual. TRACER is indeed a very powerful tool, yet complicated to operate. The software is basically a command line program written in Java. You can feed it sentences from two texts, and TRACER uses its armada of algorithms to find all kinds of parallels, be it verbatim quotations, paraphrases or allusions.

TRACER is not only capable of approximate string matching, but can also compare words based on linguistic metadata, as long as you provide it with part-of-speech tags and lists of lemmata or synonyms for the language of your source texts. This means, of course, that you need to generate this metadata in advance using some kind of Natural Language Processing software. For modern languages, this would be a simple matter, since the popular Python packages for Natural Language Processing normally include all the necessary tools (like NLTK and spaCy). But the available solutions for Latin are either still quite buggy or they do not work well with medieval and early modern Latin — like the Classical Language Toolkit (CLTK), which focusses on classical Latin and is still under development. (BTW, there are interesting projects especially in the Italian DH community that bring ancient, medieval and early modern Latin into today’s world of Natural Language Processing, e.g. the Centro Interdisciplinare di Ricerche per la Computerizzazione dei Segni dell‘Espressione, or Greta Franzini’s GitHub account.) If you invest enough time into preprocessing, you can do all kinds of sophisticated comparisons with TRACER. So far, however, I have limited myself to searching for verbatim or near-verbatim quotations.

Having installed TRACER (see the manual for details), you need to get familiar with the basic architecture of TRACER and the various steps TRACER takes to compare the texts. Every step has one or more parameters that influence its behaviour. You can change these parameters in the tracer_config.xml file which is the “control tower” of the software with over 900 lines (!) of XML. The manual provides a detailed explanation of the most important parameters. However, it is quite tedious to work through the long and sometimes cryptic XML parameters (e.g. <category name="eu.etrap.tracer.selection.globalglobal.GlobalMinFeatureFrequencySelectorImpl">). You will need to take detailed notes of the changes you make in the config file to understand the effects of these changes.

Last but not least, you have to convert your text data into csv (or rather tab-separated values), a simple text format for tabular data. TRACER expects the text data as a series of predefined segments. Therefore, you have to cut your text corpus into segments that make sense in your specific case of text re-use. TRACER compares these predefined segments (not the corpus as a whole) by calculating the degree of similarity between all the segments of the two texts according to the parameters in tracer_config.xml.

TRACER processes the segments in five steps (preprocessing, training, selection, linking, scoring). Depending on the size of your corpus, your CPU and the speed of your hard drive, the calculations will take some minutes. For every step, TRACER stores the intermediate results in a nested tree of folders on disk. The folder names reflect the corresponding parameters in config.xml. The table with the actual degrees of similarity is found in the very last folder of this tree. Therefore, the path to the folder containing the relevant result looks something like this: /tracer-master/data/corpora/Bible/TRACER_DATA/01-02-WLP-lem_true_syn_true_ssim_false_redwo_false-ngram_5-LLR_true_toLC_true_rDia_false_w2wl_false-wlt_5/01-01-01-00-00-WordBasedTrainingImpl/02-02-01-01-01-LocalMaxFeatureFrequencySelectorImpl_FeatDens=0.8/01-01-01-01-01-KJV-01-01-01-02-KJV/02-02-01-01-01-02-SelectedFeatureResemblanceSimilarityImpl_Threshold=0.5/001-TraVizPostprocessingImpl-mode=1_minfrequency=2/html/…. This sometimes leads to problems on some Windows machines when opening the visualization file, since the maximum length of a file path on Windows used to be 260 characters. So, better use MacOS (1024 characters) or Linux (4096 characters). To retrieve the results, you have to click through eleven (!) folders. And beware: some folders contain two subfolders, and one of them is a dead end, where yet another intermediate result is stored. In the very last of these nested folders, TRACER outputs an interactive visualization of the results in the so-called “Text Re-use Browser”.

This looks very nice, but if you manually check the parallels found by TRACER, you will notice that the result is quite disappointing: There definitely is a near-verbatim match between segment 1001748 and 1100516. However, the two following segments are also quite similar, but TRACER does not find this parallel:

To overcome this problem, you would have to modify the parameters in tracer_config.xml, which, as mentioned earlier, is quite complicated. I do not doubt that the perfect settings exist, though I could not find them despite a number of long trial-and-error sessions. Maybe I gave up too soon, maybe I should have a look at working examples of tracer_config.xml from successful projects of TRACER’s co-developers.

In the end, I decided to leave TRACER behind and search for alternatives. Nevertheless, I wanted to share my preprocessing workflow, the scripts that made my data TRACER-ready, and a script that executes TRACER and collects the relevant results from the nested folders.

Scripts to operate TRACER

It all starts with raw text data. The texts to be checked are already digitized by the Transkribus Spellchecker web app described in episode IV. I have added an export function to the app that does the segmentation and produces a TRACER-ready CSV file for direct download (cf. here and here on GitHub).

You can define the punctuation used for segmentation in the app (in the “separators” field). The segmentation algorithm does not cut the text after abbreviations (which I have predefined in a hard-coded list) to avoid unreasonably short segments (< 3 words). The resulting CSV file has the five column format expected by TRACER, including a unique ID for each segment, the segment itself, a time stamp, and a human-readable identifier of the book or section from which the segment is taken.

Having exported all the texts you wish to compare, the CSV files must be merged, because TRACER needs all the data in one CSV file. The first two digits of the segment ID make the texts distinguishable for TRACER. Therefore, you need to change these first two digits when joining the CSV files. You can do this by hand using an Excel spreadsheet, as described in the TRACER manual, or use a Python script (cf. my join_tracer_files.py on GitHub) .

Now that all the text data is ready, you can run TRACER from the command line. Since TRACER is a Java program, it requires the preinstalled Java Runtime Environment (Version 1.8). The command to execute TRACER it is quite long (java -Xmx600m -Deu.etrap.medusa.config.ClassConfig=conf/tracer_config.xml -jar tracer.jar), and together with the fact that you have to wait 5 minutes or more and then click through eleven nested folders to see the result, I decided to write a Python script that executes TRACER, waits until it’s done, retrieves the results from the nested folders, and copies the visualization to another location (a subfolder called tracer_output) where it is more accessible (cf. GitHub).

With the workflow shown above, it is quite fun to experiment with TRACER. But because the actual results of TRACER’s analysis were quite disappointing and I couldn’t find better parameters in tracer_config.xml (as mentioned above), I decided to give up and search for another tool.

Similarity texter: the charming 1980s are back on stage

Around 1989, Dick Grune (a lecturer in Programming Languages and Compiler Construction at the University of Amsterdam) developed a plagiarim detection algorithm optimized for plain text and various programming languages. Grune used his command line program “sim_text” to check for duplicated code and text in large software projects and to detect plagiarism in the academic and educational context he worked in. In 2016, Sofia Kalaidopoulou reimplemented the sim_text algorithm in JavaScript and designed a very nice web app called “similarity texter” that makes it easy to find verbatim parallels between two text files. Similarity texter is far simpler than TRACER and not capable of approximate string matching or other sophisticated comparisons. But it is very easy to use and immediately visualizes the results in a way that is ideal for my purposes.

Similarity texter expects the plaintext as it is, so the preprocessing steps (segmentation and conversion to CSV) are no longer necessary. Because I was too lazy to write yet another export function for my Spellchecker web app, I reused the already exported CSV files for TRACER by simply copying the second column containing the plaintext and pasting it directly into the Similarity texter web app.

Having pasted your text, just change the minimum match length, click compare and off you go:

Most of the verbatim parallels are highlighted in corresponding colors and you can easily align the two texts by clicking on one of the highlighted passages. The parallel found by TRACER in the example above shows up in a totally new light: You can immediately see that there is not just one but many parallels and that you have definitely found an interesting case of cross-denominational text re-use in the 16th century that awaits historical and theological interpretation. Still, some questions remain:

1. The result is not perfect yet: Due to the simplicity of the algorithm and the minimum match length, near-verbatim parallels are not detected:

   

   But since you definitely get a reliable impression of where to look for these near-verbatim parallels, the output is very usable and you can seamlessly continue with a manual analysis of the relevant text passages.

2. If the both texts quote from yet another text, these quotes are displayed as parallels. One comes across this problem all the time when dealing with Bible commentaries: All commentaries on —let’s say the first chapter of the Gospel of John— will have a certain amount of text in common because they all quote the same verses from the Gospel of John. The easiest way to deal with this problem is to know the biblical texts by heart, so you can spot the pseudo-parallels at first glance. Otherwise, you would have to look them up manually in the Bible.
   One technical solution would be to compare the texts in a two-step procedure: First, one compares the text in question with the Bible, then with the text from which it is presumably quoting. By placing two “similarity texter” windows next to each other, one can then identify the pseudo-quotes relatively easy. It would be nice, of course, to merge the two comparisons into one visualization, but that would be a whole new coding project (either modifying the “similarity texter” app or reimplementing its algorithm in Python…).
3. It can be difficult to determine the exact location of the passages in the book from which they are taken from. For exact citations, you have to look up manually the pages of the passages in question. It would be very convenient to have some metadata along with the text. One simple solution would be to include this metadata in the text itself (e.g. by including CTS URNs, as explained in episode V), although this would worsen the results of the comparison. You could write a new export function for the Spellchecker web app add and an optional parameter for including this metadata.
4. The workflow lacks a possibility to document the results. The “register censorship” feature allowed for a thorough documentation of the censored passages found in the sources (see episode V). However, the parallels we discover with Similarity texter remain undocumented for now. One day, an elegant solution could be to use XML to store the original text along with the censorship and the parallels following the TEI guidelines.

Including digitized texts from other sources

So far, the only source of text data is the Spellchecker web app. However, in some cases it is convenient to use texts from other sources to search for parallels. Especially if you are doing research on the history of theology, there are hundreds of digitized texts available online. In episode IV, I already mentioned the “Corpus Corporum” database which includes the “Patrologia Latina”, the “Biblia Sacra iuxta Vulgatam Clementinam” (i.e.  the Clementine Vulgate which is exactly what you need if you are studying 16th and 17th century authors), and other ancient and medieval theological texts. Using these texts, you could track down direct quotations from the Bible and the Church Fathers, etc. The only prerequisite would be to convert the HTML or XML data to plaintext files and (if necessary) perform some normalization.

Some corpora, though, are behind pay walls, e.g. the “Digital Library of the Catholic Reformation” and the “Digital Liberary of Classic Protestant Texts” on “Alexander Street”, an ongoing project between ProQuest and the Bodleian Library in Oxford which provides high quality full texts of early modern Protestant and Catholic theologians. If you are lucky, you can access the full texts from within the intranet or via the library of your university.

Databases like these sometimes do not provide an option to download the texts. You can either copy them manually (which is tedious if the texts are very long) or write a tailor-made script for scraping the page. The introduction on web scraping with Python by Monica Barget can help you to understand the basics. Using the basic concepts explained there, I wrote a script for downloading a pile of books from “Corpus Corporum” (GitHub). Sometimes, though, the Python approach would be too complicated, especially if the page uses a lot of JavaScript to dynamically load data in the background. In such cases, it is much easier to use a short but effective JavaScript function that allows you to simply save the HTML content of your browser window as a text file. This is not the same as using “Save page as…” from your browser’s menu, because the “Save page as…” command would not include the data that is dynamically loaded in the background when you visit and scroll the page. Check out the code and the comments in my JS_downloader.js script (on GitHub) to see how it works in detail. For mass downloads, you could slightly modify the script and try to turn it into a bookmarklet so that clicking the bookmark automatically executes the JavaScript code.

After you download the page content, you have to extract the raw text from the HTML files. First, investigate the structure of the downloaded HTML to find out which elements contain the actual text. (To make life easier, use a text editor with code highlighting to open the .html file!) If you know what you are looking for, you can write a short Python script to extract the text (using BeautifulSoup, lxml or the like). See my alexanderstreet_converter.py (on GitHub) and my vulgata_to_txt Jupyter Notebook (on GitHub) to get some inspiration.

The last step would be to normalize the spelling of the Latin text by replacing special characters, diacritics. etc. You can use the normalize() function of my tokenize4tracer.py script (on GitHub) that I originally wrote to feed texts from arbitrary online sources to TRACER. Just cut and paste the normalize() function into your Similarity texter workflow.

This approach is extremely flexible and allows you to make use of various online resources and to do all sorts of customization and automatization, e.g. you could convert a batch of downloaded files at once by iterating over the .html files in a specific folder.

Final thoughts

It is still a time-consuming task to identify the transconfessional parallels in Johann Wild’s texts. You have to be quite creative and lucky to find the books Wild had on his desk. (Even though the OCR of early modern books on Google Books is quite bad, it sometimes helps to identify verbatim quotes. The real superpower, however, is a thorough knowledge of the history of theology.) If you have finally found a book that Wild definitely used, you have to search for already digitized versions (and try to integrate them into your workflow) or use Transkribus to digitize it yourself. Lastly, using Similarity texter to find possible verbatim quotes is already a big step forward, but it is still quite complex to evaluate the results and draw valid conclusions on Wild’s transconfessional strategies. I really miss a feature to document and store the uncovered parallels.

Implementing this feature could be the next milestone of my personal DH journey. However, after many years of research (and programming), I really need to finish my book on Johann Wild. Because printed hardcover books is what counts in the world of Catholic theology in Germany in 2021. In the German speaking context, I have met no more than a handful of theologians who were really interested in Digital Humanities, its potential, its future. Traditionally, many theologians (or most of them?) invest their time into critisizing the negative effects of modern technology on society and philosophizing about the dangers of digitalization. The keynote of an Innsbruck conference on ”Digitalization, religion and society” in September 2021 concluded that digitalization and climate change were the two big threads of humanity.³

So what if the machines have already conquered my mind? Do I lack “prophetic scepticism”? Should I put my DH “hobby” back on the shelf? — Or put the other way around: Should I just leave academic theology behind and do my thing? Or should I take the middle road and move on, trying to use technology to further our understanding of the big clash between “Catholics” and “Protestants” in the 16th century? (At the Innsbruck conference, it was noted that technology was always more than an morally neutral “tool”. So, is digitalization a potential danger to history, theology or other humanities disciplines? On the other hand, Catholic “computer nerds” would probably invoke the Second Vatican Council to show that they are not totally wrong, cf. Gaudium et spes 35f.) Thinking about this strange ambivalence, I cannot help but smile when digital humanists regularly refer to the Jesuit Roberto Busa (1913–2011) as one of the pioneers of Digital Humanities. Of course, this is just another “creation myth” (cf. Tessa Gengnagel’s video essay), but honestly: I like it.

---

Footnotes

¹ Johann Wild, In […] euangelium secundum Ioannem […], Mainz: Franz Behem 1550 (VD16 W 2963), unpaginated fol. a3v.

² E.g. verbatim parallels between Wild’s and Huldrych Zwingli’s commentaries on Genesis: David Lerch, Isaaks Opferung christlich gedeutet. Eine auslegungsgeschichtliche Untersuchung, Tübingen 1950, p. 232–234; idem, Zur Beziehung zwischen Zwinglis und Johannes Wilds Genesisauslegung, in: Theologische Zeitschrift 8 (1952), p. 471–472.

³ https://www.katholisch.at/aktuelles/135426/glettler-der-digitalisierung-mit-prophetischer-skepsis-begegnen.

---

Featured Image: Let Python Talk to Transkribus, 2021, by Markus Müller, CC BY-SA 4.0,
build upon Wolpertinger by Rainer Zenz, CC BY-SA 3.0; Python natalensis, in: Andrew Smith: Illustrations of the zoology of South Africa, Reptilia. Smith, Elder, and Co., London 1840, PD; Markus Müller: Sammelband mit Schriften Johann Wilds, Wissenschaftliche Stadtbibliothek Mainz, Signatur 559 q 1; The ‘haunted table’ in the bar of the Black Horse Inn public house, on Nuthurst Street in Nuthurst, West Sussex, England, by Acabashi, CC BY-SA 4.0.

---

---