Before digital humanists can do things with data, they first need to collect them, and web automation (or more specific methods of web scraping) can be a quick way of gathering a large amount of data. While web automation denotes every remotely controlled action performed on the web, web scraping, web mining or web harvesting are focussed on reading and processing information (found on websites). This blog post presents useful Python packages for these tasks and explains the advantages of working with browser profiles. „Doing Digital History with Python IV: web automation“ weiterlesen
Schlagwort: Doing digital history with Python
Doing Digital History with Python III: topic modelling with Gensim, spaCy, NTLK and SciKit learn
In April 2020, we started a series of case studies to introduce researchers working with historical sources to data analysis and data visualisation with Python. Today’s blog post covers topic modelling with the Python packages Gensim, spaCy, NLTK and SciKit learn.
Topic modelling is one of the central methods of Natural Language Processing (NLP), the “automatic manipulation of natural language, like speech and text, by software.” (Jason Brownlee: What Is Natural Language Processing?, in: Deep Learning for Natural Language Processing, 22nd September 2017) In its most basic form, a “topic” modelled by software displays word co-occurrences in texts, assuming that the frequency of co-occurrences defines certain areas of meaning. „Doing Digital History with Python III: topic modelling with Gensim, spaCy, NTLK and SciKit learn“ weiterlesen
Doing Digital History with Python II: creating custom Word Clouds
by Monika Barget
In the second edition of Doing digital history with Python, I would like to address word clouds as a visual method of finding patterns in texts (see critical reflection in Basic Text Mining: Word Clouds, their Limitations, and Moving Beyond Them). Word clouds display the frequency or importance of individual keywords in individual texts or entire corpora. There are many ready-made tools in multiple languages that help you create word clouds in different designs, such as the in-built word cloud generator in Voyant Tools or browser-based tools such as Wortwolken.com. However, not all of them may be suitable for your specific use case. „Doing Digital History with Python II: creating custom Word Clouds“ weiterlesen
Doing Digital History with Python I: reading (messy) XML & JSON data
by Monika Barget
During our DH brownbag lunches at the IEG, colleagues have repeatedly asked us if we could recommend Python packages for digital history. We have therefore set up a list of packages we at the IEG DH Lab are using for the analysis of text (stored, for instance, in XML/TEI or JSON formats), the modelling of historical networks, or the creation of interactive maps.
The list Python for digital history is based on our personal experiences and, though by no means exhaustive, may serve as an appetizer for “Doing Digital History with Python”. In a series of blog posts, we will try and introduce you to some of the packages mentioned through case studies from current IEG research.
Today’s post covers the extraction of data from XML and JSON files with xml.etree.ElementTree, lxml, json(5) and beautifulsoup(4) as reading structured text is often a starting point of digital history projects. „Doing Digital History with Python I: reading (messy) XML & JSON data“ weiterlesen