Experiments and tutorials from the wide field of cultural analytics based on textual and multimodal corpora.

In this tutorial you will learn to:

• web/screen scrape relatively unstructured data from the Wikipedia
• transform unstructured data into tabular data to facilitate processing with Python
• create graph data from your data to visualize your data as networks
• export Python-created data to use it with JavaScript visualization libraries such as D3.js

Would you should already know:

• a little Python 3
• some minor HTML
• some JavaScript if you want to understand the web-based visualization at the end of the tutorial

This notebook comes with a requirements.txt file to facilitate the installation of package dependencies. To install the dependencies, launch the following command from the command line before you start the notebook:

pip3 install -r requirements.txt

In case you have Podman installed, there is a Dockerfile available in the containerdirectory.

Docker might work as well but has not been tested.

Make sure that you are running the following commands from within the containerdirectory.

podman build  -t ibi_runtime . 

This will take some time as it will build everything from scratch.

After the creation of the image, you are set to run the container by executing the following command:

podman run  -p 127.0.0.1:8888:8888 -p 127.0.0.1:8000:8000 -i -t localhost/ibi_runtime

This command will ensure that you can access the Jupyter notebook under http://localhost:8888/notebooks/WikipediaTest.ipynb and the web server that can be launched from within the notebook under port 8000. The address 127.0.0.1 is represented by localhost.

Furthermore, the command will open a terminal connection to the container in order to display all log output from Jupyter and launch the notebook automatically.

To stop the container, you can use the shut down entry from Jupyter's File menu or activate the terminal running Jupyter and press CTRL+C. You will then be asked immediately if you want to shut down the Jupyter server. Enter y and Jupyter will shut down.

After the server has been stopped, the container will exit as well.

Attention: You will also lose all your data that has been created from within the notebook!

This notebook eventually evolved into a TPDL publication. ATTENTION! The notebook is no longer maintained here. It has been moved to a separate repository.

• In this tutorial, you will learn to read metadata from an OAI-PMH data provider and how to convert the retrieved data from Dublin Core to a pandas data frame.
• Furthermore, you will carry out some basic data analysis on your data in order to find out if the data is corrupt or unclean. Based on an example, you will clean some aspects of your data using techniques borrowed from machine learning.
• Finally, you will visualize data with the help of a network graph.

In this tutorial, you will learn how to read from a unstructured and structured dataset, create a dataframe from this raw data, and to visualize characteristics from the data in order to find out whether the titles of a research library are truly neutral from a sentiment analysis perspective and how they compare to a sample from books sold by Amazon.