Home

In this wiki, we will guide you through the steps to be taken for the typical usage of the library.

Installation

First of all, make sure that you have cloned the repository and installed the requirements in the requirements.txt file. We did not include the installation of PyTorch to avoid conflicts in case you want to use a Docker instance with integrated PyTorch and CUDA, so please make sure they are installed or included in your Dockerfile before starting the process.

Create the dataset

Creating the dataset to be fed in the training script requires following multiple steps.

...

Train the models

Finally, after creating the dataset and splitting with the seeds, we can train our models. Again, using the italian for the example, we can launch the training with the following command: python 1-train.py --lang it --seeds 110,221,332 --device cuda --fp16 --eval_metric f1_macro. This command launches the script 1-train.py and it sets the language for the data as "italian" and tells the script to train 3 models with the 3 different data splits for the seeds. The trainings will be run sequentially, meaning that first, the training for the first seed is done, then the second and so on. It also tells the script to prefer the gpu over the cpu with the device argument, enables mixed precision training with the fp16 argument and chooses the macro-weighted F1 as the metric to use during the evaluation on the dev set at the end of each training epoch. The duration of a model training depends on both the hardware and the model parameters used. The defaults are good to use in an environment with limited memory, but increasing the batch size accordingly can help to reduce the duration. Mixed precision is also helpful to speed up the training.

The final best model for each run will be saved in the folder specified via the models_path argument, each in the folder of their seed. So, for example, the best model for the training of the data split relative to the seed 110 will be saved in the folder models\110\checkpoint-xxxxxx if the models_path argument is set to models.

There are other more specific arguments to be used with this script. To check them out, you can call the script like 1-train.py --help.

Evaluate the models

Once the models are trained, they can be evaluated to extract all the metrics. To do this, we can use the following command: python 2-evaluate.py --lang it --seeds 110,221,332 --device cuda --parents add. This launches the 2-evaluate.py script and once again tells it to use the Italian language and evaluate three models for the three different seeds. To do this, the script loads the model from the checkpoint folder it finds in the folder of the current seed and predicts the labels for the "test" split. After telling it to prefer the gpu, we also tell it to include the evaluation of the parent labels with --parents add. The parents argument with the add option tells the script to map both the original labels and the predicted ones at the Thesaurus Concept level to their Micro Thesaurus and Domain parents, and to calculate the metrics for both. This makes the evaluation take longer, but allows a better understanding of the classification performance of the models.

At the end of the evaluation phase, the checkpoint folder of the tested model will contain an "evaluation" folder with all the data from the evaluation, such as the JSON file containing the calculated metrics or the classification report with basic metrics for each label.

For more arguments and their options, call the script like 2-evaluate.py --help.