Skip to content

Latest commit

 

History

History
285 lines (230 loc) · 14.2 KB

README.md

File metadata and controls

285 lines (230 loc) · 14.2 KB
Raw

PyPI version PyPi downloads DOI

sentence-embedding-evaluation-german

Sentence embedding evaluation for German.

This library is inspired by SentEval but focuses on German language downstream tasks.

Downstream tasks

The available downstream tasks are listed in the table below. If you that think that a dataset is missing and should be added, please open an issue.

task type text type lang #train #test target info
TOXIC 👿 toxic comments facebook comments de-DE 3244 944 binary {0,1} GermEval 2021, comments subtask 1, 📁 📖
ENGAGE 🤗 engaging comments facebook comments de-DE 3244 944 binary {0,1} GermEval 2021, comments subtask 2, 📁 📖
FCLAIM ☝️ fact-claiming comments facebook comments de-DE 3244 944 binary {0,1} GermEval 2021, comments subtask 3, 📁 📖
VMWE ☁️ verbal idioms newspaper de-DE 6652 1447 binary (figuratively, literally) GermEval 2021, verbal idioms, 📁 📖
OL19-A 👿 offensive language tweets de-DE ??? 3031 binary {0,1} GermEval 2019, 📁 📖
OL19-B 👿 offensive language, fine-grained tweets de-DE ??? 3031 4 catg. (profanity, insult, abuse, oth.) GermEval 2019, 📁 📖
OL19-C 👿 explicit vs. implicit offense tweets de-DE 1921 930 binary (explicit, implicit) GermEval 2019, 📁 📖
OL18-A 👿 offensive language tweets de-DE 5009 3398 binary {0,1} GermEval 2018, 📁
OL18-B 👿 offensive language, fine-grained tweets de-DE 5009 3398 4 catg. (profanity, insult, abuse, oth.) GermEval 2018, 📁
ABSD-1 🤷 relevance classification 'Deutsche Bahn' customer feedback de-DE 19432 2555 binary GermEval 2017, 📁
ABSD-2 😃😐😡 sentiment analysis 'Deutsche Bahn' customer feedback de-DE 19432 2555 3 catg. (pos., neg., neutral) GermEval 2017, 📁
ABSD-3 🛤️ aspect categories 'Deutsche Bahn' customer feedback de-DE 19432 2555 20 catg. GermEval 2017, 📁
MIO-S 😃😐😡 sentiment analysis 'Der Standard' newspaper article web comments de-AT 1799 1800 3 catg. One Million Posts Corpus, 📁
MIO-O 🤷 off-topic comments 'Der Standard' newspaper article web comments de-AT 1799 1800 binary One Million Posts Corpus, 📁
MIO-I 👿 inappropriate comments 'Der Standard' newspaper article web comments de-AT 1799 1800 binary One Million Posts Corpus, 📁
MIO-D 👿 discriminating comments 'Der Standard' newspaper article web comments de-AT 1799 1800 binary One Million Posts Corpus, 📁
MIO-F 💡 feedback comments 'Der Standard' newspaper article web comments de-AT 3019 3019 binary One Million Posts Corpus, 📁
MIO-P ✉️ personal story comments 'Der Standard' newspaper article web comments de-AT 4668 4668 binary One Million Posts Corpus, 📁
MIO-A ✴️ argumentative comments 'Der Standard' newspaper article web comments de-AT 1799 1800 binary One Million Posts Corpus, 📁
SBCH-S 😃😐😡 sentiment analysis 'chatmania' app comments, only comments labelled as Swiss German are included gsw 394 394 3 catg. SB-CH Corpus, 📁
SBCH-L ⛰️ dialect classification 'chatmania' app comments gsw 748 748 binary SB-CH Corpus, 📁
ARCHI ⛰️ dialect classification Audio transcriptions of interviews in four dialect regions of Switzerland gsw 18809 4743 4 catg. ArchiMob, 📁 📖
LSDC 🌊 dialect classification several genres (e.g. formal texts, fairytales, novels, poetry, theatre plays) from the 19th to 21st centuries. Extincted Lower Prussia excluded. Gronings excluded due to lack of test examples. nds 74140 8602 14 catg. Lower Saxon Dialect Classification, 📁 📖
KLEX-P 🤔 text level Conceptual complexity classification of texts written for adults (Wikipedia), children between 6-12 (Klexikon), and beginner readers (MiniKlexikon); Paragraph split indicated by <eop> or * de 8264 8153 3 catg. 📁 📖

Download datasets

bash download-datasets.sh

Check if files were actually downloaded

find ./datasets/**/ -exec ls -lh {} \;

Usage example

Import the required Python packages.

from typing import List
import sentence_embedding_evaluation_german as seeg
import torch

Step (1) Load your pretrained model

In the following example, we generate a random embedding matrix for demonstration purposes.

# (1) Instantiate an embedding model
emb_dim = 512
vocab_sz = 128
emb = torch.randn((vocab_sz, emb_dim), requires_grad=False)
emb = torch.nn.Embedding.from_pretrained(emb)
assert emb.weight.requires_grad == False

Step (2) Specify your preprocessor function

You need to specify your own preprocessing routine. The preprocessor function must convert a list of strings batch (List[str]) into a list of feature vectors, or resp. a list of sentence embeddings (List[List[float]]). In the following example, we generate some sort of token IDs, retrieve the vectors from our random matrix, and average these to feature vectors for demonstration purposes.

# (2) Specify the preprocessing
def preprocesser(batch: List[str], params: dict=None) -> List[List[float]]:
    """ Specify your embedding or pretrained encoder here
    Paramters:
    ----------
    batch : List[str]
        A list of sentence as string
    params : dict
        The params dictionary
    Returns:
    --------
    List[List[float]]
        A list of embedding vectors
    """
    features = []
    for sent in batch:
        try:
            ids = torch.tensor([ord(c) % 128 for c in sent])
        except:
            print(sent)
        h = emb(ids)
        features.append(h.mean(axis=0))
    features = torch.stack(features, dim=0)
    return features

Step (3) Training settings

We suggest to train a final layer with bias term ('bias':True), on a loss function weighted by the class frequency ('balanced':True), a batch size of 128, an over 500 epochs without early stopping.

# (3) Training settings
params = {
    'datafolder': './datasets',
    'bias': True,
    'balanced': True,
    'batch_size': 128, 
    'num_epochs': 500,
    # 'early_stopping': True,
    # 'split_ratio': 0.2,  # if early_stopping=True
    # 'patience': 5,  # if early_stopping=True
}

Step (4) Downstream tasks

We suggest to run the following downstream tasks. FCLAIM flags comments that requires manual fact-checking because these contain reasoning, arguments or claims that might be false. VMWE differentiates texts with figurative or literal multi-word expressions. OL19-C distincts between explicit and implicit offensive language. ABSD-2 is a sentiment analysis dataset with customer reviews. These four dataset so far can be assumed to be Standard German from Germany (de-DE). MIO-P flags Austrian German (de-AT) comments if these contain personal stories. ARCHI is a Swiss (gsw), and LSDC a Lower German (nds) dialect identification task.

# (4) Specify downstream tasks
downstream_tasks = ['FCLAIM', 'VMWE', 'OL19-C', 'ABSD-2', 'MIO-P', 'ARCHI', 'LSDC']

Step (5) Run the experiments

Finally, start the evaluation. The suggested downstream tasks (step 4) with 500 epochs (step 3) might requires 10-40 minutes but it's highly dependent on your computing resources. So grab a ☕ or 🍵.

# (5) Run experiments
results = seeg.evaluate(downstream_tasks, preprocesser, **params)

Demo notebooks

Start Jupyter

source .venv/bin/activate
jupyter lab

Open an demo notebook

Appendix

Installation & Downloads

The sentence-embedding-evaluation-german git repo is available as PyPi package

pip install sentence-embedding-evaluation-german
pip install git+ssh://git@github.com/ulf1/sentence-embedding-evaluation-german.git

You need to download the datasets as well. If you run the following code, the datasets should be in a folder ./datasets.

wget -q "https://raw.githubusercontent.com/ulf1/sentence-embedding-evaluation-german/main/download-datasets.sh" -O download-datasets.sh 
bash download-datasets.sh

Development work for this package

Install a virtual environment

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt --no-cache-dir
pip install -r requirements-dev.txt --no-cache-dir
pip install -r requirements-demo.txt --no-cache-dir

(If your git repo is stored in a folder with whitespaces, then don't use the subfolder .venv. Use an absolute path without whitespaces.)

Install conda environment for GPU

conda install -y pip
conda create -y --name gpu-venv-seeg python=3.9 pip
conda activate gpu-venv-seeg
# install CUDA support
conda install -y cudatoolkit=11.3.1 cudnn=8.3.2 -c conda-forge
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
pip install torch==1.12.1+cu113 torchvision torchaudio -f https://download.pytorch.org/whl/torch_stable.html
# install other packages
pip install -r requirements.txt --no-cache-dir
pip install -r requirements-dev.txt --no-cache-dir
pip install -r requirements-demo.txt --no-cache-dir
watch -n 0.5 nvidia-smi

Python commands

  • Jupyter for the examples: jupyter lab
  • Check syntax: flake8 --ignore=F401 --exclude=$(grep -v '^#' .gitignore | xargs | sed -e 's/ /,/g')

Publish package

pandoc README.md --from markdown --to rst -s -o README.rst
python setup.py sdist 
twine upload -r pypi dist/*

Clean up

find . -type f -name "*.pyc" | xargs rm
find . -type d -name "__pycache__" | xargs rm -r
rm -r .pytest_cache
rm -r .venv

New Dataset recommendation

If you want to recommend another or a new dataset, please open an issue.

Troubleshooting

If you have troubles to get this package running, please open an issue for support.

Contributing

Please contribute using Github Flow. Create a branch, add commits, and open a pull request.

Acknowledgements

The "Evidence" project was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 433249742 (GU 798/27-1; GE 1119/11-1).

Citation

  1. You certainly need to cite the actual evaluation datasets in your paper. Please check the hyperlinks in the info column of the table above.

  2. You can cite the software package as

@software{ulf_hamster_2023_7863799,
  author       = {Ulf Hamster},
  title        = {sentence-embedding-evaluation-german},
  month        = apr,
  year         = 2023,
  publisher    = {Zenodo},
  version      = {0.1.12},
  doi          = {10.5281/zenodo.7863799},
  url          = {https://doi.org/10.5281/zenodo.7863799}
}

or cite the following arXiv preprint in which we deployed the software to benchmark sentence embeddings

@misc{hamster2023rediscovering,
      title={Rediscovering Hashed Random Projections for Efficient Quantization of Contextualized Sentence Embeddings}, 
      author={Ulf A. Hamster and Ji-Ung Lee and Alexander Geyken and Iryna Gurevych},
      year={2023},
      eprint={2304.02481},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}