In language processing, non-standard languages such as Swiss German dialects typically have to be normalized in order to achieve satisfactory results in downstream tasks such as POS-tagging.
baseline.py performs a simple normalisation algorithm on tokenized Swiss German text which is based on the most frequent normalisation in another text. The text is then POS-tagged in three different normalisation settings and the results are evaluated in evaluation.py.
- baseline.py
$ python3 baseline.py WUS_POS_data/train.txt WUS_POS_data/dev.txt WUS_POS_data/test.txt
- evaluation.py
$ python3 evaluation.py WUS_POS_data/dev_norm_out.txt WUS_POS_data/test_norm_out.txt
For this script, I worked with the WUS corpus. In the corpus, there are three tab-seperated files dev.txt, test.txt and train.txt which are each structured in the following way:
- first column: original non-standard word
- second column: manual normalisation (gold standard)
- third column: manually written POS-tag (gold standard)
Messages are seperated by empty lines.
nena nena NE
böge Böge VVFIN
wie wie KOUS
gohts geht es VVFIN+PPER
? ? $.
tip top tipptopp ADJD
.. .. $.
hett hätte VAFIN
kli klein ADJD
sushi Sushi NN
dihai daheim ADV
emojiQcatFaceWithWrySmile emojiQcatFaceWithWrySmile EMOJI
In baseline.py, the normalisation gold standard column of train.txt is used in order to generate new normalisations for dev.txt and test.txt. We can distinguish between three normalization methods:
- Unique: Words that occur in the training set exactly once get normalized in the way they were normalized in the training set
- Ambiguous: Words that have more than one normalization in the training set get the most frequent normalization or a random one in the case of a tie.
- New: Words that don't occur in the training set don't get normalized (the original non-standard word is copied.)
The output files dev_norm_out.txt and test_norm_out.txt are formated in the following way:
- 1st column: normalization strategy
- 2nd column: original non-standard word
- 3rd column: predicted normalisation (baseline)
- 4th column: manual normalisation (gold standard)
- 5th column: manually written POS-tag (gold standard)
In evaluation.py, the predicted normalisation (3rd column in dev_norm_out.txt and test_norm_out.txt) gets Part-of-speech tagged by a pre-trained tagger. The tagger is part of a spaCy model for german.
POS tags are created in three different normalisation settings:
- upper bound: the POS-tagger is applied to the manually normalized data (gold standard)
- lower bound: the POS-tagger is applied to the original non-standard data
- baseline: the POS-tagger is applied to the data that was normalised in baseline.py
The output files dev_norm_POS_out.txt and test_norm_POS_out.txt are structured in the following way:
- 1st column: normalization strategy
- 2nd column: original non-standard word
- 3rd column: predicted normalisation (baseline)
- 4th column: manual normalisation (gold standard)
- 5th column: manually written POS-tag (gold standard)
- 6th column: POS tags (lower bound)
- 7th column: POS tags (upper bound)
- 8th column: POS tags (baseline)
Finally, in evaluation.py the POS tagger gets evaluated in the different normalization settings. The newly generated POS tags that were created in the three settings are compared to the manually written POS tags (gold standard). The results of the evaluation can be found in eval_report.txt.
- Python version 3.6 or newer
- Install tabulate and spaCy and download spaCy's small model for German:
# $ pip install -U pip setuptools wheel
# $ pip install -U spacy
# $ python -m spacy download de_core_news_sm
# $ pip install tabulate
project
│ README.md
│ baseline.py
│ evaluation.py
│ eval_report.txt
│
└───WUS_POS_data
dev.txt
dev_norm_out.txt
dev_norm_POS_out.txt
test.txt
test_norm_out.txt
test_norm_POS_out.txt
train.txt
This script is licensed under the term of the MIT License, see the file LICENSE for more details.