Introducing DeReKoGram: A novel frequency dataset with lemma and part- of-speech information for German

doi:10.21203/rs.3.rs-3139640/v1

Download PDF

Research Article

Introducing DeReKoGram: A novel frequency dataset with lemma and part- of-speech information for German

https://doi.org/10.21203/rs.3.rs-3139640/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 09 Nov, 2023

Read the published version in Data →

Version 1

posted

You are reading this latest preprint version

We introduce DeReKoGram, a novel frequency dataset containing lemma and part-of-speech (POS) information for 1-, 2-, and 3-grams from the German Reference Corpus. The dataset contains information based on a corpus of 43.2 billion tokens and is divided into 16 parts based on 16 corpus folds. We describe how the dataset was created and structured. By evaluating the distribution over the 16 folds, we show that it is possible to work with a subset of the folds in many use cases (e.g. to save computational resources). In a case study, we investigate the growth of the vocabulary (as well as the number of hapax legomena) as more and more folds are included into the analysis. We cross-combine this with several cleaning stages of the dataset. We also give some guidance in the form of Python, R and Stata markdown scripts on how to work with the resource.

n-grams

frequency

dataset

German

vocabulary growth

In corpus-linguistic studies, full-text corpora are the most general resource in the sense that all the information that might become relevant during a research endeavor can be derived, especially all measures that rely on sequential information. However, a lot of interesting language resources that we want to include in corpora are subject to restrictions from national copyright law or licenses. The latter is (often) a matter of negotiations between licensees and licensors and might even differ between text sources (Kupietz et al., 2018). This not only complicates things considerably when compiling corpora but is also an obstacle for efforts regarding standards of open science (Marsden, 2019). In other words: in an ideal world shaped for corpus-linguistic research, we would be able to distribute all corpus resources freely for everyone who wants to do original research or replicate findings based on these resources.

Since – unfortunately – we do not live in this ideal world, the corpus-linguistic community often has to compromise when distributing corpora. For example, we might prevent access to the full-text corpora themselves and give users the opportunity to access them via corpus platforms that are designed to search for specific patterns (Kupietz et al., 2022) and allow for a specified set of analyses and the extraction of small parts of the corpus (e.g., keyword-in-context outputs). While opening many linguistic research avenues, there are, however, also many large-scale corpus-linguistic procedures (e.g., calculations of transition probabilities for all 2-grams in a corpus) where access paths through corpus platforms are not sufficient.

For such applications, n-gram^[1] frequency lists are another possibility to allow researchers to leverage large-scale corpora. Maybe the best-known of such lists are the Google Books corpora (Michel et al., 2011), which come with their own restrictions like a frequency threshold and other issues (cf. Koplenig, 2015; Pechenick et al., 2015; Schmidt et al., 2021). Frequency lists enable researchers to devise a range of measures (for example, the transitional probabilities mentioned above or more sophisticated language models based on n-grams) and can be used to train word-level machine learning models with a fixed context (Jurafsky & Martin, 2021). Basically, frequency lists become relevant whenever all items contained in the corpus need to be factored into the analyses. Frequency lists can also inform joint research with other linguistic disciplines. Word frequencies (or derived measures) might become relevant in choosing stimuli for psycholinguistic studies (e.g., if the corpus frequency of experimental items should be held constant or manipulated in an experimental setting). They are also being used as covariates or predictors for behavioral data such as eye movements during reading (Frisson et al., 2005; Kliegl et al., 2004) or ERP data (Hauk & Pulvermüller, 2004; Hendrix et al., 2017).

Here, we introduce new (up to 3-gram) frequency lists based on the German Reference Corpus (Deutsches Referenzkorpus, DeReKo, Kupietz et al., 2010) that contain lemma, POS and frequency information from a corpus of around 43.2 billion tokens. We will first describe which parts of DeReKo provide the basis for the frequency list and introduce the data structure. We will then evaluate the distribution of data over the 16 parts (henceforth “folds”, Section 3) and present a case study (Section 4) on vocabulary growth given several cleaning stages of the dataset. Further details and sample code for using the dataset are available at https://www.owid.de/plus/derekogram/. With this accompanying webpage, we would like to facilitate the work with DeReKoGram for as many researchers as possible. We give pointers on how to back-translate the integer codes (see Section 2 for an explanation) to human-readable wordforms and lemmas, aggregate, lower (i.e. transform all characters to lower-case), and clean the dataset and search for specific patterns based on a linguistic example from Wolfer & Hein (2022). For Python, we also show how to train smoothed n-gram language models with DeReKoGram. For Stata and R code for another linguistic application please see the supplementary material of Koplenig & Wolfer & Kupietz (2022).

DeReKo currently contains around 50 billion tokens and comprises a multitude of genres, such as (for the most part) newspaper[2] texts, fiction, or specialized texts. The corpus currently grows by around 3 billion words per year (Kupietz et al., 2018). For DeReKoGram, the dataset we are introducing here, a large subset of DeReKo has been selected, tokenized with the KorAP tokenizer (Diewald et al., 2022) and tagged with the TreeTagger (Schmid, 1994). The basis of the sample is a 2020 release of DeReKo from which we excluded sub-corpora with high proportions of foreign-language passages and high proportions of non-redacted texts (e.g., material taken from Wikipedia discussions) as well as some material where publication in this format would be legally problematic. We provide 1-, 2-, and 3-gram frequency lists with and without punctuation (as identified by the TreeTagger).[3] 2- and 3-grams cross sentence boundaries (but not article boundaries)[4].

One row in each dataset consists of integer IDs for wordform and lemma, POS tag and the associated corpus frequency. Hence, 1-gram datasets have 4, 2-gram datasets 7 and 3-gram datasets 10 columns. No frequency threshold or stopword list has been employed regarding the inclusion of n-grams. The dataset is divided into 16 folds[5] where each fold consists of 1/16 of all corpus documents (randomly sampled without replacement). The data is stored using integer IDs (Brants et al., 2007), i.e. each wordform and lemma is (independently) mapped to a unique integer. This drastically reduces file sizes and allows for faster and more resource-efficient processing for research questions where the actual word form or lemma is not required. We also provide the dictionaries necessary to map integer IDs to their respective wordforms and lemmas and routines for Python, R, and Stata to use these dictionaries.[6]

DeReKoGram is available for download under a non-commercial, academic license at http://hdl.handle.net/10932/00-0630-C8F8-FA80-F401-4.

In what follows, we evaluate the underlying sampling process (Kupietz, 2023), which was carried out using the cryptographic hash function BLAKE2b (Aumasson et al., 2014) for creating the 16 folds by comparing several frequency parameters over all folds. We do so for a corpus that has not been cleaned in any way and for raw and lowered, i.e. where all characters were converted to lower case, datasets separately.

We chose five ‘direct’ measures (number of different wordforms[7], number of hapax legomena, number of different wordforms tagged as normal nouns, function words[8] as well as the ratio between the number of name/noun tokens and finite verb tokens) and two ‘derived’ measures (type-token ratio where wordforms are treated as types) and the entropy[9] of the complete frequency distribution for wordforms. For all these measures, we calculated the difference in percentages between the minimum and maximum value over the 16 folds:
Δ1-gram = (max(x₁, x₂, …, x₁₆) – min(x₁, x₂, …, x₁₆))/max(x₁, x₂, …, x₁₆); where x_n is the value of the respective measure in the nth fold (accordingly for 2-grams). Ideally, the difference percentages should be close to zero, which would indicate no difference between the corpus folds at all. We also calculated coefficients of variation (CV, also known as relative standard deviation, RSD, also reported as percentages) defined as CV = σ_x/μ_x where σ_x is the standard deviation and μ_x the mean of the respective measure over the 16 folds.

Table 1: Difference percentages on the 1-gram level (Δ1-gram, column 2 and 3) and coefficients of variation (CV, column 4 and 5) for a number of frequency measures for the 16 raw and lowered 1-gram folds. Column 6 and 7 hold information for raw 2-gram folds (column 6: difference percentages; column 7: coefficients of variation). All values are reported as percentages.

Measure	Δ1-gram		CV		2-grams (raw only)
	raw	low	raw	low	Δ2-gram	CV
Number of wordforms/2-grams	0.22	0.23	0.05	0.06	0.21	0.05
Number of hapax legomena	0.26	0.29	0.07	0.07	0.23	0.06
Number of different wordforms tagged as nouns	0.20	0.22	0.05	0.06	-
Number of different wordforms tagged as function words	1.85	1.04	0.5	0.3	-
NE+NN tokens / fin. verb tokens	0.10		0.02		-
Entropy	0.01	0.01	< 0.01	< 0.01	0.01	< 0.01
Type-token ratio	0.17	0.17	0.05	0.05	0.13	0.03

Table 1 shows that the highest difference percentages for 1-grams and the highest coefficients of variation can be observed for the number of different wordforms tagged as function words. This is presumably because the number of different wordforms of function words is indeed much smaller than, e.g., nouns. For example, for the lowered folds, the number of different wordforms ranges from 764 to 772. On the 2-gram level, the difference percentages and the coefficients of variation are also very low.

Another way to compare the folds on the 1-gram level is to compare the frequency ranks of wordforms over all 16 corpus folds. Since each wordform has been assigned a numerical code based on its token frequency rank in the overall corpus, we can correlate the codes of the first n wordforms (ranked by their frequency in each fold) with the respective first n codes in every other fold. This yields a 16-by-16 correlation matrix with Spearman's correlation coefficients ρ. Ideally, all ρs should be very close to 1, which would indicate a perfect match of frequency ranks over all corpus folds. We calculated correlation matrices with increasing values of n for raw corpus folds without any cleaning and extracted the lowest ρ (ρ_min) for each value of n.

Table 2: Lowest spearman's correlation coefficients ρ for the first n wordform codes ranked by their in-fold frequency from 16-by-16 correlation matrices correlation all folds.

First n wordforms	*Spearman's* ρ_min
100	0.9999
1,000	0.9999
10,000	0.9997
100,000	0.9975
1,000,000	0.9660

Table 2 shows that all values for ρ_min are very close to 1 but are steadily decreasing as n increases. This is because the token frequencies in higher ranks (= lower frequencies) tend to produce more ties, i.e. wordforms with equal frequencies. In other words, wordform frequencies lose distinguishing power for higher values of n. For example, in fold 4, the token frequency for all wordforms with the in-fold token frequency ranks of 995,506 to 1,018,394 (22,889 wordforms) is 28.[10] This lack of distinguishing power in lower frequency ranges consequently leads to lower values of ρ_min.

Finally, we Spearman-correlated all POS frequency distributions in the 16 folds with all other folds, again yielding a 16-by-16 correlation matrix. Here, all ρs are larger than 0.9999.

Given these comparisons, we conclude that the sampling process produced 16 homogeneous folds. Methodologically, this means that many analyses performed using all 16 parts should produce very similar results when performed using only one fold.

First, it must be noted that we do not understand vocabulary growth here in the sense of language acquisition research, but in the sense of, for example, Baayen (1996), where it is investigated how the vocabulary develops as the corpus grows. Of course, as corpus size increases, we would expect the vocabulary to grow. However, the vocabulary growth curves should differ according to the amount and type of cleaning we apply to the frequency lists because an increasing number of wordforms is being excluded from the dataset. For this case study, we apply several cleaning stages:

A) no cleaning at all;

B) exclusion of punctuation, names, and start-end-symbols (all identified via their respective POS tags), URLs, and wordforms only consisting of numbers (both identified by regular expressions);

C) exclusion of wordforms containing numbers;

D) exclusion of wordforms that contain upper-case letters following lower-case letters[11];

E) exclusion of wordforms where the TreeTagger could not assign a lemma;

F) selection of wordforms that are themselves (or the associated lemma[12]) on a basic lemma list (BLL) of New High German standard language to identify a set of conventionalized word forms (Stadler, 2014). For more information regarding this basic lemma list, please refer to Koplenig et al. (2022, p. 2).

Cleaning stages A through D are cumulative. For example, cleaning stage D incorporates stages B and C. Stages E and F, however, both rely on stage D because they can be understood as being equivalent regarding their aim: identifying 'true' lemmas and wordforms of German. We chose these cleaning stages because they represent very general selections/exclusions in potential research projects in corpus or computational linguistics. One could also think of these cleaning stages as becoming more rigorous towards the ‘core vocabulary’ of the language in each step. Of course, the datasets provided can also be used to test other selections adapted to specific research questions (e.g., only selecting certain POS or applying frequency thresholds).

4.1 Number of wordform types

We will first examine how the number of wordform types develops when including more and more of the 16 corpus folds. Figure 1 shows that the first four cleaning stages (panels A through D) exhibit roughly the same overall pattern for raw and lowered versions: The vocabulary growth curves do not show clear signs of approaching a ceiling value. This finding replicates several studies for English language corpora that are summarized by Brysbaert et al. (2016, p. 2) who also “failed to find any flattening of the predicted linear curve, indicating that the pool of possible word types was still far from exhausted”. There is, in other words, “no indication of a stop to the growth”, which is an instantiation of Herdan’s (1964) or Heaps’ (1978) law.

The final two cleaning stages (panels E and F) quickly show asymptotical behavior, but only the vocabulary growth curves for the lowered dataset (grey lines). This is especially true for cleaning stage F where we restrict the corpus to a fixed set of wordforms. For the raw corpus version, many new forms are still observed, even approaching the full corpus.

The same data can also be visualized as percentage increases as more and more folds are included (Figure 2). There is virtually no difference between the raw vs. lowered versions for the first three cleaning stages (panels A through C) and the number of observed wordforms still increases in the last step (15 to 16 folds) by approx. 4%. This is remarkably close to the figure reported by Miller and Biber (2015, p. 41) who used a corpus of ten introductory psychology textbooks and investigated the growing lexical diversity when adding whole textbooks to their sample one after another.[13]

The results for cleaning stages E and F are different: The fourth step (4 to 5 folds) in panel E shows a percentage increase of below 1% for the lowered dataset. In panel F, it's already the second step (2 to 3 folds). The lowest percentage increase is observed for the last step of lowered corpora for cleaning stage F: 0.03% (or, in absolute numbers, 157 newly observed wordforms[14] after adding the final fold). So, for cleaning stages E and F, we can conclude that the boundless growth of vocabulary is far less pronounced than in the previous cleaning stages. This makes sense given that E and F are the cleaning stages where we tried to identify a basic set of German wordforms.

4.2 Number of hapax legomena

Hapax legomena (henceforth: HL), i.e. items appearing only once in a frequency list, are often used in calculations of quantitative linguistic measures, for example in analyses concerning productivity (Baayen, 1994). It is therefore interesting to see how the number of HL is influenced by corpus size (= number of folds), cleaning stage and lowering of the dataset.

In terms of frequency, we would expect more HL in larger corpora, especially when no or rather light (cleaning stages A through D) cleaning is performed because we observe more and more 'non-canonical' wordforms as the corpus gets larger. However, it is hard to hypothesize what the pattern looks like for the last two cleaning stages E and F.

There are no considerable differences between the first four cleaning stages (panels A through D in Figure 3) and, indeed, none of the curves show any signs of approaching a point where no new HL are being added after a specific corpus size (which would be indicated by the curve approaching a horizontal asymptote). The highest number of HL is observed for the full raw corpus without any cleaning in panel A of Figure 3 (66,149,313 wordforms, 58.0% of all wordforms).

The last two cleaning stages (panels E and F), which we consider quite ‘strict’, show a different pattern compared to the first four stages. In these cleaning stages, the trajectory of the curves for raw and lowered datasets differs. For the lowered dataset, the number of HL steadily decreases, which is what we would expect given datasets where only recognized lemmas (stage E) or elements from a well-defined word list (stage F) are allowed. Consequently, the lowest number of HL is observed for the full corpus in cleaning stage F for a lowered dataset (2,205 wordforms), which make for 0.37% of all wordforms at this point.

For the raw version it is especially noteworthy that the number of HL first decreases and then increases again. In stage E, the lowest point of the curve is reached for 4 folds (239,461 HL, 8.6% of all wordforms) with steadily rising counts until the corpus is complete (304,293 HL, 9.6%). To get an idea of where this effect comes from, we can look at the HL in the complete 16-fold dataset which were not observed in the 4-fold dataset (247,607 wordforms). These 228,156 HL had to be added somewhere between the inclusion of folds 5 to 16. 51.8% of these wordforms (n = 118,076) consist of upper-case letters only.[15] Another 7.9% (n = 18,111) begin with more than one upper-case letter (e.g., COusinen or HERZstiche), also indicating irregular capitalizations of wordforms that would not be hapax legomena if capitalized in a regular way. So, irregular capitalization seems to play a large role in the increasing number of hapax legomena after the few initial folds for the final cleaning stages of raw corpora.

35,014 HL (15,3%) have an upper-case letter in word-initial position only. Most of these wordforms turn out to be sentence-initial or nominalized forms of adjectives (Flauschigsten, Abwischbarer) and verbs (Durchtauchte, Lullten) or compound nouns (Waldgesundheitsprogramm, Dampflokomotivkessel).[16] Since all the effects reported above involve capitalization, they are not observed for lowered corpora. Hence, the diverging patterns for the respective curves in panels E and F in Figure 3.

We have presented a novel frequency list dataset for German, DeReKoGram, containing all 1-, 2-, and 3-grams in a large subset (43.2 billion tokens) of the German Reference Corpus DeReKo with lemma and POS information. We have evaluated the distribution of the corpus material over the 16 folds and concluded that, at least for a number of analyses, results obtained using one fold are generalizable to the whole corpus. This should make it more convenient for people with fewer computational resources to work with the frequency lists. In a case study, we have shown how the size of the wordform vocabulary develops if more and more corpus folds are included. We cross-combined this with five different cleaning stages and a raw vs. lowered version of the dataset. We paid particular attention to the number of hapax legomena that are found in the growing datasets. To download the dataset and get started working with DeReKoGram, you can visit the OWIDplus page at https://www.owid.de/plus/derekogram/.

Funding

No external funding was received to assist with the preparation of this manuscript. The authors have no relevant financial or non-financial interests to disclose.

Data availability

The dataset generated and analyzed in this article is available from the persistent research data repository of the Leibniz Institute for the German Language. The data as well as supplementary Python, R and Stata scripts are referenced in the article.

Human participants and/or animals

The presented research did not involve human participants or animals.

Author contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Sascha Wolfer, Alexander Koplenig and Marc Kupietz. The first draft of the manuscript was written by Sascha Wolfer and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Acknowledgments

We thank our colleague Peter Fankhauser for many fruitful discussions during the initial stages of the project. We also thank Pawel Kamocki, who assisted us in clarifying the legal conditions of publishing the dataset and Denis Arnold for publishing the datasets in the IDS repository as well as Frank Michaelis for setting up the OWIDplus page.

Aumasson, J.-P., Meier, W., Phan, R. C.-W., & Henzen, L. (2014). BLAKE2. In The Hash Function BLAKE (pp. 165–183). Springer.
Baayen, R. H. (1994). Productivity in language production. Language and Cognitive Processes, 9(3), 447–469. https://doi.org/10.1080/01690969408402127
Baayen, R. H. (1996). The Effects of Lexical Specialization on the Growth Curve of the Vocabulary. Computational Linguistics, 22(4), 455–480.
Brants, T., Popat, A. C., Xu, P., Och, F. J., & Dean, J. (2007). Large Language Models in Machine Translation. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 858–867. https://aclanthology.org/D07-1090
Brysbaert, M., Stevens, M., Mandera, P., & Keuleers, E. (2016). How Many Words Do We Know? Practical Estimates of Vocabulary Size Dependent on Word Definition, the Degree of Language Input and the Participant’s Age. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.01116
Diewald, N., Kupietz, M., & Lüngen, H. (2022). Tokenizing on scale. Preprocessing large text corpora on the lexical and sentence level. In A. Klosa-Kückelhaus, S. Engelberg, C. Möhrs, & P. Storjohann (Eds.), Dictionaries and Society. Proceedings of the XX EURALEX International Congress, 12-16 July 2022, Mannheim, Germany (pp. 208–221). IDS-Verlag.
Frisson, S., Rayner, K., & Pickering, M. J. (2005). Effects of Contextual Predicability and transitional probability on eye movements during reading. Journal of Experimental Psychology / Learning, Memory & Cognition, 31(5), 862–877.
Hauk, O., & Pulvermüller, F. (2004). Effects of word length and frequency on the human event-related potential. Clinical Neurophysiology, 115(5), 1090–1103. https://doi.org/10.1016/j.clinph.2003.12.020
Heaps, H. S. (1978). Information retrieval, computational and theoretical aspects. Academic Press.
Hendrix, P., Bolger, P., & Baayen, H. (2017). Distinct ERP signatures of word frequency, phrase frequency, and prototypicality in speech production. Journal of Experimental Psychology: Learning, Memory, and Cognition, 43(1), 128–149. https://doi.org/10.1037/a0040332
Herdan, G. (1964). Quantitative linguistics. Butterworths.
Jurafsky, D., & Martin, J. H. (2021). Speech and Language Processing (3rd ed.). https://web.stanford.edu/~jurafsky/slp3/
Kliegl, R., Grabner, E., Rolfs, M., & Engbert, R. (2004). Length, frequency, and predictability effects of words on eye movements in reading. European Journal of Cognitive Psychology, 16(1/2), 262–284.
Koplenig, A. (2015). The Impact of Lacking Metadata for the Measurement of Cultural and Linguistic Change Using the Google Ngram Data Sets—Reconstructing the Composition of the German Corpus in Times of WWII. Digital Scholarship in the Humanities, fqv037. https://doi.org/10.1093/llc/fqv037
Koplenig, A., Kupietz, M., & Wolfer, S. (2022). Testing the Relationship between Word Length, Frequency, and Predictability Based on the German Reference Corpus. Cognitive Science, 46(6). https://doi.org/10.1111/cogs.13090
Kupietz, M. (2023). Totalngrams (2.3.0). Zenodo. https://doi.org/10.5281/ZENODO.7612858
Kupietz, M., Belica, C., Keibel, H., & Witt, A. (2010). The German Reference Corpus DeReKo: A primordial sample for linguistic research. In N. Calzolari, D. Tapias, M. Rosner, S. Piperidis, J. Odjik, J. Mariani, B. Maegaar, & K. Choukri (Eds.), Proceedings of the Seventh conference on International Language Resources and Evaluation. International Conference on Language Resources and Evaluation (LREC-10) (pp. 1848–1854). European Language Resources Association (ELRA).
Kupietz, M., Diewald, N., & Margaretha, E. (2022). Building paths to Corpus Data. A multi-level least effort and maximum return approach. In D. Fišer & A. Witt (Eds.), CLARIN. The infrastructure for Language Resources. De Gruyter.
Kupietz, M., Lüngen, H., Kamocki, P., & Witt, A. (2018). The German Reference Corpus DeReKo: New Developments – New Opportunities. In N. C. (Conference chair), K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, & T. Tokunaga (Eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA).
Marsden, E. (2019). Open Science and Transparency in Applied Linguistics Research. In C. A. Chapelle (Ed.), The Encyclopedia of Applied Linguistics (1st ed., pp. 1–10). Wiley. https://doi.org/10.1002/9781405198431.wbeal1493
Michel, J.-B., Shen, Y. K., Aiden, A. P., Veres, A., Gray, M. K., The Google Books Team, Pickett, J. P., Hoiberg, D., Clancy, D., Norvig, P., Orwant, J., Pinker, S., Nowak, M. A., & Aiden, E. L. (2011). Quantitative analysis of culture using millions of digitized books. Science, 331(6014), 176–182.
Miller, D., & Biber, D. (2015). Evaluating reliability in quantitative vocabulary studies: The influence of corpus design and composition. International Journal of Corpus Linguistics, 20(1), 30–53. https://doi.org/10.1075/ijcl.20.1.02mil
Pechenick, E. A., Danforth, C. M., & Dodds, P. S. (2015). Characterizing the Google Books Corpus: Strong Limits to Inferences of Socio-Cultural and Linguistic Evolution. PLOS ONE, 10(10), e0137041. https://doi.org/10.1371/journal.pone.0137041
Schiller, A., Teufel, S., Stöckert, C., & Thielen, C. (1999). Guidelines für das Tagging deutscher Textcorpora mit STTS. Institut für Maschinelle Sprachverarbeitung. http://www.sfs.uni-tuebingen.de/resources/stts-1999.pdf
Schmid, H. (1994). Probabilistic Part-of-Speech Tagging Using Decision Trees. International Conference on New Methods in Language Processing. https://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/data/tree-tagger1.pdf
Schmidt, B., Piantadosi, S. T., & Mahowald, K. (2021). Uncontrolled corpus composition drives an apparent surge in cognitive distortions. Proceedings of the National Academy of Sciences, 118(45), e2115010118. https://doi.org/10.1073/pnas.2115010118
Stadler, H. (2014). Die Erstellung der Basislemmaliste der neuhochdeutschen Standardsprache aus mehrfach linguistisch annotierten Korpora (H. Blühdorn, M. Elstermann, & A. Klosa, Eds.). Institut für Deutsche Sprache. https://nbn-resolving.org/urn:nbn:de:bsz:mh39-29999
Wolfer, S., & Hein, K. (2022). Konsequenzen der los-Suffigierung im Deutschen: Korpushäufigkeit, emotional-affektive Effekte und konstruktionsgrammatische Perspektiven. Zeitschrift Für Wortbildung / Journal of Word Formation, 6(2), 71–99. https://doi.org/10.3726/zwjw.2022.02.03

n-grams are adjacent sequences of words where n encodes the window size. 1-grams (unigrams) are single words, 2-grams (bigrams) are adjacent sequences of two words and so on.
Newspapers from most regions of German-speaking countries are included. Also, the included newspapers vary in the size of their distribution area and their political orientation.
Note that this increases the number of lexical items contained (for example) in some 3-grams. The original corpus sequence ich glaube, dass (Engl. "I believe that") counts as a 4-gram and is thus not included in the dataset with punctuation. In the dataset without punctuation, however, the 3-gram ich glaube dass is included. This might be beneficial for all analyses where punctuation is not important.
We included begin- and end-of-article markers («START» and «END»). Please refer to Koplenig et al. (2022, fn. 7) for an explanation. Note that n-grams crossing sentence boundaries can be excluded by deleting n-grams based on the POS tag $., which identifies the end of a sentence. Hence, n-grams crossing sentence boundaries cannot be excluded from the dataset without punctuation.
16 folds have the advantage (e.g., over 10 folds) that one can create 5 datasets where the size is doubled in each step by using 1, 2, 4, 8, and 16 folds.
To enable a resource-efficient approach, we also provide dictionaries for each of the 16 folds separately.
This is the number of wordform types. One may wonder why we do not evaluate the number of word form tokens. However, this would be somewhat trivial since the sampling process randomly assigned corpus documents to folds (see Section 2). The length of the documents (measured in tokens) may therefore differ only minimally between the folds. With the evaluations presented here, we aim to show that this similarity also holds for somewhat more complex measures.
Part-of-speech tags are taken from the automatic (and non-corrected) classification of the TreeTagger, which uses the Stuttgart-Tübingen-Tagset (Schiller et al., 1999). The following tags were classified as function words: APPO, APPR, APPRART, APZR, ART, KOKOM, KON, KOUI, KOUS, PAV, PDAT, PDS, PIAT, PIS, PPER, PPOSAT, PPOSS, PRELAT, PRELS, PRF, PTKA, PTKNEG, PTKZU, PWAT, PWAV, PWS.
Entropy h is defined as
$h=-\sum _{i=1}^{N}{p}_{i}*{\text{log}}_{2}{p}_{i}$
where p_i is the relative frequency of wordform i and N is the number of wordform types in the corpus fold.
To demonstrate that a similar pattern can also be observed for other folds: in fold 13, all tokens with the in-fold frequency ranks 994,655 to 1,017,581 (22,927 tokens) share a frequency of 28.
On this cleaning stage, we exclude wordforms with non-conventional capitalization (e.g., dEr) while, at the same time, keeping capitalized abbreviations (e.g., NATO). For this cleaning stage, we only use the raw version because there cannot be any difference to the lowered version.
Note that this means that, for example, the inflected wordform Weihnachtsmannes is also included although it is not on the BLL itself, but the associated lemma Weihnachtsmann is. Another example is the wordform u. which is shorthand for the lemma und.
However, this similarity might be coincidental because Miller and Biber used lemmas (not wordforms) and a much more thematically restricted corpus in their study. Also, their overall corpus is much smaller.
One of these wordforms (habilitationsprofessur) appears 5 times, and 14 twice. Note that these numbers and the wordforms themselves depend on which fold is added last to the growing dataset. Here, we simply included folds 1 to 16 subsequently.
Note that in stage D, we only excluded wordforms where upper-case letters follow at least one lower-case letter.
We made this observation by sampling 200 of these wordforms. 128 are adjectives or verbs with a word-initial upper-case letter. 62 are compound nouns (41 with two, 18 with three and 3 with four constituent parts). Also, several forms show alternative spellings of umlauts (e.g., Gegenstaendlich instead of Gegenständlich) or ß/ss alternation (e.g., Fussten or Verkehrskompaß) or simply typos (Unwahrschienlich instead of Unwahrscheinlich).

No competing interests reported.

Download PDF

Journal Publication

published 09 Nov, 2023

Read the published version in Data →

Version 1

posted

You are reading this latest preprint version

Introducing DeReKoGram: A novel frequency dataset with lemma and part- of-speech information for German

Status:

Journal Publication

Version 1

Abstract

Figures

1. Introduction

2. Data selection and structure

3. Evaluation of fold distribution

4. Case study: Vocabulary growth

4.1 Number of wordform types

4.2 Number of hapax legomena

5. Summary

Declarations

References

Footnotes

Additional Declarations

Status:

Journal Publication

Version 1