Canonical way to determine if an entry is a lemma?

[rob-ross]

In WordNet, every LexicalEntry is a lemma. In wiktextract, a line (json object) in the file could be a lemma or not. There are many potential fields that could be examined to determine this. Is there a "cookbook" or "recipe" that answers these kinds of questions? For example, for which parts of speech does the concept of lemma/not a lemma apply? Are names (proper nouns) considered lemmas? Can compound words (hyphenate, space delimited, etc.) or phrases be lemmas? What fields, tags, keywords etc. in the json object define this categorization of a word?

Thanks!

[kristian-clausal]

We only extract data, and if there is no explicit data for whether a word is a lemma it's hard to tell what 'should' and 'shouldn't' be from the perspective of Wiktionary. There is no way to do it formally with Wiktionary data because non-lemma forms have their own entries as well. The closest you can get is by looking for form_of fields: if a word is a form of some other words, chances are that other word is the lemma.

[rob-ross]

FYI this is my current regular expression pattern for determining if a word is not a lemma. I might be too conservative with the "alt-of" tag, as here I'm only counting alt-of that are misspellings. But I think perhaps I should just include all alt-of tags.

Advice?

NOT_LEMMA_RE     =  re.compile( r'"noun form"|'
                               r'"verb form"|'
                               r'"\s(comparative|superlative)\s(adjective|adverb)[^s]"|'
                               r'"form-of"|'
                               r'"form_of"|'
                               r'"tags":\s*\["alt-of",\s*"misspelling"]|'
                               r'"tags":\s*\["abbreviation",'
                               )

[xxyzz]

I use jq to filter data that have "form-of" or "alt-of" tag: https://github.com/xxyzz/wiktionary_stardict/blob/26d9995b75d1013f33331a8bab2cf498b0d801de/en.jq#L27

[kristian-clausal]

Yeah, as someone who has tried using regexes to filter entries, I also suggest learning jq. It can be surprisingly tricky to figure out the absolutely correct syntax for certain things, but when if you succeed it can be a very powerful tool. It's basically a full-fledged (functional) programming language, after all, there's even an Exercism track... That makes you do normal, but weird things with the language.

[rob-ross]

I'm in the "getting to know the data" phase of my experience with wiktextract. So one of the things I do a lot is create subset files of the data to focus on a specific thing, like say "all pronouns that have a category entry of 'English non-lemmas'. So I'm creating a lot of extract files and I wanted to reduce the processing time as much as possible. I have learned that deserializing the json objects takes the most processing time. Iterating over lines in the 20GB file is rather fast in comparison. So whenever possible I've tried to keep the read lines as bytes objects and use Pattern[bytes]. This works well for matching the key names which are ASCII bytes (in the English version anyway.) And even when I have to do regex matching against data fields like words or glosses, etc, it's still far faster to just decode those particular fields to str and run a Pattern[str] regex on them than having to decode every line in the file. I can usually create a subset extract file in about 30 seconds doing things this way.

[xxyzz]

"alt-of" tag is not guaranteed to the first element in "tags" list, your regex pattern may mismatch some data. And jaq usually is faster than jq.