Tagging “don't” (or really two tokens, “do” and “n't”): the verb 'do' is not considered an auxiliary in the PTB scheme in the sense of having a special tag. If it’s a present form like “I don’t do X” then the first ‘do’ is VVP and the second 'do' is VV (a base form); if it’s an imperative like “Don't go!”, it’s VV for both verbs (imperative is not considered present). The negation ‘not’ and also the form ‘n’t’ is considered adverbial (compare ‘very good’ vs. ‘not good’ – both modifiers are adverbs). As a result, it’s tagged RB.
Titles of books, films, etc.: tokens are considered NP or NPS if they are capitalized, but function words are tagged as normal. So for Starship Troopers, both words are considered ‘proper’ and tagged: Starship = NP and Troopers = NPS. But for “Beauty and the Beast” we get: NP, CC, DT, NP
In cases like “more interesting”, we have two tokens - ‘more’ itself is tagged JJR, but 'interesting' is still just a normal JJ. If you're counting comparatives in the corpus, counting JJR still gets you
When a hyphen or dash appears in a number or date range, it means (and would be pronounced as) 'to', and is therefore tagged TO.
When an error (annotated with sic in markup) is a grammatically plausible construction, tag the word as it is found in the text, rather than what it “should” be:
Misspellings are tagged as if they were correctly spelled, even if the misspelling has the form of a different word. For example, if 'too' appears in a construction where only 'to' would be grammatically conceivable, it is considered a misspelling of 'to' and tagged accordingly:
A disfluent token is tagged based on what you think it would have been had it not been disfluent:
If it's hard to know with reasonable certainty what a disfluent token was “supposed” to have been, consider using UH or SYM.
Use WDT for which, as well as that when it is used as a relative pronoun:
And use WP for what, who, and whom:
But use WDT and not WP when what is modifying a noun:
Cf. PTB guidelines for more details.
No verb should have -ing in its lemma. However, nouns ending with -ing should keep the -ing in their lemma. Some words can be both nouns and verbs; categorize them based on the specific instance.
* The pronoun "em" (e.g. we saw 'em) is tagged PP and lemmatized "they" * "kinda", when used to mean "approximately", is tagged RB and lemmatized "kinda" * "gonna", "wanna" are tokenized gon + na, wan + na and tagged using standard forms, e.g. gon/VVG/go na/TO/to
If an interrupted word can be obviously reconstructed, it is given it's normal POS tag:
If the the reconstruction is uncertain, the tag UH is used:
If a multi-word construction has been lexicalized into one word (i.e. rapidly-growing rather than rapidly growing, then it must be treated as a lexicalized adjective or noun rather than a verb. Most often, these become JJs, such as
Lexicalized nouns exist too, like
The lemmas of these words keep the gerund, i.e. egg-laying and not *egg-lay.
URL should be tagged as proper noun (NP) (effectively the name of a ‘place’)