nltk-trainer / docs / analyze_tagged_corpus.rst

Analyzing a Tagged Corpus

The analyze_tagged_corpus.py script will show the following statistics about a tagged corpus:

  • total number of words
  • number of unique words
  • number of tags
  • the number of times each tag occurs

Example output can be found in Analyzing Tagged Corpora and NLTK Part of Speech Taggers.

To analyze the treebank corpus::
python analyze_tagged_corpus.py treebank
To sort the output by tag count from highest to lowest::
python analyze_tagged_corpus.py treebank --sort count --reverse
To see simplified tags, instead of standard tags::
python analyze_tagged_corpus.py treebank --simplify_tags
To analyze a custom corpus, whose fileids end in ".pos", using a TaggedCorpusReader::
python analyze_tagged_corpus.py /path/to/corpus --reader nltk.corpus.reader.tagged.TaggedCorpusReader --fileids '.+\.pos'

The corpus path can be absolute, or relative to a nltk_data directory. For example, both corpora/treebank/tagged and /usr/share/nltk_data/corpora/treebank/tagged will work.

For a complete list of usage options::
python analyze_tagged_corpus.py --help
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.