The cleanNLP package is designed to make it as painless as possible to turn raw text into feature-rich data frames. You can download the package from within R, either directly from CRAN:
Or you may grab the development version using devtools:
As described in detail below, the package has a bare-bones parser that does not require any additional system dependencies. However, to benefit from the package most users will want to load either a Python or Java backend. The Python backend uses the spaCy module, which can be installed by running the following in a terminal:
pip install -U spacy python -m spacy download en
Further instructions are available at spacy.io.
The cleanNLP package is designed to make it as painless as possible to turn raw text into feature-rich data frames. Take for example the opening lines of Douglas Adam’s Life, the Universe and Everything:
text <- c("The regular early morning yell of horror was the sound of", "Arthur Dent waking up and suddenly remembering where he", "was. It wasn't just that the cave was cold, it wasn't just", "that it was damp and smelly. It was the fact that the cave", "was in the middle of Islington and there wasn't a bus due", "for two million years.") text <- paste(text, collapse = " ")
A minimal working example of using cleanNLP consists of loading the
package, setting up the NLP backend, initalizing the backend, and running
run_annotators. Because our input is a text string we set
TRUE (the default is to assume that we are giving the
function paths to where the input data sits on the local machine”):
library(cleanNLP) cnlp_init_spaCy() obj <- run_annotators(text, as_strings = TRUE)
Here, we used the spaCy backend. A discussion of the various backend that are available are given the following section. The returned annotation object is nothing more than a list of data frames (and one matrix), similar to a set of tables within a database. The names of these tables are:
##  "coreference" "dependency" "document" "entity" "sentence" ##  "token" "vector"
The canonical way of accessing these data frames is by using functions of
get_TABLE. For example, the document table gives metadata about
each document in the corpus, which here consists of only a single
# A tibble: 1 × 5 id time version language <int> <dttm> <chr> <chr> 1 1 2017-05-20 15:24:44 1.8.2 <NA> # ... with 1 more variables: uri <chr>
The tokens table has one row for each word in the input text, giving data
about each word such as its lemmatized form and its part of speech. We
access these table with the
# A tibble: 68 × 8 id sid tid word lemma upos pos cid <int> <int> <int> <chr> <chr> <chr> <chr> <int> 1 1 1 1 The the DET DT 0 2 1 1 2 regular regular ADJ JJ 4 3 1 1 3 early early ADJ JJ 12 4 1 1 4 morning morning NOUN NN 18 5 1 1 5 yell yell NOUN NN 26 6 1 1 6 of of ADP IN 31 7 1 1 7 horror horror NOUN NN 34 8 1 1 8 was be VERB VBD 41 9 1 1 9 the the DET DT 45 10 1 1 10 sound sound NOUN NN 49 # ... with 58 more rows
The output from the
get functions are (mostly) pre-calculated. All of the hard
work is done in the
There are three “backends” for parsing text in cleanNLP. These are:
The only benefit of the R-based version is its lack of external dependencies. We supply it mainly for testing and teaching purposes when access to the machine(s) running the code are not under our control. For most use-cases we recommend using the spaCy library as it strikes a balance between features, speed, and ease of set-up. The CoreNLP backend should be used when access to the advanced annotation tasks is required or when parsing in languages not yet available in spaCy.
To use the Python-based backend spaCy, we first need to install Python and spaCy on our system. The package reticulate must also be installed; see the spaCy website for download instructions on your platform. After this is done, we can run the code snippet given in the prior section
In order to make use of the Java-based coreNLP backend, a version of Java >= 7.0 must be installed and the rJava package must be set up. This should be straightforward, but this has caused problems on some machines, particularly with macOS. For help, see the GitHub issues tracker. Once these system requirements are met, we can install the “.jar” files with
These files are large and may take several minutes to download. The Java
backend is then configured with
setup_coreNLP_backend. The easiest
interface is to specify a speed code from 0 to 3, with higher numbers
including more models but taking increasingly long to parse the text.
Setting it equal to 2 is a good balance between time and
init_coreNLP(anno_level = 2L, lib_location = lib_loc)
After the pipeline is loaded, we again call run_annotators and set the
backend to “coreNLP” (by default run_annotators will use whichever backend
for most recently initalized, so this option is technically not
needed if you just ran
obj <- run_annotators(text, as_strings = TRUE, backend = "coreNLP") obj
## ## A CleanNLP Annotation: ## num. documents: 1
The annotation object contains the same tables as the spaCy models, with slightly different fields filled in.
Once an annotation object is created there are two ways of saving the output.
saveRDS saves the output as a binary file which can be read back into
R at any time. Alternatively, the function
write_annotation saves the annotation
as a collection of comma separated files:
od <- tempfile() write_annotation(obj, od) dir(od)
 "dependency.csv" "document.csv" "entity.csv" "token.csv"  "vector.csv"
Notice that only those tables that are non-empty are saved. These may be
read back into R as an annotation object using
anno <- read_annotation(od)
Alternatively, as these are just comma separated values, the data may be read
read.csv in R or whatever other programming language or software
a user would like to work with.
This document is meant just to get users up to speed enough to starting being useful with the coreNLP package. There are many more options and helper functions available to make working with textual data as easy as working with tabular datasets. For a full example of using the package to do an analysis of a corpus see the vignette:
For more detailed information about the fields in each of the tables, users
can consult the help pages for the
get_ functions. An even more detailed
guide to the fields and the underlying philosophy is given in the vignette
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.