Exploratory Analysis
Simple summary statistics are easily computed from the table of tokens. To see the distribution of sentence length, the token table is grouped by the document and sentence id and the number of rows within each group are computed. The percentiles of these counts give a quick summary of the distribution.
anno$token %>%
group_by(doc_id, sid) %>%
summarize(sent_len = n()) %$%
quantile(sent_len, seq(0,1,0.1))## 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
## 1 10 14 18 22 26 30 36 43 56 611The median sentence has 26 tokens, whereas at least one has over 600 (this is due to a bulleted list in one of the written addresses being treated as a single sentence) To see the most frequently used nouns in the dataset, the token table is filtered on the universal part of speech field, grouped by lemma, and the number of rows in each group are once again calculated. Sorting the output and selecting the top \(42\) nouns, yields a high level summary of the topics of interest within this corpus.
anno$token %>%
filter(upos == "NOUN") %>%
group_by(lemma) %>%
summarize(count = n()) %>%
top_n(n = 42, count) %>%
arrange(desc(count)) %>%
use_series(lemma)## [1] "year" "country" "people" "law"
## [5] "time" "nation" "power" "government"
## [9] "interest" "world" "war" "citizen"
## [13] "service" "duty" "part" "system"
## [17] "peace" "right" "man" "program"
## [21] "policy" "act" "work" "condition"
## [25] "force" "subject" "legislation" "effort"
## [29] "treaty" "purpose" "land" "business"
## [33] "state" "action" "measure" "way"
## [37] "question" "relation" "consideration" "report"
## [41] "attention" "life"The result is generally as would be expected from a corpus of government speeches, with references to proper nouns representing various organizations within the government and non-proper nouns indicating general topics of interest such as “country”, “law”, and “peace”.
The length in tokens of each address is calculated similarly by grouping and summarizing at the document id level. The results can be joined with the document table to get the year of the speech and then piped in a ggplot2 command to illustrate how the length of the State of the Union has changed over time.
anno$token %>%
group_by(doc_id) %>%
summarize(n = n()) %>%
left_join(anno$document, by="doc_id") %>%
ggplot(aes(year, n)) +
geom_line(color = grey(0.8)) +
geom_point(aes(color = sotu_type)) +
geom_smooth(method="loess", formula = y ~ x) +
theme_minimal()
Here, color is used to represent whether the address was given as an oral address or a written document. The output shows that their are certainly time trends to the address length, with the form of the address (written versus spoken) also having a large effect on document length.
Finding the most used entities from the entity table over the time period of the corpus yields an alternative way to see the underlying topics. A slightly modified version of the code snippet used to find the top nouns in the dataset can be used to find the top entities.
anno$entity %>%
filter(entity_type == "LOC") %>%
group_by(entity) %>%
summarize(count = n()) %>%
top_n(n = 44, count) %>%
arrange(desc(count)) %>%
use_series(entity)## [1] "Europe" "Pacific"
## [3] "Asia" "Atlantic"
## [5] "Africa" "the Middle East"
## [7] "Central America" "South"
## [9] "West" "Earth"
## [11] "Latin America" "South America"
## [13] "East" "Mediterranean"
## [15] "District" "North"
## [17] "the Gulf of Mexico" "Western Europe"
## [19] "the Rocky Mountains" "the Pacific Ocean"
## [21] "the Western Hemisphere" "Americas"
## [23] "Gulf" "the Far East"
## [25] "the West Indies" "the Mississippi River"
## [27] "the Persian Gulf" "Governments"
## [29] "Eastern Europe" "Southeast Asia"
## [31] "Lake Erie" "Middle East"
## [33] "Caribbean" "North America"
## [35] "South Vietnam" "the Pacific Coast"
## [37] "the Southern States" "Bering Sea"
## [39] "Orient" "South Asia"
## [41] "Southwest Asia" "the Bering Sea"
## [43] "Hemisphere" "Islands"
## [45] "Mississippi" "the Caribbean Sea"
## [47] "the Indian Ocean"The ability to redo analyses from a slightly different perspective is a direct consequence of the tidy data model supplied by cleanNLP.
The top locations include some obvious and some less obvious instances. Those sovereign nations included such as Great Britain, Mexico, Germany, and Japan seem as expected given either the United State’s close ties or periods of war with them. The top states include the most populous regions but also smaller states.
One of the most straightforward way of extracting a high-level summary of the content of a speech is to extract all direct object object dependencies where the target noun is not a very common word. In order to do this for a particular speech, the dependency table is joined to the document table, a particular document is selected, and relationships of type “dobj” (direct object) are filtered out. The result is then joined to the data set word_frequency, which is included with cleanNLP, and pairs with a target occurring less than 0.5% of the time are selected to give the final result. Here is an example of this using the first address made by George W. Bush in 2001:
anno$token %>%
left_join(
anno$token,
c("doc_id"="doc_id", "sid"="sid", "tid"="tid_source"),
suffix=c("", "_source")
) %>%
left_join(anno$document, by="doc_id") %>%
filter(year == 2001) %>%
filter(relation == "dobj") %>%
select(doc_id = doc_id, start = token, word = token_source) %>%
left_join(word_frequency, by="word") %>%
filter(frequency < 0.001) %>%
select(doc_id, start, word) %$%
sprintf("%s => %s", start, word)## [1] "amount => unprecedented" "effort => recruit"
## [3] "care => lawsuits" "approach => hopeful"
## [5] "poor => disadvantaged" "meal => mentor"
## [7] "action => compassionate" "dollars => trillion"
## [9] "strategy => confront" "people => allies"
## [11] "defenses => missile" "ourselves => allies"
## [13] "ability => negotiate" "rise => drastic"
## [15] "praise => servants"Most of these phrases correspond with the ``compassionate conservatism" that George W. Bush ran under in the preceding 2000 election. Applying the same analysis to the 2002 State of the Union, which came under the shadow of the September 11th terrorist attacks, shows a drastic shift in focus.
anno$token %>%
left_join(
anno$token, c("doc_id"="doc_id", "sid"="sid", "tid"="tid_source"),
suffix=c("", "_source")
) %>%
left_join(anno$document, by="doc_id") %>%
filter(year == 2002) %>%
filter(relation == "dobj") %>%
select(doc_id = doc_id, start = token, word = token_source) %>%
left_join(word_frequency, by="word") %>%
filter(frequency < 0.001) %>%
select(doc_id, start, word) %$%
sprintf("%s => %s", start, word)## [1] "dangers => unprecedented" "debt => owe"
## [3] "terrorists => regimes" "terrorists => plotting"
## [5] "parasites => threaten" "gas => anthrax"
## [7] "gas => poison" "defenses => missile"
## [9] "America => allies" "America => allies"
## [11] "police => heroic" "police => firefighters"
## [13] "arrivals => departures" "neighborhoods => safer"
## [15] "Moutardier => attendants" "package => stimulus"
## [17] "ethic => creed" "best => emerged"
## [19] "doctors => mobilized" "efforts => recruit"
## [21] "peace => prosperity" "freedom => dignity"Here the topics have almost entirely shifted to counter-terrorism and national security efforts.
