Notebook 13 – Solutions
Getting Started
Before running this notebook, select “Session > Restart R and Clear Output” in the menu above to start a new R session. This will clear any old data sets and give us a blank slate to start with.
After starting a new session, run the following code chunk to load the libraries and data that we will be working with today.
Today’s questions are more open-ended than in some of the previous notebooks.
Time API
Let’s come back to the largest cities data. This data contains a code for the latitude and longitude of each of the 81 cities:
cities <- read_csv("../data/largest_cities.csv")
cities## # A tibble: 81 × 26
## name country city_…¹ popul…² city_…³ city_…⁴ metro…⁵ metro…⁶ urban…⁷
## <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Tokyo Japan Metrop… 37.4 13.5 2191 37.3 13452 38.5
## 2 Delhi India Nation… 28.5 16.8 1484 29 3483 28.1
## 3 Shanghai China Munici… 25.6 24.2 6341 NA NA 22.1
## 4 São Paulo Brazil Munici… 21.6 12.3 1521 21.7 7947 20.9
## 5 Mexico C… Mexico City-s… 21.6 8.92 1485 20.9 7854 20.4
## 6 Cairo Egypt Urban … 20.1 9.5 3085 NA NA 16.9
## 7 Mumbai India Munici… 20.0 12.5 603 24.4 4355 23.6
## 8 Beijing China Munici… 19.6 21.7 16411 NA NA 19.4
## 9 Dhaka Bangla… Capita… 19.6 14.4 338 14.5 NA 18.6
## 10 Osaka Japan Design… 19.3 2.72 225 19.3 13228 17.2
## # … with 71 more rows, 17 more variables: urban_area <dbl>, wiki <chr>,
## # country_code2 <chr>, country_code3 <chr>, country_name_official <chr>,
## # continent <chr>, lon <dbl>, lat <dbl>, koppen_code <chr>,
## # koppen_main <chr>, city <chr>, num <dbl>, cost_of_living <dbl>,
## # cost_rent <dbl>, cost_groceries <dbl>, cost_restaurant <dbl>,
## # local_pp <dbl>, and abbreviated variable names ¹city_definition,
## # ²population, ³city_pop, ⁴city_area, ⁵metro_pop, ⁶metro_area, …The time API we saw today has an API endpoint for looking up the time zone of a particular longitude and latitude. The URL has the following elements:
protocol: https authority: www.timeapi.io path: /api/Time/current/coordinate query parameters: longitude and latitude
Use this information to build a data set with one row for each city
in cities and six columns: the city name, the longitude,
the latitude, the name of the timezone, the current datetime, and the
current hour.
# Question 01
url_base <- modify_url("https://www.timeapi.io/api/Time/current/coordinate")
n <- nrow(cities)
output <- vector("list", length = n)
for (i in seq_len(n))
{
url_str <- modify_url(url_base, query = list(
longitude = cities$lon[i], latitude = cities$lat[i]
))
res <- dsst_cache_get(url_str, cache_dir = "cache")
obj <- content(res, type = "application/json")
output[[i]] <- tibble(
name = cities$name[i],
lon = cities$lon[i],
lat = cities$lat[i],
timezone = obj$timeZone,
datetime = ymd_hms(obj$dateTime),
hour = obj$hour
)
}
output <- bind_rows(output)
output## # A tibble: 81 × 6
## name lon lat timezone datetime hour
## <chr> <dbl> <dbl> <chr> <dttm> <int>
## 1 Tokyo 140. 35.7 Asia/Tokyo 2022-11-25 08:53:01 8
## 2 Delhi 77.2 28.6 Asia/Kolkata 2022-11-25 05:23:01 5
## 3 Shanghai 121. 31.2 Asia/Shanghai 2022-11-25 07:53:01 7
## 4 São Paulo -46.6 -23.6 America/Sao_Paulo 2022-11-24 20:53:01 20
## 5 Mexico City -99.1 19.4 America/Mexico_City 2022-11-24 17:53:01 17
## 6 Cairo 31.2 30.1 Africa/Cairo 2022-11-25 01:53:01 1
## 7 Mumbai 72.8 19.0 Asia/Kolkata 2022-11-25 05:23:01 5
## 8 Beijing 116. 39.9 Asia/Shanghai 2022-11-25 07:53:02 7
## 9 Dhaka 90.4 23.8 Asia/Dhaka 2022-11-25 05:53:02 5
## 10 Osaka 136. 34.7 Asia/Tokyo 2022-11-25 08:53:02 8
## # … with 71 more rowsNow, use the data to show a visualization of the current hours of the day in different world cities:
# Question 02
output %>%
mutate(hour = factor(hour)) %>%
ggplot(aes(lon, lat)) +
geom_point(aes(color = hour), size = 3)
Aesops Fables
Similar to the XKCD API, we can grab the text of Aesops fables from read.gov through the following URL for any id between 2 and 147.
i <- 10
url_str <- modify_url(sprintf("https://www.read.gov/aesop/%03d.html", i))This is HTML format. We can parse it just as we did the CNN Lite data to get the text in paragraph codes (p) and the title in the header tag h1.
In the following code, create a docs that contains the title and text of each fable. Please ask for help; you’ll probably need it!
# Question 03
docs <- vector("list", length = 146)
for (i in seq(1, 146))
{
url_str <- modify_url(sprintf("https://www.read.gov/aesop/%03d.html", i + 1))
res <- dsst_cache_get(url_str, cache_dir = "cache")
obj <- content(res, type = "text/html", encoding = "UTF-8")
docs[[i]] <- tibble(
doc_id = xml_text(xml_find_all(obj, "..//h1")),
train_id = "train",
text = paste0(xml_text(xml_find_all(obj, "..//p")), collapse = " ")
)
}
docs <- bind_rows(docs)
docs## # A tibble: 146 × 3
## doc_id train_id text
## <chr> <chr> <chr>
## 1 The Frogs & the Ox train "An Ox came down to a reedy …
## 2 Belling the Cat train "The Mice once called a meet…
## 3 The Town Mouse & the Country Mouse train "A Town Mouse once visited a…
## 4 The Fox & the Grapes train "A Fox one day spied a beaut…
## 5 The Wolf & the Crane train "A Wolf had been feasting to…
## 6 The Lion & the Mouse train "A Lion lay asleep in the fo…
## 7 The Gnat & the Bull train "A Gnat flew over the meadow…
## 8 The Plane Tree train "Two Travellers, walking in …
## 9 The Owl & the Grasshopper train "The Owl always takes her sl…
## 10 The Oak & the Reeds train "A Giant Oak stood near a br…
## # … with 136 more rowsTwo of the fable names are repeated. I suggest removing them:
docs <- docs %>%
filter(!duplicated(doc_id))Now, create an annotation table of the fables:
# Question 04
library(cleanNLP)
cnlp_init_udpipe("english")
docs <- filter(docs, stringi::stri_length(text) > 0)
anno <- cnlp_annotate(docs)$token## Processed document 10 of 144
## Processed document 20 of 144
## Processed document 30 of 144
## Processed document 40 of 144
## Processed document 50 of 144
## Processed document 60 of 144
## Processed document 70 of 144
## Processed document 80 of 144
## Processed document 90 of 144
## Processed document 100 of 144
## Processed document 110 of 144
## Processed document 120 of 144
## Processed document 130 of 144
## Processed document 140 of 144Now, use the data to produce a UMAP plot of the nouns and verbs along with the titles of the stories. Make the font size small (2?) to allow the plot to be more readable.
# Question 05
anno %>%
filter(upos %in% c("NOUN", "VERB")) %>%
dsst_umap() %>%
ggplot(aes(v1, v2)) +
geom_point() +
geom_text_repel(aes(label = doc_id), size = 2)## as(<dgCMatrix>, "dgTMatrix") is deprecated since Matrix 1.5-0; do as(., "TsparseMatrix") instead
Finally, print out the top five words according to the G-scores for each fable. Can you tell/remember the moral based on the words?
# Question 06
anno %>%
filter(upos %in% c("NOUN", "VERB")) %>%
dsst_metrics(docs, label_var = "doc_id") %>%
filter(count > expected) %>%
group_by(label) %>%
slice_head(n = 6L) %>%
summarize(terms = paste0(token, collapse = "; ")) %>%
mutate(out = sprintf("%30 s => %s", stri_sub(label, 1, 30), terms)) %>%
getElement("out") %>%
cat(sep = "\n")## A Raven & a Swan => feather; live; diive; drown; remain; wash
## Belling the Cat => plan; mouse; cat; enemy; know; discuss
## Hercules & the Wagoner => wagon; wheel; urge; effort; farmer; move
## Jupiter & the Monkey => baby; prize; animal; pop; provide; imagine
## Mercury & the Woodman => one; pool; div; appear; silver; fortune
## The Animals & the Plague => sacrifice; belong; crime; majesty; admit; beasts
## The Ant & the Dove => straw; stone; brook; pity; float; miss
## The Ants & the Grasshopper => store; summer; music; Grasshopper; begge; shoulders
## The Ass & His Driver => mountain; lead; side; pull; path; silly
## The Ass & His Shadow => traveler; ass; heat; shade; cast; shadow
## The Ass & the Grasshoppers => Grasshopper; s; song; drink; admiration; heart
## The Ass & the Lap Dog => master; table; stable; ass; lick; face
## The Ass & the Load of Salt => load; seashore; cross; river; ass; foot
## The Ass Carrying the Image => ass; honor; head; adorn; midst; priest
## The Ass in the Lion's Skin => bray; ass; animal; frightene; hunter; keep
## The Ass, the Fox, & the Lion => ass; hurt; promise; lead; bit; fox
## The Astrologer => star; hole; mud; earth; front; spend
## The Bat & the Weasels => Weasel; bird; mouse; go; eat; enemy
## The Bear & the Bees => swarm; log; diive; happene; clover; destroy
## The Bees & Wasps, & the Hornet => honey; case; judge; belong; build; witness
## The Birds, the Beasts, & the B => battle; cause; family; race; bird; tooth
## The Boy & the Filberts => hand; get; permission; give; boy; pitcher
## The Boy & the Nettle => mother; grasp; kiss; sting; hurt; blow
## The Boys & the Frogs => pond; play; stone; trembl; water; mean
## The Bull & the Goat => cave; insult; stormy; leave; enter; prowl
## The Bundle of Sticks => stick; break; son; example; untie; mind
## The Cat & the Birds => bird; knock; medicine; doctor; pee; neighborhood
## The Cat & the Fox => hound; trick; fox; cat; one; dodge
## The Cat & the Old Rat => cat; distance; hang; keep; celebrate; heap
## The Cat, the Cock, and the You => monster; look; creature; mother; mouse; acquaintance
## The Cock & the Fox => news; wait; cock; hear; wing; bury
## The Cock & the Jewel => lose; owner; choose; scratch; deal; find
## The Crow & the Pitcher => pitcher; pebble; water; Crow; drink; thirst
## The Dog & His Master's Dinner => basket; neighborhood; dinner; dog; duty; tempt
## The Dog & His Reflection => bone; dog; think; mirror; scramble; whom
## The Dog & the Oyster => egg; groaning; he; shell; wander; pain
## The Dog in the Manger => dog; fill; asleep; awaken; behavior; snarl
## The Dog, the Cock, & the Fox => cock; awaken; roost; tree; farmyard; wood
## The Dogs & and the Hides => hide; river; water; dog; drinking; remain
## The Dogs & the Fox => tooth; story; tear; skin; chance; laugh
## The Eagle & the Beetle => Eagle; egg; lap; nest; roll; spare
## The Eagle & the Jackdaw => eagle; child; wing; fierce; fluttering; silly
## The Eagle & the Kite => eagle; provide; reply; talon; accept; like
## The Farmer & His Sons => treasure; son; spot; father; heed; find
## The Farmer & the Cranes => Crane; sling; field; stone; air; kill
## The Farmer & the Snake => snake; stiff; warm; breath; winter; bite
## The Farmer & the Stork => bird; Crane; nature; net; plead; punishment
## The Fighting Bulls and the Fro => marsh; drive; crush; fierce; trample; tremble
## The Fighting Cocks & the Eagle => farmyard; corner; fight; crawl; rival; crow
## The Fisherman & the Little Fis => fish; basket; put; catch; fry; luck
## The Flies & the Honey => honey; foot; buz; gorge; taste; invitation
## The Fox & the Crab => meadow; crawl; shell; fare; twinkling; crab
## The Fox & the Crow => Crow; cheese; beak; breakfast; bite; search
## The Fox & the Goat => jump; drink; Goat; thirsty; leape; sense
## The Fox & the Grapes => branch; hang; jump; ripe; miss; train
## The Fox & the Hedgehog => swarm; drive; disturb; struggle; fly; swim
## The Fox & the Leopard => fox; remark; rival; beauty; bushy; good
## The Fox & the Lion => speed; hair; hiding; look; time; lion
## The Fox & the Monkey => trap; ruler; meat; king; hold; animal
## The Fox & the Pheasants => limb; instant; blink; caper; patch; performance
## The Fox & the Stork => Stork; serve; dine; jar; bill; trick
## The Fox Without a Tail => tail; fox; cut; hunt; advise; brush
## The Frog & the Mouse => mouse; pond; Frog; swoop; bank; leg
## The Frogs & the Ox => puffe; monster; crush; brother; miss; burst
## The Frogs Who Wished for a Kin => king; Frogs; rule; log; send; croak
## The Gnat & the Bull => horn; bull; flew; meadow; tip; use
## The Goatherd & the Goat => horn; Goat; patch; temp; clover; stray
## The Goatherd & the Wild Goats => feed; flock; shepherd; goat; clear; hill
## The Goose & the Golden Egg => egg; countryman; day; possess; get; count
## The Hare & His Ears => horn; command; dream; ear; animal; warren
## The Hare & the Tortoise => Tortoise; race; fun; sleep; run; nap
## The Hares & the Frogs => scurry; seek; warren; flash; fright; noise
## The Heron => fish; breakfast; swam; water; fry; morsel
## The Kid & the Wolf => kid; content; grudge; heart; shelter; jeer
## The Lark & Her Young Ones => wheat; nest; grain; frightened; work; stalk
## The Leap at Rhodes => leap; city; lands; suppose; adventure; witness
## The Lion & the Ass => anger; remark; walk; flash; stroke; bray
## The Lion & the Gnat => sting; rage; claw; buz; buzze; King
## The Lion & the Mouse => mouse; net; forest; Lion; asleep; haste
## The Lion, the Ass, & the Fox => divide; pile; ass; fox; add; heap
## The Lion, the Bear, & the Fox => seize; kid; continue; dash; wound; leape
## The Lion's Share => part; count; divide; claw; claim; left
## The Man & the Lion => man; lion; forest; clearing; mind; act
## The Man & the Satyr => man; blow; comrade; friend; warm; reply
## The Mice & the Weasels => number; attack; Weasel; mouse; distinguish; ornament
## The Milkmaid & Her Pail => milk; buy; money; dress; head; egg
## The Miller, His Son, & the Ass => sell; ride; market; carry; people; boy
## The Mischievous Dog => dog; acquaintance; attract; impress; keep; visitor
## The Miser => gold; treasure; hole; piece; cry; stone
## The Mole & his Mother => mother; pebble; sense; prove; smell; bit
## The Monkey & the Camel => dance; animal; company; monkey; leg; clumsy
## The Monkey & the Cat => paw; fire; time; pull; draw; get
## The Monkey & the Dolphin => monkey; swam; indeed; take; shore; back
## The Mother & the Wolf => child; window; hear; wolf; voice; mother
## The Mouse & the Weasel => get; basket; opening; feel; gorge; groaning
## The Mule => father; feel; downhearted; feeding; prance; rest
## The North Wind & the Sun => traveler; wrap; heat; grow; bitter; ray
## The Oak & the Reeds => bow; wind; stand; oak; blow; breeze
## The Old Lion => gas; insult; King; trample; tusk; beauty
## The Old Lion & the Fox => cave; visitor; add; health; sympathy; come
## The Owl & the Grasshopper => Grasshopper; den; oak; sleep; word; tree
## The Oxen & the Wheels => wagon; complain; draw; pull; besides; compare
## The Peacock => train; Peacock; ground; feather; accustom; barnyard
## The Peacock & the Crane => spread; Peacock; barnyard; compare; freedom; glory
## The Plane Tree => shade; seek; leave; lie; receive; bear
## The Porcupine & the Snakes => snake; stay; cave; home; permission; prick
## The Quack Toad => cure; advise; fact; medicine; try; doctor
## The Rabbit, the Weasel & the C => dispute; cat; settle; home; fact; latch
## The Rat & the Elephant => procession; notice; eye; admiration; adorn; highway
## The Rose & the Butterfly => love; kiss; charme; chase; whisper; leave
## The Serpent & the Eagle => eagle; countryman; horn; rush; chance; belt
## The Sheep & the Pig => shepherd; pasture; sheep; astonish; struggle; wool
## The Shepherd & the Lion => thief; shepherd; offer; sacrifice; cave; deserv
## The Shepherd Boy & the Wolf => Wolf; forest; run; sheep; village; wolf
## The Sick Stag => strength; food; clearing; health; Stag; starve
## The Spendthrift & the Swallow => wear; man; change; haste; left; reputation
## The Stag & His Reflection => leg; asham; admire; curse; mirror; ornament
## The Stag, the Sheep, & the Wol => answer; know; sheep; measure; runner; security
## The Swallow & the Crow => swallow; Crow; feather; plumage; quill; stiff
## The Tortoise & the Ducks => Tortoise; stick; wedding; hold; see; back
## The Town Mouse & the Country M => country; mouse; city; dainty; little; food
## The Travelers & the Purse => find; road; say; lose; reply; companion
## The Travelers & the Sea => object; ship; ride; shore; treasure; traveler
## The Two Goats => goat; mountain; meet; form; means; mighty
## The Two Pots => side; step; pace; consent; urge; crack
## The Vain Jackdaw & his Borrowe => Peacock; garden; dress; feather; envy; finery
## The Wild Boar & the Fox => danger; work; grin; suffer; tusk; fun
## The Wolf & His Shadow => shadow; lair; spirit; run; appetite; fit
## The Wolf & the Ass => wolf; ass; bushes; groan; hedge; pace
## The Wolf & the Crane => bone; reward; throat; wolf; Crane; pull
## The Wolf & the Goat => cliff; grass; steep; appetite; listen; edge
## The Wolf & the House Dog => wolf; dog; difference; mark; reply; skin
## The Wolf & the Kid => kid; tune; flock; grass; song; start
## The Wolf & the Lamb => stream; lamb; brother; year; morning; highness
## The Wolf & the Lean Dog => dog; wolf; porter; eat; promise; master
## The Wolf & the Lion => property; change; lair; make; right; steal
## The Wolf & the Sheep => drink; food; sheep; hunger; suppose; thirst
## The Wolf & the Shepherd => flock; shepherd; wolf; care; sheep; errand
## The Wolf in Sheep's Clothing => skin; even; mutton; enter; stroll; cast
## The Wolf, the Kid, & the Goat => kid; door; wolf; race; paw; show
## The Wolves & the Sheep => dog; sheep; even; lurk; persuade; Wolves
## The Young Crab & His Mother => walk; crab; son; want; turn; trip
## Three Bullocks & a lion => field; attack; watch; accustom; feeding; hope
## Two Travelers & a Bear => man; climb; company; brush; savage; touch