Question

In R, I am trying to webscrape the all working paper # (e.g, 31424, 31481, etc) of the following webpage:

https://www.nber.org/papers?facet=topics%3AFinancial%20Economics&page=1&perPage=50&sortBy=public_date

I trying to run the following code to get such:

url<-"https://www.nber.org/papers?facet=topics%3AFinancial%20Economics&page=1&perPage=50&sortBy=public_date"
page=read_html(url)
name=page%>%html_nodes(".paper-card__paper_number")%>%html_text()

However, this code returns character(0), NOT giving me the working paper # s. Is there any way I can modify this code to get the working paper # s?

Answer 1

To scrape dynamically generated content, you can use a headless browser automation tool like RSelenium, which allows you to control a real web browser programmatically. Here s how you can modify your code to achieve this:

1.First, make sure you have RSelenium and rvest installed:

install.packages("RSelenium")
install.packages("rvest")

2.Load the required libraries:

library(RSelenium)
library(rvest)

3.Start a Selenium server and open a browser:

driver <- rsDriver(browser="chrome", chromever="latest", port=4567L)
remDr <- driver[["client"]]

4.Navigate to the desired URL:

url <- "https://www.nber.org/papers?facet=topics%3AFinancial%20Economics&page=1&perPage=50&sortBy=public_date"
remDr$navigate(url)

5.Get the working paper numbers:

page_source <- remDr$getPageSource()[[1]]
page <- read_html(page_source)
name <- page %>% html_nodes(".paper-card__paper_number") %>% html_text()

6.Stop the Selenium server and close the browser:

remDr$close()
driver$server$stop()

友情链接