English 中文(简体)
针对生成的 Javarampn 内容的 Python 网络刮刮网
原标题:Python web scraping for javascript generated content
I am trying to use python3 to return the bibtex citation generated by http://www.doi2bib.org/. The url s are predictable so the script can work out the url without having to interact with the web page. I have tried using selenium, bs4, etc but cant get the text inside the box. url = "http://www.doi2bib.org/#/doi/10.1007/s00425-007-0544-9" import urllib.request from bs4 import BeautifulSoup text = BeautifulSoup(urllib.request.urlopen(url).read()) print(text) Can anyone suggest a way of returning the bibtex citation as a string (or whatever) in python?
最佳回答
You don t need BeautifulSoup here. There is an additional XHR request sent to the server to fill out the bibtex citation, simulate it, for example, with requests: import requests bibtex_id = 10.1007/s00425-007-0544-9 url = "http://www.doi2bib.org/#/doi/{id}".format(id=bibtex_id) xhr_url = http://www.doi2bib.org/doi2bib with requests.Session() as session: session.get(url) response = session.get(xhr_url, params={ id : bibtex_id}) print(response.content) Prints: @article{Burgert_2007, doi = {10.1007/s00425-007-0544-9}, url = {http://dx.doi.org/10.1007/s00425-007-0544-9}, year = 2007, month = {jun}, publisher = {Springer Science $mathplus$ Business Media}, volume = {226}, number = {4}, pages = {981--987}, author = {Ingo Burgert and Michaela Eder and Notburga Gierlinger and Peter Fratzl}, title = {Tensile and compressive stresses in tracheids are induced by swelling based on geometrical constraints of the wood cell}, journal = {Planta} } You can also solve it with selenium. The key trick here is to use an Explicit Wait to wait for the citation to become visible: from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC driver = webdriver.Firefox() driver.get( http://www.doi2bib.org/#/doi/10.1007/s00425-007-0544-9 ) element = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, //pre[@ng-show="bib"] ))) print(element.text) driver.close() Prints the same as the above solution.
问题回答

暂无回答




相关问题
Can Django models use MySQL functions?

Is there a way to force Django models to pass a field to a MySQL function every time the model data is read or loaded? To clarify what I mean in SQL, I want the Django model to produce something like ...

An enterprise scheduler for python (like quartz)

I am looking for an enterprise tasks scheduler for python, like quartz is for Java. Requirements: Persistent: if the process restarts or the machine restarts, then all the jobs must stay there and ...

How to remove unique, then duplicate dictionaries in a list?

Given the following list that contains some duplicate and some unique dictionaries, what is the best method to remove unique dictionaries first, then reduce the duplicate dictionaries to single ...

What is suggested seed value to use with random.seed()?

Simple enough question: I m using python random module to generate random integers. I want to know what is the suggested value to use with the random.seed() function? Currently I am letting this ...

How can I make the PyDev editor selectively ignore errors?

I m using PyDev under Eclipse to write some Jython code. I ve got numerous instances where I need to do something like this: import com.work.project.component.client.Interface.ISubInterface as ...

How do I profile `paster serve` s startup time?

Python s paster serve app.ini is taking longer than I would like to be ready for the first request. I know how to profile requests with middleware, but how do I profile the initialization time? I ...

Pragmatically adding give-aways/freebies to an online store

Our business currently has an online store and recently we ve been offering free specials to our customers. Right now, we simply display the special and give the buyer a notice stating we will add the ...

Converting Dictionary to List? [duplicate]

I m trying to convert a Python dictionary into a Python list, in order to perform some calculations. #My dictionary dict = {} dict[ Capital ]="London" dict[ Food ]="Fish&Chips" dict[ 2012 ]="...

热门标签