Question

I am trying to scrap some data from a website. The scripts I am trying to write, should get the content of the page:

http://www.atpworldtour.com/Rankings/Singles.aspx

Should simulate the user going trough every option for Additional Standings and the dates and simulate clicking on Go then after fetching the data should use the back function.

For now I have been trying to just select this option for Additional Standing:

            <option value="101" >101-200</option>

Here is my (poor) attempt to try to do this:

from mechanize import Browser
from BeautifulSoup import BeautifulSoup
import re
import urllib2



br = Browser();
br.open("http://www.atpworldtour.com/Rankings/Singles.aspx");
br.select_form(nr=0);
br["r"] = "101";

response = br.submit();

However it just fails on the select_form(nr=0) which should be selecting the first form.

This is the log returned by Python:

>>> from mechanize import Browser
>>>
>>> from BeautifulSoup import BeautifulSoup
>>> import re
>>> import urllib2
>>>
>>>
>>>
>>> br = Browser();
>>> br.open("http://www.atpworldtour.com/Rankings/Singles.aspx");
<response_seek_wrapper at 0x311bb48L whose wrapped object = <closeable_response
at 0x311be88L whose fp = <socket._fileobject object at 0x0000000002C94408>>>
>>> br.select_form(nr=0);
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "builddist.win-amd64eggmechanize\_mechanize.py", line 505, in select_
form
  File "builddist.win-amd64eggmechanize\_html.py", line 546, in __getattr__
  File "builddist.win-amd64eggmechanize\_html.py", line 559, in forms
  File "builddist.win-amd64eggmechanize\_html.py", line 228, in forms
mechanize._html.ParseError

I could not find a proper explanation of all the functions in the mechanize home page. Can anyone either point me to a proper tutorial for using forms and Mechanize or help me on this particular issue ?

Anthony

Answer 1

I think you are using the library correctly, but the parser seems to be having trouble with that particular page. I m using the library in the same way on another page ("http://flashcarddb.com/login") and it does not raise an error.

Answer 2

I just ran into this same problem. The page I was accessing passed W3C validation, so I didn t think it was a markup problem. However, html tidy complained that the page had a within a . Once I fixed that, mechanize started working.

Also, I saw a reply to this issue on a mailing list. I d just like to note that adding factory=mechanize.RobustFactory() to mechanize.Browser() didn t change the result.

Answer 3

Clue : define MORE about your mechanize.Browser()

友情链接