I am developing a rather large automation application to scrape various abandoned property information from various state databases, in order to find specific properties. I have already developed search scripts for about 8 state websites, using various forms of automation. I prefer to use something like ruby s Mechanize library to perform the automation, because it is the most stable method I have come across so far. In some cases, I am unable to automate the scraping with Mechanize and must fall back to something like Watir (or, more specifically, the branch of Watir called Vapir). Vapir is needed specifically when a source requires javascript to be searched, since Mechanize only makes HTTP requests and does not deal with JS interpretation.
My problem is with Vapir automating an instance of Internet Explorer. In some cases, after prolonged searches (some of these searches are for lists of 4,000+ search terms), IE locks up. I assume it is an issue with the OLE engine. The error I receive is as follows:
failed to create WIN32OLE object from `InternetExplorer.Application HRESULT error code:0x80004005 Unspecified error
I cannot find anything to resolve this issue.
My question is if anyone knows of any solution or work-around to an automated OLE instance that locks up? To fix the error, I have to manually kill all of the IE processes and restart the automated search.
Alternatives that I am aware of are to automate Firefox through Vapir in the back-end (rather than IE), or possibly switch over to something like PhantomJS. Does anybody have an opinion on either of these options?