As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
大家好,我已经写了小型网页爬行功能。但我对多字阅读很新,无法优化它。我的代码是:
alreadySeenURLs = dict() #the dictionary of already seen crawlers
candidates = set() #the set of URL candidates to crawl
def initializeCandidates(url):
#gets page with urllib2
page = getPage(url)
#parses page with BeautifulSoup
parsedPage = getParsedPage(page)
#function which return all links from parsed page as set
initialURLsFromRoot = getLinksFromParsedPage(parsedPage)
return initialURLsFromRoot
def updateCandidates(oldCandidates, newCandidates):
return oldCandidates.union(newCandidates)
candidates = initializeCandidates(rootURL)
for url in candidates:
print len(candidates)
#fingerprint of URL
fp = hashlib.sha1(url).hexdigest()
#checking whether url is in alreadySeenURLs
if fp in alreadySeenURLs:
continue
alreadySeenURLs[fp] = url
#do some processing
print url
page = getPage(url)
parsedPage = getParsedPage(page, fix=True)
newCandidates = getLinksFromParsedPage(parsedPage)
candidates = updateCandidates(candidates, newCandidates)
正如人们所看到的,在特定的时间里,这需要来自候选人的一分骨灰。我想把这个剧本多读,这样,它至少可以从候选人中取出N URLs, 并且做这个工作。有人能指导我吗? 提供链接或建议吗?