Question

我从网站下载网页列表时写了一个脚本。我经常收到以下错误( 秒数是变量) :

The bwshare module will refuse your requests for the next 7 seconds. You have downloaded data too rapidly.

I found when using sleep(2) in the loop, it works much better, however the time delay is too expensive. What s the best way how to deal with this module? Should I scrape it without any delay and if the response will be similar to the above message simply use sleep for the requested number of seconds?

Answer 1

这都取决于在错误消息之前您可以得到多少页。

尝试测量您平均可以得到多少页。

最起码须在体重共享信息之前 < 强> 4页。

如果您在 < / em> 下载到 4 页之前收到错误消息 < em>, il 将在每次下载后< em> faster 入睡(2) 。

Answer 2

这样试试... 可能会有帮助

  $requestTime = 0.1; // s/connection

    foreach(/* blah */) {
        $start = microtime(true);
        // Do your stuff to here.. get_file_content($url) and other processing .........

        if($timeTaken = microtime(true)-$start < $requestTime) {
            usleep(($requestTime-$timeTaken)*1000000);
        }
    }

如果问题解决了,请公布答案,以便其他人也受益。

友情链接