English 中文(简体)
• 如何阻止403和404个网站:面板和(或)复读物的错误
原标题:How to stop 403 and 404 http errors from facebook bots and/or refresh share cache

我在几周前改变我的服务器的目录结构后,从面板上抽出403和404个错误。 当然,在这种局面中,这种错误应当预料到,直到ache清为止,而且在大多数情况下都是如此。 然而,对于一组选定的股份链接,我仍然发现这一错误。 我一再试图通过相应网页的“ de”工具清除海滩,一切都看着产出是完美的,但坏要求不断提出。 (Lint tool: lint tool url。) 我的记录中有一些例子:

HTTP 查阅记录:

69.171.224.251 - - 443 [13/Jan/2012:06:22:01 -0500] "GET /web/user/images/b0/b0ahhSjq1C1oEX0TBS5gLAmcSX4wKdPT.240.jpg HTTP/1.1" 403 338

http://www.un.org

[Fri Jan 13 05:55:01 2012] [error] [client 69.171.228.249] File does not exist: /var/xxx/www/html/web/user/images/1/ab/abSIktLHDs3rcUPYyFtxsP8J9u7vvaVr.240.jpg

这些IP地址是背书。

也许,我对错误的ur弄? 我如何发现这些要求属于什么? t是否在一段时期后停止要求,并恢复其切身? 上周的第二次错误是每天25次。

(At this point I would not consider url rewriting.)

问题回答

你们应当设立301个永久方向。 或者做某种ur。 这两种方式都是行之有效的。

您也可以确信,您的报告能够渗透到耳机/cra机的用户中间。

www.un.org/Depts/DGACM/index_spanish.htm 什么时候Facebook就报废了我的网页?

Facebook needs to scrape your page to know how to display it around the site.

Facebook scrapes your page every 24 hours to ensure the properties are up to date. The page is also scraped when an admin for the Open Graph page clicks the Like button and when the URL is entered into the Facebook URL Linter. Facebook observes cache headers on your URLs - it will look at "Expires" and "Cache-Control" in order of preference. However, even if you specify a longer time, Facebook will scrape your page every 24 hours.

The user agent of the scraper is: "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"





相关问题
Logic for Implementing a Dynamic Web Scraper in C#

I am looking to develop a Web scraper in C# window forms. What I am trying to accomplish is as follows: Get the URL from the user. Load the Web page in the IE UI control(embedded browser) in ...

Building a NetHack bot: is Bayesian Analysis a good strategy?

A friend of mine is beginning to build a NetHack bot (a bot that plays the Roguelike game: NetHack). There is a very good working bot for the similar game Angband, but it works partially because of ...

Updating a rss-feed continuously

I m creating a bot in PHP that continuously updates an RSS-feed and gathers information. Every loop takes around 0.1 sec but sometimes it takes up to 9 sec to finish the cycle. Why does this happen ...

ERROR occurs in Bots open source EDI Software

I developing a very big project in which we have to use a "Bots open source EDI tranlation tool." Bots uses a pythen script to convert a edi file to specified file (i.e. xml, csv, x12, database, etc)....

Is there a list of known web crawlers? [closed]

I m trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I m not sure, they may or may not be ...

热门标签