What does robots.txt file do in PHP project?
Not really PHP, see:
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it s both meaningless to you and a waste of your site s bandwidth. "Robots.txt" lets you tell Google just that.
robots.txt is related to search engines robots and not specific to PHP projects. it contains instructions for robots that crawl you website such as which urls you dont want to be crawled by the search engines and so on.
see http://www.webconfs.com/what-is-robots-txt-article-12.php
It is used to instruct webcrawlers like google and yahoo to crawl only parts of the page or not at all. See http://www.robotstxt.org for more information.
robots.txt lives in the / of your website.
When a search engine bot finds your website it first requests the robots.txt to find out which URIs it CAN and CANNOT index.
This is to prevent (it s a CONVENTION, so robots CAN ignore it) certain parts of your website from popping up in Google / Yahoo (for example your shoutbox...)
More info http://en.wikipedia.org/wiki/Robots_exclusion_standard
i have a blog build in wordpress, And my domain name is like example.com (i can t give you the original name, because some times the editors will mark this question as SPAM :( , and if any one really ...
I m developing a website that helps people understand rap lyrics. Users see the lyrics of a rap song and can click certain lyrics to see an explanation: (click here for more) As you can see, each ...
Somebody has posted an hour ago or so a question that was about the drupal search engine and was about like this: I know drupal should index anything that is returned by node_view() but this is not ...
All, This would seem like a fairly basic asp.net question - but in all my years of coding, I ve never really thought about it. Say you have a asp.net 2.0 site with only a masterpage and a default....
I am working in PHP(with Symfony Framework) and i want to create a search based on multiple values selected from multiple selection element, i.e there will be multiple selection dropdown elements for ...
I have created several Q about correctly mapping a website with a database so that google can index it properly. However, need more info. My website is a classifieds website (PHP). Users can search ...
When we visit Stack Overflow, the IE Instant Search Dropdown button is turning orange indicating that SO can be added as a search provider. What code should I put inside my web page for IE to detect ...
I am making an opensearch plugin to be used as a search engine provider in the browser. http://library.curtin.edu.au/services/toolbar/#searchPlugin As there is no way to specify any ajax call in the ...
