Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: Crawl-delay to stop crawler overload the server. |
|
| Author |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
|
 Posted: Fri 28 Jun '13 10:35 Post subject: Crawl-delay to stop crawler overload the server. |
 |
|
When you want that Google, Bing, Yandex etc. index your site(s) and it is overloading the server then you can use in robot.txt the Crawl-delay directive.
See for expample for an explanation: www.bing.com/blogs/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx
Steffen
From the Yandex help:
If the server is overloaded and does not have enough time to process downloading requests, use the Crawl-delay directive. It enables you to specify the minimum interval (in seconds) for a search robot to wait after downloading one page, before starting to download another. To achieve compatibility with robots that somewhat deviate from standard behaviour when processing robots.txt, the Crawl-delay directive must be added to the group that starts from the 'User-Agent' entry, right after the 'Disallow'('Allow') directive(s).
Yandex search robot supports fractional values for Crawl-Delay, e.g. 0.5. It does not mean that the search robot will access your site every half a second, but it gives the robot more freedom and may speed up the site processing. |
|
| Back to top |
|
|
|
|
|
|
