June 5, Friday near 14.00 our server based on HP Proliant (config above the average, not the top) with the regional media website on board (~ 100,000 pages in the index, up to 25,000 hosts per day, Google News, Yandex News and others. ) was droped.
The reason - a huge number of hits and downloads (primarily images) to the subnets belonging to Yandex.
Fixed a prompt, just remembering the directive Crawl Delay for robots.
During my practice (about 5 years) it was happend for the first time....
Google and Crawl-dealy
from Google http://ift.tt/1MIjlwK
via IFTTT
Google and Crawl-dealy
Google search engine
Out Of Topic Show Konversi KodeHide Konversi Kode Show EmoticonHide Emoticon