Inside this one article, the spider that how will Gu day arrogant guide search engine to everybody introduction calls in and dominate spider action alone.

One, to the search engine refers website map

Before how letting a search engine knows our website and expedite reptile does capture come? The method has a lot of kind of course, we are for instance OK with other website advocate undertake linking exchange, search engine reptile passes the link on website of all previous the other side, crawl thereby our website. Also can all alone to search engine entrance refers network address directly, perhaps refer website map to wait to search engine. New website and link of other site exchange are not easy, because this refers website map,give search engine can yet be regarded as a good method.

The website interior structure that refers website map to be able to let very simply search engine to identify you. Search engine capture in order to go to the lavatory, accelerate index.

1. refers website map Sitemap:h to GoogleTtp://www.google.com/addurl/ ? Continue=/addurl is referred through network address management;

Each big search engine offers corresponding website government tool to give head of a station people use. For example the website government tool of Google is OK set capture frequency, refer website map, delete invalidation link.

2. to Yahoo! Refer website map Sitemap:
Through network address Http://siteexplorer.search.yahoo.com management is referred;

3. refers website map Sitemap to MSN: Refer directly with URL: Http://api.moreover.com/ping? U=http://your.domainname/sitemap.xml

4. Baidu does not support Sitemap now. But the network address that can refer you through Http://www.baidu.com/search/url_submit.html, baidu can index by oneself.

2, the search that will control each to search engine greatly through compiling Robots.txt document

The spider that search engine group comes out if not capture our website, so our website cannot get effective index, the netizen also alls alone in search without method engine searchs our website, but if be opposite website capture frequency is exorbitant, also can bring about server speed to produce a problem, large-scale capture can produce the result that is similar to virus attack even, make the visit speed of the website becomes extremely slow.

Accordingly, SEO should know the spider capture frequency that controls each search engine, among them especially the spider that Yahoo searchs is hard to control most.

The spider that how can just let Yahoo (Yahoo! Slurp) reduce capture frequency?

The product manager car of Yahoo the proposal east is Crawl-delay increases in the Robots.txt of the website: Setting, this setting is Slurp is at present particular, with will tell Slurp the spider visits the interval between 2 times, the unit is a second.

The question that leave should be installed with respect to the value that is Crawl-delay how old, it is OK that this should see website oneself susceptive is laden, assume you hope every 10 seconds come to Slurp visit, this setting is:

User-agent: *
Crawl-delay: 10

But in fact Yahoo United States and Yahoo China have 2 spiders moving, because this actual visit is possible,twice across undertakes. Fortunately every spider abides by Crawl-delay interval, accordingly to reduce Slurp pressure further, the hope reachs the channel that 10 seconds request, actual Crawl-delay configuration is about *2.

And the spider of Yahoo is distributing to go up in a lot of servers, and the capture frequency that does not coordinate same website each other also is between many servers group. Car Slurp of the respecting in Dong Zaiwen chapter originatesNet of 40 many IP paragraph, with the net paragraph the spider inside coordinates capture frequency.

Accordingly: Actual Slurp is installed, need setting is second of 10*40 = 400, every 10 seconds visit the Slurp that just can reach expectation frequency.

The setting means with other and specific Robots.txt, everybody can consult SEO200 is secret: The secret of Robots.txt file, there is very detailed introduction inside.

Bibliographical reference:

Yahoo searchs a daily record: Http://ysearchblog.cn/2006/07/yahoo_slurp.html, car east.