robots.txt’s crawl errors, as a webmaster should at least weekly check for updates about whether robots.txt can be updated normally. After we look under the "web pages" error:
The above example is
2 server, abnormal
is larger, the general will accelerate the use of CDN, but some sites not only on the "equipment" using CDN acceleration, but also used for Spider acceleration. What is the final result? If the CDN node is not stable, so to the website spider, this problem will be fatal.
in Shanghai love robots.txt update, if repeatedly click Detect and update "will appear often can be updated, but often cannot update problem. So, should not be included in the prohibition of things, robots.txt is included, and delete is normal. What is it the problem? Not excessive server load, but because of the wrong part of the Baiduspider firewall will be blacklisted.
1, wrong ban
server does not say, we all know, Kita Kamihiro is generally good. But there are some special server, most owners do not know? For example, Western Digital "Hong Kong server" is very interesting, it is Hong Kong? What kind of room itself in China, in order to avoid filing and Taiwan? With a Taiwan IP, all the data in china.
He Guijiang: once a diagnosis of qianwanji included site, but the index has always been repeatedly removed, and repeatedly included, enterprises have been unable to find. But when we are at the beginning of the site inspection found a strange phenomenon:
so what is wrong? We will find the site server through the CDN, even if it is a picture you upload, will display the "302 state", is to enhance the access speed up, but so to Shanghai dragon? Ha ha. I don’t know the West as a large domestic IDC service providers is how to use
Site , the ignorant?
3, fail to obtain the real IP
does not capture how included, how have the ranking? However such an obvious problem, there are a lot of the site will be ignored. Shanghai Longfeng diagnostic services in A5 clients here, including "20%" website will have to grab mistake, direct and serious impact on the site’s growth effect. This article, if you are fortunate enough to see, hope can be read to the end of this article, and share out, because it will be of great value.