Shanghai dragon ER must pay attention to love Shanghai crawling experience

so, a good URL rules, and the code rules, can greatly improve the level of Shanghai grab love experience, raise your chances.

A spider case analysis of Loghao

few days ago, he love Shanghai website spider status code 40% is 302? The author studied the link structure of his site, is /xxxx at the end of the visit, after a link back to automatically add / change to /xxxx/. Love Shanghai spiders access /xxxx through 302 jump to /xxxx/. Excuse me, this jump visit, love Shanghai crawling experience is how? The answer is very clear, causing many spiders visited once, if the site has 1 million pages, the spider will visit 1 million times, in order to capture target content.

what is love Shanghai crawling experience? Good code structure, good structure URL. I believe that many owners have collected a lot of CMS experience, with the capture function.

love Shanghai crawling experience and you write acquisition rules is the same, a site has good code structure, the structure of URL is good, you can easily write acquisition rules, the successful completion of the acquisition of website content.

code structure, a list of web pages and content pages, the code is very complex. There may be word editorial content, directly copied to the website editor, publish directly. A list page may be JS, iframe embedded page, code structure, how to write acquisition rules? You may directly give up this site collection. The love of spiders in Shanghai too, may plan to 10 thousand crawl, because the analysis efficiency is too low, it may be only 3000 times.

ER if Shanghai Longfeng programming experience, through the analysis of the complex code, for it is no problem to write the site collection rules. So, love Shanghai love Shanghai spider spider? How to grasp the content, can not love spiders in Shanghai for a website to write acquisition rules.


URL, a web site if the URL is very complex, after several times of 302 or 301 after the jump to reach the destination page. This URL, if you want to write rules collected by CMS, is not increased the difficulty of rule acquisition. Some URL even through the JS jump, to reach the content of the page, the URL link, ask how to write acquisition rules.

love Shanghai crawling experience, sound is a new term, in fact it and user experience, appeared very early in Shanghai dragon industry. Shanghai love crawling experience, may be lurking a word in the mind of the senior Shanghai dragon ER.

Recommended Reading


Your email address will not be published. Required fields are marked *