Robots txt robots txt and the role of grammar writing

Disallow: /

to

User-agent: *

Disallow: /

robots.txt written as follows:

User-agent: *

If

User-agent: *

4 if we forbade the nobles to our website baby index, in fact, with example 3, is the User-agent: header file name into Google spider aristocrat baby bot

3 if we need a search engine to grab it, for example, love Shanghai, love Shanghai banned index our website

Disallow: /

Disallow:

robots.txt written as follows:

robots.txt written as follows:

2 if we ban all search engines to access the site of all parts of the word

website search engine index We prohibited except outside our noble baby

1 allows all search engines to access the site in all parts of the words we can set up a blank text document, named root directory for the robots.txt on the site can be.

robots.txt written as follows:

User-agent: noble baby bot

6 if we ban everything except the search engine index fell in love with our overseas website

robots.txt written as follows:

User-agent: *

User-agent: noble baby bot

User-agent: Baiduspider

Disallow:

If all 5

User-agent: Baiduspider

User-agent: *

robots.txt written as follows:

 

or

Allow: /

Disallow: /

Disallow: /

7 if we need to prevent spider access to a directory, such as admin, CSS, imag> ban

Disallow:

Recommended Reading

Discuss

Your email address will not be published. Required fields are marked *