Shanghai dragon concept the difference between novice crawling and indexing

(2): duplicate content with the same content of the page may not be indexed.

index and crawling is the biggest, the index is not a specific URL crawl, but many songs in the page URL, simply means that when the search engine to determine which page index after all it will URL this page for processing and capture, in the process of grab spider will appropriate treatment according to the content of the page, a simple example, in some of the page content although there is a URL but the URL has added a NOINDEX instruction, then the search engine will not crawl the page.

(1) encoding link is Javascript, because this is called "spider trap", when crawling the site, the spider will automatically shield crawling these links.

(3): the age and other reasons to link link popularity

site pages in non indexed state:

when the search engine spiders in web crawling, the site has one and only one URL was crawling, search engines can track from other web pages crawling to the site, so the crawling is the search engine to crawl in accordance with the understanding of the specific links on the site url.

indexThe difference between

(1) containing instructions in a NOINDEX page (

(4) link in the page is a page includes a chain nofollow instruction



said although the crawling and indexing are spiders crawl the site, but they contain party >

if the novice want to know which pages have been indexed, you can use "site+ domain" approach to retrieval, by this way, the search engine will show how many pages have been cited and what page spider.

) crawling


(2) robots.txt link is marked as "no instruction.

The concept of

crawl and index are two professional terms Shanghai dragon, when many novice contact Shanghai dragon, they will be the two words mixed use, that is the index crawl. Today I will tell you carefully and the difference between some good crawling index is what

may be above abstract, we illustrate briefly, for example, in most of the time when the spider crawling site, some links in the site would not be able to capture, for example:


(3) isolated links (no one hung in the case of the sitemaps.xml, including the

(5) is supposed to capture the link, but because the server is not stable, unable to grasp.

Leave a Reply

Your email address will not be published. Required fields are marked *