Spidering

What is it? The process of surfing the web, storing URLs and indexing keywords, links and text. Typically, even the largest search engines cannot spider all of the pages on the net. This is due to the huge amount of data available, the speed at which the new data appears, the use of politeness windows and practical limits on the number of pages that can be visited in a given time. The search engines have to make compromises in order to visit as many sites as possible, and they do this in different ways. For example some only index the home pages of each site, some only visit sites theyre explicitly told about, and some make judgments about the importance of sites (from number and quality of inbound links) before digging deeper into the subpages of a site.

Added By: Daniel

Send To Friends!

If you'd like to send the Spidering definition to yourself or to your friends/colleagues, just enter the e-mail addresses in the boxes below -

We hope you now understand the meaning of Spidering. If you need any more information on this term, please don't hesitate to contact us.