...Beauty, cleaning, DIY tips and more - free to join!
   Login   Contact us   Site map   Puzzle Club   Ask a question    Newsletter

How does a web crawler work?

Question asked by: knowitall

A web crawler or so-called robot is a computer program that goes through websites and indexes the material on those sites.
They tend to work by recognising links on web pages, and then following those links and so on until they reach a dead end or end up at a place they have been before.

They can be sophisticated or quite simple and there are different levels of crawler you can get depending what number of links they follow and whether they stay on site or follow any link they find.

They work through pattern matching for links and then usually recursive calls to a function that fetches the material on a page linked to and then extracts links and then calls itself on those links, possibly ad nauseam.

By: Unknown
star star star star
Average rating for this answer is 4 / 5

Rate Answer
Comment or provide your answer to this question
No comments have been added to this question "How does a web crawler work?".
Ask a New Question

Find out more about Computing

Computing Questions and Answers

search engines Questions and Answers

Next question: How can I make my website accessible?

Become a Member! It's Free >>>

Share on Facebook: On Twitter: TwitterTweet this!

Question Keywords

crawler  web  

More Questions:

How Can I Sell More On Auction Sites: Tips Wanted?
What Is Googlewhacking?
How Can I Remove Unwanted Elements From An Image?
What Is Social Networking?
How Do I Know If Customers Are Finding My Website Easy To Use?