How does a web crawler work?
Question asked by: knowitall
A web crawler or so-called robot is a computer program that goes through websites and indexes the material on those sites.
They tend to work by recognising links on web pages, and then following those links and so on until they reach a dead end or end up at a place they have been before.
They can be sophisticated or quite simple and there are different levels of crawler you can get depending what number of links they follow and whether they stay on site or follow any link they find.
They work through pattern matching for links and then usually recursive calls to a function that fetches the material on a page linked to and then extracts links and then calls itself on those links, possibly ad nauseam.
By: Unknown
Average rating for this answer is 4 / 5
Rate Answer
Comment or provide your answer to this question
No comments have been added to this question "How does a web crawler work?".
No comments have been added to this question "How does a web crawler work?".
Ask a New Question
Find out more about Computing
Computing Questions and Answers search engines Questions and Answers Next question: How can I make my website accessible?
Become a Member! It's Free >>>
Puzzles
Question Keywords
crawler web
More Questions:How Do I Make An Image Smaller So I Can Email It?
Is Twitter A Social Networking Site?
How Do I Remove An Old Email Address?