How does a web crawler work?
Question asked by: knowitall
A web crawler or so-called robot is a computer program that goes through websites and indexes the material on those sites.
They tend to work by recognising links on web pages, and then following those links and so on until they reach a dead end or end up at a place they have been before.
They can be sophisticated or quite simple and there are different levels of crawler you can get depending what number of links they follow and whether they stay on site or follow any link they find.
They work through pattern matching for links and then usually recursive calls to a function that fetches the material on a page linked to and then extracts links and then calls itself on those links, possibly ad nauseam.
By: Unknown
Average rating for this answer is 4 / 5
Rate Answer
Comment or provide your answer to this question
No comments have been added to this question "How does a web crawler work?".
No comments have been added to this question "How does a web crawler work?".
Ask a New Question
Find out more about Computing
Find out more about Computing
Computing Questions and Answers
search engines Questions and Answers
Next question: How can I make my website accessible?
Become a Member! It's Free >>>
Share on Facebook: 
On Twitter:
Tweet this!
Question Keywords
crawler web
More Questions:How Can I Amend The Content A User Enters In A Text Box?
What Happens If The Web Browser Does Not Have Javascript?
How Can I Make My Website Perform Better In Search Engines?
What Is HTML?
I Buy On Internet Auctions What Should I Look For?