...Beauty, cleaning, DIY tips and more - free to join!
   Login   Contact us   Site map   Puzzle Club   Ask a question    Newsletter

How does a web crawler work?

Question asked by: knowitall

A web crawler or so-called robot is a computer program that goes through websites and indexes the material on those sites.
They tend to work by recognising links on web pages, and then following those links and so on until they reach a dead end or end up at a place they have been before.

They can be sophisticated or quite simple and there are different levels of crawler you can get depending what number of links they follow and whether they stay on site or follow any link they find.

They work through pattern matching for links and then usually recursive calls to a function that fetches the material on a page linked to and then extracts links and then calls itself on those links, possibly ad nauseam.

By: Unknown
star star star star
Average rating for this answer is 4 / 5

Rate Answer
Comment or provide your answer to this question
No comments have been added to this question "How does a web crawler work?".
Ask a New Question

Find out more about Computing

Computing Questions and Answers

search engines Questions and Answers

Next question: How can I make my website accessible?

Become a Member! It's Free >>>

Share on Facebook: On Twitter: TwitterTweet this!

Question Keywords

crawler  web  

More Questions:

How Can I Make My Website Accessible?
How Can I Get My Caps Lock Beep Back?
Is Twitter A Social Networking Site?
Why Do Robots Add Friends On Twitter?
How To Organize A Website Development Team?