|
A program that visits web pages and reads
their contents, usually on behalf of a search engine responding to a
request from a website owner. Once read, the information is returned to the
search engine, indexed and made generally available to the outside world. A
crawler cannot penetrate a firewall, which limits its effectiveness. |
