crawler

A program run by a search engine that downloads pages and follows links to build a search index.

In more detail

Crawlers read the HTML and status code a server returns. Streaming output is indexed normally, and a not-found page returns 404 or a noindex tag so crawlers do not index it.

Related terms

Taught in

References