search engine crawler

A program run by a search engine that downloads pages and reads their structure to index what they are about.

In more detail

Many crawlers now run JavaScript, but content present in the initial HTML is still found most reliably. Crawlers use semantic elements and headings to understand which content matters, so good structure helps both accessibility and search.

Related terms

Taught in

References