crawler
A program run by a search engine that downloads pages and follows links to build a search index.
In more detail
Crawlers read the HTML and status code a server returns. Streaming output is indexed normally, and a not-found page returns 404 or a noindex tag so crawlers do not index it.
Related terms
- SEO — Search engine optimisation: making pages easy for search engines to find, understand and rank.
- 404 Not Found — The HTTP status meaning there is no resource at the requested address.
Taught in
- Streaming SSR: Progressive HTML & Chunked Transfers — Next.js 15 & Modern App Router Architecture
- Not Found Handling: not-found.tsx & Programmatic Exits — Next.js 15 & Modern App Router Architecture