DM1: How search engines find and rank pages

Lesson 2 of 5 · reading 15–25 minutes; practice and project time is on the module page

Crawling: links, robots rules and status codes

🎯You'll be able to

Explain how crawlers discover pages, write a robots.txt that allows what should be crawled, and read HTTP status codes.

🏠The everyday picture

A postal worker finds houses by following roads and addresses. A "No entry" sign at the start of a lane keeps them out of every house on it, even the ones expecting letters.

📘What it really is

Crawlers (Googlebot, Bingbot) fetch a page, read its links, and add new addresses to a queue. Pages nobody links to, and that aren't in a sitemap, may not be found for a long time.
robots.txt sits at the root of a site (/robots.txt) and tells crawlers which paths not to fetch:

User-agent: *
Disallow: /checkout/
Sitemap: https://www.mittimart.example/sitemap.xml

Disallow: /product/ blocks every address starting /product/. Disallow: / blocks the whole site.
robots.txt controls crawling, not listing. A blocked page can still appear in results (without a description) if other sites link to it. To keep a page out of results, let it be crawled and use noindex (Lesson 3).
robots.txt isn't security. Anyone can read it, and bad bots ignore it. Private pages need a login.
Status codes tell the crawler what happened:
200 OK; 301 moved permanently (passes the page's signals to the new address); 302 moved for now;
404 not found and 410 gone (the page drops out of the index over time);
5xx server errors (crawlers slow down and retry; long outages can drop pages).
Broken internal links waste crawls and frustrate visitors. Redirect removed products to the nearest useful page, or remove the link.

📝A worked example

Input: MittiMart's starter robots.txt:

# STARTER: copied from the staging site (Project 3)
User-agent: *
Disallow: /product/

Steps:

Read it: every product page is blocked, and there's no Sitemap line.
Rewrite it: User-agent: *, Allow: /, and Sitemap: https://www.mittimart.example/sitemap.xml.
Run npm run test:seo and read test 4.

Expected output: test 4 no longer reports "robots.txt must not block /product/" (it will still ask for noindex on the cart, checkout and thank-you pages until those are fixed).

🌍Where it's used

Large shops block internal search results and endless filter combinations from crawling, so crawlers spend time on real products.
Site migrations use 301 redirects from every old address to its new one.
Hosting teams watch 5xx errors, because a slow or failing server also hurts crawling.

📝Case scenario

A travel site deleted 2,000 old tour pages and returned 404 for all of them. Many had links from blogs. The team redirected the 200 most-linked pages to the closest current tours with 301, and let the rest return 410. Visits from those links came back.

🖱️Try it

Open /robots.txt on three sites you use (for example a shop, a news site and a government portal). Note one path each one disallows and guess why. Success: three paths with a sensible reason for each.

⚠️Common mistakes

Copying staging rules to the live site. Sign: Disallow: / after a launch. Fix: check robots.txt in the launch checklist.
Blocking a page to remove it from results. Sign: a disallowed page still listed. Fix: allow crawling and use noindex.
Redirecting everything to the home page. Sign: all old addresses go to /. Fix: redirect each to the closest equivalent page.

🛠️For your project

Project 3's tests 4 and 6 check robots.txt and broken links; the starter blocks /product/ and links to a removed lamp page.

✅Quick check

Does Disallow keep a page out of search results?

Show the answer

Not reliably. It stops crawling; a linked page can still be listed. Use noindex on a crawlable page.

Which status code tells a crawler a page has moved for good?

Show the answer

301.