Glossary
Robots.txt: Definition and Examples
Robots.txt is a plain text file placed at the root of a website that tells search engine crawlers which sections they may or may not request. It manages crawling, not indexing, so a blocked page can still appear in results if other sites link to it.
Why it matters for businesses in Port Moody
A small mistake in this file can hide an entire site from search engines. For businesses in Port Moody, reviewing it after a redesign or migration is a basic safeguard, especially when a staging site's block rules are copied to the live site.
Example
A site owner blocks the admin area and internal search results so crawlers spend their time on service pages. The file also lists the location of the XML sitemap.
Practical tips
- Check the file after every redesign, migration or platform change.
- Do not use it to hide private content, because it is publicly readable.
- Test rules in Google Search Console before publishing changes.
Can robots.txt remove a page from Google?
Not reliably. It stops crawling, but the address may remain indexed without content. To keep a page out of results, use a noindex tag or password protection, and let the page stay crawlable so the tag can be read.