Does robots.txt keep a page out of Google?

No. Google can still list a disallowed URL it discovers through links, often with little or no snippet — and it can't see a noindex on a page it's blocked from crawling. To exclude a public page, allow crawling and use a noindex tag or X-Robots-Tag header; the change lands after the page is recrawled. For confidential content, require authentication — noindex is not access control.

What's the difference between robots.txt, noindex, and canonical?

robots.txt controls which paths compliant crawlers may request; noindex tells supporting search engines not to index a page they can crawl; a canonical annotation signals your preferred URL among duplicate or very similar pages, with the search engine making the final selection. All three can be useful on one site — but never disallow a URL whose noindex or canonical the crawler needs to read.

How do I know what's actually indexed?

Use Search Console's Page indexing report to spot patterns and URL Inspection to check specific known URLs. Neither is a complete inventory — the report covers URLs Google knows about and samples its lists — and a site: search is only a rough diagnostic. Protect private content with access control and audit it separately; an SEO tool is not a privacy check.

Can search engines discover and index the right pages?

Getting found in search is two jobs: letting crawlers discover your pages, and getting the right ones indexed while keeping the wrong ones out. A robots.txt file is one small part of that, and treating it as the whole story is how sites either hide themselves by accident or leak pages they meant to keep private.

The tools, and what each one is for

robots.txt — crawl guidance. It requests that compliant crawlers skip certain paths and points them to your sitemap. It is not a way to exclude a URL from results — a disallowed URL found through links can still be listed, and Google never sees any noindex on a page it's blocked from crawling — and it's not security: everything you "block" stays publicly reachable. Its rules also apply only to the exact protocol, hostname and port serving the file, so check every publicly reachable hostname.
XML sitemap — discovery. A list of the canonical, indexable URLs you want search engines to discover and consider crawling. A sitemap is a hint, not a guarantee of crawling or indexing. Point to it from robots.txt.
noindex — indexing exclusion. A meta tag or header that keeps a crawlable page out of supported search results — the right tool for a thank-you page or a page you've judged shouldn't be listed. It takes effect after the engine recrawls the page, applies only to crawlers that honor it, and does nothing about who can open the URL: confidential content needs authentication, not noindex.
canonical — preferred-URL signal. A <link rel="canonical"> on duplicate or very similar URLs, signaling which version you prefer. The search engine makes the final selection — keep the annotation consistent with your redirects, internal links and sitemap so the signals agree.

The classic trap: blocking what you want found

A dangerous launch mistake is carrying Disallow: / from staging into production: compliant crawlers can't fetch anything, so a new site fails to get properly indexed — while previously known URLs may linger in results with little or no snippet. Close behind: blocking CSS/JS that pages need to render. These mistakes can go unnoticed until indexing or traffic changes, so test robots.txt and inspect important URLs before and after launch. For contrast, a healthy minimal robots.txt blocks nothing and points to your sitemap:

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

And verify the basics: open /robots.txt in a private window and confirm it returns the intended plain-text file — not a login page, redirect loop or server error.

Check what's actually indexed

Don't assume your configuration worked — verify against reality. Search Console's Page indexing report shows indexing patterns, and URL Inspection shows Google's status for a specific URL you name. Use them to investigate your important pages — knowing the report only covers URLs Google has seen, and samples its lists. A site: search is a rough diagnostic at best.

Takeaway: Manage discovery and indexing as a whole — sitemap and robots.txt to guide crawling, noindex to exclude, canonicals to state your preference — then verify what Google actually holds, and never mistake any of it for access control.

What to do

Publish an XML sitemap of your canonical, indexable URLs, reference it from /robots.txt, and confirm the file answers correctly on every public hostname.
Use robots.txt only to steer crawlers away from true junk — internal search results, cart URLs — testing each rule against representative URLs first, so a broad pattern doesn't also catch useful pages or required resources. The file is public: don't advertise admin or private paths in it; protect those with authentication.
Use noindex, not Disallow, for crawlable pages that shouldn't be listed — and never combine the two on one URL, or the tag goes unseen.
Add accurate canonical annotations to duplicate or very similar URLs, and use the preferred URL consistently in internal links and the sitemap — the engine still makes the final call.
Never block CSS/JS the page needs to render.
Check reality in Search Console — the Page indexing report and URL Inspection — before and after changes, and especially right after launch.