Robots.txt controls crawling
Robots.txt tells supporting crawlers which paths they may crawl on a particular host and protocol. It is not password protection and is not a universal removal mechanism. A blocked URL can still be known through links.
To exclude a public page from search, a crawlable noindex directive is usually the relevant mechanism. Truly private information needs access controls. A crawler cannot be expected to read a noindex instruction inside a page that robots.txt prevents it from fetching.
A sitemap lists preferred discoverable URLs
An XML sitemap helps search engines discover pages. Include current canonical URLs that should be indexed. Old redirects, errors, staging addresses and intentionally excluded pages can send conflicting signals.
A successfully processed sitemap does not guarantee indexing. Useful pages also need coherent internal discovery and content suited to their purpose.
Check robots.txt systematically
- Open the production file and inspect its HTTP response.
- Identify the relevant crawler group and the affected page path.
- Look for an accidental site-wide block left after testing or migration.
- Check that required rendering resources are not unnecessarily blocked.
- After editing, verify the published file rather than the editor alone.
Do not copy a universal collection of Disallow rules without understanding the site. A rule suitable for one shop may block valuable pages on another.
Check sitemap entries against real pages
Sample the homepage, a service, an article and their language versions. Open the URLs from the XML and confirm the final page is the intended one. Review canonical tags, unintended noindex directives and image URLs where included.
Look particularly for staging domains, outdated paths and missing files. A small sample can reveal a pattern before you validate the complete list automatically.
Interpret a PageSpeed timeout correctly
A robots.txt fetch timeout means that a particular test did not retrieve the resource in time. It does not establish invalid file syntax. Investigate repeatability, server behaviour, security controls and available crawl information in Search Console.
Repeated failures affecting Google itself deserve attention. One Lighthouse message, however, is not enough to explain all ranking problems on the site.
Verify the outcome
Clear the relevant cache, check actual responses and resubmit the sitemap if appropriate. Repeated daily submission is not a substitute for fixing the cause. For an excluded page, use the indexing diagnostic guide; for duplicate addresses, review canonicals.