GSC status

How to fix “blocked by robots.txt”

Google can see the URL — from a sitemap or a link — and robots.txt forbids the fetch. Until that Disallow is gone, every submit is asking Googlebot to do something your file refuses.

What this status actually means

Discovery without crawl

Blocked URLs often also appear as discovered-not-indexed until you look at the robots reason. The fetch never happens, so quality and canonical tables do not apply yet.

How we test it

diagnose_robots_blocks($rules, $path, $botUa) loads the Googlebot group if present, otherwise *, then prefix-matches each non-empty Disallow. That is the same check the diagnose worker uses on tracked URLs.

Googlebot vs * — and empty Disallow

This is a fix page for a GSC status, not a robots.txt tester. The table is the Googlebot-group semantics the diagnose parser already implements.

Cause Check Fix
User-agent: * Disallow matching this path Fetch /robots.txt. For the * group, look for a Disallow whose value is a prefix of the path (/blog blocks /blog/a). Our parser treats that as blocked. Remove or narrow the Disallow. After the file is crawlable, request indexing on the URL.
User-agent: Googlebot blocks while * allows If a Googlebot group exists, Google uses that group alone. Check it even when * is wide open. Diagnose with bot UA Googlebot vs * will disagree in this case. Delete the Googlebot Disallow, or add an Allow for this path in that group. Do not assume * saved you.
Empty Disallow: (allow all) mistaken for a block A line that is exactly Disallow: with no path is allow. diagnose_robots_blocks skips it. If that is your only rule, this GSC status has another cause — a different group, or a stale crawl of an older file. Do not “fix” an empty Disallow by deleting it. Look for a Googlebot group or a leftover Disallow: /.
Disallow: / published by accident Diagnose on path / returns blocked. The whole site is off-limits, including URLs in the sitemap. Delete that line. This is the fastest way to earn a sitewide blocked-by-robots.txt report.
Path still listed in the sitemap while Disallowed Sitemap contains the href; robots.txt forbids it. Google can discover and still refuse to crawl — often adjacent to discovered, currently not indexed. Either allow the crawl or drop the URL from the sitemap. Do not submit blocked URLs.

Unblock, then ask for a fetch

1

Read the Googlebot group first

Only if it is absent do you fall back to *. Empty Disallow is allow.
2

Ship a crawlable robots.txt

Google recrawls robots.txt on its own schedule. After you change it, URL Inspection → Test live URL will show whether the block is gone.
3

Request indexing once it is allowed

Then you may leave this status for crawled-not-indexed or indexed. Related: how to submit a URL.

Common questions

Related: All indexing issues · Duplicate without user-selected canonical · Alternate page with proper canonical tag · Crawled — currently not indexed · Discovered — currently not indexed · Soft 404 · Excluded by noindex tag

Is an empty Disallow a block?

No. Disallow: with nothing after the colon means allow all for that group. Our parser skips empty Disallow values. A block requires a non-empty prefix that matches the path.

I allowed * but Googlebot is still blocked. Why?

If a User-agent: Googlebot group exists, those rules are used for Googlebot instead of the * group — they are not merged. A Googlebot Disallow can block a path that * allows.

Should I title a page “robots.txt tester” for this keyword?

No. This is a Search Console status with a specific fix. A tester is a different template. The check you need is: does the Googlebot group (or * if there is no Googlebot group) Disallow this path?

Once I unblock it, is indexing guaranteed?

No. We submit and measure. Nobody can guarantee Google indexing. Unblocking only lets Googlebot fetch. After the fetch you may land on crawled-not-indexed instead.

Submit URLs that Googlebot is actually allowed to fetch.

We record robots-blocked evidence on diagnose runs and coverage on inspect. 25 free, no card.

Start free