Google: "Blocked by ROBOTS.TXT"
Recently, Google has become more transparent regarding what they show in Google Search Console.
Users have reported seeing an increase in errors, specifically Blocked by ROBOTS.TXT.
What is ROBOTS.TXT?
It's simply a text file that's uploaded to your website, that tells web crawlers such as Google and Bing which portions of the website they are allowed to visit.
Which portions of the website do we tell ROBOTS.TXT not to visit?
There are quite a few, but common ones you'll see as errors or warnings in Google are:
| URL Path | Page on store |
|---|---|
/notify | Stock Notification |
/ExpressCheckout | PayPal Express Checkout |
/Feeds | Product Feeds e.g. Google Feeds |
/ForgotPwd | Customer Account: Forgot Password |
/Forms | Certain pages with forms |
/logout | Customer Account: Logout |
/MyAccount | Customer Account: Dashboard |
/preview/ | Preview Store URLs |
/CartService | Cart Service |
/cart/* | All Cart Pages |
/account/* | All Customer Account pages |
/checkout | The Checkout |
/checkout/* | All Checkout pages |
/confirm/* | Confirm Payment Page |
/unsubscribe/* | Unsubscribe from Newsletter |
/review/* | Review Product |
/SidebySideCartView | Cart Page |
/SidebySideCheckout | Start of Checkout page |
/SidebySideCheckoutForm | Checkout Form page |
/SidebySideConfirm | Confirm Payment page |
Why do we block these pages?
We typically do not want Google and other crawlers to crawl and rank these pages on their search engine.
For example, if a customer is on your store on the checkout, they will have a specific URL for their checkout page such as /cart/*. If Google did crawl this and rank it in their search engines, that link would be public, and anyone could visit the checkout and see what's in that customer's cart.
These pages offer zero SEO value to your store, and if ranked in Google you'd see a large bounce increase.
What should you do with the errors/ warnings?
In general, nothing.
It doesn't hurt your SEO or ranking at all. The rest of your store will be crawled as normal, and the pages and products that you do want to be visible on the search engines will be.
Last updated