No Randy, that’s not the”best practice” use for the Robots.txt file anymore.


I was wasting some time on Reddit recently when this question popped up on one of the SEO related sub-reddits, and since I have seen this situation several times recently I thought I’d share the situation and my take on it:

Hi, our website fails the mobile friendly test but when it’s searched on a mobile device the search results say that it’s mobile friendly. Our developer says that it fails the test because they have a robots.txt file in place which blocks several JS and CSS files from being indexed and that I shouldn’t be concerned since it shows up in search as mobile friendly.

Can someone explain the discrepancy and whether this will have a negative impact on organic search results?

I’d really appreciate any insight into whether I should be concerned about failing the test since it does say mobile friendly on mobile device searches. Thanks very much!

Randy, the developer said that they do have a .txt file in place which blocks some of JS and CSS from indexing and that this was normal and best practice in development. Thanks very much again for your assistance!

No, Randy. It may be “normal” to block JS & CSS with Robots.txt but it’s certainly not best practice!

“Let it crawl, let it crawl!

Let Googlebot crawl your site!

Let it crawl, let it crawl,

Then you will get indexed right!

I don’t care,

What they’re going to say,

Crawl Errors never bothered me anyways.”

Google hates blocked resources, it even has a tool in Search Console, to help you identify them. There is no SEO benefit to blocking Java and CSS. None.

Robots.txt is a paper tiger anyways. Any bot that REALY wants to crawl your site will ignore it. So the ONLY time it should be used is if there are pages you REALY do not want indexed and cannot actually remove them from existence.

Nice web, Mr.Crack Spider

Like Mr. Crack spider removed his competition…

The mobile friendly test is actually a very specific set of  four criteria, and while you do generally need to have those items on working in order to pass the test to get the badge in SERPs, they’re not one and one. Especially if the reason you’re failing is because you’re blocking resources, then you could be putting yourself unnecessarily in harm’s way.

It’s pretty damn clear, Randy.

“Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.”

Leave a comment

Your email address will not be published. Required fields are marked *