Fork me on GitHub
973x250
  • Posted by skenow
  • Posted on 30/7/2015

Dear Webmaster, Google says "We can't access all your JS and CSS files" - Really!

Yesterday, webmasters around the globe were getting emails that looked like they were from Google, which many overlooked or treated with suspicion. I did at first - then, I signed into my Google Webmaster Tools (GWT) account and found they were, indeed, from Google.

Google robots messageYesterday, webmasters around the globe were getting emails that looked like they were from Google, which many overlooked or treated with suspicion. I did at first - then, I signed into my Google Webmaster Tools (GWT) account and found they were, indeed, from Google.

The message read-

"Google systems have recently detected an issue with your homepage that affects how well our algorithms render and index your content. Specifically, Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file. These files help Google understand that your website works properly so blocking access to these assets can result in suboptimal rankings."

[more]A little bit of searching around the internet and I found other webmasters wondering what to do - especially those that use a content management system (CMS) to power their sites. The use of a robots.txt file has been part of ImpressCMS from the beginning, so now was a good time to look it over.

Google robots message

The idea behind a robots.txt file is to tell the search bots which directories on your server should not be included in their indexes - test folders, temporary folders, cache folders, code, alternate (duplicate) versions of content. For ImpressCMS, and other CMSs, folders like /class/, /cache/, /include/, /themes/, and /templates_c/ are all folders that could get indexed if not excluded.

Well, that all changes now, at least for Google. They want access to all the files that get rendered or sent to the browser, which includes javascript, stylesheets, and images. If they cannot access those files, they will warn you, if you are using GWT. If they continue to have problems, your site and pages could find themselves dropping in the search results.

What do we do? There is a simple little addition to the bottom of the robots.txt file that will address Google's complaint -

User-agent: Googlebot
Allow: /*.js*
Allow: /*.css*



You can test your site, or any page on it, using Fetch and Render as Googlebot - https://www.google.com/webmasters/tools/googlebot-fetch

You can test your robots.txt file - https://www.google.com/webmasters/tools/robots-testing-tool

Once we get past this little update mandate from Google, we can look at what the best use of robots.txt, if indeed there is one anymore.

Just a little something you can do for your sites and clients to make them a little bit better than the masses.

*/

The comments are owned by the poster. We aren't responsible for their content.