Here you can find the one that will be available in the next version https://github.com/arastta/arastta/blob/1.2-dev/robots.txt

Please note that since the default robots.txt is disallowing access to the hole catalog folder the site might not pass the Mobile Friendly test at Google, since javascripts and css also is disallowed by this: https://www.google.com/webmasters/tools/mobile-friendly/?url=http%3a%2f%2fdemo.arastta.org%2f

You might like to add a allow to it, for the hole view folder:

Allow: /catalog/view/

Or if more "paranoid", restrict it to the specific scripts reported by the test, or per sub folder needed.

Adding the following at the top seems to have resolved the issue:

User-Agent: Googlebot

Allow: *.js

Allow: *.css

Probably not a bad idea Denis, but that only helps for Google then, and not Bing etc.:
https://www.bing.com/webmaster/tools/mobile-friendliness

Also the original robots.txt specification doesn't define any wildcards, so it might not be supported by all. And also on the other hand, I'm not sure it's good practise to allow crawling all js and css in all folders, including admin, should probably at least be restricted to the view folder?

Have been reading around a bit, and it seems like it would be better to add your allows to the bottom, without restricting it to Googleboot:

User-agent: *

Disallow: /admin/

Disallow: /catalog/

Disallow: /download/

Disallow: /system/

Disallow: /upload/

Disallow: /vqmod/



Allow: *.css

Allow: *.js

EDIT: Actually... I had to do the following to get the previews right in Google's Mobile Test, and no warnings (take a closer look at the report for your demo, there are still blocked resources):

User-agent: *

Disallow: /admin/

Disallow: /catalog/

Disallow: /download/

Disallow: /system/

Disallow: /upload/

Disallow: /vqmod/



Allow: /catalog/view/*.css

Allow: /catalog/view/*.js

Allow: /catalog/view/*.woff

https://www.google.com/webmasters/tools/mobile-friendly/?url=http%3a%2f%2fdemo.arastta.no%2f

But this would probably also work well, including those not reading wildcards:

User-agent: *

Disallow: /admin/

Disallow: /catalog/controller/

Disallow: /catalog/event/

Disallow: /catalog/language/

Disallow: /catalog/model/

Disallow: /catalog/view/*.php

Disallow: /catalog/view/*.tpl

Disallow: /download/

Disallow: /system/

Disallow: /upload/

Disallow: /vqmod/

The following one sounds good Rune, would you mind to open a PR for this?

User-agent: *

Disallow: /admin/

Disallow: /catalog/

Disallow: /download/

Disallow: /system/

Disallow: /upload/

Disallow: /vqmod/



Allow: /catalog/view/*.css

Allow: /catalog/view/*.js

Allow: /catalog/view/*.woff

Will do, but not yet. Still testing Bing, which is a pies of s.... in all ways. Having it's own way of reading etc., and caching even when testing...

Seems like this is the only way to make it work for most by default:

User-agent: *

Disallow: /admin/

Disallow: /catalog/

Allow: /catalog/view/javascript/

Allow: /catalog/view/theme/default/image/

Allow: /catalog/view/theme/default/javascript/

Allow: /catalog/view/theme/default/stylesheet/

Disallow: /download/

Disallow: /system/

Disallow: /upload/

Disallow: /vqmod/

Surely it will need to be adjusted when using different templates, but that's normal.

Bing doesn't understand this:

Allow: /catalog/view/*.css

Allow: /catalog/view/*.js

Allow: /catalog/view/*.woff

Nor does it understand:

Allow: /catalog/view/theme/*/image/

Allow: /catalog/view/theme/*/javascript/

Allow: /catalog/view/theme/*/stylesheet/

Also Bing honours only one set of directives, so if adding one for Bing alone all from the general has to be repeated and adjusted.
Pies of s....