Do someone has a robots.txt file for Arastta?
In SEO
Friday, December 04 2015, 03:13 PM
Share this post:
Responses (8)
  • Accepted Answer

    Wednesday, January 27 2016, 12:08 AM - #Permalink
    Seems like this is the only way to make it work for most by default:
    User-agent: *
    Disallow: /admin/
    Disallow: /catalog/
    Allow: /catalog/view/javascript/
    Allow: /catalog/view/theme/default/image/
    Allow: /catalog/view/theme/default/javascript/
    Allow: /catalog/view/theme/default/stylesheet/
    Disallow: /download/
    Disallow: /system/
    Disallow: /upload/
    Disallow: /vqmod/

    Surely it will need to be adjusted when using different templates, but that's normal.

    Bing doesn't understand this:
    Allow: /catalog/view/*.css
    Allow: /catalog/view/*.js
    Allow: /catalog/view/*.woff

    Nor does it understand:
    Allow: /catalog/view/theme/*/image/
    Allow: /catalog/view/theme/*/javascript/
    Allow: /catalog/view/theme/*/stylesheet/

    Also Bing honours only one set of directives, so if adding one for Bing alone all from the general has to be repeated and adjusted.
    Pies of s....
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, January 24 2016, 08:51 PM - #Permalink
    Will do, but not yet. Still testing Bing, which is a pies of s.... in all ways. Having it's own way of reading etc., and caching even when testing... :(
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, January 24 2016, 12:12 AM - #Permalink
    The following one sounds good Rune, would you mind to open a PR for this?
    User-agent: *
    Disallow: /admin/
    Disallow: /catalog/
    Disallow: /download/
    Disallow: /system/
    Disallow: /upload/
    Disallow: /vqmod/

    Allow: /catalog/view/*.css
    Allow: /catalog/view/*.js
    Allow: /catalog/view/*.woff
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, January 21 2016, 09:12 PM - #Permalink
    Have been reading around a bit, and it seems like it would be better to add your allows to the bottom, without restricting it to Googleboot:
    User-agent: *
    Disallow: /admin/
    Disallow: /catalog/
    Disallow: /download/
    Disallow: /system/
    Disallow: /upload/
    Disallow: /vqmod/

    Allow: *.css
    Allow: *.js


    EDIT: Actually... I had to do the following to get the previews right in Google's Mobile Test, and no warnings (take a closer look at the report for your demo, there are still blocked resources):
    User-agent: *
    Disallow: /admin/
    Disallow: /catalog/
    Disallow: /download/
    Disallow: /system/
    Disallow: /upload/
    Disallow: /vqmod/

    Allow: /catalog/view/*.css
    Allow: /catalog/view/*.js
    Allow: /catalog/view/*.woff

    https://www.google.com/webmasters/tools/mobile-friendly/?url=http%3a%2f%2fdemo.arastta.no%2f

    But this would probably also work well, including those not reading wildcards:
    User-agent: *
    Disallow: /admin/
    Disallow: /catalog/controller/
    Disallow: /catalog/event/
    Disallow: /catalog/language/
    Disallow: /catalog/model/
    Disallow: /catalog/view/*.php
    Disallow: /catalog/view/*.tpl
    Disallow: /download/
    Disallow: /system/
    Disallow: /upload/
    Disallow: /vqmod/
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, January 20 2016, 09:58 AM - #Permalink
    Probably not a bad idea Denis, but that only helps for Google then, and not Bing etc.:
    https://www.bing.com/webmaster/tools/mobile-friendliness

    Also the original robots.txt specification doesn't define any wildcards, so it might not be supported by all. And also on the other hand, I'm not sure it's good practise to allow crawling all js and css in all folders, including admin, should probably at least be restricted to the view folder?
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, January 20 2016, 09:31 AM - #Permalink
    Adding the following at the top seems to have resolved the issue:
    User-Agent: Googlebot
    Allow: *.js
    Allow: *.css
    The reply is currently minimized Show
  • Accepted Answer

    Monday, January 18 2016, 11:06 PM - #Permalink
    Please note that since the default robots.txt is disallowing access to the hole catalog folder the site might not pass the Mobile Friendly test at Google, since javascripts and css also is disallowed by this: https://www.google.com/webmasters/tools/mobile-friendly/?url=http%3a%2f%2fdemo.arastta.org%2f

    You might like to add a allow to it, for the hole view folder:
    Allow: /catalog/view/

    Or if more "paranoid", restrict it to the specific scripts reported by the test, or per sub folder needed.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, December 04 2015, 11:32 PM - #Permalink
    Here you can find the one that will be available in the next version https://github.com/arastta/arastta/blob/1.2-dev/robots.txt
    The reply is currently minimized Show
Your Reply