Mercurial > hg > rr-repo
annotate robots.txt @ 0:ff03f76ab3fe
initial version
| author | danieleb <danielebarchiesi@me.com> |
|---|---|
| date | Wed, 21 Aug 2013 18:51:11 +0100 |
| parents | |
| children |
| rev | line source |
|---|---|
| danielebarchiesi@0 | 1 # |
| danielebarchiesi@0 | 2 # robots.txt |
| danielebarchiesi@0 | 3 # |
| danielebarchiesi@0 | 4 # This file is to prevent the crawling and indexing of certain parts |
| danielebarchiesi@0 | 5 # of your site by web crawlers and spiders run by sites like Yahoo! |
| danielebarchiesi@0 | 6 # and Google. By telling these "robots" where not to go on your site, |
| danielebarchiesi@0 | 7 # you save bandwidth and server resources. |
| danielebarchiesi@0 | 8 # |
| danielebarchiesi@0 | 9 # This file will be ignored unless it is at the root of your host: |
| danielebarchiesi@0 | 10 # Used: http://example.com/robots.txt |
| danielebarchiesi@0 | 11 # Ignored: http://example.com/site/robots.txt |
| danielebarchiesi@0 | 12 # |
| danielebarchiesi@0 | 13 # For more information about the robots.txt standard, see: |
| danielebarchiesi@0 | 14 # http://www.robotstxt.org/wc/robots.html |
| danielebarchiesi@0 | 15 # |
| danielebarchiesi@0 | 16 # For syntax checking, see: |
| danielebarchiesi@0 | 17 # http://www.sxw.org.uk/computing/robots/check.html |
| danielebarchiesi@0 | 18 |
| danielebarchiesi@0 | 19 User-agent: * |
| danielebarchiesi@0 | 20 Crawl-delay: 10 |
| danielebarchiesi@0 | 21 # Directories |
| danielebarchiesi@0 | 22 Disallow: /includes/ |
| danielebarchiesi@0 | 23 Disallow: /misc/ |
| danielebarchiesi@0 | 24 Disallow: /modules/ |
| danielebarchiesi@0 | 25 Disallow: /profiles/ |
| danielebarchiesi@0 | 26 Disallow: /scripts/ |
| danielebarchiesi@0 | 27 Disallow: /themes/ |
| danielebarchiesi@0 | 28 # Files |
| danielebarchiesi@0 | 29 Disallow: /CHANGELOG.txt |
| danielebarchiesi@0 | 30 Disallow: /cron.php |
| danielebarchiesi@0 | 31 Disallow: /INSTALL.mysql.txt |
| danielebarchiesi@0 | 32 Disallow: /INSTALL.pgsql.txt |
| danielebarchiesi@0 | 33 Disallow: /INSTALL.sqlite.txt |
| danielebarchiesi@0 | 34 Disallow: /install.php |
| danielebarchiesi@0 | 35 Disallow: /INSTALL.txt |
| danielebarchiesi@0 | 36 Disallow: /LICENSE.txt |
| danielebarchiesi@0 | 37 Disallow: /MAINTAINERS.txt |
| danielebarchiesi@0 | 38 Disallow: /update.php |
| danielebarchiesi@0 | 39 Disallow: /UPGRADE.txt |
| danielebarchiesi@0 | 40 Disallow: /xmlrpc.php |
| danielebarchiesi@0 | 41 # Paths (clean URLs) |
| danielebarchiesi@0 | 42 Disallow: /admin/ |
| danielebarchiesi@0 | 43 Disallow: /comment/reply/ |
| danielebarchiesi@0 | 44 Disallow: /filter/tips/ |
| danielebarchiesi@0 | 45 Disallow: /node/add/ |
| danielebarchiesi@0 | 46 Disallow: /search/ |
| danielebarchiesi@0 | 47 Disallow: /user/register/ |
| danielebarchiesi@0 | 48 Disallow: /user/password/ |
| danielebarchiesi@0 | 49 Disallow: /user/login/ |
| danielebarchiesi@0 | 50 Disallow: /user/logout/ |
| danielebarchiesi@0 | 51 # Paths (no clean URLs) |
| danielebarchiesi@0 | 52 Disallow: /?q=admin/ |
| danielebarchiesi@0 | 53 Disallow: /?q=comment/reply/ |
| danielebarchiesi@0 | 54 Disallow: /?q=filter/tips/ |
| danielebarchiesi@0 | 55 Disallow: /?q=node/add/ |
| danielebarchiesi@0 | 56 Disallow: /?q=search/ |
| danielebarchiesi@0 | 57 Disallow: /?q=user/password/ |
| danielebarchiesi@0 | 58 Disallow: /?q=user/register/ |
| danielebarchiesi@0 | 59 Disallow: /?q=user/login/ |
| danielebarchiesi@0 | 60 Disallow: /?q=user/logout/ |