http://forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/ News on forums, blogs, wikis, cms, etc Sat, 05 May 2012 14:44:33 +0000 hourly 1 http://wordpress.org/?v=3.3.1 http://forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/comment-page-1/#comment-3350 Ryan Sat, 29 Nov 2008 01:45:51 +0000 http://www.forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/#comment-3350 Going through my Google results I found that they were indexing quite a few pages that were the wap2 version, which I didn't want. So I added line 35 above to block those requests. We'll see whether it helps. Going through my Google results I found that they were indexing quite a few pages that were the wap2 version, which I didn’t want. So I added line 35 above to block those requests. We’ll see whether it helps.

]]> http://forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/comment-page-1/#comment-16 Ryan Thu, 14 Jun 2007 00:18:11 +0000 http://www.forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/#comment-16 Thanks for the comment granec. I will say that I haven't had much luck getting my SMF forums indexed so this is just part of what I'm trying. Thanks for the comment granec. I will say that I haven’t had much luck getting my SMF forums indexed so this is just part of what I’m trying.

]]> http://forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/comment-page-1/#comment-12 granec Mon, 11 Jun 2007 14:35:46 +0000 http://www.forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/#comment-12 Thank you for posting this! I'll give it a try and use it as a startpoint. Thank you for posting this! I’ll give it a try and use it as a startpoint.

]]> http://forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/comment-page-1/#comment-8 robots.txt for vBulletin » Forums Blogs Wikis Mon, 04 Jun 2007 14:41:02 +0000 http://www.forumsblogswikis.com/2007/06/03/my-robotstxt-file-for-simple-machines/#comment-8 [...] a short follow-up to the robots.txt for SMF posting I did yesterday, here is a similar article over at TheAdminZone.com on robots.txt for [...] [...] a short follow-up to the robots.txt for SMF posting I did yesterday, here is a similar article over at TheAdminZone.com on robots.txt for [...]

]]>