Wordpress SEO : using robots.txt to avoid content duplication

By fiLi • Mar 10th, 2007 • Category: SEO, Wordpress

Google really doesn’t like content duplication on sites and so it is advisable to prevent the Google crawler from reaching the same content on your site from more than one url. Since Wordpress does offer many ways of reaching your content, you should block certain URL and URL paths by defining the right robots.txt.

Here’s my suggestion for the Wordpress robots.txt :

User-agent:  Googlebot
 
# Disallow all directories and files within
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
 
# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
 
# Disallow parsing individual post feeds, categories and trackbacks..
Disallow: */trackback/
Disallow: */feed/
Disallow: /category/*

Be extremely careful when implementing this. For example, some Wordpress installations have Gallery2 embedded which - for reasons unknown - likes to run with main.php in the url (even with url-rewrite enabled!). Furthermore, if your blog is in a sub-directory in your domain and you change the robots.txt for the entire domain note that you might block essential pages in other sub-directories. I imagine this is the reason why robots.txt isn’t included as part of the default wordpress installation.
As explained by my fellow bloggers who trackbacked, you also need to take care with the agents you block, and it would be wise to target bots specifically instead of using the problematic * symbol in the "user-agent" field.

Tagged as: ,

32 Responses »

  1. Make Easy Money with Google and AdSense on March 17th, 2007 at 1:52 pm:

    AdSense and robots.txt (Part 2)…

    In AdSense and robots.txt (Part 1) I described the basic syntax for the robots.txt file. Today we look at how the robots.txt file can affect your AdSense income if you’re not careful with how you declare the exclusion rules.

    Mediabot: The AdSense Cra…

  2. filination on March 18th, 2007 at 8:30 am:

    You’re absolutely right. Thanks for the helpful comments.

  3. Frostfox on March 24th, 2007 at 5:21 pm:

    I tried your suggestion about the robot.txt. I am not sure if it was the reason, but my Google page rank went from a 2 to a 4. The only other reason I can think of is that I moved my blog from the URL http://www.blog.frostfox.com to http://www.frostfox.com/blog.

  4. Lonnie on March 25th, 2007 at 2:53 am:

    Consolidating your incoming links ups your page rank score. Google.com/webmaster has some tools to help you with consolidation….

    I generally avoid any exclude tags on the site…The dangers outweigh the advantages for me…Any case studies on this one?

  5. filination on March 25th, 2007 at 3:02 am:

    Frostfox - Yeah, I believe Lonnie is right, but you should know that pagerank only updates once every 3-4 month, so it’s not something that you get immediate results on. But redirecting a few reachable urls into one, is very good practice, especially if Google penalized you for duplicate content.

    Lonnie - Yeah, there are, and those are all over the SEO blogosphere. You can start off by SEOBook and see his self-report as well as incoming trackbacks. As long as you closely monitor the robots.txt performance in Google webmasters tools, I believe you’ll be o’right.

  6. Frostfox on March 25th, 2007 at 5:51 am:

    I have been playing around with the Google webmaster thing for a bit now.
    2 things about your robot.txt, one you have a spelling mistake “indididual” should be “individual”, and isn’t disallowing files that end with .php a bad idea? Your index for your site is index.php.

  7. filination on March 25th, 2007 at 5:59 am:

    Thanks for the spelling correction :P :$

    Your question about index.php is actually what content duplication is about. Some blogs allow the exact same page to appear through index.php and their main blog path “/” and that’s something you want to avoid.

  8. Apache Gal on March 27th, 2007 at 6:00 am:

    Nice post dude.. You will want to check out Wordpress robots.txt for more examples.

  9. עבודות SEO על בלוג וורדפרס · הבלוג של אח”י דקר on March 28th, 2007 at 7:53 am:

    [...] גוגל לא אוהב טסטים כפולים ומוריד לכם את ה-PR על דפים שהוא מאנדקס מספר פעמים, כמו כן יש דפים שאתם בעצם ממש לא רוצים שגוגל יאנדקס לכם. כתוצאה מכך אתם בהחלט רוצים למנוע מגוגל להגיע לדפים שונים דרך יותר מכתובת אחת (וורדפרס דווקא מאפשר זאת) וזאת ניתן לעשות בעזרת הקובץ robots.txt. טיפים לשינוי הקובץ ניתן לקרוא בפוסט “איך להשתמש בקובץ robots.txt על מנת למנוע כפילות תוכן”. [...]

  10. fiLi on March 28th, 2007 at 6:56 pm:

    Yeah, I later found your post through various bloggers on the net (JohnTP etc.). That’s a good comprehensive post you wrote there…

  11. Content duplication, redirecting and other tips for using Feedburner feeds » fiLi’s tech on April 10th, 2007 at 4:25 am:

    [...] Using Feedburner? then you should disable your feeds from search engine indexing. After you’re done tweaking your robots.txt to avoid content duplication and sorting your .htaccess to setup all the redirects you should also take care of the content duplication happening with your Feedburner feed. [...]

  12. Mark Wilson on April 17th, 2007 at 10:27 am:

    Hey Fili - thanks for the advice; unfortunately by blocking all PHP files I stopped Google from accessing my home page (the Google Webmaster Tools said that

    ).

    I read your comment above to Frostfox, do you have any advice for dealing with the situation where http://www.markwilson.co.uk/blog/ and http://www.markwilson.co.uk/blog/index.php are actually one and the same?

    TIA, Mark

  13. fiLi on April 17th, 2007 at 1:50 pm:

    Glad I could help.

    I believe this next plugin will take care of that problem for you (and a few other duplicate content issues) :
    Permalink Redirect

  14. Mark’s (we)Blog » Removing duplicate search engine content using robots.txt on April 17th, 2007 at 3:43 pm:

    [...] following fiLi’s advice for using robots.txt to avoid content duplication, I started to edit my robots.txt file. I won’t list the file contents here - suffice to say [...]

  15. Mark Wilson on April 17th, 2007 at 4:12 pm:

    Thanks again fiLi - that plugin looks really useful. M

  16. Wordpress SEO : Using excerpt, robots.txt and noindex meta-tag for duplicate content in index, archives and categories | fiLi’s tech on May 27th, 2007 at 1:02 pm:

    [...] just about the same thing, but might be a little more tricky to set up correctly. Check out “Wordpress SEO : using robots.txt to avoid content duplication” for [...]

  17. Promoting the Israeli independent blogosphere - the Wordpress SEO review project | fiLi’s tech on May 28th, 2007 at 2:54 pm:

    [...] Robots.txt : Either robots.txt doesn’t exist at all (LT), it has something that does nothing (MKS), or it’s useless (TCI). Use robots.txt to avoid content duplication. [...]

  18. Concentrating on robots.txt specifically for Wordpress on June 8th, 2007 at 2:07 pm:

    [...] kinda like that, but it doesn’t seem to cover everything. Fili’s Tech has an article on wordpress seo for wordpress too, and I like his ideas. So I ended up with something like this: # Disallow all directories and [...]

  19. WordPress no es “google-friendly” » blogpocket 6.0 on July 1st, 2007 at 11:01 pm:

    [...] Wordpress SEO : using robots.txt to avoid content duplication [...]

  20. AskApache on July 11th, 2007 at 3:15 pm:

    Great blog and nice post Fili.. I like how you are keeping it simple, I recently changed my robots.txt from WordPress robots.txt example, to a simpler version, perhaps its time for a followup article???

  21. WordPress SEO 之 Robots.txt at 无知博客 on July 30th, 2007 at 12:22 am:

    [...] 刚才看到 filination.com上面提到的一个robots文件，并且提供了简短的解释。刚刚接触robots.txt文件的朋友，可以参考一下： User-agent: Googlebot [...]

  22. Ask Apache on August 10th, 2007 at 4:04 pm:

    Hey you should check out the Updated WordPress SEO robots.txt!

  23. How to Create an Effective Internal Linking Structure in WordPress Pt. 3 | Bracing Your Brand on January 15th, 2008 at 12:47 am:

    [...] a combination between the files on this robots.txt post at Connected Internet and this robots.txt post from Filination. If you use these you’ll manage to maximise your Page Rank on all of your [...]

  24. Bollywood Actress on February 9th, 2008 at 5:37 pm:

    This robots.txt file looks very simple and thanks for some of the points you made. Now I again have to go through all the site I visited for the robots.txt file and create my own file based on the convincing information provided by them. Hopefully, after experimenting a little I would come to know which is best for a wordpress site. Thank you for the information.

  25. Rss Feeds Txt on March 8th, 2008 at 1:49 am:

    If an RSS feed is the Yahoo backdoor, is a Blog Google’s?…

    Though the answer is in a book I wrote this July, the question is still asked of me repeatedly. Why does it work for some sites and not others? And how come some blogs get indexed in a day and then are dropped, and others stay in Google indefinitely?…

  26. Will the WordPress “All in One SEO Pack” Plugin Help or Hinder Your Blog? | SEONoobs on March 8th, 2008 at 12:35 pm:

    [...] one suggestion for the WordPress robots.txt from Fili’s Tech: User-agent: [...]

  27. Add Robots.txt to get traffic increase | Sha Money Maker on March 13th, 2008 at 1:07 am:

    [...] to the last line in the file. There has also been other bloggers such as Everton, 20 steps and FiLi who have created a Robots.txt and saw a marked increase in their blog traffic. Trust me, Google [...]

  28. ביקורת בלוגים - לטאת האמבט | הבלוג של אח"י דקר on April 7th, 2008 at 2:52 am:

    [...] robots.txt - שוב, שימוש בקובץ זה יכול לעזור להמנע מבעיות כפילות תכנים. שווה לקרוא את הפוסט של פילי בנושא. [...]

  29. The Web Robots Pages - Labpress on June 5th, 2008 at 8:11 pm:

    [...] Dica: O arquivo robots.txt controla o que os robôs dos mecanismos de busca podem ou não indexar em seu blog. Esse artigo contém o robots.txt ideal para um blog Wordpress. Leia mais… [...]

  30. Romano on July 5th, 2008 at 12:54 am:

    Many thanks for your preciuos info..
    Grazie mille and greetings from Italy.

  31. How to Setup a WordPress Blog | Niche Store Strategies on July 31st, 2008 at 1:57 pm:

    [...] WordPress SEO: using robots.txt to avoid content duplication [...]

Leave a Reply