Optimize your Robots.txt WordPress

Every WordPress site must have a robots.txt file.

That file tells to the search engine robots to index or not index some parts of your site.

In a WordPress site we want to index all post but we don’t want to index the admin section (wp-admin) or core files (wp-includes).

So How to optimize your robots.txt file in WordPress?

1.

Create the robots.txt file in your root installation folder if you don’t have it already.

2.

Fill the robots.txt file with this lines:

Please notice the last line is the URL to your XML Sitemap. Create it if you don’t have it.

This lines will allow to index all your site but will disallow to index the wp-includes and the wp-admin folders.

Some WordPress developers also block the wp-content folder. It’s up to you to index that folder.

Here is my robots.txt http://php.quicoto.com/robots.txt

Do you have a robots.txt file?

Tags:

6 Responses to “Optimize your Robots.txt WordPress”

  1. Ashutosh May 3, 2010 at 11:45 am #

    I don’t have robot.txt but will include it now thanks for info.

    • quicoto May 3, 2010 at 11:48 am #

      You’re welcome.

      The most important part is where you tell add the Sitemap url.

      Thanks for your comment Ashutosh.

      Regards

  2. Alan May 7, 2010 at 3:02 am #

    Hi Quicoto, I was gladly surprised to find that you’ve become a web developer, I’m a fan of your photography blog 🙂 and it’s nice to see that you’ve chosen to walk this path.

    I use a robots.txt file a little bit more complex; it’s nothing but restrictions to google so it doesn’t index dupe content (pagination links, author and category pagination too). You’re free to check it out anytime you want 🙂

    Hope that helps.

    • quicoto May 7, 2010 at 8:25 am #

      Hey Alan,

      Thank you for your words 😉

      Would be nice to see to arround here, an other expert eye to learn from 😛

      I’m linking your robots here interesting example

      http://tecnokun.com/robots.txt

      • Alan May 8, 2010 at 4:08 am #

        Hi,

        I’m not an expert, not even close to that but I love web design/development. I’m a huge fan of blogs like this and be sure that I will be checking out new content very often 🙂

        Thanks for the kind words.
        Regards

  3. Wachira October 11, 2011 at 5:18 am #

    It very easy robots.txt code. and very easy to use.

Leave a Reply

Add <code> Some Code </code> by using this tags.