Optimize your Robots.txt WordPress

Every WordPress site must have a robots.txt file.

That file tells to the search engine robots to index or not index some parts of your site.

In a WordPress site we want to index all post but we don't want to index the admin section (wp-admin) or core files (wp-includes).

So How to optimize your robots.txt file in WordPress?


Create the robots.txt file in your root installation folder if you don't have it already.


Fill the robots.txt file with this lines:

Please notice the last line is the URL to your XML Sitemap. Create it if you don't have it.

This lines will allow to index all your site but will disallow to index the wp-includes and the wp-admin folders.

Some WordPress developers also block the wp-content folder. It's up to you to index that folder.

Here is my robots.txt http://php.quicoto.com/robots.txt

Do you have a robots.txt file?

About Rick

Senior Front-end Software Engineer from Barcelona, Haidong Gumdo Instructor (korean martial art of the sword), street photographer, travel lover, TV addict, Boston Red Sox fan, and privacy advocate.
  • I don’t have robot.txt but will include it now thanks for info.

  • Hi Quicoto, I was gladly surprised to find that you’ve become a web developer, I’m a fan of your photography blog 🙂 and it’s nice to see that you’ve chosen to walk this path.

    I use a robots.txt file a little bit more complex; it’s nothing but restrictions to google so it doesn’t index dupe content (pagination links, author and category pagination too). You’re free to check it out anytime you want 🙂

    Hope that helps.

    • Hey Alan,

      Thank you for your words 😉

      Would be nice to see to arround here, an other expert eye to learn from 😛

      I’m linking your robots here interesting example


      • Hi,

        I’m not an expert, not even close to that but I love web design/development. I’m a huge fan of blogs like this and be sure that I will be checking out new content very often 🙂

        Thanks for the kind words.

  • It very easy robots.txt code. and very easy to use.

Leave a Reply

Add <code> Some Code </code> by using this tags.