What Is robots.txt?

A gentle explanation of what robots.txt is, including its purpose, how to write it, and how to use it.

Completed Medium
Start
August 3, 2025
End
August 3, 2025
Period
1Days
Last Updated
August 3, 2025
Tags
SEO

🔰 1. What is robots.txt?

robots.txt is like a “rulebook” placed on your website.

It tells search engine robots (crawlers) like Google which pages they’re allowed or not allowed to access.

For example:

  • Pages you’re okay with being crawled → ✅ OK
  • Pages you don’t want crawled → 🚫 STOP

You can set rules like these!

📁 2. Where Is robots.txt Located?

The robots.txt file should be placed in the root directory of your website.

For example, if your website is https://example.com then the location of the robots.txt file will be: 👉 https://example.com/robots.txt

📄 3. Basic Syntax of robots.txt Rules

You can use robots.txt to give instructions like “Don’t allow this bot to access this area.”

🧠 Common Directives:

DirectiveDescription
User-agent:Specifies which bot the rule applies to
Disallow:Blocks access to the specified path
Allow:Grants access to the specified path
Sitemap:Indicates the location of your sitemap (site map)

✏ 4. Simple Examples of How to Write It

✅ To block all crawlers from everything:

User-agent: *
Disallow: /

This means “Don’t allow any pages to be crawled.”

📂 To block only a specific folder:

User-agent: *
Disallow: /calendar/

This means “Don’t crawl the ‘calendar’ folder.”

📄 To block one specific page:

User-agent: *
Disallow: /secret.html

💡 5. Why Use robots.txt?

  • When you have pages not ready to show yet
  • When you want to keep some pages private
  • When you want to avoid unnecessary pages being crawled by Google

⚠ 6. Important Notes

❌ robots.txt does NOT guarantee privacy!

Even if you disallow a page via robots.txt, it may still appear in search results. If you want to hide a page completely, use a password or access restrictions.

🌍 It’s Public!

Since robots.txt is public, anyone can see it online. So never include confidential info or URLs in this file.

🛠 7. How to Create robots.txt

Step 1: Create a file named robots.txt

Use Notepad or any text editor and save it in UTF-8 format.

Step 2: Upload it to the root folder of your site

For example, in WordPress, this would be public_html, www, etc.

Step 3: Check the file via your browser

Go to: https://your-website.com/robots.txt to confirm the file is correctly accessible.

🧪 8. Want to Test It?

Use the robots.txt Tester in Google Search Console to verify your rules are working properly.

🧠 Summary: robots.txt is Your Site’s Gatekeeper!

  • robots.txt is a rulebook for search engine robots
  • You can use it to say “STOP!” to folders or pages you don’t want crawled
  • But it’s not suitable for hiding private content
  • It’s easy to create and totally free to use!

Reference URL:

Back to projects
Back