🔰 1. What is robots.txt?

robots.txt is like a “rulebook” placed on your website.

It tells search engine robots (crawlers) like Google which pages they’re allowed or not allowed to access.

For example:

• Pages you’re okay with being crawled → ✅ OK
• Pages you don’t want crawled → 🚫 STOP

You can set rules like these!

📁 2. Where Is robots.txt Located?

The robots.txt file should be placed in the root directory of your website.

For example, if your website is https://example.com then the location of the robots.txt file will be: 👉 https://example.com/robots.txt

📄 3. Basic Syntax of robots.txt Rules

You can use robots.txt to give instructions like “Don’t allow this bot to access this area.”

🧠 Common Directives:

✏ 4. Simple Examples of How to Write It

✅ To block all crawlers from everything:

User-agent: *
Disallow: /

This means “Don’t allow any pages to be crawled.”

📂 To block only a specific folder:

User-agent: *
Disallow: /calendar/

This means “Don’t crawl the ‘calendar’ folder.”

📄 To block one specific page:

User-agent: *
Disallow: /secret.html

💡 5. Why Use robots.txt?

• When you have pages not ready to show yet
• When you want to keep some pages private
• When you want to avoid unnecessary pages being crawled by Google

⚠ 6. Important Notes

❌ robots.txt does NOT guarantee privacy!

Even if you disallow a page via robots.txt, it may still appear in search results. If you want to hide a page completely, use a password or access restrictions.

🌍 It’s Public!

Since robots.txt is public, anyone can see it online. So never include confidential info or URLs in this file.

🛠 7. How to Create robots.txt

Step 1: Create a file named robots.txt

Use Notepad or any text editor and save it in UTF-8 format.

Step 2: Upload it to the root folder of your site

For example, in WordPress, this would be public_html, www, etc.

Step 3: Check the file via your browser

Go to: https://your-website.com/robots.txt to confirm the file is correctly accessible.

🧪 8. Want to Test It?

Use the robots.txt Tester in Google Search Console to verify your rules are working properly.

🧠 Summary: robots.txt is Your Site’s Gatekeeper!

• robots.txt is a rulebook for search engine robots
• You can use it to say “STOP!” to folders or pages you don’t want crawled
• But it’s not suitable for hiding private content
• It’s easy to create and totally free to use!

Reference URL:

• How to Create and Submit a robots.txt File – Google Search Central