The Ultimate Robots.txt Guide for Webmasters

You've probably heard the words robots.txt file thrown around in some small business Facebook group, or by a mate who reckons they "do SEO", and felt a bit lost. Maybe you've even opened your website's files, spotted one sitting there, and had no idea what it actually does or whether you're allowed to touch it. Sound familiar? You're not alone.

When I had my own trade business in Perth, I ignored mine for years because it looked too technical to bother with. The truth is, a robots.txt file is one of the simpler parts of your website to understand once someone walks you through it properly. That's exactly what this guide does, without the jargon and without assuming you're a developer.

What a Robots.txt File Actually Does

A robots.txt file is a small text file that sits on your website and gives instructions to search engine crawlers, the automated programs (like Googlebot) that visit your site to read its pages. Think of it like a note stuck to your shop door, telling delivery drivers which rooms they can walk into and which ones are off limits.

Every website can have one, though not every website needs one. The file doesn't hide pages from people browsing your site. It tells crawlers which parts of your site you'd rather they skip, so they spend their crawl budget, the number of pages a crawler will bother visiting in one session, on the pages that actually matter to your business.

This matters more for larger sites with hundreds of pages than it does for a five-page café website, but it's still worth understanding for every site owner.

How Search Engine Crawlers Read Your Robots.txt File

Before a crawler works through your website, it checks for a robots.txt file first. If it finds one, it reads the rules inside before deciding where to go next. If there's no file at all, most crawlers assume they can access everything on the site.

It's worth knowing that these rules are a request, not a lock. Well-behaved crawlers from Google, Bing, and similar search engines generally respect the file. Poorly behaved bots, the kind scraping content or hunting for security gaps, often ignore it completely. That's an important distinction: robots.txt manages crawling, it doesn't protect private information. If you have something genuinely sensitive on your site, password protection is the right tool, not a text file.

How to Create a Robots.txt File

This is the part most business owners get stuck on, so let's go through it step by step.

You don't need any special software to create a robots.txt file. It's a plain text file, the same kind you'd make in Notepad or TextEdit, saved with the exact filename "robots.txt" in lowercase letters. Nothing fancy, no coding program required.

Inside the file, you write simple instructions using a small set of terms: which crawler the rule applies to, and which folders or pages it should avoid. A very basic file for a small business site might look like this:

That first line applies the rule to every crawler. The second line blocks the admin folder, the backend area where you log in to manage your site, something no search visitor ever needs to see. The third line points crawlers toward your XML sitemap, a file listing every page you want indexed.

Once your file is written, it needs to sit in the right spot for crawlers to find it, which we'll cover shortly. If you run your site through WordPress, Shopify, or a similar platform, there's often a built-in tool for editing this file without touching any code directly, which makes the whole process far less intimidating than it sounds.

Understanding Robots.txt Syntax and Directives

Once you know the basic robots.txt syntax, the handful of terms used to write rules, the rest becomes far less confusing. Here's a simple breakdown of what each one does.

The User-agent line always comes first, followed by one or more rules underneath it. A Disallow directive blocks access, while an Allow directive works the opposite way, letting a specific page through even if it sits inside a folder you've otherwise blocked. You might use this if you've blocked an entire images folder but still want one product photo crawled for Google Images.

Robots.txt Rules: Pages You Should and Shouldn't Block

Deciding what to include in your file is really the whole point of learning robots.txt rules in the first place. Some pages are worth keeping away from crawlers, while others should always stay open, even if blocking them feels tempting.

Pages commonly worth blocking include:

• Admin or login areas that customers never see
• Internal search result pages, which create endless duplicate URLs
• Staging or test versions of your site
• Shopping cart and checkout steps that don't need to appear in search results
• Duplicate filtered or sorted product listing pages on online stores

Pages you should almost never block include your homepage, your main service or product pages, your blog posts, and the CSS and JavaScript files that help your site display properly. Blocking those by mistake is one of the most common issues small business owners run into, which brings us to the next section.

Common Robots.txt Mistakes That Hurt Small Business Websites

Most robots.txt problems come from good intentions gone slightly wrong. A boutique owner trying to tidy up her site once blocked her entire images folder, not realising her product photos would stop showing up in Google Image searches as a result. It's an easy mistake to make when you're not sure what each line actually controls.

The most damaging error is blocking important pages by accident, often through a rule that's broader than intended, like disallowing an entire folder when only one page inside it needed hiding. Another common slip is forgetting to update the file after a website redesign, leaving old rules pointing at folders that no longer exist, or worse, blocking new pages that should be visible.

Robots.txt vs the Meta Robots Tag

These two get confused constantly, so it's worth clearing up quickly. A robots.txt file controls whether a crawler visits a page at all. A meta robots tag, a small piece of code placed inside an individual page, controls whether that page gets indexed once the crawler has already read it.

If you block a page in robots.txt, Google generally won't crawl it, but it can sometimes still appear in search results without any description if other sites link to it. If you want a page fully removed from Google's results, the meta robots tag is usually the better tool for that specific job.

Where to Put Your File and How to Test It

Your robots.txt file needs to live in the root directory of your website, meaning the main top-level folder, not tucked inside a subfolder. It should be reachable at yourdomain.com.au/robots.txt exactly, nothing extra in the address.

Once it's uploaded, it's worth taking a moment to test your robots.txt before assuming everything's fine. Google Search Console includes a simple checking tool for this, letting you paste in a URL and see whether your current rules block or allow it. Running this check after any changes takes a couple of minutes and can save you from an accidental mistake sitting unnoticed for months.

Do You Even Need a Robots.txt File?

For a lot of small, straightforward websites, the honest answer is that a basic file helps but isn't strictly essential. If your site has no admin folders, no duplicate pages, and nothing you'd rather keep away from crawlers, a simple file pointing to your sitemap is often enough.

Larger sites, online stores, and anything built on a platform that generates lots of automatic pages tend to benefit more, since there's usually more worth tidying up behind the scenes.

FAQs