The Ultimate Robots.txt Guide for Webmasters
Aug 13,2026
By SEO ANALYSER
Crawling and indexing
Master robots.txt with a practical guide to file creation, syntax, allow and disallow rules, crawler control, testing, and common SEO mistakes.
Short answer
Key takeaway
Sam Holloway, Small Business SEO Educator Sam Holloway spent five years running his own trade business in Perth before teaching himself SEO
What a Robots.txt File
What a Robots.txt File Actually Does
Arobots.txt fileis a small text file that sits on your website and gives instructions tosearch engine crawlers, the automated programs (like Googlebot) that visit your site to read its pages. Think of it like a note stuck to your shop door, telling delivery drivers which rooms they can walk into and which ones are off limits.
Every website can have one, though not every website needs one. The file doesn't hide pages from people browsing your site. It tells crawlers which parts of your site you'd rather they skip, so they spend theircrawl budget, the number of pages a crawler will bother visiting in one session, on the pages that actually matter to your business.
This matters more for larger sites with hundreds of pages than it does for a five-page café website, but it's still worth understanding for every site owner.
Search Engine Crawlers
How Search Engine Crawlers Read Your Robots.txt File
Before a crawler works through your website, it checks for a robots.txt file first. If it finds one, it reads the rules inside before deciding where to go next. If there's no file at all, most crawlers assume they can access everything on the site.
It's worth knowing that these rules are a request, not a lock. Well-behaved crawlers from Google, Bing, and similar search engines generally respect the file. Poorly behaved bots, the kind scraping content or hunting for security gaps, often ignore it completely. That's an important distinction: robots.txt manages crawling, it doesn't protect private information. If you have something genuinely sensitive on your site, password protection is the right tool, not a text file.
Related reading: manage search engine access, and canonical tags.
How to Create
How to Create a Robots.txt File
This is the part most business owners get stuck on, so let's go through it step by step.
You don't need any special software tocreate a robots.txt file. It's a plain text file, the same kind you'd make in Notepad or TextEdit, saved with the exact filename " robots.txt" in lowercase letters. Nothing fancy, no coding program required.
Inside the file, you write simple instructions using a small set of terms: which crawler the rule applies to, and which folders or pages it should avoid. A very basic file for a small business site might look like this:
That first line applies the rule to every crawler. The second line blocks the admin folder, the backend area where you log in to manage your site, something no search visitor ever needs to see. The third line points crawlers toward yourXML sitemap, a file listing every page you want indexed.
Once your file is written, it needs to sit in the right spot for crawlers to find it, which we'll cover shortly. If you run your site through WordPress, Shopify, or a similar platform, there's often a built-in tool for editing this file without touching any code directly, which makes the whole process far less intimidating than it sounds.
Understanding Robots.txt Syntax
Understanding Robots.txt Syntax and Directives
Once you know the basicrobots.txt syntax, the handful of terms used to write rules, the rest becomes far less confusing. Here's a simple breakdown of what each one does.
TheUser-agent linealways comes first, followed by one or more rules underneath it. ADisallow directiveblocks access, while anAllow directiveworks the opposite way, letting a specific page through even if it sits inside a folder you've otherwise blocked. You might use this if you've blocked an entire images folder but still want one product photo crawled for Google Images.
Robots.txt Rules Pages
Robots.txt Rules: Pages You Should and Shouldn't Block
Deciding what to include in your file is really the whole point of learningrobots.txt rulesin the first place. Some pages are worth keeping away from crawlers, while others should always stay open, even if blocking them feels tempting.
Pages commonly worth blocking include:
- Admin or login areas that customers never see
- Internal search result pages, which create endless duplicate URLs
- Staging or test versions of your site
- Shopping cart and checkout steps that don't need to appear in search results
- Duplicate filtered or sorted product listing pages on online stores
Pages you should almost never block include your homepage, your main service or product pages, your blog posts, and the CSS and JavaScript files that help your site display properly. Blocking those by mistake is one of the most common issues small business owners run into, which brings us to the next section.
Related reading: crawl and indexing, and block duplicate content.
Common Robots.txt Mistakes
Common Robots.txt Mistakes That Hurt Small Business Websites
Most robots.txt problems come from good intentions gone slightly wrong. A boutique owner trying to tidy up her site once blocked her entire images folder, not realising her product photos would stop showing up in Google Image searches as a result. It's an easy mistake to make when you're not sure what each line actually controls.
The most damaging error isblocking important pagesby accident, often through a rule that's broader than intended, like disallowing an entire folder when only one page inside it needed hiding. Another common slip is forgetting to update the file after a website redesign, leaving old rules pointing at folders that no longer exist, or worse, blocking new pages that should be visible.
Robots.txt vs the Meta
Robots.txt vs the Meta Robots Tag
These two get confused constantly, so it's worth clearing up quickly. A robots.txt file controls whether a crawler visits a page at all. Ameta robots tag, a small piece of code placed inside an individual page, controls whether that page gets indexed once the crawler has already read it.
If you block a page in robots.txt, Google generally won't crawl it, but it can sometimes still appear in search results without any description if other sites link to it. If you want a page fully removed from Google's results, the meta robots tag is usually the better tool for that specific job.
Related reading: noindex tag.
Where to Put Your
Where to Put Your File and How to Test It
Your robots.txt file needs to live in theroot directoryof your website, meaning the main top-level folder, not tucked inside a subfolder. It should be reachable at yourdomain.com.au/robots.txt exactly, nothing extra in the address.
Once it's uploaded, it's worth taking a moment totest your robots.txtbefore assuming everything's fine. Google Search Console includes a simple checking tool for this, letting you paste in a URL and see whether your current rules block or allow it. Running this check after any changes takes a couple of minutes and can save you from an accidental mistake sitting unnoticed for months.
Do You Even Need
Do You Even Need a Robots.txt File?
For a lot of small, straightforward websites, the honest answer is that a basic file helps but isn't strictly essential. If your site has no admin folders, no duplicate pages, and nothing you'd rather keep away from crawlers, a simple file pointing to your sitemap is often enough.
Larger sites, online stores, and anything built on a platform that generates lots of automatic pages tend to benefit more, since there's usually more worth tidying up behind the scenes.
SEO Analyser
Where Website Audit fits robots.txt QA
SEO Analyser includes a module named Website Audit. Use it after robots.txt changes to confirm money pages remain crawlable and that Disallow rules did not overreach.
Summary
A robots.txt file doesn't need to feel technical or intimidating once you understand what it's actually doing: guiding search engine crawlers toward the pages that matter and away from the ones that don't. Getting the basics right, correct placement in your root directory, sensible rules, and a quick test after any changes, covers most of what a small business site will ever need.
You don't need to overhaul everything at once. Start with a simple file, block only what you're confident about, and build from there as your site grows. Understanding robots.txt properly means you're already ahead of plenty of business owners who've never looked at their file at all, and that's a solid step toward a site that search engines can read the way you intended.
References
Sources and further reading
- SEO Analysermanage search engine access
- SEO Analyserblock duplicate content
- SEO Analysernoindex tag
- SEO Analysercanonical tags
- SEO Analysercrawl and indexing
