Donaldson R F
WordPress
Create & Configure Robots.txt
Set up a robots.txt file that correctly controls what search engines can and can't crawl.
The Problem
A missing robots.txt isn't a disaster, but a wrong one can accidentally block search engines from your entire site — a single misplaced rule has tanked more than one site's traffic overnight.
About this problem
The robots.txt file sits at the root of the domain and tells crawlers which paths they should not fetch. WordPress generates a virtual one by default, which SEO plugins or a physical file can override. Because it applies site-wide, one wrong line, such as a leftover Disallow rule from a staging site, can stop search engines crawling everything.
People usually look for this after a launch or migration when pages vanish from search, when Search Console reports pages blocked by robots.txt, or when they want to keep admin and internal search pages out of crawling. Many are unsure which rules are safe.
What's Included
- Auditing what currently should and shouldn't be crawlable
- Writing a correct robots.txt reflecting that
- Making sure it doesn't conflict with your sitemap or SEO plugin settings
- Verifying with Search Console that it's being read correctly
What's NOT Included
- Fixing indexing issues unrelated to robots.txt
- Blocking/unblocking specific pages beyond standard crawl rules (can be added)
- Ongoing SEO strategy
How It Works
- I fetch the current robots.txt and note whether it is virtual, a physical file or generated by a plugin.
- I list the sections that should and should not be crawlable, such as wp-admin, internal search and parameter URLs, while keeping assets needed for rendering available.
- I write the rules and place them through the correct mechanism so a plugin does not overwrite them.
- I add the sitemap reference and check it does not conflict with SEO plugin settings.
- I test important URLs against the file with a robots.txt testing tool.
- I check Search Console to see that the updated file is fetched and read correctly.
In practice: you buy the service, send over whatever access or details the job needs, I investigate and do the work, and you confirm it's resolved before we call it done.
Frequently Asked Questions
- What should a WordPress robots.txt contain?
- Typically a rule keeping crawlers out of wp-admin, an exception for admin-ajax.php and a link to the sitemap. The details depend on your site.
- Can robots.txt remove a page from Google?
- Not reliably. It stops crawling, not indexing, so pages that must be kept out need noindex handling instead.
- Why is my whole site blocked?
- Often a Disallow rule for everything, or the WordPress setting that discourages search engines, left on after a launch. I check for both.
- Do you fix other indexing problems?
- No, only those related to robots.txt.
- Can you block or unblock specific pages?
- Standard crawl rules are included. Specific page-level blocking beyond them can be added.
- Does this include SEO strategy?
- No, ongoing SEO strategy is not part of this service.