Owning and maintaining a personal or business website can be quite a struggle. There are many issues or problems that can arise that can be difficult to get a straight answer. Throughout my experiences designing and managing websites, one of the most common problems I have encountered is adding and editing a robots. txt file to my sites. Without a robots.txt file, search engines like Google, Yahoo and Bing will be unable to index your website. If you want your websites to be indexed and ranked by search engines, the addition of a robots.txt file is essential.
Why Does Google Say It is Blocked by my Robots.txt File?
If you are a website manager and you are having trouble crawling your site using Google Search Console, you are not alone. I had this problem many times myself, at first I would contact customer service at my hosting service but it was hit or miss whether they would understand what the issue was. So I decided to look into the problem myself.
When I tried to crawl my website’s URL the status message would always come back saying “Blocked”. I did not understand why. My first thought was that maybe the reading settings in WordPress were discouraging search engines from indexing my website. Upon further inspection this was not the problem.
So what was going on?
I discussed the problem with one of my good friends who is also manages a number of websites. He informed me that the problem may be that my website does not have a robots.txt file at all.
This got me thinking…
As soon as I got a chance I tried to see if my website had a robots.txt file. This is where I ran into my second roadblock. I had no idea where to even look for a robots.txt file.
Where to Add a Robots.txt File to Your WordPress Website
It took some tinkering but after a few days of trial and error I finally figured out exactly where to add a robots.txt file so that search engines would be able to crawl my websites.
A website’s Robots.txt file should be located in the website’s root folder. Your host should allow you to access and edit your website’s root folder fairly easily. When you access the root folder look for the Document Root.
Once you are in the Document Root you can add an empty folder and name it “robots.txt.” Once you add this blank folder, search engine algorithms will be able to access, index, and rank the website in search engine results pages. Once the folder is added the document root folder should look like this:
After the folder has been added test that the problem has been solved by crawling your site using Google search console. Use fetch as Google. If the folder was added correctly you should be able to crawl immediately. Adding the robots.txt folder is an instant fix.
Comment below if you have any problems adding a Robots.txt file to your website or fixing issues you have encountered with your robots.txt file.