Search engines are the source for most traffic to a website on the Internet. However, there are times when you might not want sites like Google to crawl your content. In those cases, perhaps you want to prevent WordPress from being indexed in search results.
After all, not everything you create online needs to immediately acquire traffic.
In this tutorial, I’ll show you how to stop search engines from indexing a site built with WordPress.
Why Would You Want to Prevent WordPress from Being Indexed?
Although most websites thrive on bringing traffic from search results, there are several reasons why you might want to prevent this. In reality, not everything on the Internet needs to show up in Google.
In some cases, you would want to discourage search engines from indexing the site. This would help keep the content away from the general public.
For instance, perhaps:
- You’re in the middle of website development.
Whether you’re creating a site for yourself or a client, Google doesn’t need to index something that is a work in progress. - You’re creating a private personal or family blog.
Perhaps you want to create a central hub for your family and friends to interact. You could also create an online journal for your own personal needs. - You’re setting up an intranet for your business.
WordPress is a powerful tool for building an intricate platform for any business. An intranet could easily benefit everyone under your employ. - You don’t want to share certain files or information with the rest of the world.
The whole world doesn’t need access to every bit of information you want to host on a website. You could simply want an easy-to-use method to share video or audio files in a team environment.
These are only a handful of reasons why some would want to stop search engines from indexing a website. From company information to sharing blog posts about your nephews birthday party, not everyone needs access to that content.
Here are six methods which may help you prevent the site from being crawled by search engines.
1. Search Engine Visibility
WordPress already has a built-in method to help stop search engines from indexing the site. However, this doesn’t always stop all search engines. Some of them may simply ignore the request.
However, the major engines like Google and Bing will.
Go to Settings and click, “Reading.”
Click the box for “Search Engine Visibility.”
Click the “Save Changes” button.
2. Using Yoast SEO Plugin
Yoast SEO has a lot of great tools even if you don’t plan on using it for search engine optimization. For instance, I love the content readability analysis and use it on several private blogs.
If you use Yoast SEO, you can disable crawling for pages, posts, and archives.
To find these settings, go to the Yoast icon on the left and click, “Search Appearance.”
Click the “Content Types” option on the top.
In this page, you can click “No” to show posts in search results. You also have the option to disable searching of pages as well on this page.
Click the “Archives” tab on the top.
In this screen, you can change how the archives are displayed on the site as well as in search engines. For example, you can choose to not show author archives in search results.
When you expand the “Date archives settings,” you’ll see the date archives is disabled by default for search engines.
If you do make any changes to Yoast, make sure you click the “Save Changes” button available on each settings page.
3. Using Password Protected Plugin
Many people will choose to use the Password Protected plugin available in WordPress. This locks down the website with a single password. Unfortunately, it doesn’t prevent access to images or uploaded files.
This means if someone knew the exact URL of those resources, they can still see them. But for a simple solution to lock the site when you don’t have uploaded imagery, this plugin is quite useful.
Install and activate the “Password Protected” plugin.
Go to Settings and click, “Password Protected.”
In this screen, you can set up the status as well as different permissions for the site. You can also whitelist specific IP addresses to access the website. This makes it convenient in a client development setting.
Once you’re done, just click the “Save Changes” button to lock the site.
4. Making Your Posts and Pages Private in WordPress
If you’re not too worried about people seeing the overall layout of the site but want to keep posts and pages secure, you can always password protect them. You can also change the content to private if you wish.
This is a built-in feature of WordPress and is easy to set up. It’s part of the publish function for posts and pages, actually.
Open any post or page in WordPress.
Under the Publish section of the post, you’ll see an option for “Visibility.” By default, this will show as Public.
Click the “Edit” link next to Visibility. This will expand a few additional options for you.
Click the radio button of how you want to secure the content. You can choose to password protect or privatize the post or page.
What’s the difference?
Password protecting the content means users will have to know the password in order to view the material. Changing it to private will still show the post or page to those who are logged into the system and with certain permissions.
Choose which privacy option you want and click, “OK.”
Now when you update or publish the post or page, the privacy settings will go into effect.
5. Modifying Your robots.txt File
Another method some will use to stop search engines from indexing the site is to edit the robots.txt file. This instructs search engines to ignore the contents of the website completely.
But like the option above for setting Search Engine Visibility in WordPress, the crawler can ignore the robots.txt file.
On the upside, the most popular of engines, like Google, Bing, and Yahoo, honor the file and will skip the site.
Edit the robots.txt file with the following lines:[ht_message mstyle=”info” title=”” show_icon=”” id=”” class=”” style=”” ]User-agent: *
Disallow: /[/ht_message]
6. Creating a Completely Private Blog
Setting up a completely private blog isn’t all that difficult in WordPress. With the right combinations of solid plugins and setting adjustments, you can easily lock down the site.
This includes elements like creating custom login URLs, excluding certain parts of the site, landing page redirection and more.
You can also set the site to govern over user registration making it easier for others to be members. This is great for family, personal and business intranet websites.
Search Engines Don’t Need to Know Everything
Out of the millions upon millions of websites on the Internet, a lot of them are set to prevent WordPress from being indexed in sites like Google and Bing. Not every situation calls for an influx of visitor traffic.
What kind of reasons would you use a private site? How often do you find yourself privatizing posts or pages on your site?