WordPress has announced a change in the way it will block search engines when indexing sites . This change abandons the traditional Robots.txt solution in favor of the Robots Meta Tag approach. The reason for this is the desire to ensure that blocked pages do not actually appear in Google’s search results.

This is the Robots Meta Tag that the new version of WordPress will use:

<meta name = ‘robots’ content = ‘noindex, nofollow’ />

It has long been common practice to use Robots.txt file to block a website from being indexed.

The word indexing refers to Googlebot’s search for a website. On the basis of this search, Google understands the content of the site and indexes the pages. By using the Robots.txt blocking feature , you could stop Google from downloading the page in question and, it was assumed, that Google would not be able to display your pages in search results. But the robots.txt directive only prevented Google from searching the site. Google was still free to add it to its index if it was able to detect the URL.

WordPress has announced a change in version 5.3

Starting with version 5.3, WordPress has announced an important change in the way it will block search engines from indexing sites. This change abandons the traditional Robots.txt solution in favor of the Robots Meta Tag approach. A more reliable approach will be adopted by using the Robots Meta Tag that prevents the site from being indexed. This change is a great improvement and help for anyone who doesn’t want certain parts of their site to be indexed. WordPress users can be more confident when they know that blocked websites won’t show up in Google’s search results.

WordPress relied on Robots.txt as the standard way to block site indexing that everyone used. Yet, as explained, it was an unreliable approach.

The most effective method to exclude development sites from crawling by search engines is to include the X-Robots-Tag HTTP Header: noindex, nofollow when uploading images, PDFs, videos, and other content.

The release of the new version is scheduled for November 2019. Take a look at the official announcement of WordPress here.

Source: searchenginejournal.com