{"id":26549,"date":"2024-11-14T14:59:48","date_gmt":"2024-11-14T13:59:48","guid":{"rendered":"https:\/\/www.digitizer.rs\/news-stories\/the-new-standard-for-robots-txt-files\/"},"modified":"2026-01-09T15:33:34","modified_gmt":"2026-01-09T14:33:34","slug":"the-new-standard-for-robots-txt-files","status":"publish","type":"post","link":"https:\/\/www.digitizer.rs\/en\/news-stories\/the-new-standard-for-robots-txt-files\/","title":{"rendered":"A new standard for robots.txt files"},"content":{"rendered":"
[vc_row full_width=”stretch_row” css=”.vc_custom_1487086373501{padding-top: 80px !important;padding-bottom: 140px !important;}”][vc_column][vc_column_text css=”.vc_custom_1584717269620{padding-top: 30px !important;padding-bottom: 30px !important;}”]Google<\/em> has proposed introducing a new Internet standard when it comes to rules related to robots.txt files. <\/span><\/p>\n

 <\/p>\n

These rules, enshrined in the Robots Exclusion Protocol<\/em> (REP),<\/em> have been an unofficial standard for the past 25 years.<\/span><\/p>\n

 <\/p>\n

Although the REP<\/em> has been adopted by search engines, it is not yet official, which means that it is open to interpretation by developers. Further, it has never been updated to cover today’s use cases. <\/span><\/p>\n

 <\/p>\n

According to Google<\/em>, this creates a challenge for website<\/em> owners because the protocol is ambiguously written, and has made it difficult to write rules correctly.<\/span>[\/vc_column_text][vc_column_text css=”.vc_custom_1584717361969{padding-top: 30px !important;}”]\n

New rules for robots.txt<\/strong><\/span><\/h1>\n[\/vc_column_text][vc_column_text css=”.vc_custom_1584717282284{padding-top: 30px !important;padding-bottom: 30px !important;}”]To eliminate this challenge, Google<\/em> has documented how REP<\/em> is being used on the modern web<\/em> and submitted it to the Internet Engineering Task Force (IETF) for consideration.<\/span><\/p>\n

 <\/p>\n

Google<\/em> explains what’s included in the draft:<\/span><\/p>\n

 <\/p>\n

“The proposed draft REP reflects over 20 years of real-world experience relying on robots.txt rules, which are also used by Googlebot<\/em> and other major search engines, as well as about half a billion sites that rely on REP.<\/em> These modified controls give the publisher the ability to decide what they would like to be indexed on their site and potentially displayed to interested users. “<\/span><\/p>\n

 <\/p>\n

The draft does not change any of the rules established in 1994, it has only been updated for the modern web<\/em>. <\/span>[\/vc_column_text][vc_column_text]Some of the updated policies include:<\/span><\/p>\n

 <\/p>\n