Blogging News Hubb
Advertisement
  • Home
  • News
  • Blogging
  • Content Marketing
  • SEO
  • Mobile SEO
  • Contact
No Result
View All Result
  • Home
  • News
  • Blogging
  • Content Marketing
  • SEO
  • Mobile SEO
  • Contact
No Result
View All Result
Blogging News Hubb
No Result
View All Result
Home SEO

New open source robots.txt projects  |  Google Search Central Blog  |  Google Developers

by admin
June 17, 2022
in SEO

[ad_1]

Monday, September 21, 2020

Last year we released the
robots.txt parser and matcher that we use in
our production systems to the open source world. Since then, we’ve seen people build new tools
with it,
contribute to the
open source library (effectively improving our production systems- thanks!), and release new
language versions like golang and
rust, which make it easier for
developers to build new tools.

With the intern season ending here at Google, we wanted to highlight two new releases related to
robots.txt that were made possible by two interns working on the Search Open Sourcing team,
Andreea Dutulescu and
Ian Dolzhanskii.

Robots.txt Specification Test

First, we are releasing a
testing framework for robots.txt
parser developers, created by Andreea. The project provides a testing tool that can validate
whether a robots.txt parser follows the Robots Exclusion Protocol, or to what extent. Currently
there is no official and thorough way to assess the correctness of a parser, so Andreea built a
tool that can be used to create robots.txt parsers that are following the protocol.

Java robots.txt parser and matcher

Second, we are releasing an official
Java port of the C++ robots.txt parser,
created by Ian. Java is the
3rd most popular programming language
on GitHub and it’s extensively used at Google as well, so no wonder it’s been the most requested
language port. The parser is a 1-to-1 translation of the C++ parser in terms of functions and
behavior, and it’s been thoroughly tested for parity against a large corpora of robots.txt
rules. Teams are already planning to use the Java robots.txt parser in Google production
systems, and we hope that you’ll find it useful, too.

As usual, we welcome your contributions to these projects. If you built something with the
C++ robots.txt parser or with these new
releases, let us know so we can potentially help you spread the word! If you found a bug, help
us fix it by opening an issue on GitHub or directly contributing with a pull request. If you
have questions or comments about these projects, catch us on
Twitter!

It was our genuine pleasure to host Andreea and Ian, and we’re sad that their internship is
ending. Their contributions help make the Internet a better place and we hope that we can
welcome them back to Google in the future.


Posted by Edu Pereda and
Gary, Google Search Open Sourcing team



[ad_2]

Source link

Previous Post

Slot machine power stars trucchi, bc slots

Next Post

Googlebot will soon speak HTTP/2  |  Google Search Central Blog  |  Google Developers

Next Post

Googlebot will soon speak HTTP/2  |  Google Search Central Blog  |  Google Developers

Trending

17 Most Followed Celebrities on Instagram|eAskme

by admin
September 13, 2022

Digital Transformation & Content Marketing Technology

by admin
September 13, 2022

Content Syndication Explained: How, What, Who, When, Where, & Why|eAskme

by admin
September 13, 2022

How to Find Popular Keywords Using Google Insight Search in 2022|eAskme

by admin
September 13, 2022

How to Use Ego Bait Content To Build Links and Generate Leads|eAskme

by admin
September 13, 2022

What is Visual Content Marketing (with Benefits)

by admin
September 13, 2022

© 2022 Blogging News Hubb All rights reserved.

Use of these names, logos, and brands does not imply endorsement unless specified. By using this site, you agree to the Privacy Policy.

Navigate Site

  • Home
  • News
  • Blogging
  • Content Marketing
  • SEO
  • Mobile SEO
  • Contact

Newsletter Sign Up

No Result
View All Result
  • Home
  • News
  • Blogging
  • Content Marketing
  • SEO
  • Mobile SEO
  • Contact

© 2022 Blogging News Hubb All rights reserved.

SUBSCRIBE TO OUR WEEKLY NEWSLETTERS
Our Spring Sale Has Started

You can see how this popup was set up in our step-by-step guide: https://wppopupmaker.com/guides/auto-opening-announcement-popups/