python-clinic / Doc / library / urllib.robotparser.rst

:mod:`urllib.robotparser` --- Parser for robots.txt

This module provides a single class, :class:`RobotFileParser`, which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the :file:`robots.txt` file. For more details on the structure of :file:`robots.txt` files, see

This class provides methods to read, parse and answer questions about the :file:`robots.txt` file at url.

The following example demonstrates basic use of the RobotFileParser class.

>>> import urllib.robotparser
>>> rp = urllib.robotparser.RobotFileParser()
>>> rp.set_url("")
>>> rp.can_fetch("*", "")
>>> rp.can_fetch("*", "")