pbot /

Filename Size Date modified Message
pbot
2.0 KB
480 B

Description:

Pbot contains two modules, Bot and Spider

Bot is a simple helper, created to save request state (cookies, referrer) between http requests. Also, it provides addional methods for adding cookies. With no dependencies this module is easy to use when you need to simulate browser.

Spider it's pbot, armed by lxml (required). Provides addional methods for easy website crawling, see below.

Bot is very easy to use:

from pbot.pbot import Bot
bot = Bot(proxies={'http': 'localhost:3128'}) # You can provide proxies, during bot creation, or set later as bot.proxies
bot.add_cookie({'name': 'sample', 'value': 1, 'domain': 'example.com'})
response = bot.open('http://example.com') # Open with cookies and empty referrer
bot.follow('http://google.com') # Open google with example.com as a referrer
response = bot.response # Response saved, and can be read later
bot.follow('http://example.com', post={'q': 'abc'}) # You can provide post and get as keyword arguments
bot.refresh_connector() # Flush cookies and referrer

Spider gives you special features:

from pbot.spider import Spider
bot = Spider() # or Spider(force_encoding='utf-8') to force encoding for parser
bot.open('http://example.com')
bot.tree.xpath('//a') # lxml tree can be accessed by .tree, response will be automatically readed and parsed by lxml.html
form = bot.xpath('//form[@id="main"]') # xpath shortcut for bot.tree.xpath
bot.submit(form) # Submit lxml f§orm
#
# Crawler, recursively crawl from target page yielding xml_tree, query_url, real_url (real_url - url after all redirects).
bot.crawl(self,
    url=None, # Target url to start crawling
    check_base=True, # Yield pages only on domain from url
    only_descendant=True, # Yield only pages that urls starts with url
    max_level=None, #Maximum level
    allowed_protocols=('http:', 'https:'),
    ignore_errors=True,
    ignore_starts=(), # Tuple/array,  ignore urls that starts with ignore_starts (exclude some parts of site)
    check_mime=())
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.