Commits

Author Commit Message Labels Comments Date
Evgeniy Tatarkin
downlader.get returnr tuple of (url, page_content)
Evgeniy Tatarkin
add logs
Evgeniy Tatarkin
add simple threaded downloader
Evgeniy Tatarkin
use versiontoos
Evgeniy Tatarkin
py3k support
Evgeniy Tatarkin
ignore cibox files
Evgeniy Tatarkin
add simple downloader, example
Evgeniy Tatarkin
chain urls for _width_first_method
Evgeniy Tatarkin
width/depth crawl method
Evgeniy Tatarkin
move iterator to utils
Evgeniy Tatarkin
remove collector from engine
Evgeniy Tatarkin
add pipelines
Evgeniy Tatarkin
recursive crawl
Evgeniy Tatarkin
add item and field
Evgeniy Tatarkin
base crawler and downloader
Evgeniy Tatarkin
init repo