authspider

A Powerful Spider(web crawler) System built in Python.

Features

• Find login URL and automate auth process by inserting username and password
• Extract random data from a webpage
• Extract data from members only section

Thanks:

We used ToScrape.com, a free Web Scraping Sandbox to implement scraping techniques.

Mechanize: https://github.com/python-mechanize/mechanize

BeautifulSoup4: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

For learning purpose only!