This is a generic website crawler created by ATHENA R.C.
Given a website it collects all html data from the domain.
The crawler operates on a Breadth-First-Search manner and stops after a specific number of crawled pages.
In order to run the crawler:
python3.6 crawler.py --inpath=data2crawl.json --out_dir=./output/ --max_pages_to_visit=1000
The data will be collected in files under the director "output"
