About

This is a personal, utilize Scrapy and Selenium to automate fetching hotels' price and storing in database. Client side is included to filter and display data.

Tech Stack

1. Backend: Django
2. Frontend: React - Redux - TypeScript
3. Database: Heroku Postgresql
4. Frameworks: Scrapy - Selenium - Pandas

Usage

1. Deployment

Project is live here

2. Local usage

• Clone this repository git clone https://github.com/nannadao/webspyder-fs.git
• Install python packages pip install
• Install npm packages npm install
• Generate Secret Key for Django Project python manage.py shell from django.core.management.utils import get_random_secret_key print(get_random_secret_key())
• Create .env file with following values: SECRET_KEY, DATABASE_NAME, DATABASE_USER, DATABASE_PASSWORD, DATABASE_HOST to save scrapped data in local database
• Generate client build npm run build
• Run project python manage.py runserver