Skip to content

Navigation Menu

scrapy

Explore
By company size
By use case
By industry
View all solutions
Topics
- AI
- DevOps
- Security
- Software Development
- View all
Explore
- GitHub Sponsors
  Fund open source developers
- The ReadME Project
  GitHub community articles
Repositories
- Enterprise platform
  AI-powered developer platform
Available add-ons
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

545 followers
https://scrapy.org

Overview
Repositories
Projects
Packages
People

More

Overview
Repositories
Projects
Packages
People

Pinned Loading

scrapy scrapy Public

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python 53.3k 10.6k
scrapy.org scrapy.org Public

The scrapy.org website

HTML 60 139

Repositories

Loading

Type

Select type

All Public Sources Forks Archived Mirrors Templates

Language

Select language

All C++ DIGITAL Command Language HTML Python Shell

Sort

Select order

Last updated Name Stars

Showing 10 of 27 repositories

scrapy Public
Scrapy, a fast high-level web crawling & scraping framework for Python.

scrapy/scrapy’s past year of commit activity

Python 53,280 BSD-3-Clause 10,575 452 (21 issues need help) 217 Updated Nov 25, 2024
scrapy.org Public
The scrapy.org website

scrapy/scrapy.org’s past year of commit activity

HTML 60 139 1 1 Updated Nov 25, 2024
itemadapter Public
Common interface for data container classes

scrapy/itemadapter’s past year of commit activity

Python 62 BSD-3-Clause 13 6 4 Updated Nov 19, 2024
protego Public
A pure-Python robots.txt parser with support for modern conventions.

scrapy/protego’s past year of commit activity

DIGITAL Command Language 56 BSD-3-Clause 28 5 (1 issue needs help) 1 Updated Nov 15, 2024
scrapyd Public
A service daemon to run Scrapy spiders

scrapy/scrapyd’s past year of commit activity

Python 2,975 BSD-3-Clause 569 7 0 Updated Nov 11, 2024
parsel Public
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

scrapy/parsel’s past year of commit activity

Python 1,150 BSD-3-Clause 146 29 (1 issue needs help) 12 Updated Oct 16, 2024
w3lib Public
Python library of web-related functions

scrapy/w3lib’s past year of commit activity

Python 393 BSD-3-Clause 104 11 (1 issue needs help) 4 Updated Oct 16, 2024
itemloaders Public
Library to populate items using XPath and CSS with a convenient API

scrapy/itemloaders’s past year of commit activity

Python 45 BSD-3-Clause 16 17 4 Updated Oct 16, 2024
cssselect Public
CSS Selectors for Python

scrapy/cssselect’s past year of commit activity

Python 291 61 17 4 Updated Oct 16, 2024
queuelib Public
Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

scrapy/queuelib’s past year of commit activity

Python 270 BSD-3-Clause 54 3 2 Updated Oct 16, 2024

View all repositories

People

Top languages

Loading…

Most used topics

hacktoberfest python chromium gurl scraping

Footer

© 2024 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Docs
Contact

You can’t perform that action at this time.