GitList

Crawling  A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing

📂Crawling Frameworks

  Scrapy, a fast high-level web crawling & scraping framework for Python.

Python39.84 k
scrapy/scrapy

  A Powerful Spider(Web Crawler) System in Python.

Python13.91 k
binux/pyspider

  Elegant Scraper and Crawler Framework for Golang

Go13.55 k
gocolly/colly

  Redis-based components for Scrapy.

Python3.19 k
rmax/scrapy-redis
📂Spider Application

  新浪微博爬虫(Scrapy、Redis)

Python2.87 k
LiuXingMing/SinaSpider

  DHT Spider + BitTorrent Client = P2P Spider

Go2.81 k
fanpei91/p2pspider

  微信公众号爬虫

Python2.46 k
bowenpay/wechat-spider

  基于 webmagic 的 Java 爬虫应用

Java2.09 k
brianway/webporter

  豆瓣读书的爬虫

Python1.85 k
lanbing510/DouBanSpider

  🍥 Bilibili 用户爬虫

Python1.66 k
airingursb/bilibili-user
Suggest tags
#scraper#python#go#scrapy#python3#selenium#crawling#crawler#scraping#spider
Projects under this category

  Scrapy, a fast high-level web crawling & scraping framework for Python.

Python39.84 k
scrapy/scrapy

  Create agents that monitor and act on your behalf. Your agents are standing by!

Ruby31.33 k
huginn/huginn

  👾 Fast, simple and clean video downloader

Go14.09 k
iawia002/annie

  Elegant Scraper and Crawler Framework for Golang

Go13.55 k
gocolly/colly

  Python爬虫代理IP池(proxy pool)

Python12.07 k
jhao104/proxy_pool

  News, full-text, and article metadata extraction in Python 3. Advanced docs:

Python10.09 k
codelucas/newspaper

  一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Python9.87 k
shengqiangzhang/examples-of-web-crawlers

  A scalable web crawler framework for Java.

Java9.73 k
code4craft/webmagic

  An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

Python9.72 k
twintproject/twint

Copyright © GitList.top 2021.