Skip to content

Latest commit

 

History

History
79 lines (53 loc) · 2.12 KB

README.md

File metadata and controls

79 lines (53 loc) · 2.12 KB
Raw

Scylla Build Status codecov Documentation Status PyPI version Docker Build Status

An intelligent proxy pool for humanities, only supports Python 3.6. Key features:

  • Automatic proxy ip crawling and validation
  • Easy-to-use JSON API
  • Simple but beautiful web-based user interface (eg. geographical distribution of proxies)
  • Get started with only 1 command minimally
  • Straightforward programmable API
  • (Additional) Headless browser crawling

Get started

Install with Docker (highly recommended)

docker run -it --name scylla wildcat/scylla

Install directly via pip

pip install scylla
scylla # Run the cralwer and web server for JSON API

Use from source

git clone https://github.com/imWildCat/scylla.git
cd scylla

pip install -r requirements.txt

npm install # or yarn install
npm build # or yarn build

python -m scylla

Programmatically

For more details, please read the API Documentation.

Roadmap

  • Basic RESTFul API
  • Advanced RESTFul API (Sorting, paging, conditions)
  • Programmable API
  • Docker integration
  • Smart web proxy server
  • More statistical views of proxies
  • Quality statistics for different providers
  • Over 10 data sources

Development and Contribution

git clone https://github.com/imWildCat/scylla.git
cd scylla

pip install -r requirements.txt

make build-assets

Testing

pip install -r tests/requirements-test.txt
pytest -n 10 tests

License

Apache License 2.0. For more details, please read the LICENSE file.