Scylla

An intelligent proxy pool for humanities, only supports Python 3.6. Key features:

• Automatic proxy ip crawling and validation
• Easy-to-use JSON API
• Simple but beautiful web-based user interface (eg. geographical distribution of proxies)
• Get started with only 1 command minimally
• Straightforward programmable API
• (Additional) Headless browser crawling

Get started

Install with Docker (highly recommended)

docker run -d -p 8899:8899 -v /var/www/scylla:/var/www/scylla --name scylla wildcat/scylla

Install directly via pip

pip install scylla
scylla # Run the cralwer and web server for JSON API

Use from source

git clone https://github.com/imWildCat/scylla.git
cd scylla

pip install -r requirements.txt

npm install # or yarn install
make build-assets # or yarn build

python -m scylla

Programmatically

For more details, please read the API Documentation.

Roadmap

Please see Projects.

Development and Contribution

git clone https://github.com/imWildCat/scylla.git
cd scylla

pip install -r requirements.txt

npm install # or `yarn install`
make build-assets

Testing

pip install -r tests/requirements-test.txt
pytest -n 15 tests

License

Apache License 2.0. For more details, please read the LICENSE file.