variety bug fix #142
variety bug fix #142
Conversation
Dockerfile
Outdated
|
|
||
| RUN pip install scylla | ||
| RUN pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ && python setup.py install |
There was a problem hiding this comment.
We don't need this because GH Actions are running overseas.
I will find a way to fix the CI.
|
|
||
| FROM python:3.9-slim | ||
| FROM python:3.9-slim as prod |
There was a problem hiding this comment.
I added a docker-compose file which is under testing, prod is used for specific target in docker-compose file's build
There was a problem hiding this comment.
oops, changes for testing are sending by pr 😭
There was a problem hiding this comment.
please just review b299692 😭😭😭😭
There was a problem hiding this comment.
Yes, of course, give one night and I'll merge them into another pr.
There was a problem hiding this comment.
Sure! Let me know if your PR is ready. You could also mark this one as a draft PR if needed.
|
|
||
| FROM python:3.9-slim | ||
| FROM python:3.9-slim as prod |
scylla/scheduler.py
Outdated
| @@ -111,6 +111,7 @@ def feed_from_db(): | |||
| class Scheduler(object): | |||
|
|
|||
| def __init__(self): | |||
| self.manager = Manager() | |||
There was a problem hiding this comment.
Could you explain what's the purpose of this change?
There was a problem hiding this comment.
A legacy change. When I face UnpickleableError: Cannot pickle <type 'thread.lock'> objects, I thought it is due to multiprocessing.Queue object can not be picked.
But I finally found the problem is caused by initialize provider in the main process.
It could be removed.
scylla/scheduler.py
Outdated
| @@ -111,6 +111,7 @@ def feed_from_db(): | |||
| class Scheduler(object): | |||
|
|
|||
| def __init__(self): | |||
| self.manager = Manager() | |||
|
|
||
| FROM python:3.9-slim | ||
| FROM python:3.9-slim as prod |
There was a problem hiding this comment.
Sure! Let me know if your PR is ready. You could also mark this one as a draft PR if needed.
|
Hi, commits are neat now. |
| *.egg | ||
| MANIFEST | ||
|
|
||
| # PyInstaller |
There was a problem hiding this comment.
We're not using PyInstaller. Could you remove it?
There was a problem hiding this comment.
Since .dockerignore is just a copy of .gitignore I do not review it🤣
So, maybe we could remove it from both files?
There was a problem hiding this comment.
Not a blocker for this PR. We can clean it up later.
Dockerfile
Outdated
|
|
||
| FROM python:3.9-slim as build |
There was a problem hiding this comment.
maybe python-build for clarity?
| @@ -35,14 +35,14 @@ def parse(self, document: PyQuery) -> [ProxyIP]: | |||
| def urls(self) -> [str]: | |||
| ret = [] | |||
| first_url = 'http://proxy-list.org/english/index.php?p=1' | |||
| sub = first_url[0:first_url.rfind('/')] # http://proxy-list.org/english | |||
| # sub = first_url[0:first_url.rfind('/')] # http://proxy-list.org/english | |||
There was a problem hiding this comment.
Since of commenting out of this line, could we just remove it since we have history?
|
@PaleNeutron I'm merging this one. |
Proposed Changes
PyQuery.findreturnsHtmlElementnotPyQueryUnpickleableError: Cannot pickle <type 'thread.lock'> objects