Unable to resume pretraining

[erotavlas]

I was running pretraining on a machine that crashed so I wanted to resume where the training left off. However when I attempt to do so I get this error in spacy 3.0.5
(in a conda environment on windows 10 machine)

(spacy3) PS C:\users\username\documents> python -m spacy pretrain config_pretrain.cfg output_pretrain --paths.raw_text rawtext_preprocessed.jsonl -r "C:\Users\username\Documents\output_pretrain" --epoch-resume 219 --gpu-id 0 --code tokenizer_rules.py
⚠ Output directory is not empty.
If you're resuming a run in this directory, the old weights for the consecutive
epochs will be overwritten with the new ones.
ℹ Using GPU: 0
ℹ Loading config from: config_pretrain.cfg
✔ Saved config file in the output directory
ℹ Resume training tok2vec from: C:\Users\username\Documents\output_pretrain
Traceback (most recent call last):
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\__main__.py", line 4, in <module>
    setup_cli()
    command(prog_name=COMMAND)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\typer\main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\cli\pretrain.py", line 70, in pretrain_cli
    pretrain(
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\training\pretrain.py", line 44, in pretrain
    _resume_model(model, resume_path, epoch_resume, silent=silent)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\training\pretrain.py", line 99, in _resume_model
    with resume_path.open("rb") as file_:
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\pathlib.py", line 1218, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\pathlib.py", line 1074, in _opener
    return self._accessor.open(self, flags, mode)
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\username\\Documents\\output_pretrain'

So then I tried to move the output files to another directory in C:\temp folder, but I got the same error. The last file that was written is model219.bin so I thought maybe I have to start at 220 instead of 219, but that failed with the same error as well. Then I decided to upgrade to spacy 3.0.0 in case it was an issue that was fixed in the most recent release. But then after upgrade I tried again and get a different error

(spacy3) PS C:\users\username\documents> python -m spacy pretrain config_pretrain.cfg output_pretrain --paths.raw_text rawtext_preprocessed.jsonl -r C:\temp\resume --epoch-resume 219 --gpu-id 0 --code tokenizer_rules.py
⚠ Output directory is not empty.
If you're resuming a run in this directory, the old weights for the consecutive
epochs will be overwritten with the new ones.
ℹ Using GPU: 0
ℹ Loading config from: config_pretrain.cfg
✔ Saved config file in the output directory
Traceback (most recent call last):
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\__main__.py", line 4, in <module>
    setup_cli()
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\cli\_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\typer\main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\cli\pretrain.py", line 70, in pretrain_cli
    pretrain(
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\training\pretrain.py", line 40, in pretrain
    model = create_pretraining_model(nlp, P)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy\training\pretrain.py", line 149, in create_pretraining_model
    tok2vec.initialize(X=[nlp.make_doc("Give it a doc to infer shapes")])
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\thinc\model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\thinc\layers\chain.py", line 86, in init
    layer.initialize(X=curr_input, Y=Y)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\thinc\model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\thinc\layers\chain.py", line 86, in init
    layer.initialize(X=curr_input, Y=Y)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\thinc\model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\thinc\layers\concatenate.py", line 132, in init
    layer.initialize(X=X, Y=Y)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\thinc\model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "C:\Users\username\Miniconda3\envs\spacy3\lib\site-packages\spacy_legacy\layers\staticvectors_v1.py", line 89, in init
    model.set_param("W", init_W(model.ops, (nO, nM)))
TypeError: do_registration() takes 1 positional argument but 2 were given

[polm]

The permissions issue doesn't have anything to do with spaCy, not sure what's up with that.

What versions of spaCy are you actually using? First you say you're using 3.0.5, then you say you upgraded to 3.0.0, so I'm not sure what's going on.

do_registration is a function in catalogue, so it sounds like you may have a version mismatch somewhere. Can you provide the output of pip freeze?

[adrianeboyd]

The init_W thing is a bug. You should be able to upgrade to spacy-legacy==3.0.5 to fix it in a minute or two.

If you're training from scratch, it would be a good idea to switch to the updated .v2 versions of the relevant architectures here instead for models, too (which also avoids this bug): https://spacy.io/api/legacy#HashEmbedCNN_v1

[erotavlas]

I'm still getting the permissions error. There is actually no permissions issue on this machine or the account I'm logged in as. I am able to run pretraining successfully (just tested) without any errors. It is able to write to the folders I specified. However it only fails when I try resume training. Below is my environment. I also tested one weights file prior to the last one in case the last was corrupted but same result.

blis==0.7.4
catalogue==2.0.3
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
cupy-cuda111==8.5.0
cymem==2.0.5
en-core-web-trf @ https://github.com/explosion/spacy-models/releases/download/en_core_web_trf-3.0.0/en_core_web_trf-3.0.0-py3-none-any.whl
fastrlock==0.6
filelock==3.0.12
idna==2.10
Jinja2==2.11.3
joblib==1.0.1
MarkupSafe==1.1.1
murmurhash==1.0.5
numpy==1.20.2
packaging==20.9
pathy==0.5.2
Pillow==8.2.0
preshed==3.0.5
pydantic==1.7.3
pyparsing==2.4.7
regex==2021.4.4
requests==2.25.1
sacremoses==0.0.45
six==1.15.0
smart-open==3.0.0
spacy==3.0.6
spacy-alignments==0.8.3
spacy-legacy==3.0.5
spacy-transformers==1.0.2
srsly==2.4.1
thinc==8.0.3
tokenizers==0.10.2
torch==1.8.1+cu111
torchaudio==0.8.1
torchvision==0.9.1+cu111
tqdm==4.60.0
transformers==4.5.1
typer==0.3.2
typing-extensions==3.7.4.3
urllib3==1.26.4
wasabi==0.8.2
wincertstore==0.2

[erotavlas]

I just tested this again on a different machine where training and pretraining work fine (no issues reading writing to folders) However when I try resuming pretraining it fails with the same error.

PermissionError: [Errno 13] Permission denied

[polm]

This is kind of a shot in the dark, but we've recently had weird permissions related issues on Windows that were resolved by using pip 21.0 instead of 21.1. The issues were in a pretty different context, and since you're in a conda env that may not be relevant at all, but maybe you could try using that pip version?

[erotavlas]

I tried this but it didn't help. Same error. I also tried to uninstall and reinstall miniconda to a different location as well as use python 3.7 instead of python 3.8.5 in my conda environment but nothing - same permission error.

Does this actually work for your team in Windows? If so are you using conda or python virtual env? Also what does your command look like when you tried it successfully?

Are you sure there isn't somewhere in spacy source code where its trying to read from folder that doesn't exist?

[erotavlas]

As I suspected it looks like the resume path is wrong. When I print it out it contains only the full path to the folder without the model.bin file. so when it gets to the line 99 in pretrain.py it fails because its trying to open a directory as a file instead of an actual file.

with resume_path.open("rb") as file_:

If I have to specify an actual file in the resume path, then the documentation is a bit misleading because it doesn't actually state if a file is required to be part of the path and also the command requires that we specify the --epoch-resume parameter with a number which is kind of redundant if we supplied the model.bin file from which to resume in --resume-path.

When I specify the --resume-path including the model.bin file it works. I just made sure to specify the model219.bin file so it matched with the number from --epoch-resume option. Although I'm not sure if the --resume-path can be any of the model.bin files

Some clarification on the correct usage of this command is definitely needed.

[polm]

so when it gets to the line 99 in pretrain.py it fails because its trying to open a directory as a file instead of an actual file.

Ah, that explains a lot. I didn't realize that Windows gives "permission denied" in this case - on Linux there's a specific IsADirectoryError.

We'll take a look at the docs/errors. We should be able to check if the argument is a directory and give a helpful message.

[polm]

the --epoch-resume parameter with a number which is kind of redundant if we supplied the model.bin file from which to resume in --resume-path.

Also, as a note here, if your model name uses the default pattern like model219.bin you don't have to specify the --epoch-resume parameter, the number will be inferred from the filename.