use prodigy with mixedin data and pretrained spacy ner model get error.

[YanLiang1102]

I created an issue in Spacy for this.

[YanLiang1102]

explosion/spaCy#2496

[YanLiang1102]

well I hard coded and put the model path inside of spacy source code in here
File "/usr/local/lib/python3.5/dist-packages/spacy-2.0.12.dev0-py3.5-linux-x86_64.egg/spacy/util.py", line 119, in load_model
but after training for like 30 mins got and segmentation fault error.....:(

[YanLiang1102]

well that is becasue it uses up all the memory since the big pretrained vectors,
so use the our own vector and finally get it trained,
I made virtual environment and rebuild all the dependecy
and finally get it to run.
here is the perfomance with only the ldcdaat+prodigy data not perfect, since it is using ontoNotes orginal format, it dose not take advantage of the tag clustering stuff, we have and the anercorp data is not merged in too.

@ahalterman

[YanLiang1102]

@andy what is that BEFORE accuracy mean Andy? the 0.029 the mixed in ratio I am using is 5
so prodigy has 801 records, we will mix in the first 4005 records of onto_notes to the augmented data set.

[YanLiang1102]

also I need to manually copy the ar model into the dist-package forder for the virtual environment in order to fix the problem

[ahalterman]

The BEFORE is built in to Prodigy, I think. It just applies the existing model to the entities and reports the accuracy. So that must be the accuracy on model9.

[YanLiang1102]

I mean that accuracy is definitely not right on model9 it should be around 55% I guess that is with "the new format data" on model9?