More options for Early stopping criteria

[aaaallleen]

I previously created a merge request for using metrics such as BLEU, TER as early stopping metrics. The current metrics perplexity and accuracy might be insufficient, in a lot of scenarios, ppl and acc might be deteriorating but the BLEU and TER metrics are still improving.

After going through the code base, I noticed that the implementation of early stopping is the same. If you guys feel that this could be useful, I could create a new PR and push my code for review.

[francoishernandez]

Seems ok to me. The BLEU/TER scoring codepath might evolve at some point, but that should not conflict with the early stopping logic which only accesses the stats. Thanks for contributing!

[aaaallleen]

The code logic is super simple, check if TER/BLEU is specified as an argument by user for validation, if so, obtain the values then use it for early stopping.

[vince62s]

tbh the community's consensus is now to use Comet so I think if one needs to use early stopping this is the best criteria IMO.

[aaaallleen]

I saw the PR and am looking at a COMET implementation, is anyone working on it?

[vince62s]

AFAIK no one is working on it.
However there are 2 ways to approach it:

1. with the "official Unbabel" API which would require to import the library
2. with a EOLE trained model (we support encoder with estimator hence similar to COMET models see the recipe)
   Both should be viable but maybe official COMET is an easier starting point.

[aaaallleen]

OK! I'll try and look into the details.