Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug in RNNT Joint WER calculation for fused batch #8587

Merged
merged 1 commit into from
Mar 5, 2024
Merged

Conversation

titu1994
Copy link
Collaborator

@titu1994 titu1994 commented Mar 5, 2024

What does this PR do ?

Fix metric calculation in the case fused batch size > 1 during training / validation calculation.

Previously, the RNNT metric was stateful while the CTC one was not (r1.22.0, r1.23.0)

Therefore this calculation in the RNNT joint for fused operation worked properly. However with the unification of metrics in r1.23.0, a bug was introduced where only the last sub-batch of metrics calculates the scores and does not accumulate.

Collection: [ASR]

Changelog

  • Preserve the list of metric outputs from each sub-batch
  • take average of batch wer, compute running sum of numerator and denominator

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

@github-actions github-actions bot added the ASR label Mar 5, 2024
@titu1994 titu1994 requested a review from tbartley94 March 5, 2024 17:57
@titu1994
Copy link
Collaborator Author

titu1994 commented Mar 5, 2024

jenkins

Copy link
Collaborator

@nithinraok nithinraok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@titu1994 titu1994 merged commit 53f3add into main Mar 5, 2024
12 checks passed
@titu1994 titu1994 deleted the fix_rnnt_wer branch March 5, 2024 18:35
@tbartley94 tbartley94 mentioned this pull request Mar 5, 2024
2 tasks
@titu1994 titu1994 mentioned this pull request Mar 5, 2024
pzelasko pushed a commit that referenced this pull request Mar 7, 2024
zpx01 pushed a commit to zpx01/NeMo that referenced this pull request Mar 8, 2024
Agoniii pushed a commit to Agoniii/NeMo that referenced this pull request Mar 15, 2024
JRD971000 pushed a commit that referenced this pull request Mar 15, 2024
pablo-garay pushed a commit that referenced this pull request Mar 19, 2024
pzelasko added a commit that referenced this pull request Apr 16, 2024
…t-only) dataloading (#8581)

* wip

Signed-off-by: Piotr Żelasko <[email protected]>

* Partially working config groups

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with abasic group in the input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with nested groups in input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with specifying a YAML path for input_cfg

Signed-off-by: Piotr Żelasko <[email protected]>

* a very rough example of text dataloading via lhotse

Signed-off-by: Piotr Żelasko <[email protected]>

* Cleaner integration of multimodal audio/text loading that allows to control the effective audio vs text size (requires latest lhotse)

Signed-off-by: Piotr Żelasko <[email protected]>

* remove obsolete test

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix an import in export_utils.py (#8571)

Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Yttm deprecation (#8322)

* yttm deprecation init commit

Signed-off-by: AlexGrinch <[email protected]>

* removed tests

Signed-off-by: AlexGrinch <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bug fix

Signed-off-by: AlexGrinch <[email protected]>

* path fix

Signed-off-by: AlexGrinch <[email protected]>

* fixing path

Signed-off-by: AlexGrinch <[email protected]>

* updated tests to spm

Signed-off-by: AlexGrinch <[email protected]>

* updated Jenkinsfile

Signed-off-by: AlexGrinch <[email protected]>

* new model with spm in tests

Signed-off-by: AlexGrinch <[email protected]>

* yttm removed

Signed-off-by: AlexGrinch <[email protected]>

* updated aayn config

Signed-off-by: AlexGrinch <[email protected]>

---------

Signed-off-by: AlexGrinch <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed missing copy import in rnnt_decoder.py (#8580)

* Added copy import to rnnt_decoding.py

Signed-off-by: Isaac McFadyen <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Isaac McFadyen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix bug in RNNT Joint WER calculation for fused batch (#8587)

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed Context Parallel HtoD sync (#8557)

* Fixed cp HtoD sync

Signed-off-by: Selvaraj Anandaraj <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* change default and add key to config files (#8594)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix triton import guards (#8552)

* Fix triton import guards

Signed-off-by: Michal Futrega <[email protected]>

* Update attention.py

Signed-off-by: Michal Futrega <[email protected]>

---------

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Add config key for dropout position in LoRA adapter (#8583)

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* fix ia3 mlp infused adapter (#8597)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Prevent Redundant Gather for LoRA Sequence Parallel (#8602)

* enable layernorm output gathered

Signed-off-by: Chen Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chen Cui <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Accelerate `transcribe_speech.py` for short-form data: pre-sorting support (#8564)

* POC using bucketing in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* extend to multi task aed

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes for aed multi task text/lang field selectors

Signed-off-by: Piotr Żelasko <[email protected]>

* remove assert

Signed-off-by: Piotr Żelasko <[email protected]>

* fix

Signed-off-by: Piotr Żelasko <[email protected]>

* expose option for bucket buffer size

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes, ctc support

Signed-off-by: Piotr Żelasko <[email protected]>

* support pre-sorting manifests in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* cleanup

Signed-off-by: Piotr Żelasko <[email protected]>

* reorder transcriptions back to original manifest order

Signed-off-by: Piotr Żelasko <[email protected]>

* remove bucketing entirely

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes--amend

Signed-off-by: Piotr Żelasko <[email protected]>

* refactor text_field/lang_field passing

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix reordering bug; disable presorting for multi task for now

Signed-off-by: Piotr Żelasko <[email protected]>

* Add support for presort + multi task model

Signed-off-by: Piotr Żelasko <[email protected]>

* Code reviews

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix jenkins tests, add user-friendly error msg for canary

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>

* fix tests

Signed-off-by: Piotr Żelasko <[email protected]>

* Bump min required lhotse version

Signed-off-by: Piotr Żelasko <[email protected]>

* Add some documentation about this config format and the multimodal features

Signed-off-by: Piotr Żelasko <[email protected]>

* Add caution about multiple shards

Signed-off-by: Piotr Żelasko <[email protected]>

* Address Tom's code review

Signed-off-by: Piotr Żelasko <[email protected]>

* Add copyright header

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix (hopefully) issue with forced ascii encoding in CI

Signed-off-by: Piotr Żelasko <[email protected]>

* Support resolving input_cfg path into config contents

Signed-off-by: Piotr Żelasko <[email protected]>

* Code review changes in docs

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix unicode decode error

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>
Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: AlexGrinch <[email protected]>
Signed-off-by: Isaac McFadyen <[email protected]>
Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Selvaraj Anandaraj <[email protected]>
Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Michal Futrega <[email protected]>
Co-authored-by: w4-jinhyeonkim <[email protected]>
Co-authored-by: Aleksey Grinchuk (Oleksii Hrinchuk) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Isaac McFadyen <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Chen Cui <[email protected]>
Co-authored-by: Michal Futrega <[email protected]>
Co-authored-by: Pablo Garay <[email protected]>
xingyaoww pushed a commit to xingyaoww/NeMo that referenced this pull request Apr 23, 2024
…t-only) dataloading (NVIDIA#8581)

* wip

Signed-off-by: Piotr Żelasko <[email protected]>

* Partially working config groups

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with abasic group in the input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with nested groups in input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with specifying a YAML path for input_cfg

Signed-off-by: Piotr Żelasko <[email protected]>

* a very rough example of text dataloading via lhotse

Signed-off-by: Piotr Żelasko <[email protected]>

* Cleaner integration of multimodal audio/text loading that allows to control the effective audio vs text size (requires latest lhotse)

Signed-off-by: Piotr Żelasko <[email protected]>

* remove obsolete test

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix an import in export_utils.py (NVIDIA#8571)

Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Yttm deprecation (NVIDIA#8322)

* yttm deprecation init commit

Signed-off-by: AlexGrinch <[email protected]>

* removed tests

Signed-off-by: AlexGrinch <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bug fix

Signed-off-by: AlexGrinch <[email protected]>

* path fix

Signed-off-by: AlexGrinch <[email protected]>

* fixing path

Signed-off-by: AlexGrinch <[email protected]>

* updated tests to spm

Signed-off-by: AlexGrinch <[email protected]>

* updated Jenkinsfile

Signed-off-by: AlexGrinch <[email protected]>

* new model with spm in tests

Signed-off-by: AlexGrinch <[email protected]>

* yttm removed

Signed-off-by: AlexGrinch <[email protected]>

* updated aayn config

Signed-off-by: AlexGrinch <[email protected]>

---------

Signed-off-by: AlexGrinch <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed missing copy import in rnnt_decoder.py (NVIDIA#8580)

* Added copy import to rnnt_decoding.py

Signed-off-by: Isaac McFadyen <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Isaac McFadyen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix bug in RNNT Joint WER calculation for fused batch (NVIDIA#8587)

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed Context Parallel HtoD sync (NVIDIA#8557)

* Fixed cp HtoD sync

Signed-off-by: Selvaraj Anandaraj <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* change default and add key to config files (NVIDIA#8594)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix triton import guards (NVIDIA#8552)

* Fix triton import guards

Signed-off-by: Michal Futrega <[email protected]>

* Update attention.py

Signed-off-by: Michal Futrega <[email protected]>

---------

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Add config key for dropout position in LoRA adapter (NVIDIA#8583)

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* fix ia3 mlp infused adapter (NVIDIA#8597)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Prevent Redundant Gather for LoRA Sequence Parallel (NVIDIA#8602)

* enable layernorm output gathered

Signed-off-by: Chen Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chen Cui <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Accelerate `transcribe_speech.py` for short-form data: pre-sorting support (NVIDIA#8564)

* POC using bucketing in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* extend to multi task aed

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes for aed multi task text/lang field selectors

Signed-off-by: Piotr Żelasko <[email protected]>

* remove assert

Signed-off-by: Piotr Żelasko <[email protected]>

* fix

Signed-off-by: Piotr Żelasko <[email protected]>

* expose option for bucket buffer size

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes, ctc support

Signed-off-by: Piotr Żelasko <[email protected]>

* support pre-sorting manifests in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* cleanup

Signed-off-by: Piotr Żelasko <[email protected]>

* reorder transcriptions back to original manifest order

Signed-off-by: Piotr Żelasko <[email protected]>

* remove bucketing entirely

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes--amend

Signed-off-by: Piotr Żelasko <[email protected]>

* refactor text_field/lang_field passing

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix reordering bug; disable presorting for multi task for now

Signed-off-by: Piotr Żelasko <[email protected]>

* Add support for presort + multi task model

Signed-off-by: Piotr Żelasko <[email protected]>

* Code reviews

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix jenkins tests, add user-friendly error msg for canary

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>

* fix tests

Signed-off-by: Piotr Żelasko <[email protected]>

* Bump min required lhotse version

Signed-off-by: Piotr Żelasko <[email protected]>

* Add some documentation about this config format and the multimodal features

Signed-off-by: Piotr Żelasko <[email protected]>

* Add caution about multiple shards

Signed-off-by: Piotr Żelasko <[email protected]>

* Address Tom's code review

Signed-off-by: Piotr Żelasko <[email protected]>

* Add copyright header

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix (hopefully) issue with forced ascii encoding in CI

Signed-off-by: Piotr Żelasko <[email protected]>

* Support resolving input_cfg path into config contents

Signed-off-by: Piotr Żelasko <[email protected]>

* Code review changes in docs

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix unicode decode error

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>
Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: AlexGrinch <[email protected]>
Signed-off-by: Isaac McFadyen <[email protected]>
Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Selvaraj Anandaraj <[email protected]>
Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Michal Futrega <[email protected]>
Co-authored-by: w4-jinhyeonkim <[email protected]>
Co-authored-by: Aleksey Grinchuk (Oleksii Hrinchuk) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Isaac McFadyen <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Chen Cui <[email protected]>
Co-authored-by: Michal Futrega <[email protected]>
Co-authored-by: Pablo Garay <[email protected]>
alxzhang-amazon pushed a commit to alxzhang-amazon/NeMo that referenced this pull request Apr 26, 2024
…t-only) dataloading (NVIDIA#8581)

* wip

Signed-off-by: Piotr Żelasko <[email protected]>

* Partially working config groups

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with abasic group in the input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with nested groups in input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with specifying a YAML path for input_cfg

Signed-off-by: Piotr Żelasko <[email protected]>

* a very rough example of text dataloading via lhotse

Signed-off-by: Piotr Żelasko <[email protected]>

* Cleaner integration of multimodal audio/text loading that allows to control the effective audio vs text size (requires latest lhotse)

Signed-off-by: Piotr Żelasko <[email protected]>

* remove obsolete test

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix an import in export_utils.py (NVIDIA#8571)

Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Yttm deprecation (NVIDIA#8322)

* yttm deprecation init commit

Signed-off-by: AlexGrinch <[email protected]>

* removed tests

Signed-off-by: AlexGrinch <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bug fix

Signed-off-by: AlexGrinch <[email protected]>

* path fix

Signed-off-by: AlexGrinch <[email protected]>

* fixing path

Signed-off-by: AlexGrinch <[email protected]>

* updated tests to spm

Signed-off-by: AlexGrinch <[email protected]>

* updated Jenkinsfile

Signed-off-by: AlexGrinch <[email protected]>

* new model with spm in tests

Signed-off-by: AlexGrinch <[email protected]>

* yttm removed

Signed-off-by: AlexGrinch <[email protected]>

* updated aayn config

Signed-off-by: AlexGrinch <[email protected]>

---------

Signed-off-by: AlexGrinch <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed missing copy import in rnnt_decoder.py (NVIDIA#8580)

* Added copy import to rnnt_decoding.py

Signed-off-by: Isaac McFadyen <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Isaac McFadyen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix bug in RNNT Joint WER calculation for fused batch (NVIDIA#8587)

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed Context Parallel HtoD sync (NVIDIA#8557)

* Fixed cp HtoD sync

Signed-off-by: Selvaraj Anandaraj <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* change default and add key to config files (NVIDIA#8594)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix triton import guards (NVIDIA#8552)

* Fix triton import guards

Signed-off-by: Michal Futrega <[email protected]>

* Update attention.py

Signed-off-by: Michal Futrega <[email protected]>

---------

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Add config key for dropout position in LoRA adapter (NVIDIA#8583)

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* fix ia3 mlp infused adapter (NVIDIA#8597)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Prevent Redundant Gather for LoRA Sequence Parallel (NVIDIA#8602)

* enable layernorm output gathered

Signed-off-by: Chen Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chen Cui <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Accelerate `transcribe_speech.py` for short-form data: pre-sorting support (NVIDIA#8564)

* POC using bucketing in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* extend to multi task aed

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes for aed multi task text/lang field selectors

Signed-off-by: Piotr Żelasko <[email protected]>

* remove assert

Signed-off-by: Piotr Żelasko <[email protected]>

* fix

Signed-off-by: Piotr Żelasko <[email protected]>

* expose option for bucket buffer size

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes, ctc support

Signed-off-by: Piotr Żelasko <[email protected]>

* support pre-sorting manifests in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* cleanup

Signed-off-by: Piotr Żelasko <[email protected]>

* reorder transcriptions back to original manifest order

Signed-off-by: Piotr Żelasko <[email protected]>

* remove bucketing entirely

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes--amend

Signed-off-by: Piotr Żelasko <[email protected]>

* refactor text_field/lang_field passing

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix reordering bug; disable presorting for multi task for now

Signed-off-by: Piotr Żelasko <[email protected]>

* Add support for presort + multi task model

Signed-off-by: Piotr Żelasko <[email protected]>

* Code reviews

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix jenkins tests, add user-friendly error msg for canary

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>

* fix tests

Signed-off-by: Piotr Żelasko <[email protected]>

* Bump min required lhotse version

Signed-off-by: Piotr Żelasko <[email protected]>

* Add some documentation about this config format and the multimodal features

Signed-off-by: Piotr Żelasko <[email protected]>

* Add caution about multiple shards

Signed-off-by: Piotr Żelasko <[email protected]>

* Address Tom's code review

Signed-off-by: Piotr Żelasko <[email protected]>

* Add copyright header

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix (hopefully) issue with forced ascii encoding in CI

Signed-off-by: Piotr Żelasko <[email protected]>

* Support resolving input_cfg path into config contents

Signed-off-by: Piotr Żelasko <[email protected]>

* Code review changes in docs

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix unicode decode error

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>
Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: AlexGrinch <[email protected]>
Signed-off-by: Isaac McFadyen <[email protected]>
Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Selvaraj Anandaraj <[email protected]>
Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Michal Futrega <[email protected]>
Co-authored-by: w4-jinhyeonkim <[email protected]>
Co-authored-by: Aleksey Grinchuk (Oleksii Hrinchuk) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Isaac McFadyen <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Chen Cui <[email protected]>
Co-authored-by: Michal Futrega <[email protected]>
Co-authored-by: Pablo Garay <[email protected]>
suiyoubi pushed a commit that referenced this pull request May 2, 2024
…t-only) dataloading (#8581)

* wip

Signed-off-by: Piotr Żelasko <[email protected]>

* Partially working config groups

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with abasic group in the input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with nested groups in input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with specifying a YAML path for input_cfg

Signed-off-by: Piotr Żelasko <[email protected]>

* a very rough example of text dataloading via lhotse

Signed-off-by: Piotr Żelasko <[email protected]>

* Cleaner integration of multimodal audio/text loading that allows to control the effective audio vs text size (requires latest lhotse)

Signed-off-by: Piotr Żelasko <[email protected]>

* remove obsolete test

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix an import in export_utils.py (#8571)

Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Yttm deprecation (#8322)

* yttm deprecation init commit

Signed-off-by: AlexGrinch <[email protected]>

* removed tests

Signed-off-by: AlexGrinch <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bug fix

Signed-off-by: AlexGrinch <[email protected]>

* path fix

Signed-off-by: AlexGrinch <[email protected]>

* fixing path

Signed-off-by: AlexGrinch <[email protected]>

* updated tests to spm

Signed-off-by: AlexGrinch <[email protected]>

* updated Jenkinsfile

Signed-off-by: AlexGrinch <[email protected]>

* new model with spm in tests

Signed-off-by: AlexGrinch <[email protected]>

* yttm removed

Signed-off-by: AlexGrinch <[email protected]>

* updated aayn config

Signed-off-by: AlexGrinch <[email protected]>

---------

Signed-off-by: AlexGrinch <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed missing copy import in rnnt_decoder.py (#8580)

* Added copy import to rnnt_decoding.py

Signed-off-by: Isaac McFadyen <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Isaac McFadyen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix bug in RNNT Joint WER calculation for fused batch (#8587)

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed Context Parallel HtoD sync (#8557)

* Fixed cp HtoD sync

Signed-off-by: Selvaraj Anandaraj <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* change default and add key to config files (#8594)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix triton import guards (#8552)

* Fix triton import guards

Signed-off-by: Michal Futrega <[email protected]>

* Update attention.py

Signed-off-by: Michal Futrega <[email protected]>

---------

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Add config key for dropout position in LoRA adapter (#8583)

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* fix ia3 mlp infused adapter (#8597)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Prevent Redundant Gather for LoRA Sequence Parallel (#8602)

* enable layernorm output gathered

Signed-off-by: Chen Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chen Cui <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Accelerate `transcribe_speech.py` for short-form data: pre-sorting support (#8564)

* POC using bucketing in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* extend to multi task aed

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes for aed multi task text/lang field selectors

Signed-off-by: Piotr Żelasko <[email protected]>

* remove assert

Signed-off-by: Piotr Żelasko <[email protected]>

* fix

Signed-off-by: Piotr Żelasko <[email protected]>

* expose option for bucket buffer size

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes, ctc support

Signed-off-by: Piotr Żelasko <[email protected]>

* support pre-sorting manifests in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* cleanup

Signed-off-by: Piotr Żelasko <[email protected]>

* reorder transcriptions back to original manifest order

Signed-off-by: Piotr Żelasko <[email protected]>

* remove bucketing entirely

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes--amend

Signed-off-by: Piotr Żelasko <[email protected]>

* refactor text_field/lang_field passing

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix reordering bug; disable presorting for multi task for now

Signed-off-by: Piotr Żelasko <[email protected]>

* Add support for presort + multi task model

Signed-off-by: Piotr Żelasko <[email protected]>

* Code reviews

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix jenkins tests, add user-friendly error msg for canary

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>

* fix tests

Signed-off-by: Piotr Żelasko <[email protected]>

* Bump min required lhotse version

Signed-off-by: Piotr Żelasko <[email protected]>

* Add some documentation about this config format and the multimodal features

Signed-off-by: Piotr Żelasko <[email protected]>

* Add caution about multiple shards

Signed-off-by: Piotr Żelasko <[email protected]>

* Address Tom's code review

Signed-off-by: Piotr Żelasko <[email protected]>

* Add copyright header

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix (hopefully) issue with forced ascii encoding in CI

Signed-off-by: Piotr Żelasko <[email protected]>

* Support resolving input_cfg path into config contents

Signed-off-by: Piotr Żelasko <[email protected]>

* Code review changes in docs

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix unicode decode error

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>
Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: AlexGrinch <[email protected]>
Signed-off-by: Isaac McFadyen <[email protected]>
Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Selvaraj Anandaraj <[email protected]>
Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Michal Futrega <[email protected]>
Co-authored-by: w4-jinhyeonkim <[email protected]>
Co-authored-by: Aleksey Grinchuk (Oleksii Hrinchuk) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Isaac McFadyen <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Chen Cui <[email protected]>
Co-authored-by: Michal Futrega <[email protected]>
Co-authored-by: Pablo Garay <[email protected]>
Signed-off-by: Ao Tang <[email protected]>
rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request Jun 25, 2024
rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request Jun 25, 2024
…t-only) dataloading (NVIDIA#8581)

* wip

Signed-off-by: Piotr Żelasko <[email protected]>

* Partially working config groups

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with abasic group in the input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with nested groups in input config

Signed-off-by: Piotr Żelasko <[email protected]>

* Working test with specifying a YAML path for input_cfg

Signed-off-by: Piotr Żelasko <[email protected]>

* a very rough example of text dataloading via lhotse

Signed-off-by: Piotr Żelasko <[email protected]>

* Cleaner integration of multimodal audio/text loading that allows to control the effective audio vs text size (requires latest lhotse)

Signed-off-by: Piotr Żelasko <[email protected]>

* remove obsolete test

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix an import in export_utils.py (NVIDIA#8571)

Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Yttm deprecation (NVIDIA#8322)

* yttm deprecation init commit

Signed-off-by: AlexGrinch <[email protected]>

* removed tests

Signed-off-by: AlexGrinch <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* bug fix

Signed-off-by: AlexGrinch <[email protected]>

* path fix

Signed-off-by: AlexGrinch <[email protected]>

* fixing path

Signed-off-by: AlexGrinch <[email protected]>

* updated tests to spm

Signed-off-by: AlexGrinch <[email protected]>

* updated Jenkinsfile

Signed-off-by: AlexGrinch <[email protected]>

* new model with spm in tests

Signed-off-by: AlexGrinch <[email protected]>

* yttm removed

Signed-off-by: AlexGrinch <[email protected]>

* updated aayn config

Signed-off-by: AlexGrinch <[email protected]>

---------

Signed-off-by: AlexGrinch <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed missing copy import in rnnt_decoder.py (NVIDIA#8580)

* Added copy import to rnnt_decoding.py

Signed-off-by: Isaac McFadyen <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Isaac McFadyen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix bug in RNNT Joint WER calculation for fused batch (NVIDIA#8587)

Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fixed Context Parallel HtoD sync (NVIDIA#8557)

* Fixed cp HtoD sync

Signed-off-by: Selvaraj Anandaraj <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* change default and add key to config files (NVIDIA#8594)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Fix triton import guards (NVIDIA#8552)

* Fix triton import guards

Signed-off-by: Michal Futrega <[email protected]>

* Update attention.py

Signed-off-by: Michal Futrega <[email protected]>

---------

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Add config key for dropout position in LoRA adapter (NVIDIA#8583)

Signed-off-by: Michal Futrega <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* fix ia3 mlp infused adapter (NVIDIA#8597)

Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>

* Prevent Redundant Gather for LoRA Sequence Parallel (NVIDIA#8602)

* enable layernorm output gathered

Signed-off-by: Chen Cui <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chen Cui <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Piotr Żelasko <[email protected]>

* Accelerate `transcribe_speech.py` for short-form data: pre-sorting support (NVIDIA#8564)

* POC using bucketing in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* extend to multi task aed

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes for aed multi task text/lang field selectors

Signed-off-by: Piotr Żelasko <[email protected]>

* remove assert

Signed-off-by: Piotr Żelasko <[email protected]>

* fix

Signed-off-by: Piotr Żelasko <[email protected]>

* expose option for bucket buffer size

Signed-off-by: Piotr Żelasko <[email protected]>

* fixes, ctc support

Signed-off-by: Piotr Żelasko <[email protected]>

* support pre-sorting manifests in transcribe_speech.py

Signed-off-by: Piotr Żelasko <[email protected]>

* cleanup

Signed-off-by: Piotr Żelasko <[email protected]>

* reorder transcriptions back to original manifest order

Signed-off-by: Piotr Żelasko <[email protected]>

* remove bucketing entirely

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes

Signed-off-by: Piotr Żelasko <[email protected]>

* code review changes--amend

Signed-off-by: Piotr Żelasko <[email protected]>

* refactor text_field/lang_field passing

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix reordering bug; disable presorting for multi task for now

Signed-off-by: Piotr Żelasko <[email protected]>

* Add support for presort + multi task model

Signed-off-by: Piotr Żelasko <[email protected]>

* Code reviews

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix jenkins tests, add user-friendly error msg for canary

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>

* fix tests

Signed-off-by: Piotr Żelasko <[email protected]>

* Bump min required lhotse version

Signed-off-by: Piotr Żelasko <[email protected]>

* Add some documentation about this config format and the multimodal features

Signed-off-by: Piotr Żelasko <[email protected]>

* Add caution about multiple shards

Signed-off-by: Piotr Żelasko <[email protected]>

* Address Tom's code review

Signed-off-by: Piotr Żelasko <[email protected]>

* Add copyright header

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix (hopefully) issue with forced ascii encoding in CI

Signed-off-by: Piotr Żelasko <[email protected]>

* Support resolving input_cfg path into config contents

Signed-off-by: Piotr Żelasko <[email protected]>

* Code review changes in docs

Signed-off-by: Piotr Żelasko <[email protected]>

* Fix unicode decode error

Signed-off-by: Piotr Żelasko <[email protected]>

---------

Signed-off-by: Piotr Żelasko <[email protected]>
Signed-off-by: w4-jinhyeonkim <[email protected]>
Signed-off-by: AlexGrinch <[email protected]>
Signed-off-by: Isaac McFadyen <[email protected]>
Signed-off-by: smajumdar <[email protected]>
Signed-off-by: Selvaraj Anandaraj <[email protected]>
Signed-off-by: Chen Cui <[email protected]>
Signed-off-by: Michal Futrega <[email protected]>
Co-authored-by: w4-jinhyeonkim <[email protected]>
Co-authored-by: Aleksey Grinchuk (Oleksii Hrinchuk) <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Isaac McFadyen <[email protected]>
Co-authored-by: Somshubra Majumdar <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Selvaraj Anandaraj <[email protected]>
Co-authored-by: Chen Cui <[email protected]>
Co-authored-by: Michal Futrega <[email protected]>
Co-authored-by: Pablo Garay <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants