Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parameters of Linear should be swapped #414

Closed
vineetk1 opened this issue Dec 17, 2018 · 0 comments
Closed

parameters of Linear should be swapped #414

vineetk1 opened this issue Dec 17, 2018 · 0 comments

Comments

@vineetk1
Copy link
Contributor

File: https://github.com/pytorch/fairseq/blob/master/fairseq/models/lstm.py
Seems that in Line 269
self.input_proj = Linear(input_embed_dim, output_embed_dim, bias=False)
the parameters input_embed_dim and output_embed_dim should be swapped as follows:
self.input_proj = Linear(output_embed_dim, input_embed_dim, bias=False)

If Line 269 is not changed then Lines 277 and 280:
x = self.input_proj(input)
attn_scores = (source_hids * x.unsqueeze(0)).sum(dim=2)
should be replaced with the following two lines:
source_hids = self.input_proj(source_hids)
attn_scores = (source_hids * input.unsqueeze(0)).sum(dim=2)

moussaKam pushed a commit to moussaKam/language-adaptive-pretraining that referenced this issue Sep 29, 2020
Summary: Pull Request resolved: facebookresearch#470

Differential Revision: D13803964

Pulled By: myleott

fbshipit-source-id: 91b66599e9a539833fcedea07c608b349ba3b449
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant