Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preserve element type in stateless layers #554

Closed
wants to merge 1 commit into from

Conversation

staticfloat
Copy link
Contributor

This should fix issues where Float32 weights and activations get suddenly switched to Float64 on the backwards pass. MWE to show current problem:

using Flux

x = param(randn(Float32, 10, 10, 4, 4)
mp = MeanPool((10,10))
Flux.back!(Flux.mse(mp(x), mp(x)))

This currently errors out with:

ERROR: MethodError: no method matching ∇meanpool(::Array{Float64,4}, ::Array{Float32,4}, ::Array{Float32,4}, ::Tuple{Int64,Int64}; pad=(0, 0), stride=(10, 10))

This is because the backward pass of mse() is spuriously generating Float64's. I don't really understand why it is doing that, but this PR seems to fix it.

This should fix issues where Float32 weights and activations get
suddenly switched to Float64 on the backwards pass
@staticfloat
Copy link
Contributor Author

It's possible this will fix #501

@DhairyaLGandhi
Copy link
Member

For further reference, #514, #538, #533 might be relevant.

@KristofferC
Copy link
Contributor

KristofferC commented Jan 14, 2019

This is because the backward pass of mse() is spuriously generating Float64's. I don't really understand why it is doing that, but this PR seems to fix it.

It does so because it divides by an Int and inv(::Integer) gives a Float64 (in the backwards pass). I think #538 is a cleaner solution.

@staticfloat
Copy link
Contributor Author

Agreed; #538 is the cleaner solution here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants