Carrierwave::Downloader::Base does not properly handle valid urls with colon characters in query

[coneybeare]

Valid Google Maps URI with colons:

https://maps.googleapis.com/maps/api/staticmap?format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:https://cdn.viewing.nyc/maps/[email protected]|scale:2|House+of+Wax%2C+445+Albee+Square+W+%234410%2C+Brooklyn%2C+NY+11201&scale=2&sensor=false&size=160x160

Error when running that URI through Carrierwave::Downloader::Base#process_uri:

Addressable::URI::InvalidURIError: Invalid scheme format: format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon
from /Users/coneybeare/.rvm/gems/ruby-2.6.2@viewingnyc/gems/addressable-2.7.0/lib/addressable/uri.rb:901:in `scheme='
[46] pry(main)> uri_parts

The problem exists in master, in the Downloader code here

What is happening here is that the colons in the query path are being interpreted as scheme delineators, and the Addressable::URI.encode(uri_parts.join('?')) then fails to encode.

This is a regression and was introduced with this change to the Carrierwave::Downloader::Base

Attempting to encode these colons as %3A caracters in the url before this method allows us to pass through without error, however the resulting url is different and not working:

https://maps.googleapis.com/maps/api/staticmap?format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon%253Ahttps%253A//cdn.viewing.nyc/maps/[email protected]%7Cscale%253A2%7CHouse+of+Wax%252C+445+Albee+Square+W+%25234410%252C+Brooklyn%252C+NY+11201&scale=2&sensor=false&size=160x160

One potential solution is to not use Addressable::URI.encode for the query, but the normal URI.encode:

[97] pry(main)> static_map_url
=> "https://maps.googleapis.com/maps/api/staticmap?format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:http://cdn.viewing.nyc/maps/[email protected]|scale:2|House+of+Wax%2C+445+Albee+Square+W+%234410%2C+Brooklyn%2C+NY+11201&scale=2&sensor=false&size=160x160"

[98] pry(main)> Addressable::URI.encode(static_map_url)
=> "https://maps.googleapis.com/maps/api/staticmap?format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:http://cdn.viewing.nyc/maps/[email protected]%7Cscale:2%7CHouse+of+Wax%252C+445+Albee+Square+W+%25234410%252C+Brooklyn%252C+NY+11201&scale=2&sensor=false&size=160x160"

[99] pry(main)> Addressable::URI.encode(static_map_url.split('?').last)
Addressable::URI::InvalidURIError: Invalid scheme format: format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon
from /Users/coneybeare/.rvm/gems/ruby-2.6.2@viewingnyc/gems/addressable-2.7.0/lib/addressable/uri.rb:901:in `scheme='

[100] pry(main)> URI.encode(static_map_url.split('?').last)
=> "format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:http://cdn.viewing.nyc/maps/[email protected]%7Cscale:2%7CHouse+of+Wax%252C+445+Albee+Square+W+%25234410%252C+Brooklyn%252C+NY+11201&scale=2&sensor=false&size=160x160"

Another potential solution is to split on : chars, then join again after encoding its parts:

[103] pry(main)> static_map_url
=> "https://maps.googleapis.com/maps/api/staticmap?format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:http://cdn.viewing.nyc/maps/[email protected]|scale:2|House+of+Wax%2C+445+Albee+Square+W+%234410%2C+Brooklyn%2C+NY+11201&scale=2&sensor=false&size=160x160"

[104] pry(main)> uri_parts = static_map_url.split('?')
=> ["https://maps.googleapis.com/maps/api/staticmap", "format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:http://cdn.viewing.nyc/maps/[email protected]|scale:2|House+of+Wax%2C+445+Albee+Square+W+%234410%2C+Brooklyn%2C+NY+11201&scale=2&sensor=false&size=160x160"]

[105] pry(main)> encoded_uri = Addressable::URI.parse(uri_parts.shift).normalize.to_s
=> "https://maps.googleapis.com/maps/api/staticmap"
"2|House+of+Wax%2C+445+Albee+Square+W+%234410%2C+Brooklyn%2C+NY+11201&scale=2&sensor=false&size=160x160"]

[106] pry(main)> query = uri_parts.join('?').split(':').map{|uri_part| Addressable::URI.encode(uri_part)}.join(':')
=> "format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:http://cdn.viewing.nyc/maps/[email protected]%7Cscale:2%7CHouse+of+Wax%252C+445+Albee+Square+W+%25234410%252C+Brooklyn%252C+NY+11201&scale=2&sensor=false&size=160x160"

[107] pry(main)> encoded_uri << '?' << query.gsub('%5B', '[').gsub('%5D', ']') if query.length
=> "https://maps.googleapis.com/maps/api/staticmap?format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:http://cdn.viewing.nyc/maps/[email protected]%7Cscale:2%7CHouse+of+Wax%252C+445+Albee+Square+W+%25234410%252C+Brooklyn%252C+NY+11201&scale=2&sensor=false&size=160x160"

[108] pry(main)> URI.parse(encoded_uri)
=> #<URI::HTTPS https://maps.googleapis.com/maps/api/staticmap?format=jpg&key=AIzaSyDPxZod4KFMi-SHCqCiMDWcbkg1zE-Gaf8&maptype=roadmap&markers=icon:http://cdn.viewing.nyc/maps/[email protected]%7Cscale:2%7CHouse+of+Wax%252C+445+Albee+Square+W+%25234410%252C+Brooklyn%252C+NY+11201&scale=2&sensor=false&size=160x160>

There may be more solutions.

[atitan]

We're experiencing similar issue with double-escaped characters.

Original URL:
https://my.image.resize.proxy/resize?height=600&nocrop=false&stripmeta=true&type=auto&url=http%3A%2F%2Fs3.amazonaws.com%2Fuploads%2Frecipe%2Fcover%2F212001%2F5d5abf197eaee26e.jpg&width=800

After process_uri
https://my.image.resize.proxy/resize?height=600&nocrop=false&stripmeta=true&type=auto&url=http%253A%252F%252Fs3.amazonaws.com%252Fuploads%252Frecipe%252Fcover%252F212001%252F5d5abf197eaee26e.jpg&width=800