jsonify no longer causes Content-Length to be set on returned Response object

[mbra]

I am not sure if this was an intentional change, as it is only triggered by jsonify by passing an iterable to flask.wrappers.Response/werkzeug.wrappers.Response.

In commit d9402fc a newline was added to the json response, as per a request from the httpbin project.

When werkzeug.wrappers.Response is passed a non text_type iterable, the self.set_data codepath, responsible for setting the Content-Length header, is not used anymore.

Ironcially this breaks httpbins "response-headers" route that uses the response object in a somewhat
creative way and breaks while trying to access response.headers['Content-Length']

If this considered a backwards incompatible change that might need to be undone, or is this considered new and intended behaviour?

[untitaker]

This is unintended behavior.

[untitaker]

From a first look the best fix is to explicitly set content-length in jsonify.

[mbra]

I do not have a good grasp of the Encoding Implications, but would it not help to just concat the newline onto the generated json string so we trigger the intended behaviour in werkzeug.wrappers.Response?

After all setting somewhat "special" headers might introduce other undesired side-effects over time.

[untitaker]

I'm not sure about the performance implications of concatenating strings like that. We already are O(n) (at least!), but I think somebody would report a performance regression because of this.

[ThiefMaster]

foo = json.dumps(...)
...
rv.headers['Content-length'] = len(foo) + 1

something like this could work (you know \n is 1 byte and i'd expect len() on a string to be O(1) in python)

[untitaker]

The point by @mbra was that if the behavior of Response changed wrt to the content-length header, jsonify wouldn't change.

Also we need to be careful here: len(u"ö") != len(u"ö".encode("utf-8")).

[ThiefMaster]

ah damn.. json dumps still returns a unicode string at least in python3. not sure if it'll contain any non-ascii chars though. AFAIK it escapes them all

but yeah, might be a bit fragile nonetheless

edit: just checked the docs:

If ensure_ascii is True (the default), all non-ASCII characters in the output are escaped with \uXXXX sequences, and the result is a str instance consisting of ASCII characters only.

So I guess it'd be safe since it's documented behavior to always return plain ascii

[mbra]

Well, I did not consider the performance penalty, the reallocations from the string concatenation could really be harmful, for large responses.

Perhaps calling werkzeug.wrappers.ResponseBase.calculate_content_length and setting the header is a good enough compromise, after all the real logic is kept in werkzeug. But I do not
know how often the additional _ensure_sequence Codepath would be, but at a first glance this
should not be as much of an issue as concating a large json object and a newline.

Also, the bad feeling I have about setting the Content-Length is that I do not know if and where a decision if made to use Transfer-Encoding: chunked and how that plays with manually setting Content-Length.

[mbra]

Mhh, after reading up on chunked Transfer and WSGI, this seems not to be an issue...

[untitaker]

Chunked encoding doesn't make much sense since we're not streaming. Content-Length is required by some clients, if it is suddenly missing I'm sure at least some code will break.

[untitaker]

The only alternative I could see involves StringIO, and calling json.dump on it.

@mitsuhiko Do you think this is breakage?

[mbra]

Mhh, perhaps this should be handled by werkzeug. The docs for automatically_set_content_length says: "Should this response object automatically set the content-length header if possible? This is true by default."

Perhaps I am missing something again, but calculating from a tuple should always be possible, right?
I know this just moves the same problem down the callchain but at least the Content-Length calculation is kept in one place.

[davidism]

Not sure when this got fixed, but I can't reproduce this. Examining the response from a client shows the correct content length. (headers['Content-Length'] isn't available, but it's eventually set.)