Store fog of war as a PNG image instead of TGA

[akortunov]

An alternative for #2580.

Just store fog of war as a PNG image (with its palette and compression) instead of TGA, which uses 4 bytes per pixel.
As a result, fog of war occupies about 1.5% of savegame instead of 20% without noticable performance difference (my testing save shrinked from 29.3MB to 23.9MB with this PR, and to 24.7MB with the raw data storage).

We do not need any additional 3d-party dependencies since we already store world map in savegames as a PNG image.

Basically, we will need to decide which data format would be better for us.

[AnyOldName3]

If we start compressing saves (which I think we should do), then a zipped PNG might well end up bigger than a zipped TGA, especially if it's a luminance-only TGA. Thoughts?

[wareya]

PNG will probably compress better if the save format uses a very fast compression format.

[AnyOldName3]

Why would it? Compression of compressed data usually makes it bigger because of the pigeonhole principle.

[wareya]

In terms of space, compressing already-compressed data is virtually free. The overhead is so tiny that it doesn't matter. You only lose significant space on container size, which is not an issue when everything is packed together.

PNG's preprocessing makes image data with a lot of fuzz or gradients compress better than it would be compressed by DEFLATE alone.

If I take this fog of war image:

and save it as a four-channel tga, then compress that tga in a zip file, it comes out to 1.15KB. The equivalent four-channel PNG is 1.06KB.

If I take this fog of war image:

(which is the same thing, but in brightness instead of alpha) then the three-channel PNG is 1.19KB, the three-channel TGA is 1.48KB in a zip file, and the four-channel TGA is 1.62KB in a zip file.

This is not attempting to make the point that PNGs will definitely be smaller, but rather that they will probably compress better, at least in complicated scenarios. In situations where e.g. the top or bottom half of the fog of war image are completely blank then DEFLATE'd TGAs might be better, but it's not clear without testing on a full mid-game save file whether one way of doing it is better than the other or not, and my intuition points towards PNGs being good.

EDIT: at some point github screwed up the first image in this post, it's supposed to be all alpha-channel with the color channels being black.

[AnyOldName3]

We're not talking about compressing files individually, though. They're as a part of a much longer stream of mostly dissimilar data. It's easier for a compression algorithm to see there are no changes it needs to make if it's not just compressed a bunch of other data really effectively. If it's already built up a dictionary or Markov chain weights or whatever, then it might well try using them, which is bad for PNG as the actual image data will resemble random noise, but good for TGA as the same stuff that worked for any previous TGAs will work again for the current one. (Note: I'm writing this without any specific details on the implementation of any particular compression algorithm and it might be the case that we don't end up hitting the degenerate case I'm envisioning here with what we end up using.)

Also, luminance-only or alpha-only images would be a much better comparison than multi-channel ones, as multi-channel gives a false advantage to PNG's domain-specific details.

[wareya]

Please read my message all the way through to the end including the final paragraph.

[AnyOldName3]

I already did and it changes very little. I'm explaining why my intuition is telling me that the reasoning behind your intuition is probably not completely sound.

What we need to do is get someone's mid-game save and edit it so we've got a version with optimised PNGs (i.e. whichever settings make them best, which I guess is that this PR does) and another with optimised TGAs (i.e. single-channel, potentially with the 18-byte header chopped off, which is effectively what #2580 does), and then compress them as a whole with all the algorithms we might use. Helpfully, we've already got two PRs which will actually edit the saves to have those image formats, and then it's just a case of throwing 7-zip/gzip/whatever at them.

[wareya]

I specifically went out of my way to say

• In terms of space, compressing already-compressed data is virtually free. The overhead is so tiny that it doesn't matter. You only lose significant space on container size, which is not an issue when everything is packed together.

and

• it's not clear without testing on a full mid-game save file whether one way of doing it is better than the other or not

which cover something like half of both of your posts. Don't get disturbed just because I wrote the word "intuition" somewhere. I posted actual numbers too. It's not hocus pocus.

Also, luminance-only or alpha-only images would be a much better comparison than multi-channel ones, as multi-channel gives a false advantage to PNG's domain-specific details.

General purpose compression algorithms are already very much prepared to handle arrays of 32-bit integers efficiently, which means that if the R, G, and B values are all zero (which they are in an alpha-only image) then it's not going to have a significant effect. If it had a significant effect then it would be shameful on the part of whoever designed the compression algorithm. In fact, this is exactly what I see. Stripping the RGB values from my 32-bit TGA file before compressing results in a compressed size of 1.56KB as opposed to 1.62KB. The difference is 58 bytes.

and then it's just a case of throwing 7-zip/gzip/whatever at them.

7-zip's LZMA is slow. You would not want to use it on game save data even if it saves 5% or 10% space over DEFLATE.

[akortunov]

BTW, why do you take in account overall savegame comression at all?
In out case it is just a way to hide design flaws (ineffective fog format, full data for every object in every visited cell, etc.)
For example, Morrowind does not need a compression at all since savegames have size < 5MB even after dozens of hours of gameplay, while in OpenMW such saves have size about 30MB .

[AnyOldName3]

I specifically went out of my way to say

In terms of space, compressing already-compressed data is virtually free. The overhead is so tiny that it doesn't matter. You only lose significant space on container size, which is not an issue when everything is packed together.

and

it's not clear without testing on a full mid-game save file whether one way of doing it is better than the other or not

which cover something like half of both of your posts. Don't get disturbed just because I wrote the word "intuition" somewhere. I posted actual numbers too. It's not hocus pocus.

It's almost like it was those specific points that half of both my posts were addressing. 😛

While writing this post, I tried coming up with a hypothetical reasonable-sounding compression scheme that had the behaviour I was concerned about, but then I fairly quickly came up with a minor change that mostly resolved it. I'm now convinced that any non-stupid compression scheme won't make a stream of mixed compressible and uncompressible data significantly bigger than its input (although it might end up bigger than if it had been given the two parts separately and the results had been concatenated). I no longer think this is a major concern for PNGs.

I was agreeing with you that we should try testing a mid-game save done both ways.

Also, luminance-only or alpha-only images would be a much better comparison than multi-channel ones, as multi-channel gives a false advantage to PNG's domain-specific details.

General purpose compression algorithms are already very much prepared to handle arrays of 32-bit integers efficiently, which means that if the R, G, and B values are all zero (which they are in an alpha-only image) then it's not going to have a significant effect. If it had a significant effect then it would be shameful on the part of whoever designed the compression algorithm. In fact, this is exactly what I see. Stripping the RGB values from my 32-bit TGA file before compressing results in a compressed size of 1.56KB as opposed to 1.62KB. The difference is 58 bytes.

I was expecting that to potentially make a bigger difference than that, despite knowing that it would already be something compression algorithms would handle pretty well. I was thinking that things like gradients might be more obvious without the other channels, but if the algorithm's considering each pixel as a 32-bit value rather than four separate channels of an image, there's nothing making patterns less obvious. It's amazing how much better brains work when they're properly awake.

and then it's just a case of throwing 7-zip/gzip/whatever at them.

7-zip's LZMA is slow. You would not want to use it on game save data even if it saves 5% or 10% space over DEFLATE.

7-zip can work with other compression schemes. I'm not going to use WinZip to apply DEFLATE.

BTW, why do you take in account overall savegame comression at all?
In out case it is just a way to hide design flaws (ineffective fog format, full data for every object in every visited cell, etc.)
For example, Morrowind does not need a compression at all since savegames have size < 5MB even after dozens of hours of gameplay, while in OpenMW such saves have size about 30MB .

You're talking to someone with double-digit gigabytes of Skyrim saves, despite most of them being in 7z files. We certainly have other, more pressing issues with the save format, but when mroe complicated setups become more common (e.g. TR or support for the later games) compression will still help even when we're keeping track of exactly as little data as possible.

I wanted a longer line break here, but it didn't appear, so you're getting a header

By the way, the example PNG used earlier is 1093 bytes long, whereas a luminance-only TGA would be 1042, even before compression. I know some of this will be because the TGA header is so tiny, and the PNG will probably be smaller if we chop the non-image data off both versions, but I'd expect there to be more opportunity to compress the TGA data then the PNG data with later steps.

[wareya]

I was agreeing with you that we should try testing a mid-game save done both ways.

I was half asleep when I wrote my posts too. In particular I definitely should have used a better word than "disturbed". It should've been something like "thrown off", plus my tone of voice could use some work.

I was expecting that to potentially make a bigger difference than that, despite knowing that it would already be something compression algorithms would handle pretty well. I was thinking that things like gradients might be more obvious without the other channels, but if the algorithm's considering each pixel as a 32-bit value rather than four separate channels of an image, there's nothing making patterns less obvious. It's amazing how much better brains work when they're properly awake.

Speaking of the gradients, it's very much possible that the fog of war texture I used happens to be a good case for PNG. Normal fog of war images might have significantly more fully visible or fully invisible area.

By the way, the example PNG used earlier is 1093 bytes long, whereas a luminance-only TGA would be 1042, even before compression. I know some of this will be because the TGA header is so tiny, and the PNG will probably be smaller if we chop the non-image data off both versions, but I'd expect there to be more opportunity to compress the TGA data then the PNG data with later steps.

You know, speaking of overhead, we could probably just store the fog of war textures as raw bitmaps with the height and width before them instead of actual image files if it turns out that PNG is less efficient for the average fog of war texture after all. Does OpenMW have a direct dependency on a DEFLATE implementation yet? Does Boost have one?

[AnyOldName3]

we could probably just store the fog of war textures as raw bitmaps with the height and width before them instead of actual image files

That's basically what a TGA is.

[wareya]

TGA has more going on in its header than just that, I mean it literally!

[AnyOldName3]

#2580 stores just the raw data without even the size as a prefix

[psi29a]

I like that we are discussing this, but I wonder where we draw the line on bike-shedding when we have two open PRs from @akortunov

I'm okay with making all saved images PNGs anyway.

I'm also okay with adding an additional library for compression, we'll need that anyway when compressing the whole save.

[AnyOldName3]

I think we're currently a little confused as the test image we were discussing ends up smaller with a #2580 approach, but akortunov said this one gave better results.

[psi29a]

Does this PR also allow for reading back old saves (backwards compatible)?

This is blocked until work on 0.47 begins.

[akortunov]

Does this PR also allow for reading back old saves (backwards compatible)?

It is backward compatible, but not forward compatible.
You may notice that this PR has the "savegame format label" and increases a savegame version.

[psi29a]

Yeah, i did notice that which is why I'm not touching it until 0.47 ;)

[AnyOldName3]

Would the raw data version be backwards compatible if we prepended it with a tga header? Even though OSG's tga writer can't write that format, it's reader can read it, so it should just be a question of determining how long ago I made it able to do that and if the last release could handle it.

[akortunov]

Would the raw data version be backwards compatible if we prepended it with a tga header?

It is already backwards compatible, it is not forward compatible, and will not be forward compatible even with header.

[AnyOldName3]

Care to explain why it wouldn't be? It's a luminance only tga of you give it the header of a luminance only tga.

Also, I belive I got the backwards and forwards the right way round as I was talking about the format rather than a release. It's OpenMW 0.45 that might not be forwards compatible if the new format isn't backwards compatible.

[akortunov]

Care to explain why it wouldn't be?

Such change is not something that can be merged just before release, so we will keep it for 0.47 branch. There are a lot of pending changes for savegave format, so 0.46 will not read saves from 0.47 anyway. As a result, manually added TGA header will not save us - a forward compatibility will be broken almost immediately, even if this hack technically works.

BTW, 0.45 already does not read fog from 0.46's saves on my hardware for some reason.

[Capostrophic]

BTW, 0.45 already does not read fog from 0.46's saves on my hardware for some reason.

Still works fine for me, curiously. 🤷‍♂️

[psi29a]

Since we already store maps in png, then we should be consistent. Thanks @akortunov