Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

osmpbf: direct decoding and ability to skip element types #18

Merged
merged 3 commits into from
Jan 9, 2021
Merged

Conversation

paulmach
Copy link
Owner

wrote https://github.com/paulmach/protoscan to provide the low-level ability to walk through protobuf data. This allows for:

  1. fast decoding with lower memory usage since the intermediate of the decoding proto objects can be skipped. This resulted in 45% less allocs and 20% less memory usage on the decoding benchmark.
  2. ability to skip certain types by moving past the encoded proto data directly. This skips all the work of decoding and building object that are not needed. However, it still requires the blocks be decompressed, so there still is some work.

I feel pretty good about this change, however writing code using protoscan is similar to handwriting the autogenerated protobuf code which could cause errors.

benchmark              old ns/op     new ns/op     delta
BenchmarkDecode-12     335335378     308820202     -7.91%

benchmark              old allocs     new allocs     delta
BenchmarkDecode-12     4544353        2473404        -45.57%

benchmark              old bytes      new bytes     delta
BenchmarkDecode-12     1245168211     983822340     -20.99%
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant