Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Remove usage of nested lmdb transaction #1084

Merged
merged 1 commit into from
Feb 28, 2023
Merged

perf: Remove usage of nested lmdb transaction #1084

merged 1 commit into from
Feb 28, 2023

Conversation

chubei
Copy link
Contributor

@chubei chubei commented Feb 28, 2023

Spotted this in the flamegraph.

Test method

Tested using data and config of taxi-objectstore in https://github.com/getdozer/dozer-samples-internal/commit/e4b719153a9d82f5fba55bb9637d44cd946f6304. Run following command:

rm -r .dozer; time ~/dozer/target/release/dozer

When trips_cache count reaches 200,000, press Ctrl + C and wait for dozer to quit.

Result

With code in 0c549a2:

~/dozer/target/release/dozer 28.25s user 23.84s system 101% cpu 51.526 total

With code in https://github.com/chubei/dozer/commit/ac46db09aca5b0751b07aaf69bb28acda872ee48:

~/dozer/target/release/dozer 13.62s user 10.19s system 115% cpu 20.578 total

Saves 60%.

@snork-alt
Copy link
Contributor

nice!

@coveralls
Copy link

Pull Request Test Coverage Report for Build 4289337951

  • 3 of 3 (100.0%) changed or added relevant lines in 1 file are covered.
  • 160 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.09%) to 70.571%

Files with Coverage Reduction New Missed Lines %
dozer-cache/src/cache/lmdb/indexer.rs 40 75.98%
dozer-core/src/record_store.rs 57 82.78%
dozer-sql/src/pipeline/planner/projection.rs 63 60.38%
Totals Coverage Status
Change from base Build 4284179721: -0.09%
Covered Lines: 27145
Relevant Lines: 38465

💛 - Coveralls

@chubei chubei merged commit a74386e into getdozer:main Feb 28, 2023
@chubei chubei deleted the perf/nested branch February 28, 2023 05:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants