Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDDS-10926. Block deletion should update container merkle tree. #6875

Merged
merged 28 commits into from
Aug 6, 2024
Merged

HDDS-10926. Block deletion should update container merkle tree. #6875

merged 28 commits into from
Aug 6, 2024

Conversation

errose28
Copy link
Contributor

@errose28 errose28 commented Jun 27, 2024 •
edited
Loading

What changes were proposed in this pull request?

When a block is deleted from a container, it should be marked as deleted in the container merkle tree. This tells peer datanodes that the data be skipped if they have not yet processed the deletion but find there is an error with this block. The chunk, block, and container hashes in the tree should not change.

This test also moves some existing test methods to a test util class since more tests, including those in this patch, need to use them.

What is the link to the Apache JIRA

HDDS-10926

How was this patch tested?

Unit tests added

@errose28
Initial outline of checksum manager and merkle tree with proto
bf0dd6e
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into mt-manager-wip
408cff9 
* HDDS-10239-container-reconciliation:
  HDDS-10372. SCM and Datanode communication for reconciliation (apache#6506)
  HDDS-10239. Storage Container Reconciliation. (apache#6121)
@errose28
Add config key for lock stripes
3adbe5a
@errose28
Keep deleted block list sorted
e02a609
@errose28
Change chunk checksum proto, add the first passing test
3680a2e
@errose28
Use config defaults for test setup
0d5800c
@errose28
Finish tests for ContainerMerkleTree
eb01da4
@errose28
Add tests for checksum maanger and standardize proto names
f35effd
@errose28
Updates after reviewing diff
21dd870
@errose28
Rename checksum manager and file. Fix findbugs and Rat
47cd213
@errose28
Make checksum file read/write for the API based on files, not protos
6460fe7
@errose28
Initially mark where blocks will be added
ff55f27
@errose28
Block deleting task supports updating the file
fffeba6
@errose28
Basic block delete test passes
4822941
@errose28
Add test for block delete commands retried
6087fa3
@errose28
Rename data tree to container tree
9714259
@errose28
Merge branch 'HDDS-10887-mt-manager' into HDDS-10926-mt-delete
aacac83 
* HDDS-10887-mt-manager:
  Rename data tree to container tree
@errose28
Use generic container type in checksum manager
3161ad8
@errose28
Checkstyle
041c5c2
@kerneltime
Copy link
Contributor

Will give another look post addressing of review comments and rebase.

@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10926-mt…
838eb59 
…-delete

* HDDS-10239-container-reconciliation:
  HDDS-10887. Implement a basic Merkle Tree Manager. (apache#6778)
  HDDS-10923. Container Scanner should still scan unhealthy containers. (apache#6809)

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
@errose28 errose28 mentioned this pull request Jul 26, 2024
Merged
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10926-mt…
be95b85 
…-delete

* HDDS-10239-container-reconciliation: (184 commits)
  HDDS-10373. Implement framework for capturing Merkle Tree Metrics. (apache#6864)
  HDDS-11188. Initial setup for new UI layout and enable users to switch to new UI (apache#6953)
  HDDS-11120. Rich rebalancing status info (apache#6911)
  HDDS-11187. Fix Event Handling in Recon OMDBUpdatesHandler to Prevent ClassCastException. (apache#6950)
  HDDS-11213. Bump commons-daemon to 1.4.0 (apache#6971)
  HDDS-11212. Bump commons-net to 3.11.1 (apache#6973)
  HDDS-11211. Bump assertj-core to 3.26.3 (apache#6972)
  HDDS-11210. Bump log4j2 to 2.23.1 (apache#6970)
  HDDS-11150. Recon Overview page crashes due to failed API Calls (apache#6944)
  HDDS-11183. Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot (apache#6946)
  HDDS-11198. Fix Typescript configs for Recon (apache#6961)
  HDDS-11180. Simplify HttpServer2#inferMimeType return statement (apache#6963)
  HDDS-11194. OM missing audit log for upgrade (apache#6958)
  HDDS-10389. Implement a search feature for users to locate open keys within the Open Keys Insights section. (apache#6231)
  HDDS-10561. Dashboard for delete key metrics (apache#6948)
  HDDS-11192. Increase SPNEGO URL test coverage (apache#6956)
  HDDS-11179. DBConfigFromFile#readFromFile result of toIOException not thrown (apache#6957)
  HDDS-11186. First container log missing from bundle (apache#6952)
  HDDS-10844. Clarify snapshot create error message. (apache#6955)
  HDDS-11166. Switch to Rocky Linux-based ozone-runner (apache#6942)
  ...
@errose28
Use block name in task
245844c
@errose28
Update test description
51e7020
@errose28
Copy link
Contributor Author

errose28 commented Jul 31, 2024 •
edited
Loading

For tests to pass I had to add the same metrics unregister workaround from #6945. HDDS-10239 HDDS-11254 will properly close the checksum tree manager when everything is connected and this will no longer be necessary.

@errose28 errose28 marked this pull request as ready for review July 31, 2024 19:25
@errose28
Copy link
Contributor Author

Run on my fork is green, waiting for verification here as well.

aswinshakil
aswinshakil approved these changes Aug 6, 2024
View reviewed changes
Copy link
Member

@aswinshakil aswinshakil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @errose28. LGTM+1

@errose28 errose28 merged commit d7f302e into apache:HDDS-10239-container-reconciliation Aug 6, 2024
50 checks passed
@smengcl
Copy link
Contributor

smengcl commented Aug 6, 2024

Thanks @errose28 . Nice commit messages btw :D

errose28 added a commit to errose28/ozone that referenced this pull request Aug 13, 2024
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-11254-re…
6f68df6 
…p-supervisor

Merge conflicts are resolved but the change does not yet build.

* HDDS-10239-container-reconciliation: (183 commits)
  HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945)
  HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024)
  HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043)
  HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040)
  HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036)
  HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003)
  HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724)
  HDDS-10926. Block deletion should update container merkle tree. (apache#6875)
  HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021)
  HDDS-11272. Statistics some node status information (apache#7025)
  HDDS-11278. Move code out of Hadoop util package (apache#7028)
  HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones
  HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026)
  HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032)
  HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990)
  HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023)
  HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007)
  HDDS-11224. Increase hdds.datanode.handler.count (apache#7011)
  HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012)
  HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
errose28 added a commit to errose28/ozone that referenced this pull request Aug 16, 2024
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-11253-co…
8601118 
…rrupt-files

* HDDS-10239-container-reconciliation: (183 commits)
  HDDS-10376. Add a Datanode API to supply a merkle tree for a given container. (apache#6945)
  HDDS-11289. Bump docker-maven-plugin to 0.45.0 (apache#7024)
  HDDS-11287. Code cleanup in XceiverClientSpi (apache#7043)
  HDDS-11283. Refactor KeyValueStreamDataChannel to avoid spurious IDE build issues (apache#7040)
  HDDS-11257. Ozone write does not work when http proxy is set for the JVM. (apache#7036)
  HDDS-11249. Bump ozone-runner to 20240729-jdk17-1 (apache#7003)
  HDDS-10517. Recon - Add a UI component for showing DN decommissioning detailed status and info (apache#6724)
  HDDS-10926. Block deletion should update container merkle tree. (apache#6875)
  HDDS-11270. [hsync] Add DN layout version (HBASE_SUPPORT/version 8) upgrade test. (apache#7021)
  HDDS-11272. Statistics some node status information (apache#7025)
  HDDS-11278. Move code out of Hadoop util package (apache#7028)
  HDDS-11274. (addendum) Replace Hadoop annotations/configs with Ozone-specific ones
  HDDS-11274. Replace Hadoop annotations/configs with Ozone-specific ones (apache#7026)
  HDDS-11280. Add Synchronize in AbstractCommitWatcher.addAckDataLength (apache#7032)
  HDDS-11235. Spare InfoBucket RPC call in FileSystem#mkdir() call. (apache#6990)
  HDDS-11273. Bump commons-compress to 1.26.2 (apache#7023)
  HDDS-11225. Increase ipc.server.read.threadpool.size (apache#7007)
  HDDS-11224. Increase hdds.datanode.handler.count (apache#7011)
  HDDS-11259. [hsync] DataNode should verify HBASE_SUPPORT layout version for every PutBlock. (apache#7012)
  HDDS-11214. Added config to set rocksDB's max log file size and num of log files (apache#7014)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kerneltime kerneltime kerneltime approved these changes

@aswinshakil aswinshakil aswinshakil approved these changes

Assignees
No one assigned
Labels
container-reconciliation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@errose28 @kerneltime @smengcl @aswinshakil