Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KAFKA-3968: fsync the parent directory of a segment file when the file is created #10405

Merged
merged 27 commits into from
Apr 3, 2021
Merged

KAFKA-3968: fsync the parent directory of a segment file when the file is created #10405

Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
459facb
[KAFKA-3968] fsync the parent directory of a segment file when the fi…
ccding Mar 25, 2021
b4c8284
move import
ccding Mar 26, 2021
b260169
address comments (except the topic partition one)
ccding Mar 29, 2021
ba086e9
remove import
ccding Mar 29, 2021
2a19e0e
reuse the function in utils.java
ccding Mar 30, 2021
40a1abe
simplify logic
ccding Mar 30, 2021
1ac80b6
default changeFileSuffixes flush to true
ccding Mar 30, 2021
09cac0b
flush when mkdirs
ccding Mar 30, 2021
5be95aa
revert accidential change
ccding Mar 30, 2021
c9448c8
atomicMoveWithFallback
ccding Mar 30, 2021
daeb698
no flush parent dir in test
ccding Mar 30, 2021
0d4800b
check null pointer
ccding Mar 31, 2021
95a6c3f
fix unit test error
ccding Mar 31, 2021
8c859f3
set flag after flush
ccding Apr 1, 2021
fdc1faa
disable flushing on renameTo
ccding Apr 1, 2021
6795ec9
address comments based on offline discussion with Jun
ccding Apr 1, 2021
55ae3bc
Merge branch 'trunk' into fsync
ccding Apr 1, 2021
e653af4
check hadCleanShutdown for open FileRecord
ccding Apr 1, 2021
85861ee
address comments
ccding Apr 1, 2021
1578678
fix default values
ccding Apr 1, 2021
fffc353
more default value
ccding Apr 1, 2021
f66c545
do flush in the LogSegment class
ccding Apr 2, 2021
56be9d8
Merge branch 'trunk' into fsync
ccding Apr 2, 2021
1ecf94b
remove parameter from FileRecord.open
ccding Apr 2, 2021
080a79a
default to false
ccding Apr 2, 2021
61eee4a
add param to javadoc
ccding Apr 2, 2021
7543938
during flush -> during the next flush
ccding Apr 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
simplify logic
  • Loading branch information
@ccding
ccding committed Mar 30, 2021
commit 40a1abe825a9c1539031ad9ff12b79114103916b
7 changes: 2 additions & 5 deletions clients/src/main/java/org/apache/kafka/common/record/FileRecords.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,11 +208,8 @@ public void flush() throws IOException {
* Flush the parent directory of a file to the physical disk, which makes sure the file is accessible after crashing.
*/
public void flushParentDir() throws IOException {
try {
Utils.flushParentDir(file.toPath());
} finally {
needFlushParentDir.set(false);
}
needFlushParentDir.set(false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, we want to flush the parent dir first before setting needFlush to false.

Copy link
Contributor Author

@ccding ccding Apr 1, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Setting flag first to prevent other threads from calling flush concurrently.

Utils.flushParentDir(file.toPath());
}

/**
Expand Down