Skip to content

Commit

Permalink
[Fix](hive-writer) Fix s3 file commiter not working. (#35502)
Browse files Browse the repository at this point in the history
### Issue
`The specified upload does not exist. The upload ID may be invalid, or
the upload may have been aborted or completed. (Service: S3, Status
Code: 404, Request ID: 66557027E897233333FFC198)`

### Root cause
#35311 adjusted the order of building hive partition update information.
This change caused the update id to be unable to be obtained, causing
the s3 file committer to not work properly.

### Solution
Because uploading to s3 occurs after s3 file writer close(), close()
must be called first, and then the build hive partiton update
information is called.
  • Loading branch information
kaka11chen authored and dataroaring committed May 31, 2024
1 parent 3f50783 commit 30db5b0
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions be/src/vec/sink/writer/vhive_partition_writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,6 @@ Status VHivePartitionWriter::open(RuntimeState* state, RuntimeProfile* profile)
}

Status VHivePartitionWriter::close(const Status& status) {
if (status.ok()) {
_state->hive_partition_updates().emplace_back(_build_partition_update());
}
if (_file_format_transformer != nullptr) {
Status st = _file_format_transformer->close();
if (!st.ok()) {
Expand All @@ -156,6 +153,9 @@ Status VHivePartitionWriter::close(const Status& status) {
LOG(WARNING) << fmt::format("Delete file {} failed, reason: {}", path, st.to_string());
}
}
if (status.ok()) {
_state->hive_partition_updates().emplace_back(_build_partition_update());
}
return Status::OK();
}

Expand Down

0 comments on commit 30db5b0

Please sign in to comment.