You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Background
We started to store pipeline state and schemas in filesystem destination to be able to restore them. In case of state table each load may produce a new file. We should truncate old states ie. to just keep 100 newest.
Requirements
when load is completed, delete old files in the _dlt_pipeline_state table
keep N last state files
delete only the state files that corresponds to finished loads (they have corresponding completed entry). this is to prevent a rate case when we have 100 unsuccessful partial loads and we delete the last right state
make this mechanism on by default and allow it to be disabled with a flag
The text was updated successfully, but these errors were encountered:
Background
We started to store pipeline state and schemas in
filesystem
destination to be able to restore them. In case of state table each load may produce a new file. We should truncate old states ie. to just keep 100 newest.Requirements
_dlt_pipeline_state
tableThe text was updated successfully, but these errors were encountered: