Commit#

Interface#

class FileStoreCommit#

Interface for commit operations in a file store.

The FileStoreCommit class provides interfaces for committing changes, expiring old snapshots, dropping partitions, and retrieving commit metrics.

Note

Direct file-system commits support append-only and primary-key tables on non-object-store paths. Object-store paths require REST catalog commit mode: enable it with CommitContextBuilder::UseRESTCatalogCommit(), call Commit() or FilterAndCommit(), and then retrieve the request with GetLastCommitTableRequest().

Public Functions

virtual ~FileStoreCommit() = default#

virtual Status Commit(const std::vector<std::shared_ptr<CommitMessage>> &commit_messages, int64_t commit_identifier = BATCH_WRITE_COMMIT_IDENTIFIER, std::optional<int64_t> watermark = std::nullopt) = 0#

Commit changes to the file store.

Parameters:

commit_messages – A vector of commit messages to be committed.
commit_identifier – An optional identifier for the commit operation. Default is BATCH_WRITE_COMMIT_IDENTIFIER.
watermark – An optional event-time watermark used to indicate the progress of data processing. Default is std::nullopt.

Returns:

Status indicating the success or failure of the commit operation.

virtual Result<int32_t> FilterAndCommit(const std::map<int64_t, std::vector<std::shared_ptr<CommitMessage>>> &commit_identifier_and_messages, std::optional<int64_t> watermark = std::nullopt) = 0#

Filter out all std::vector<CommitMessage> which have been committed and commit the remaining ones.

Compared to commit, this method will first check if a commit_identifier has been committed, so this method might be slower. A common usage of this method is to retry the commit process after a failure.

Parameters:

commit_identifier_and_messages – A map containing all CommitMessages in question. The key is the commit_identifier.
watermark – An optional event-time watermark used to indicate the progress of data processing. Default is std::nullopt.

Returns:

Number of std::vector<CommitMessage> committed.

virtual Status Overwrite(const std::map<std::string, std::string> &partition, const std::vector<std::shared_ptr<CommitMessage>> &commit_messages, int64_t commit_identifier, std::optional<int64_t> watermark = std::nullopt) = 0#

Overwrite from manifest committable and partition.

Parameters:

partition – A single partition maps each partition key to a partition value. Depending on the user-defined statement, the partition might not include all partition keys. Also note that this partition does not necessarily equal to the partitions of the newly added key-values. This is just the partition to be cleaned up.
commit_messages – Description of the commit messages.
commit_identifier – Unique identifier.
watermark – An optional event-time watermark used to indicate the progress of data processing. Default is std::nullopt.

Returns:

Result of the operation.

virtual Result<int32_t> FilterAndOverwrite(const std::map<std::string, std::string> &partition, const std::vector<std::shared_ptr<CommitMessage>> &commit_messages, int64_t commit_identifier, std::optional<int64_t> watermark = std::nullopt) = 0#

This is a temporary interface for internal use.

It will be removed in a future version. Please do not rely on it for long-term use.

Parameters:

partition – Description of the partition.
commit_messages – Description of the commit messages.
commit_identifier – Unique identifier.
watermark – An optional event-time watermark used to indicate the progress of data processing. Default is std::nullopt.

Returns:

Result of the operation.

virtual Result<std::string> GetLastCommitTableRequest() = 0#

If user want to use REST catalog commit, please set CommitContextBuilder::UseRESTCatalogCommit(), then call Commit() (or FilterAndCommit()) normally, then call this method to get the last commit table request, which is a JSON string that can be used to send to REST catalog server.

Note

Temporary interface for internal use, will be removed in the future.

Returns:: A Result containing a JSON string which including snapshot and statistics, but excluding tableId.

virtual Result<int32_t> Expire() = 0#

Expire old snapshot in the file store.

Returns:: Result<int32_t> indicating the number of expired items or an error status.

virtual Status DropPartition(const std::vector<std::map<std::string, std::string>> &partitions, int64_t commit_identifier) = 0#

Drop specified partitions from the file store.

Parameters:

partitions – A vector of partitions to be dropped.
commit_identifier – An identifier for the commit operation.

Returns:

Status indicating the success or failure of the drop partition operation.

virtual Status TruncateTable(int64_t commit_identifier) = 0#

Truncate the whole table by overwriting all partitions with empty data.

The generated snapshot has commit kind OVERWRITE.

Parameters:: commit_identifier – An identifier for the commit operation.
Returns:: Status indicating the success or failure of the truncate operation.

virtual Status Abort(const std::vector<std::shared_ptr<CommitMessage>> &commit_messages) = 0#

Abort an unsuccessful commit.

The data and index files described by the given commit messages will be deleted on a best-effort basis (delete failures are ignored).

Parameters:: commit_messages – A vector of commit messages whose files should be cleaned up.
Returns:: Status indicating the success or failure of the abort operation.

virtual Result<bool> RollbackToAsLatest(int64_t target_snapshot_id) = 0#

Roll back to the target snapshot and materialize it as the latest snapshot.

Reads the surviving files of both the current latest snapshot and the target snapshot, then commits an OVERWRITE snapshot whose visible state equals the target.

Parameters:: target_snapshot_id – The snapshot id to roll back to.
Returns:: Result<bool>; true if the atomic commit succeeded. Returns an error status if there is no latest snapshot or the target snapshot does not exist.

virtual FileStoreCommit &RowIdCheckConflict(std::optional<int64_t> row_id_check_from_snapshot) = 0#

Configure row-id conflict checking from a specific snapshot id.

If set to a snapshot id, commit conflict detection will additionally validate row-id conflicts against snapshots after that id. Passing std::nullopt disables this behavior.

Parameters:: row_id_check_from_snapshot – Snapshot id to start row-id conflict checks from, or std::nullopt to disable.
Returns:: Current commit object for chaining.

virtual std::shared_ptr<Metrics> GetCommitMetrics() const = 0#

Retrieve metrics related to commit operations.

Returns:: A shared pointer to a Metrics object containing commit metrics.

Public Static Functions

static Result<std::unique_ptr<FileStoreCommit>> Create(std::unique_ptr<CommitContext> context)#

Create an instance of FileStoreCommit.

Parameters:: context – A unique pointer to the CommitContext used for commit operations.
Returns:: A Result containing a unique pointer to the FileStoreCommit instance.

class CommitContextBuilder#

CommitContextBuilder used to build a CommitContext, has input validation.

Public Functions

CommitContextBuilder(const std::string &root_path, const std::string &commit_user)#

Constructs a CommitContextBuilder with required parameters.

Parameters:

root_path – The root path of the Paimon table.
commit_user – The user identifier for the commit operation.

~CommitContextBuilder()#

CommitContextBuilder &SetOptions(const std::map<std::string, std::string> &options)#

Set a configuration options map to set some option entries which are not defined in the table schema or whose values you want to overwrite.

Note

The options map will clear the options added by AddOption() before.

Parameters:: options – The configuration options map.
Returns:: Reference to this builder for method chaining.

CommitContextBuilder &AddOption(const std::string &key, const std::string &value)#

Add a single configuration option which is not defined in the table schema or whose value you want to overwrite.

If you want to add multiple options, call AddOption() multiple times or use SetOptions() instead.

Parameters:

key – The option key.
value – The option value.

Returns:

Reference to this builder for method chaining.

CommitContextBuilder &IgnoreEmptyCommit(bool ignore_empty_commit)#

Sets whether to ignore empty commits (default is true).

When set to true, commits that don’t contain any actual data changes will be ignored.

Parameters:: ignore_empty_commit – True to ignore empty commits, false otherwise.
Returns:: Reference to this builder for method chaining.

CommitContextBuilder &UseRESTCatalogCommit(bool use_rest_catalog_commit)#

Sets whether to use REST catalog commit (default is false).

Note

Temporary interface, will be removed in the future.

Parameters:: use_rest_catalog_commit – True to use REST catalog commit, false otherwise.
Returns:: Reference to this builder for method chaining.

CommitContextBuilder &AppendCommitCheckConflict(bool append_commit_check_conflict)#

Sets whether append commits should perform conflict checking (default is false).

Parameters:: append_commit_check_conflict – True to enable append conflict checks.
Returns:: Reference to this builder for method chaining.

CommitContextBuilder &WithMemoryPool(const std::shared_ptr<MemoryPool> &memory_pool)#

Sets the memory pool to be used for memory allocation during commit operations.

Parameters:: memory_pool – Shared pointer to the memory pool instance.
Returns:: Reference to this builder for method chaining.

CommitContextBuilder &WithExecutor(const std::shared_ptr<Executor> &executor)#

Sets the executor to be used for asynchronous operations during commit.

Parameters:: executor – Shared pointer to the executor instance.
Returns:: Reference to this builder for method chaining.

CommitContextBuilder &WithFileSystem(const std::shared_ptr<FileSystem> &file_system)#

Sets a custom file system instance to be used for all file operations in this commit context.

This bypasses the global file system registry and uses the provided implementation directly.

Note

If not set, use default file system (configured in Options::FILE_SYSTEM)

Parameters:: file_system – The file system to use.
Returns:: Reference to this builder for method chaining.

Result<std::unique_ptr<CommitContext>> Finish()#

Build and return a CommitContext instance with input validation.

Returns:: Result containing the constructed CommitContext or an error status.

class CommitContext#

CommitContext is some configuration for commit operations.

Please do not use this class directly, use CommitContextBuilder to build a CommitContext which has input validation.