Page History

...

Each commit operation on the table generates a CommitEvent regardless of whether it succeeds or fails. including three commit types: APPEND, COMPACT, and OVERWRITE. Many features such as independent compaction service and advancing downstream tasks based on watermark rely on CommitEvent.

public class CommitEvent implements Event{

    private List<ManifestEntry> tableFiles;
    private List<ManifestEntry> changelogFiles;
    private List<IndexManifestEntry> indexFiles;
    private long identifier;
    private Long watermark;
    private Map<Integer, Long> logOffsets;
    private Snapshot.CommitKind commitKind;


    private Throwable throwable;

 	/** The file path of the table in the filesystem. */
 	private String path; 
}

TriggerCompactEvent & CompactEvent

Since the compaction task is likely to take a long time to execute, a TriggerCompactEvent will be generated before the compaction task begins in order to perceive its execution status. A CompactEvent will be generated for every compaction finish regardless of whether it succeeds or fails. Through these two events, users can accurately know the start time and file status of each task, which can provide reference for users to judge whether the merge task is healthy. In addition, most of the compaction-related information that users are concerned about can be obtained through these two events. 

public class TriggerCompactEvent implements Event{

    private final boolean fullCompaction;

    private final BinaryRow partition;

    private final int bucket;

    private final List<DataFileMeta> inputs;

	/** The file path of the table in the filesystem. */
	private final String path;
}

CompactEvent

A CompactEvent will be generated for every compaction, regardless of whether it succeeds or fails.

public class CompactEvent implements Event{

    private final boolean fullCompaction;

    private final BinaryRow partition;

    private final int bucket;

    private final List<DataFileMeta> beforeFiles;

    private final List<DataFileMeta> afterFiles;

    private final Throwable throwable;

	/** The file path of the table in the filesystem. */
    private final String path;
}

CreateTableEvent

...

& DropTableEvent & RenameTableEvent & AlterTableEvent

These four events are generated when a table is successfully created, dropped, altered, or renamed in the catalog. In features like an independent compaction service, it is necessary to timely perceive these status changes of the table. For example, when an option of a table is modified, the new compaction parameters need to be checked in time to determine whether to immediately perform compaction on the table. Create and drop events can promptly notify the compaction service of table creation and deletion, which affects its scheduling strategy. The rename event can prompt the compaction service to correspond the execution record information of the historical table with the new table name.Without a listener mechanism, it would be necessary to continuously scan tables through polling, which would result in high resource consumption and poor timelinessAfter successfully creating a table, a CreateTableEvent will be generated.

public class CreateTableEvent implements Event {

	/** The file path of the table in the filesystem. */
    private String path;

    private Schema schema;
}

DropTableEvent

...

public class Drop
TableEvent implements Event {

	/** The file path of the table in the filesystem. */
    private String path;
}

...

After successfully renaming a table, a RenameTableEvent will be generated.

public class RenameTab
leEvent implements Event{

 	/** The old file path of the table in the filesystem. */
    private String oldPath;

 	/** The new path of the table in the filesystem. */
    private String newPath;
}

AlterTableEvent

...

public class AlterTableEvent implements Event{

 	/** The file path of the table in the filesystem. */
    private String path;

    private List<SchemaChange> schemaChanges;
}

...