You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Motivation

In data warehouse, it's a common requirement to query historical data. So we need to periodically merge incremental data with historical data to generate complete data of current time. For example, Hive 2.2 supports using MERGE INTO statement to merge source table's records into target table by performing update, insert and delete according to different filter, so a good way is taking historical table as target and incremental table as source, then using MERGE INTO to perform update (for existing records) and insert (for non-existing records). But when there exists a large amount of data, the merging may cost much time.

Paimon's snapshots can provide another way to query historical data without cost of merging. Paimon's data storage will generate snapshot for each commit, so we can find historical data at any snapshot (usually called time travel query in SQL). 

But in most scenarios, a table will generate too many snapshots, so Paimon lets users to configure snapshot expiration time to clean old, unused snapshots. Thus the snapshot of a specified time point may have expired when user want to query it. So we want to introduce savepoint for Paimon to persist full data in a time point.

Public Interfaces

We need a entrance class `Savepoint` to store the information of a savepoint:

Savepoint
public class Savepoint {
	/** Used for identify a savepoint. */
	private final long id;
	
	/** Which snapshot does this savepoint save. */
	private final long savedSnapshotId;

	/** Id of the current table schema of the records in this savepoint. */
	private final long schemaId;

	/** The manifest list of all data of this savepoint. */
	private final long fullManifestList;

	/** How many records in this savepoint. */
	private final long recordCount;

	/** Getters. */
	
	/** Some util methods. */

	/** Return all {@link ManifestFileMeta} instances for full data manifests in this snapshot. */
 	public List<ManifestFileMeta> dataManifests(ManifestList manifestList);
	
	/** Serialize to json. */
	public String toJson();

	/** Deserialize from json. */
	public static Savepoint fromJson(String json);

	/** Get a savepoint from path. */
	public static Savepoint fromPath(FileIO fileIO, Path path);
}

We need a `SavepointManager` to manage the savepoints (similar to `SnapshotManager`).

SavepointManager
public class SavepointManager {
	/** Return the root Directory of savepoints. */
	public Path savepointDirectory();
 	
	/** Return the path of a savepoint. */
	public Path savepointPath(long savepointId);

 	/** Get a savepoint instance. */
	public Savepoint savepoint(long savepointId);

 	/** Check if a savepoint exists. */
	public boolean savepointExists(long savepointId);
 	
	/** Return id of the earlist savepoint. */
 	public @Nullable Long earlistSavapointId();

 	/** Return id of the latest savepoint. */
    public @Nullable Long latestSavepointId();
}







  • No labels