THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Current state: "Under Discussion"
Discussion thread:here
JIRA: here
Released: Yet To Be Determined
...
All calls to Timeline Service for fetching the view will include a latest completed instant and timeline signature (hash) as known by the client. The timeline service will then be able to dynamically detect if its view is up to date or needs syncing. In case if one or more instants gets added, the timeline service lists the metadata folder and reads new metadata incrementally
Completed Action | Incremental Syncing |
Commit/Delta-Commit | Metadata part of instant file contains partitions and files touched. Timeline Server updates the latest file-slices based with this new information |
Cleans | Metadata part of instant file contains partitions and files removed. These are old file slices which are removed from the view without listing the file-system again |
Rollback old commits/delta-commits | Metadata part of instant file contains partitions and files removed. These files are removed and the corresponding file-slices merged. Again, this is done from the view without listing the file-system again. |
The advantage of the above approach in reducing the file-system calls will be greatly felt when the embedded timeline-service instance is servicing more than one write calls (ingest/compact/clean/rollback) and in the case of running this service in standalone mode.
...
Alternatives Considered for Timeline Caching :
In-Memory File System View Sent to each executor:
In this case, a file-system view is built in driver as a regular Java Map and sent to Spark executors as Spark closure. While this is a very simple model to implement, it does not scale well for large datasets as shown above. This increase in memory requirement is not sustainable for running ingestion/compaction at scale.
File System View Managed as Spark Broadcast Variable:
In this case, the file system view manager is again built in driver and distributed as read-only broadcast variables to executors for file-system lookup. Even though, this would amortize the memory overhead costs when multiple executors run on a single physical machine, there is no guarantee that the cluster manager will place the executors to tap this savings. Also, this will be a spark-only solution and cannot be extended in general.
...