This document is a work in progress. I will be updating it in the coming days. |
The partition concept is very closely related to logical date. In fact, logical date, and execution date before it, has effectively always been a kind of partition. And their reason for existing in the way that the do is to support real partition-driven workloads e.g. hive ETL.
Ultimately it would probably make sense to make it so logical date was internally a partition. Do we need to do this now? Or can it be done in backward-compatible way in the future?
There are a number of questions to resolve here and I will work on enumerating them in the coming days.