Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The partition concept is very closely related to logical date.  In fact, logical date, and execution date before it, has effectively always been a kind of partition.  And their reason for existing in the way that the do is to support real partition-driven workloads e.g. hive ETL.

Ultimately it would probably make sense to make it so logical date was internally a partition.  Do we need to do this now?  Or can it be done in backward-compatible way in the future?

There are a number of questions to resolve here and I will work on enumerating them in the coming days.