This document is a work in progress. I will be updating it in the coming days. |
The partition concept is very closely related to logical date. In fact, logical date, and execution date before it, has effectively always been a kind of partition. And their reason for existing in the way that the do is to support real partition-driven workloads e.g. hive ETL.
Ultimately it would probably make sense to make it so logical date was internally a partition. Do we need to do this now? Or can it be done in backward-compatible way in the future?
It may seem like an obvious "yes". But normal cron dags have a logical date, not a partition key, and there isn't an "obvious" way to map a logical date to a partition key. E.g. should it be a date string? Date plus hour? Month date? Week date?
So the user needs both to opt in and specify the format, unless the dag's schedule is somehow driven by a partition scheme.
One option is to force the user to define a partition scheme.
Another option is to provide an interface to format the logical date as a partition key.
The current answer to this is no.