DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
This document is a work in progress. I will be updating it in the coming days.
The partition concept is very closely related to logical date. In fact, logical date, and execution date before it, has effectively always been a kind of partition. And their reason for existing in the way that the do is to support real partition-driven workloads e.g. hive ETL.
Ultimately it would probably make sense to make it so logical date was internally a partition. Do we need to do this now? Or can it be done in backward-compatible way in the future?
Can a normal cron dag update asset partitions?
It may seem like an obvious "yes". But normal cron dags have a logical date, not a partition key, and there isn't an "obvious" way to map a logical date to a partition key. E.g. should it be a date string? Date plus hour? Month date? Week date?
So the user needs both to opt in and specify the format, unless the dag's schedule is somehow driven by a partition scheme.
One option is to force the user to define a partition scheme.
Another option is to provide an interface to format the logical date as a partition key.
Should logical date be populated when using a timetable in partition mode?
The current answer to this is no.