Page History

...

Data intervals are not a specific target of this proposal.  But, in any of these solutions we have to think about what are the implications for data intervals and what to do about them.  And, in some of the solutions considered, the proposal is to not have data intervals defined in some scenarios.  Generally speaking, I think they are confusing and misleading and we would probably do well, as is planned for assets, to move away from them.

With that let's briefly discuss some of the  issues.

...

Data intervals were layered onto the existing (albeit implicit) partitioning concept represented by execution_date / logical_date.  They assume more about what the user is doing than they really need to.  So e.g. rather than saying this execution_date is simply the partition key for this run (whatever that may mean) they say this logical date represents an interval.  But what if a M-F dag really does process only the M-F data, i.e. really only deals witth with one date at a time, and doesn't care about weekends?

...