Page History

...

NONE = None
QUEUED = "queued"
SCHEDULED = "scheduled"
REMOVED = "removed"
RUNNING = "running"
SUCCESS = "success"
SHUTDOWN = "shutdown"  # External request to shut down
FAILED = "failed"
UP_FOR_RETRY = "up_for_retry"
UPSTREAM_FAILED = "upstream_failed"
SKIPPED = "skipped"

The scheduler processes tasks that have a state of NONE, QUEUED, and UP_FOR_RETRY. NONE is a newly created TaskInstance, QUEUED is a task that is waiting for a slot in an executor and UP_FOR_RETRY means a task that failed before but needs to be retried. At the moment the scheduler will not change the state of a Task. This creates the possibility of a race condition: if the executor cannot execute a task quickly enough its state will not be updated and it can get scheduled again. Due to the loop times being quite large in complex cases, this won't occur too often. As people try to limit scheduler loop times, or just prefer LocalExecutor more than the CeleryExecutor, they will run multiple schedulers at the same time, which increases the chances of this happening.If the task has a state of NONE it will be set to SCHEDULED when to the scheduler sends it to the executor. 

Unfortunately a race condition remains for UP_FOR_RETRY tasks as another scheduler can pick those up. To eliminate this the check for UP_FOR_RETRY needs to migrate from the TI to the scheduler. However, For this reason a "SCHEDULED" state is proposed in one of the PRs. This will not fully close the door to this race condition due to the fact that TaskInstance evaluate their own state in case of UP_FOR_RETRY (ie. checking if they should run now). Normally one would move this check to the scheduler was it not for that fact that we have backfills... (see below)

Better state handling

Work in progress

In order to remove race conditions and to be able to kill any of airflows components and still be able to continue where we left off, better state handling needs to be done. This means that at handover to a different process only certain states can get set by each process. For example the scheduler should only have an outgoing state of "SCHEDULED". The executor should set a state of "LAUNCHED". A task instance can set UP_FOR_RETRY, RUNNING, UPSTREAM_FAILED, SUCCEEDED but only handles LAUNCHED.

Queues for workers should be handled in the worker, for pools most likely in the scheduler

Pools

All tasks used to start with State.NONE. Unknown User (bolke)'s latest PR (AIRFLOW-128) introduces State.SCHEDULED when the TI has been created sent to the executor by the scheduler but not yet run. When the executor loads launches a task in a separate process a TI, it eventually calls TI.run() which does one of two things:

...