Test cases AIP-78 Scheduler-managed backfill

AIP78_TC_001

Verify user can create a backfill using /public/backfill endpoint

1. Send a POST request to create a backfill job for a specific DAG using API with valid parameters.

{
  "dag_id": "string",
  "from_date": "2025-02-20T08:55:22.239Z",
  "to_date": "2025-02-20T08:55:22.239Z",
  "run_backwards": false,
  "dag_run_conf": {},
  "reprocess_behavior": "none",
  "max_active_runs": 10
} 

1. Verify backfill response and check backfill has been created
2. Verify in database in backfill and backill dag_run_table

Positive

Pass

AIP78_TC_002

Verify user is able to create backfills for different time ranges

1. Create multiple backfill jobs for the same DAG using different combinations of start_date and end_date via API.
2. Fetch each created backfill job to verify the specified time ranges.

Backfill jobs are created successfully with the correct time ranges.

Positive

Pass

AIP78_TC_003

Verify user is still able to run backfills using CLI

1. Run the backfill command using CLI for a specific DAG with valid parameters.
2. Verify that parameters mentioned here are not available.

CLI backfill runs successfully with only the supported parameters.

Positive

Pass

AIP78_TC_004

Verify user is able to perform a dry run using backfill dry_run endpoint

1. Send a POST request to dry_run endpoint with below request

   {
     "dag_id": "string",
     "from_date": "2025-02-20T08:55:22.239Z",
     "to_date": "2025-02-20T08:55:22.239Z",
     "run_backwards": false,
     "dag_run_conf": {},
     "reprocess_behavior": "none",
     "max_active_runs": 10
   } 

1. The response does list the backfill date details which will be created

Positive

Pass

AIP78_TC_005

Verify user can create, monitor, clear, and manage backfill jobs from UI

1. Create a backfill job from the API or CLI.
2. Access the web UI and monitor the backfill job status.
3. Clear or manage the backfill job using available UI options.

Users can create, monitor, and manage backfill jobs from the UI without issues.

Not Implemented

AIP78_TC_006

Verify BackfillRunDagRun table correctly maps Backfill job with DAG run

1. Create a backfill job and verify entries in the BackfillRunDagRun table.
2. Verify that each entry correctly associates a backfill job with the corresponding DAG run.

BackfillRunDagRun table correctly maps each backfill job with the appropriate DAG runs.

Positive

Pass

AIP78_TC_007

Verify backfill DAG runs respect concurrency limitations

1. Create a backfill job with multiple DAG runs.
2. Verify that the scheduler respects defined concurrency limitations when creating and scheduling DAG runs.

Backfill DAG runs are created and managed within defined concurrency limitations.

Positive

Pass

AIP78_TC_008

Perform scheduler benchmarking (task throughput tests)

1. Run backfill jobs under different scenarios (e.g., large backfill range, high number of tasks).
2. Measure scheduler throughput and observe any performance degradation.

Scheduler maintains acceptable performance under increased load due to backfill jobs.

Performance

Pass

AIP78_TC_009

Verify backfills for different types of DAGs (using dynamic mapping, datasets, setup & teardown, etc.)

1. Create backfill jobs for different types of DAGs (e.g., dynamic mapped tasks, setup & teardown tasks).
2. Monitor and verify the execution and outcome of each backfill.

Backfill jobs work correctly with various DAG types without errors.

Positive

Pass

AIP78_TC_010

Verify user can list backfill using List backfill endpoint.

1. Provide DAG ID in List Endpoint request.

All the backfill details listed in response from the backfill table

Positive

Pass

AIP78_TC_011

Verify user can get details of specific backfill ID

1. Make a get call to backill endpoint with backfill id 

Details of a specific backfill id are listed in the backfill table

Positive

Pass

AIP78_TC_012

Verify user can pause running backfill

1. Execute a backfill with many dates
2. Make a call to pause endpoint with backfill id

Running backfills should be paused successfully

Positive

Pass

AIP78_TC_013

Verify user can unpause running backfill

1. Make a call to unpause endpoint with any of paused backfill id

Paused backfills should be resumed successfully

Positive

Pass

AIP78_TC_014

Verify user can cancel running backfill

1. Make a call to cancel endpoint with any of running backfill id

Backfill should be cancelled successfullt

Positive

Pass

AIP78_TC_015

Verify user can create backfill with `run_backwards` as True

1. Send a POST request to create a backfill job for a specific DAG using API with valid parameters.

{
  "dag_id": "string",
  "from_date": "2025-02-20T08:55:22.239Z",
  "to_date": "2025-02-20T08:55:22.239Z",
  "run_backwards": True,
  "dag_run_conf": {},
  "reprocess_behavior": "none",
  "max_active_runs": 10
} 

1. Verify backfill response and check backfill has been created
2. Verify in database in backfill and backill dag_run_table
3. Verify backfill should be executed in backwards way

Positive

Pass

1. Send a POST call to backfill create endpoint with reprocess behavior as none. Also ensure for specific dates run should be in success state

1. Backfills should not be created. Verify in backfill dag run table there should be new record with exception as `already exits`

AIP78_TC_017

Verify user can create backfill with reprocess behavior as Completed.

1. Ensure there are runs of specific dates with success state.
2. Try creating backfill for same dates with reprocess behavior as completed

Positive

Pass

AIP78_TC_018

Verify user can create backfill with reprocess behavior as Failed.

1. Ensure there are runs of specific dates with failed state.
2. Try creating backfill for same dates with reprocess behavior as failed

Positive

Pass

AIP78_TC_019 

Verify user perform dry run using CLI

1. execute CLI command with dry-run command option

Positibe

Pass