| Test Scope | Test Scenarios |
|---|
| Flows to test | - Backup of a new store
- Restore on a new host
- Incremental backup/restore of a store
- Cleanup for a new container/job
- Cleanup called after a failed container restart
- Cleanup called during commit sequence
- Partial failure of backup in different stages of commit lifecycle (commit, upload, persist, cleanup).
- Ensure that the partial state is cleaned up.
- Failure scenarios of restore in different stages of commit lifecycle (commit, upload, persist, cleanup).
- Ensure that a container restarting on the same host after partial failure can cleanup/recover and restore correctly (does not get stuck).
- Failure scenarios of blob store (missing index blobs, file blobs, retry etc.)
- Test cleanup/commit sequence during taskinstance init works with transactional kafka and blob store
|
| End-to-end testing | - Test that checkpoint read/write version configurations are validated before job launch.
- Test upgrade and rollback compatibility for samza versions with and without blob store backend.
- How do we enforce that backup and restore managers aren't both enabled for the first deployment?
|
| BlobStoreTaskBackupManager |
init init with checkpoint V1 init with no/null checkpoint Test init cleans up unused stores correctly.
upload No previous checkpoint (first upload) Previous checkpoint passed during init (subsequent upload) Test upload handles logged / non-logged / persistent / durable stores correctly (document expectation here). Test upload calculates diff from previous checkpoint correctly (during initial start and during post-startup commits) Test upload returns snapshot blob id and records previous snapshot blob id in the snapshot correctly.
cleanup Test Cleanup removes TTL of remote snapshot and associated files Test Cleanup deletes old remote snapshot Test Cleanup deletes files/subdirs to remove from current checkpoint Test Cleanup cleans stores removed from config Cleanup failed container/job restart
|
| BlobStoreRestoreManager | init Test init fails for checkpoint V1. Test init works for no/null checkpoint. Test init returns blob store backend store scms if present in checkpoint. Test util method to get snapshot indexes from checkpoint. Test container fails to start with meaningful error message if init fails.
restore Test that restore restore to the correct store directory depending on store type.
Test that it ignores any files that are not present when upload is called (e.g. offset files).
Test restore handles logged / non-logged / durable / persistent stores correctly. Test logic for checking if checkpoint directory is identical to remote snapshot. Test restore handles stores with missing SCM in checkpoint correctly. Test restore handles multiple stores correctly. Test restore always deletes main store dir. Test restore uses previous checkpoint directory if identical to remote snapshot. Test restore restores from remote snapshot if no previous checkpoint dir. Test restore restores from remote snapshot if checkpoint dir not identical to remote snapshot. Test restore recreates subdirs correctly. Test restore recreates recursive subdirs correctly Test restore creates empty files correctly. Test restore creates empty dirs correctly. Test restore creates empty sub-dirs / recursive subdirs correctly. Test restore restores multi-part file contents completely and in correct order. Test restore verifies checksum for files restored if enabled.
|
| BlobStoreStateBackendUtil | Test throws exception for checkpoint v1. Test no-op for null / empty checkpoint. Test works correctly for missing blob store backend factory entry. Test works correctly for missing blob store backend factory store entry. Test throws exception on sync and async blob store errors. Test gets the right blobid from remote store. Test returns the correct pair of scm and snapshot index. Test blocks once at the end for all futures instead of blocking for each store.
|
| Concurrency and Retries | - Test CompletableFutureUtil methods.
- Test that all operations use an explicit and expected executor (no default executor).
- Verify future composition (allOf, toMap etc) and blocking (individual vs collected vs nonblocking) for all async methods.
- Verify that there is no blocking on caller threads. Document and justify exceptions (e.g. restore thread)
- Test BlobStoreManager Impl/BlobStoreUtil error handling and retries.
- Test completionexception unwrapping to identify actual cause.
- Test callback order (par/seq dep graph) for all chained operations.
- Test async retriable exceptions are transformed correctly.
- Test/verify put / get / delete futures always complete (handle sync / async errors correctly).
- Test retries for get / put create new input / output streams.
- Test error handling for get (sync, future, callback errors).
- Test TaskInstance commit flow.
- Test async commit stage fails if upload/checkpoint write/cleanup fails.
- Verify all async stage operations execute on a separate threadpool.
- Test async commit succeeds and ublocks future commits if all async operations succeed.
- Test async commit stage fails if any async operations failed.
- Verify async commit stage operations are chained correctly.
- Test exceptions in asyc commit stage are propagated to next sync commit stage.
- Test sync commit fails if a previous async commit fails.
- Test commit skips if previous async commit in progress and < max delay.
- Test commit blocks if previous async commit in progress and > max delay
- Test that sync commit times out if previous async commit does not complete within max commit delay.
- Test BackupManager/RestoreManager flow
- Test all async stage operations execute on a separate threadpool.
- Verify/Test error propagation, handling and operation chaining (par/seq dep graph).
- Verify/Test timeouts for blocking operations. Document and justify blocking operations.
- Test handling of retriable / ignorable (410s) / unrecoverable errors.
- Verify/Test idempotency of cleanup / delete / ttl operations.
|
