Status

Current state: Accepted

Vote thread: here

Discussion thread: here

JIRA: KAFKA-6435 – Application Reset Tool might delete incorrect internal topics

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Users may want to specify which internal topics should be deleted. At present, the streams reset tool deletes all topics that start with "<application.id>-" and there are no options to control it.

The `--internal-topics` option is especially useful when there are prefix conflicts between applications, e.g. "app" and "app-v2". In this case, if we want to reset "app", the reset tool's default behaviour will delete both the internal topics of "app" and "app-v2" (since both are prefixed by "app-"). With the `--internal-topics` option, we can provide internal topic names for "app" and delete the internal topics for "app" without deleting the internal topics for "app-v2".

Public Interfaces

Add an option to the streams reset tool.

--internal-topics <String: list>      Comma-separated list of internal topics
                                      to delete. Must be a subset of the
                                      internal topics marked for deletion by
                                      the default behaviour (do a dry-run without this
                                      option to view these topics).

Proposed Changes

Add an internal-topics option to streams reset tool.

The `--internal-topics` option will take a comma-separated list of internal topics to be deleted. These topics must be a subset of those flagged as internal during a `--dry-run` without the `internal-topics` option. If omitted, the tool will default to the original behaviour, i.e. delete all topics that start with "<application.id>-".

Usage

Usage of the "internal-topic" options will involve:

  1. Run `kafka-streams-application-reset.sh` with `--dry-run` to check which internal topics will be deleted.
  2. If everything looks okay,  run the script again without `--dry-run`. Otherwise, run it with `--internal-topics` to specify the internal topics to be deleted.

Compatibility, Deprecation, and Migration Plan

If the new option is not specified, the stream resetter's behaviour will remain unchanged.

Rejected Alternatives

Disallow "dash" in the application.id

This fixes the example given above in the motivation, but does not allow users to specify the internal topics. Also, it will raise a lot of backward compatibility issues.


Have a "negative prefix" option

This may be more convenient than having to list out relevant internal topics which may be tedious. However, the internal-topics option is a lot more explicit and it's usage will be more familiar because of it's similarity to other options (e.g. input-topics and intermediate-topics).  Having displayed the internal-topics marked for deletion in the dry-run, it is more straightforward to simply specify the subset to be deleted instead of finding a prefix (or prefixes) to omit everything by.


  • No labels