Page History

...

# create a new virtual cluster
virtual-clusters.sh create
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster

# create a topic link in it
virtual-clusters.sh alter
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster
  --add
  --topic my-topic --link test-topic

# assign a user to the virtual cluster
virtual-clusters.sh alter
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster
  --add
  --user jane-doe

# list the virtual clusters
virtual-clusters.sh list
  --bootstrap-server localhost:9092

# describe the virtual cluster
virtual-clusters.sh describe
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster

# delete the virtual-cluster
virtual-clusters.sh delete
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster

ACL Migration

Users that aren’t created freshly before adding them to a virtual cluster, probably have some ACLs applied on them. With the current proposal we can add and remove ACLs but the community would benefit from a more user-friendly tool that migrates the user’s ACLs into the virtual cluster.

A full migration could look like this:

1. Create a VC
2. Create topic links inside a VC
3. Move a user and its resources to a VC:
1. Assign its resources (groups, delegation tokens, transactional IDs) to the VC
2. Create ACLs in the VC for these resources
3. Assign the user to the VC
4. Remove the old ACLs

This is a complex migration process and also while it can be complex to do in one command, we can help cluster operators to automate b., c. and d. the following way:

• When a user is assigned to a virtual cluster, the administrator would have to specify the --copy-acls-from <source> option if ACLs needed to be migrated. This would copy all ACLs from the source virtual-cluster that targets the user in the command. If the kafka-cluster source is specified, then the migration utility will look for ACLs not belonging to any virtual clusters.
• In the same command or in a separate one by specifying the --remove-old-acls-from <source> option, the administrator from source can remove any ACLs that are already migrated. This would copy all ACLs from the source virtual-cluster that targets the user in the command. If the kafka-cluster source is specified, then the migration utility will look for ACLs not belonging to any virtual clusters.
  

During migration, topic references will be translated to the appropriate link references by querying the links on the broker side.

# assign a user to the virtual cluster and also copy its ACLs that exist in the global space (kafka-cluster)
virtual-clusters.sh alter
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster
  --add
  --user jane-doe
  --migrate-acls-from kafka-cluster

# assign a user to the virtual cluster and also copy its ACLs that exist in another VC (my-old-virtual-cluster)
virtual-clusters.sh alter
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster
  --add
  --user jane-doe
  --copy-acls-from my-old-virtual-cluster

# remove any ACLs for the user that are in the specified VC (if the user is already added, it won't readd it, but if it isn't added yet, the command fails)
virtual-clusters.sh alter
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster
  --add
  --user jane-doe
  --remove-old-acls-from kafka-cluster

# assign a user to the virtual cluster and also copy its ACLs that exist in another VC (my-old-virtual-cluster), then remove the old ones in the same command - this should be used only if downtime isn't an issue
virtual-clusters.sh alter
  --bootstrap-server localhost:9092
  --virtual-cluster my-virtual-cluster
  --add
  --user jane-doe
  --migrate-acls-from my-old-virtual-cluster
  --remove-old-acls-from my-old-virtual-cluster 

Extending kafka-acls.sh

Administrators will need to set virtual cluster specific ACLs for existing users as well, besides managing virtual clusters themselves. We plan to add the --virtual-cluster option to the kafka-acls.sh command to enhance it with VC capabilities. The existing options would work well with this extra option. We would like to introduce another new parameter to control the pattern of the virtual cluster specified. This new --virtual-cluster-resource-pattern-type could apply to VCs only to provide more robust, generic specification of ACLs. It would default to literal if not specified explicitly.

...