Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Clarified that bad syntax results in text being passed unchanged.

...

  • The UnderlyingValues interface defines the map-like interface that the input to SubstitutableValues must implement.  When a reference is made from one underlyng value to another – via the defaultKey=<Key> or fromValueOfKey modifiers as described later – the reference is resolved within the underlying values provided to the SubstitutableValuesinstance.
  • Once an instance of SubstitutableValues retrieves an underlying value and its calculated value -- whether different from the raw underlying value due to a substitution or not -- is determined, the instance of SubstitutableValues will not retrieve the underlying value again; the calculated value will be used if it is referred to. This means if the underlying values are expected to change then to see those changes a new instance of SubstitutableValues must be allocated.
  • A parsing error within recognized delimiters results in the delimiters and intervening text being passed through unchanged.  This is to help prevent accidental substituion occuring in existing passwords (for example).  See Compatibility below.

Code Block
languagejava
titleorg.apache.kafka.common.security.substitutions.UnderlyingValues
collapsetrue
package org.apache.kafka.common.security.substitutions;
 
/**
 * The map-like interface that the input to {@link SubstitutableValues} must
 * implement.
 */
 
public interface UnderlyingValues {
    /**
     * Return the value associated with the given key, if any, otherwise null
     * 
     * @param key
     *            the mandatory key
     * @return the value associated with the given key, if any, otherwise null
     */
    Object get(String key);
}
Code Block
languagejava
titleorg.apache.kafka.common.security.substitutions.SubstitutableValues
collapsetrue
package org.apache.kafka.common.security.substitutions;

/**
 * Adds support for substitution within the values of an instance of
 * {@link UnderlyingValues}. Substitution is accomplished via delimited text
 * within a value of the following form:
 *
 * <pre>
 * &lt;OPENING_DELIMITER&gt;&lt;TYPE&gt;&lt;OPTIONAL_MODIFIERS&gt;=&lt;OPTIONAL_IDENTIFIER&gt;&lt;CLOSING_DELIMITER&gt;
 * </pre>
 *
 * Where the above elements are defined as follows:
 *
 * <pre>
 * OPENING_DELIMITER: $[, $[[, $[[[, $[[[[, or $[[[[[
 * CLOSING_DELIMITER:  ],  ]],  ]]],  ]]]], or  ]]]]] (number of brackets must match)
 *
 * TYPE: everything up to (but not including) the first punctuation character
 *
 * OPTIONAL_MODIFIERS: the optional section immediately after the TYPE, starting with any
 *                     punctuation character except for the equal sign (=), and ending
 *                     with the same punctuation character followed immediately by an
 *                     equal sign. The same punctuation character delimits individual
 *                     modifiers, which come in two flavors: flags, which do not contain
 *                     an equal sign, and name=value arguments, which do.
 *
 * OPTIONAL_IDENTIFIER: the optional section immediately after any modifiers and the equal
 *                      sign (=).
 * </pre>
 *
 * For example:
 *
 * <pre>
 * $[envVar=THE_ENV_VAR]
 * $[envVar/notBlank/redact/=THE_ENV_VAR]
 * $[envVar/defaultValue = theDefaultValue/=THE_ENV_VAR]
 * $[envVar/defaultKey = theDefaultKey/=THE_ENV_VAR]
 * $[file|redact|notBlank|=/the/path/to/the/file]
 * </pre>
 *
 * Once an underlying value is retrieved and its calculated value -- whether
 * different from the raw underlying value due to a substitution or not -- is
 * determined, this instance will not retrieve the underlying value again; the
 * calculated value will be used if it is referred to. This means if the
 * underlying values are expected to change then to see those changes a new
 * instance of this class must be allocated.
 * <p>
 * Working left to right, once the delimiters are defined for a value (for
 * example, {@code $[} and {@code ]}), only those delimiters are recognized for
 * the rest of that value (and they are always recognized as meaning
 * substitution for the rest of that value). A special "empty" substitution does
 * nothing except, when it appears to the left of every other occurrence of
 * matching delimiters, it serves to force the delimiter for that value to the
 * one indicated. For example, to force the delimiter to {@code $[[} and
 * {@code ]]} (and prevent {@code $[} and {@code ]} from causing substitution)
 * for a value:
 *
 * <pre>
 * someKey = "$[[]]These $[ and ] delimiters do not cause substitution"
 * </pre>
 *
 * The following built-in substitution types are supported, though it is
 * straightforward to add others (see below):
 *
 * <ul>
 * <li>{@code envVar}: substitute an environment variable, typically indicated
 * by the identifier</li>
 * <li>{@code sysProp}: substitute a system property, typically indicated by the
 * identifier</li>
 * <li>{@code file}: substitute the contents of a file, typically indicated by
 * the identifier</li>
 * <li>{@code keyValue}: substitute the contents of another key's value,
 * typically indicated by the identifier (and that key's value has substitution
 * performed on it if necessary)</li>
 * </ul>
 *
 * The built-in substitution types support the following flags, which are
 * trimmed and may be redundantly specified:
 * <ul>
 * <li>{@code redact}: prevent values from being logged</li>
 * <li>{@code notEmpty}: the value must not be empty</li>
 * <li>{@code notBlank}: the value must not be blank (i.e. consisting only of
 * whitespace); implies {@code notEmpty}.</li>
 * <li>{@code fromValueOfKey}: provides a level of indirection so that the
 * identifier, instead of being used directly (i.e. read the indicated file, or
 * the indicated environment variable), identifies another key whose value is
 * used as the identifier instead. This allows, for example, the filename,
 * system property name, etc. to potentially be generated from multiple
 * substitutions concatenated together.</li>
 * </ul>
 * The built-in substitution types support the following arguments, whose names
 * are trimmed but whose values are not; it is an error to specify the same
 * named argument multiple times (even if the values are identical):
 * <ul>
 * <li>{@code defaultValue=<value>}: substitute the given literal value if the
 * substitution cannot otherwise be made (either because the identifier
 * indicates something that does not exist or the determined value was
 * disallowed because it was empty or blank). The substituted default value must
 * satisfy any {@code notBlank} or {@code notEmpty} modifiers that act as
 * constraints, otherwise it is an error.</li>
 * <li>{@code defaultKey=<key>}: substitute the value associated with the
 * indicated key if the substitution cannot otherwise be made (either because
 * the identifier indicates something that does not exist or the determined
 * value was disallowed because it was empty or blank). The value that is
 * ultimately substituted must satisfy any {@code notBlank} or {@code notEmpty}
 * modifiers that act as constraints, otherwise it is an error.</li>
 * </ul>
 *
 * To add new substitutions beyond the built-in ones mentioned above simply
 * define a key/value pair of the following form:
 *
 * <pre>
 * [optionalTypeDefinitionKeyPrefix]&lt;type&gt;SubstituterType = "fully.qualified.class.name"
 * </pre>
 *
 * For example:
 *
 * <pre>
 * fooSubstituterType = "org.example.FooSubstituterType"
 * </pre>
 *
 * The indicated class must implement {@link SubstituterType}, and you can
 * invoke the substitution in a value like this:
 *
 * <pre>
 * $[foo/optional/modifiers/=optionalValue]
 * </pre>/=optionalValue]
 * </pre>
 *
 * The type definition prefix is defined at construction time and may be empty.
 *
 * A parsing error within recognized delimiters results in the delimiters and
 * the intervening text that could not be parsed being left alone. For example,
 * the following text would be passed through unchanged because the delimited
 * text cannot be parsed as a valid substitution request:
 * {@code qw$[asd_4Q!]uH6}.
 *
 * @see SubstituterType
 * @see SubstituterTypeHelper
 */
public class SubstitutableValues {
    /**
     * Constructor where the type definition key prefix is empty
     *
     * @param underlyingMap
     *            the mandatory underlying map. It is not copied, so it should be
     *            immutable. Results are unspecified if it is mutated in any manner.
     */
    public SubstitutableValues(UnderlyingValues underlyingValues) {
        this("", underlyingValues);
    }

    /**
     * Constructor with the given type definition key prefix
     *
     * @param typeDefinitionKeyPrefix
     *            the mandatory (but possibly empty) type definition key prefix
     * @param underlyingMap
     *            the mandatory underlying map. It is not copied, so it should be
     *            immutable. Results are unspecified if it is mutated in any manner.
     */
    public SubstitutableValues(String typeDefinitionKeyPrefix, UnderlyingValues underlyingValues) {
        // etc...
    }
    /**
     * Return the always non-null (but possibly empty) type definition key prefix
     *
     * @return the always non-null (but possibly empty) type definition key prefix
     */
    public String typeDefinitionKeyPrefix() {
        return typeDefinitionKeyPrefix;
    }


    /**
     * Return the underlying values provided during construction
     *
     * @return the underlying values provided during construction
     */
    public UnderlyingValues underlyingValues() {
        return underlyingValues;
    }
    /**
     * Return an unmodifiable map identifying which keys have been processed for
     * substitution and the corresponding result (if any). A key is guaranteed to
     * have been processed for substitution and its name will appear as a key in the
     * returned map only after {@link #getSubstitutionResult(String)} has been
     * invoked for that key either directly or indirectly because some other key's
     * substitution result depends on the substitution result of the key.
     *
     * @return an unmodifiable map identifying which keys have been processed for
     *         substitution and the corresponding result (if any)
     */
    public Map<String, RedactableObject> substitutionResults() {
        // etc...
    }

    /**
     * Perform substitution if necessary and return the resulting value for the
     * given key
     *
     * @param key
     *            the mandatory requested key
     * @param requiredToExist
     *            if true then the requested key is required to exist
     * @return the given key's substitution result, after any required substitution
     *         is applied, or null if the key does not exist and it was not required
     *         to exist
     * @throws IOException
     *             if a required substitution cannot be performed, including if the
     *             given (or any other) required key does not exist
     */
    public RedactableObject getSubstitutionResult(String key, boolean requiredToExist) throws IOException {
        // etc...
    }

    // etc...
}

...

Compatibility, Deprecation, and Migration Plan

Anchor
Compatibility
Compatibility
There is a possibility that existing usernames or (more likely) passwords in existing client JAAS configurations could contain the "$[" and "]" delimiters.  This would cause a substitution to be attempted, which of course would fail and potentially raise an exception.  This risk is low, but it nonetheless does need to be mitigated; therefore any already-existing login modules where substitution support is to be added (namely, the ones mentioned above) will only enable substitution if a key/value pair is explicitly added to the JAAS configuration as follows:

...

Existing behavior will remain unchanged in the absence of this explicit opt-in key/value pair.  Even with this opt-in, though, we still do not want unintended substitutions to occur, so if delimiters are recognized but a parsing error occurs (e.g. a value such as qw$[asd_4Q!]uH6 would cause a substitution to be attempted but does not ultimately meet the required syntax) the delimiters and intervening text will be passed unchanged.

Rejected Alternatives

This KIP does not define Callback or CallbackHandler implementations because configuration values are typically retrieved without using them (this is the case with PlainLoginModule and ScramLoginModule).

...