Page History

...

Goals

• Provide the ability for users to parameterize the values of any Processor Property in the flow, including sensitive properties
• Support parameterization of properties without the developer needing to enable it
• Improve the User Experience of importing a flow so that all parameters can be set in one place

...

A Parameter Context is a named set of Parameters. The Parameter Context is the unit at which Policies are applied. There will exist a Policy that allows users to create a Parameter Context. Once created, policies to Read and Write a specific Parameter Context may be applied. Parameter Contexts live at the "controller level," meaning that that do not belong to any Process Group but rather are globally defined / accessible for the NiFi instance.

Concepts / Rules

A user with appropriate permissions can create a Parameter Context and then add Parameters to it. That Parameter Context can have its Read/Write Policies updated so that only appropriate users can read/write to the context.

Any property can be configured to reference a Parameter instead of explicitly defining a value, with the following caveats:
 - A sensitive property may only reference a Sensitive Parameter. This rule exists because when a sensitive property references a Parameter, we want to store that reference in the Flow Registry. If we allowed referencing a non-sensitive Parameter, this means that we would expose both the reference and the value, resulting in the exposure of the sensitive property's value.
 - A non-sensitive property may only reference a non-Sensitive Parameter. When a developer marks a property as sensitive, it means that the s/he is they are taking responsibility to treat the value of the property as such, and should not be logging the value, etc. If the property is not marked as sensitive, the developer is making no such assertion about how the value will be treated and, as such, the value of a Sensitive Parameter should not be exposed to the property.
 - Properties that reference Controller Services will not be able to use Parameters.

...

Because it is such a common occurrence for a user to want to parameterize different property values, the User Experience must make it easy for the Flow Designer to quickly and easily add a new Parameter. So, if a Parameter Context has already been set for a Process Group, and the user has the WRITE policy for it, the UI should provide the user the ability, when editing a Property Value, to click some sort of "Create Parameter..." button and create a new variable Parameter in-line. AdditionallyThis should set the value of the property to `#{<Parameter Name>}` and add the Parameter to the Parameter Context. Additionally, the UI should provide auto-complete capabilities so that the user can type something like `#{a` and then have auto-complete provide the ability to select the `abc` parameter. The auto-complete should show a tool-tip that indicates the Property's Description, Sensitivity Flag, and Value (if not sensitive).

...

Sensitive Properties may reference Sensitive Parameters. However, if doing so, the value of the property must be set precisely to a single Parameter reference. For example, a value of `#{password}123` will not be allowed. Additionally, a value of `#{password}#{suffix}` will not be allowed. This will be enforced because we want to be able to indicate in the Flow Registry that the property references a Parameter by including the value of a Sensitive Property if and only if the value is a reference to a sensitive parameter. Allowing a value of `#{password}123` makes it difficult to understand what will be sent to the Flow Registry. Should we send nothing? What if the user inadvertently has white space? That would lead to significant confusion. We cannot send `#{password}123` because that would lead to exposing part of the Sensitive Property's value. Additionally, this type of pattern would encourage the templating of passwords, etc.

From the Hamburger menu, a user should be able to go to a Parameter Context Manager screen where s/he is they are able to add a new context, and change the parameters and policies of a given context (assuming appropriate permissions). When modifying the values of Parameters, the user should be able to update the values of multiple Parameters Additionally, if a given Parameter is used in any Property that supports Allowable Values, the editor should show only the values that are allowed. If used from multiple properties, each with different Allowable Values, the UI should indicate that the value cannot be changed because there are no acceptable values for this Parameter due to this fact. When modifying the values of Parameters, the user should be able to update the values of multiple Parameters and then be able to click "Apply" to apply changes to a Parameter Context atomically. I.e., do not stop processors, apply change, validate, restart for each Parameter change but rather once for the change of the Parameter Context. When the user changes the value of a Parameter, the UI should kick off a background process to validate all components that reference that Parameter. If any component changes from being valid to invalid, that should result in a warning in the UI. The UI should not allow the user to apply changes to a Parameter Context until s/he has they have received confirmation that it does not invalidate any components or received warning that the change set will in fact invalidate components (and the corresponding validation errors).

When creating a new Parameter Context, the user should be given ability to create a new, empty Context or copy an existing Context that s/he has READ they have READ permissions to. This makes it far easier to design a flow, then deploy several instances of the flow, in which each instance has mostly the same parameters but slightly different.

...

1. Open Parameter Context Manger from the hamburger menu.
2. Create Parameter Context named "My Context"
3. Add Parameters to Context
     - db.url
     - db.table
     - db.username
     - db.password
     - kafka.brokers
     - kafka.topic
4. Create Process Group
5. Configure Process Group
     - Set Parameter Context to "My Context"
6. Add ExecuteSQL Processor to Process Group
7. Configure ExecuteSQL
     - Set Property 'Database Url' to `#{db.url}`
     - Set Property 'Database Password' to `db.password``#{db.password}`
     - etc.
8. Create PublishKafkaRecord Processor
9. Configure PublishKafkaRecord
     - Set Property 'Kafka Brokers' to `#{kafka.brokers}`
     - Set Property 'Kafka Topic' to `#{kafka.topic}`
10. Start Version Control on Process Group
      - Name Flow in Registry "Replicate DB to Kafka"
      - Choose which Parameter Context to Export
          - Choose existing Parameter Context (may or may not be the one selected for the Process Group to use during runtime).
          - Create new (provides user ability to populate values for all Parameters in the context and send those values to the Flow Registry)
          - Create new from existing (provides user ability to choose an existing Parameter Context, make a copy of it, and then provide different values to use for populating Flow Registry)
          - Set empty, which exports parameter metadata needed for "guided tour" experience for consumers, but not parameter values
      - Exports flow including parameter context.
      - Everything in parameter context gets written to Registry except the values of sensitive parameters. (If the Parameter Context includes Parameters that the flow does not use, those are not included.)

...

The following story depicts the steps required for to import the flow that was developed above into a new environment and make use of it.

...

Note that this user experience for importing the flow addresses both of the key use cases: Migration of flow from dev to test to prod, as well as the need to create parameterized flows (i.e., import the same flow multiple times into the same environment, each with different parameters). Each time the user imports the flow, s/he can they can choose a new set of parameters to use.

Future Enhancements

The following are improvements to the feature being proposed here, that are not necessarily required for the initial release / MVP of the feature.

Shareable Parameter Contexts (centralized, versioned)

• Users will be able to store parameter contexts in a central service
    - perhaps it is an interface and provider model, where one of the providers could be a product that is designed for storing sensitive information
    - perhaps NiFi Registry gets a "Parameter Context" bucket item type, and the NiFi Registry can have multiple provider impls
• Users will be able to version control Parameter Contexts
• Users will be able to reference local or remote Parameter Contexts by coordinates, so anywhere we allow setting/selecting a context in the MVP, users will also be able to set a URL:bucket:context:version

• Parameter values can reference values from other contexts, allowing a user to create a context that references values from other contexts using context:parameter coordinates
• Parameter contexts can inherit from other contexts (i.e., Context A has 10 Parameters defined and I can create Context B that inherits from it and overrides one Parameter)
• We introduce the concept of _Composite Parameter Contexts_ in which you define it as a list of other contexts, with an ordering so that Parameter value overrides get resolved deterministically.

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Change Flow Version

After importing the flow as in the story above, many times, a user will change the flow and add a new version to the Registry. If doing so results in a new Parameter being added, that must be addressed when an instance of the flow is updated to the new version. The following story depicts the steps required to change the version of the flow when new Parameters have been added to the Parameter Context, or Parameters that were previously not referenced become referenced.

1. Right-click Process Group on Canvas. Choose 'Version → Change Version'
2. Select the Registry, Bucket, and Flow from the list.
3. Choose version 2 of the flow and click Change.
4. User will be presented with a dialog that shows the Parameters that were added (along with their values). The user will be given the option of how to account for the new Parameters:
   1. Add new Parameters to existing/selected Parameter Context (only if user has the WRITE policy for the Parameter Context). User is prompted to specify values for the new Parameters.
   2. Create a new Parameter Context whose values are copied from the currently selected Parameter Context (only if user has the WRITE policy for Parameter Contexts / ability to create Parameter Contexts)
   3. Create a new Parameter Context whose values are copied from the values in the Flow Registry (only if user has the WRITE policy for Parameter Contexts / ability to create Parameter Contexts)
   4. Cancel 'Change Version' action (this requires no special policies / permissions for the user)
5. User clicks button to complete 'Change Version' action

Future Enhancements

The following are improvements to the feature being proposed here, that are not necessarily required for the initial release / MVP of the feature.

Shareable Parameter Contexts (centralized, versioned)

• Users will be able to store parameter contexts in a central service
    - perhaps it is an interface and provider model, where one of the providers could be a product that is designed for storing sensitive information
    - perhaps NiFi Registry gets a "Parameter Context" bucket item type, and the NiFi Registry can have multiple provider impls
• Users will be able to version control Parameter Contexts
• Users will be able to reference local or remote Parameter Contexts by coordinates, so anywhere we allow setting/selecting a context in the MVP, users will also be able to set a URL:bucket:context:version

• Parameter values can reference values from other contexts, allowing a user to create a context that references values from other contexts using context:parameter coordinates
• Parameter contexts can inherit from other contexts (i.e., Context A has 10 Parameters defined and I can create Context B that inherits from it and overrides one Parameter)
• We introduce the concept of _Composite Parameter Contexts_ in which you define it as a list of other contexts, with an ordering so that Parameter value overrides get resolved deterministically.

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Not Doing