Goal

Enable Parameter Providers to automatically refresh parameter values from external sources at configurable intervals, allowing NiFi flows to automatically adapt to rotating secrets, updated configuration values, and other externally managed parameters without manual intervention.

Background and Current State

Parameter Providers Overview

Parameter Providers were introduced in Apache NiFi 1.18.0 as extension points that allow parameters to be fetched from external sources such as:

Parameter Providers enable automatic creation and population of Parameter Contexts from these external sources, providing powerful configurability and portability for NiFi flows.

Current Manual Operation

The current Parameter Provider workflow requires manual intervention:

  1. Initial Setup: Users configure a Parameter Provider and map parameter groups to Parameter Contexts
  2. Manual Refresh: When external parameter values change (e.g., secret rotation), users must:

Note that new values would also be retrieved in case of a NiFi restart. The parameter values coming through a parameter provider are NOT persisted by NiFi and are kept in memory. In case of restart, NiFi would fetch again all of the values for existing parameters that are defined through parameter providers.

Limitations of Manual Approach

The manual approach creates operational overhead and delays in several scenarios:

Current Workaround: NiFi CLI Scripting

Users can partially automate parameter updates using the NiFi CLI fetch-params command:

bash

# Fetch and apply parameters for a specific provider
./nifi.sh fetch-params --parameterProviderId <provider-id> --apply

# Script example for periodic execution
#!/bin/bash
PROVIDER_ID="vault-secrets-provider"
./nifi.sh fetch-params --parameterProviderId $PROVIDER_ID --apply
if [ $? -eq 0 ]; then
    echo "Parameters updated successfully"
else
    echo "Parameter update failed"
    # Handle error (alerting, logging, etc.)
fi

Limitations of CLI Approach:

Implementation Approaches

Approach 1: Framework-Level Scheduling Integration

Overview

Extend the existing NiFi scheduling framework to support Parameter Providers as schedulable components, similar to how Reporting Tasks are currently handled.

Key Requirements

  1. Backward Compatibility: Existing Parameter Providers must continue working unchanged
  2. Opt-in Behavior: Automatic refresh must be explicitly enabled (disabled by default)
  3. Selective Updates: Only update existing parameters, ignore new parameters (new parameters may be sensitive or not, and it is better to leave this decision to a user)
  4. Audit Trail: Log all automatic parameter changes for security and compliance
  5. Error Handling: Graceful handling of external system failures
  6. Schedule Flexibility: Support both timer-driven and CRON-driven scheduling

High-Level Architecture Changes


1. Framework Scheduler Integration

2. Node-Level Scheduling (Following ReportingTaskNode Pattern)

java

public interface ParameterProviderNode extends ComponentNode {
    // Scheduling methods - identical to ReportingTaskNode
    void setSchedulingStrategy(SchedulingStrategy schedulingStrategy);
    SchedulingStrategy getSchedulingStrategy();
    void setSchedulingPeriod(String schedulingPeriod);
    String getSchedulingPeriod();
    
    // Parameter Provider specific settings
    void setAutoApplyChanges(boolean autoApply);
    boolean isAutoApplyChanges();
    
    // Standard node methods
    ParameterProvider getParameterProvider();
    void verifyCanSchedule();
    void verifyCanStop();
}

3. Framework Integration (Mirror ReportingTask Architecture)

The default value for scheduling period would be "0 sec". This specific default would be used as a way to disable the automatic refresh of the parameter values and would ensure backward compatibility.

java

public class ParameterProviderNode {
    private boolean automaticRefreshEnabled = false;  // Default: disabled
    private SchedulingStrategy schedulingStrategy = SchedulingStrategy.TIMER_DRIVEN;
    private String schedulingPeriod = "0 sec";        // Default: disabled (special value)
    private boolean autoApplyChanges = false;         // Default: manual approval required
}

4. UI Components

5. REST API Extensions

Required Code Changes

Core Framework:

Parameter Provider Infrastructure:

Pros

Cons

Approach 2: Provider-Level Background Threading

Overview

Implement automatic refresh capability within the AbstractParameterProvider class using provider-managed background threads, without modifying the core scheduling framework.

High-Level Architecture Changes

1. Abstract Provider Enhancement

java

public abstract class AbstractParameterProvider implements ParameterProvider {
    private boolean autoRefreshEnabled = false;
    private String refreshInterval = "0 sec";
    private ScheduledExecutorService scheduledExecutor;
    private volatile boolean running = false;
    
    protected void startAutoRefresh() {
        if (autoRefreshEnabled && !"0 sec".equals(refreshInterval)) {
            // Start background thread for periodic refresh
        }
    }
    
    protected abstract void onAutoRefresh();
}

2. Provider-Specific Configuration

3. Thread Management

Required Code Changes

Parameter Provider Module:

Configuration Persistence:

Pros

Cons