THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
C2 Protocol Introduction
Command and control, herein C2, consists of a C2 server and C2 agents. MiNIFi agents must adhere to the C2 protocols to have successful communications. C2 communications occur over a variety of protocols. Currently an HTTP/HTTPS RESTFul paradigm exists to support C2 capabilities to MiNiFi C2 agents. In the future additional protocols may become available for use. Note that when the phrasing "C2 designator" is used, this implies the C2 server, server, or agent that is designated as a responder to the C2 agent(s). All protocols must support the following operations:
| Operation Name | Description |
|---|---|
ACKNOWLEDGE | Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation |
CLEAR | Clears flow connection queues or repositories on the C2 agent |
| CONSUME | Consumes a heartbeat with an agent to avoid sending from other agents. Allows multiplexing responses from a condensed agent response. |
DESCRIBE | Currently Unused |
| EXECUTE | Executes commands on the agent's operating system. This feature may be disabled for any agent. |
HEARTBEAT | Heartbeat provides status and operational capabilities to C2 server(s) |
| PAUSE | Pauses the execution of flows on the C2 agent |
| REPLICATE | Replicates agent state between agents, with the ability to place agents in standby mode until they are needed. |
RESTART | Restarts C2 agents |
| RESUME | Resumes the execution of flows on the C2 agent |
START | Starts components within the C2 agents |
| STOP | Stops components within the C2 agent |
| SUBSCRIBE | Allows servers or agents to subscribe to an agent's heartbeat , requesting specific information for the next heartbeat. |
| TRANSFER | Transfers an object between the C2 agent and C2 designator. |
| UPDATE | Updates components of the C2 agent or the flow configuration. |
C2 Requirements
The requirements are an evolving list that have grown organically from an implementation. Any other portions of a heartbeat are considered optional.
| C2 Requirement | Justification and Purpose |
|---|---|
| C2 agents shall report C2 status at regular intervals through a heartbeat message | Agents must employ heartbeat messages that follow an interval that is favorable to the agent ( power ) |
| C2 agents shall report the flow version within the heartbeat message at regular intervals | Agents must report flow version to the C2 server |
| C2 agents shall report queue status within the heartbeat message at regular intervals | Agents must report queue status to the C2 server at regular intervals |
| C2 agents shall execute acknowledge commands sent via a heartbeat response | Agents must execute and acknowledge commands from the C2 server |
| C2 agents shall apply requested changes and inform the C2 server of success or failure | Agents must apply and acknowledge updates from the C2 server, responding with a success or failure |
| C2 agents shall implement clear, update, restart, start, stop, and transfer commands | Agents must implement the prescribed commands. |
C2 Messages
Heartbeats
Primary communications are carried over a C2 heartbeat. The heartbeat contains operational information about the C2 agent and can occur at a configurable frequency. The heartbeat provides status information to the C2 server. The response from the heartbeat contains requested operations from the C2 server. These operations are then acknowledged if/when they are completed. This means that the heartbeat is the only operation initiated by the C2 agent and the C2 server responds directly to these heartbeats. Version four of the heartbeat will allow a subscription model to be used for heartbeats to avoid sending unnecessary information. Though Describe can provide parts of the same information the aggregate produced for a heartbeat allows the agent to flush messaging queues to ensure subscribed heartbeats have the most up to date information. Heartbeats are intended to be lightweight structures with minimal information; however, the subscription model supports differing wire protocols and deployment scenarios that support larger payloads.
Protocols
HTTP/S Protocol
The HTTP/S protocol supports a url for heartbeating and acknowledging operations. These endpoints support the JSON structures defined below. C2 agents must send a heartbeat, defined above, to update the C2 server of its status and to receive operations. The frequency of these calls are up to the C2 agent to define.
CoAP Protocol
The CoAP protocol (https://coap.technology/) supports a constrained protocol for smaller devices. In the case of CoAP, the base requirements, as listed above, are fulfilled in each message. While the heartbeat structure, below, contains optional elements, the CoAP protocol implemented in Apache NiFi MiNiFi C++ contains minimal information per heartbeat.
Heartbeat structure
Heartbeats consist of a POST of the following Schema to the C2 heartbeat url. Metrics is a configurable list of metrics that can be returned, so the entirety of that object is optional. AvailableClasses in the structure below is optional.
Update: current work uses a flow identifier to help identify the currently running flow within DeviceInfo.
Responses to the heartbeats have the following structure
Key
Value
AgentInformation
Key
Value
AvailableClasses
SupportsDynamicProperties
Class_name
Properties
false
org::apache::nifi::minifi::processors::AppendHostInfo
property
false
org::apache::nifi::minifi::processors::AppendHostInfo
property
BuildInformation
Key
Value
Build_date
date
Build_rev
string
Build_version
string
Compiler
string
Compiler_flags
string
Device_id
string
Extensions
extension1
extension2
extensionn
NetworkInfo
Key
Value
Deviceid
string
Flowid
string
Hostname
string
Ip
string
SystemInformation
Key
Value
Machinearch
string
Physicalmem
string
Vcores
string
Components
Key
Value
FlowController
enabled
ProcessorName
enabled/disabled
Metrics
Key
Value
ProcessMetrics
Key
Value
CpuMetrics
Key
Value
Involcs
string
MemoryMetrics
Key
Value
Maxrss
string
QueueMetrics
Key
Value
Connection
Key
Value
Datasize
string
Datasizemax
string
Queued
string
Queuedmax
string
RepositoryMetrics
Key
Value
Flowfile
Key
Value
Full
1/0
Running
1/0
Size
string
Provenance
Key
Value
Full
1/0
Running
1/0
Size
string
Operation
heartbeat
State
Key
Value
Running
true/false
Uptime
string
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
Content
string
string
string
String
string
Operation schemas
The following are the schema definitions for each operation that is contained within the requested operations of a heartbeat response. It is expected that C2 agents adhere to this structure
Clear
The clear operation uses name of connection or repositories to clear either the connections or the repositories. In the case of a connection the content contains the operation arguments, in which the value defines the connection name to clear.
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
Content
connection
string
clear
Unique_map_key
connection_name
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
repositories
string
clear
Update
Update allows the C2 server to update either the c2 agent or provide a URI from which we download the new flow configuration through a GET request.
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
Content
configuration
string
update
Location
HTTP or HTTPS URL
The following activity diagram depicts the flow of updating an agent from failure to success.
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
Content
c2
string
update
Option_name
option_value
Start
Start starts a previously stopped command. If a start is called on a component that is already started, nothing should occur other than an acknowledgement. Name defines the component to start.
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
component name
string
start
Stop
Stop stops a component that is started. Components can be the FlowController, processors, or RPGs
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
component name
string
stop
Transfer
Transfer will download an object from the C2 designator ( server or other location ). The location URI will be provided by the update JSON. This transfer will not be a JSON
response but will instead be the binary object. The hash of the object will be the acknowledgement ID for the transfer.
Key
Value
Operation
acknowledge
Operationid
hash of object
Restart
Attempts to restart the component defined within name
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
component name
string
restart
Pause
Pauses the execution of flows on the C2 agent (if the agent is running and is not in paused state), while the agent keeps running and heartbeating.
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
component name
string
pause
Resume
Resumes the execution of flows on the C2 agent if the agent is in paused state.
Key
Value
Operation
heartbeat
Requested_operations
Name
Operationid
Operation
component name
string
resume
Acknowledgements.
Acknowledgements occur through a separate URL. This URL will receive a POST that contains the following payload, which acknowledges that the operation ID was received and executed.
Key
Value
Operation
acknowledge
Operationid
string
MQTT Protocol
MQTT can be used as a connecting protocol in lieu of a RESTFul Service. Additionally, MQTT can be used within an enclave and then as conversion to RESTFul to support MQTT → HTTP comms.
Operations and versioning
Apache NIFi MiNiFI C++ automatically supports versions 4, 3, and 2. Version one is not supported by the agent. Agents are backwards compatible and forward compatible.
C2 role delegation
Since C2 defines operations for agents and a conversion protocol, agents can be used for role delegation. This means that Agents can act as a C2 server. This design is intentional, and supports the eventual automation of capabilities. Version four
begins implementing facilities that will be used for autonomous control. Agents can control each other and execute commands based on defined policies. Policies are controller services and configurations that enable activities to be that allow
autonomous and near-autonomous control of subnets of agents.
C2 Policies
This sections defines the sub pages for C2 policies defined as controller services.
Operations and their operands for agents (Version 4* Not released)
| Operation Name | Description | operand/name | content/args |
|---|---|---|---|
ACKNOWLEDGE | Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation | N/A | |
| CLEAR | Clear repositories | repositories | N/A |
CLEAR | Clears the connection queues | connection | connection1=<connection name>, connection2=<connection 2> ... Will also accept a list <connection name1>,<connection name2>, ... |
| CONSUME | Consumes a heartbeat within an agent to avoid sending from other agents | N/A | N/A |
DESCRIBE | Return metrics | metrics | metricsClass=<metric class to obtain> |
| DESCRIBE | configuration | N/A | N/A |
| DESCRIBE | manifest | N/A | N/A |
| DESCRIBE | policy events – Based on the defined policies | ||
| EXECUTE | Executes commands per the agent's defined policies | command | arguments |
| HEARTBEAT | heartbeat operation – may contain embedded heartbeats. | ||
| HEARTBEAT | nonce of combined heartbeats | ||
| PAUSE | Pauses C2 agents | N/A | N/A |
| REPLICATE | Replicates an Agent's state to another agent; with standby true the replicant is paused and awaits restart. | agent | standby=true/false |
| REPLICATE | Tells agents to replicate state to nearby agents | server | |
RESTART | Restarts C2 agents | N/A | N/A |
| RESUME | Resumes C2 agents | N/A | N/A |
START | Starts components within the C2 agents | <name of component to start> | N/A |
| STOP | Stops components within the C2 agent | <name of component to stop> | N/A |
| SUBSCRIBE | Subscripts a C2 server to internal respondables ( Metrics , configuration, and policy/audit events ) . These will be placed into the heartbeat | enable/disable | subscribe=metrics, subscribe=configuration, subscribe=auditevents |
| TRANSFER | Transfers an object between the C2 agent and C2 designator. | N/A | N/A |
UPDATE | Update flow | configuration | location=<URL to updated flow file> |
| UPDATE | Update c2 properties | c2 | configkey1=configvalue1, configkey2=configvalue2 ... *configkey1 is a configuration option that is updated and its new value |
| UPDATE | Update configuration options defined within agent policies | ||
| UPDATE | Update agent | agent | location=<URL to agent binary or diff> partial=true/false ( optional) |
Operations and their operands for agents (Version 3)
| Operation Name | Description | operand/name | content/args |
|---|---|---|---|
ACKNOWLEDGE | Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation | N/A | |
| CLEAR | Clear repositories | repositories | N/A |
CLEAR | Clears the connection queues | connection | connection1=<connection name>, connection2=<connection 2> ... Will also accept a list <connection name1>,<connection name2>, ... |
DESCRIBE | Return metrics | metrics | metricsClass=<metric class to obtain> |
| DESCRIBE | configuration | N/A | N/A |
| DESCRIBE | manifest | N/A | N/A |
UPDATE | Update flow | configuration | location=<URL to updated flow file> |
| UPDATE | Update c2 properties | c2 | configkey1=configvalue1, configkey2=configvalue2 ... *configkey1 is a configuration option that is updated and its new value |
| UPDATE | Update agent | agent | location=<URL to agent binary or diff> |
RESTART | Restarts C2 agents | N/A | N/A |
START | Starts components within the C2 agents | <name of component to start> | N/A |
| STOP | Stops components within the C2 agent | <name of component to stop> | N/A |
| TRANSFER | Transfers an object between the C2 agent and C2 designator. | N/A | N/A |
Operations and their operands for agents (Version 2)
| Operation Name | Description | operand/name | content/args |
|---|---|---|---|
ACKNOWLEDGE | Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation | N/A | |
| CLEAR | Clear repositories | repositories | N/A |
CLEAR | Clears the connection queues | connection | connection1=<connection name>, connection2=<connection 2> ... Will also accept a list <connection name1>,<connection name2>, ... |
DESCRIBE | Return metrics | metrics | metricsClass=<metric class to obtain> |
| DESCRIBE | configuration | N/A | N/A |
UPDATE | Update flow | configuration | location=<URL to updated flow file> |
| UPDATE | Update c2 properties | c2 | configkey1=configvalue1, configkey2=configvalue2 ... *configkey1 is a configuration option that is updated and its new value |
| UPDATE | Update agent | agent | location=<URL to agent binary or diff> |
RESTART | Restarts C2 agents | N/A | N/A |
START | Starts components within the C2 agents | <name of component to start> | N/A |
| STOP | Stops components within the C2 agent | <name of component to stop> | N/A |
Operations and their operands for agents (Version 1)
| Operation Name | Description | operand/name | content/args |
|---|---|---|---|
UPDATE | Update flow | configuration | location=<URL to updated flow file> |
Future Work
Future architecture of C2 should be open to the discussion of distributed architectures and multiple heads ( i.e. in a client server multiple client/servers in the case where we can talk to geographically distributed agents ).
- GPS based C2 server – using location information to identify and locate C2 servers.
- C2 specific DNS responses. could we use DNS to simply provide us the closest C2 server?
- Multiple C2 servers providing separate keys to act as security arbiters.