-
Table of Contents |
---|
C2 Protocol Introduction
...
Operation Name | Description | ||
---|---|---|---|
ACKNOWLEDGE | Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation | ||
CLEAR | Clears flow connection queues or repositories on the C2 agent | ||
CONSUME | Consumes a heartbeat with an agent to avoid sending from other agents. Allows multiplexing responses from a condensed agent response. | ||
DESCRIBE | Currently Unused | EXECUTE | Executes commands on the agent's operating system. This feature may be disabled for any agent. |
HEARTBEAT | Heartbeat provides status and operational capabilities to C2 server(s) | ||
PAUSE | Pauses the execution of flows on | UPDATE | Updates components of the C2 agent or the flow configuration. |
REPLICATE | Replicates agent state between agents, with the ability to place agents in standby mode until they are needed. | ||
RESTART | Restarts C2 agents | ||
RESUME | Resumes the execution of flows on the C2 agent | RESTART | Restarts C2 agents |
START | Starts components within the C2 agents | ||
STOP | Stops components within the C2 agent | SUBSCRIBE | Allows servers or agents to subscribe to an agent's heartbeat , requesting specific information for the next heartbeat. |
TRANSFER | Transfers an object between the C2 agent and C2 designator. | ||
UPDATE | Updates components of the C2 agent or the flow configuration. |
C2 Requirements
The requirements are an evolving list that have grown organically from an implementation. Any other portions of a heartbeat are considered optional.
...
Primary communications are carried over a C2 heartbeat. The heartbeat contains operational information about the C2 agent and can occur at a configurable frequency. The heartbeat provides status information to the C2 server. The response from the heartbeat contains requested operations from the C2 server. These operations are then acknowledged if/when they are completed. This means that the heartbeat is the only operation initiated by the C2 agent and the C2 server responds directly to these heartbeats. Version four of the heartbeat will allow a subscription model to be used for heartbeats to avoid sending unnecessary information. Though Describe can provide parts of the same information the aggregate produced for a heartbeat allows the agent to flush messaging queues to ensure subscribed heartbeats have the most up to date information. Heartbeats are intended to be lightweight structures with minimal information; however, the subscription model supports differing wire protocols and deployment scenarios that support larger payloads.
Protocols
HTTP/S Protocol
The HTTP/S protocol supports a url for heartbeating and acknowledging operations. These endpoints support the JSON structures defined below. C2 agents must send a heartbeat, defined above, to update the C2 server of its status and to receive operations. The frequency of these calls are up to the C2 agent to define.
CoAP Protocol
The CoAP protocol (https://coap.technology/) supports a constrained protocol for smaller devices. In the case of CoAP, the base requirements, as listed above, are fulfilled in each message. While the heartbeat structure, below, contains optional elements, the CoAP protocol implemented in Apache NiFi MiNiFi C++ contains minimal information per heartbeat.
...
Advanced Tables - JSON Table | ||||||
---|---|---|---|---|---|---|
| ||||||
{ "Components" : { "FlowControlleroperation" : "enabledheartbeat", "ProcessorName" : "enabled/disabled" }, "AgentInformation" deviceInfo": { "NetworkInfosystemInfo" : { "deviceid" : "stringcpuUtilization", "hostname" : "string: "device cpu usage since last heartbeat <float>", "ip" : "string", "flowid" : "string" }, "SystemInformation" : { "machinearch" : "stringmachine arch <string>", "physicalmem" "memoryUsage": "stringdevice memory usage in bytes <int>", "vcores" "operatingSystem": "string<string>" }, "AvailableClasses "physicalMem": [ "device memory in bytes {<int>", "vCores": "device vCores <int>" }, "propertiesnetworkInfo": [{ "hostname": "<string>", "ipAddress": "<string>" }, "propertyidentifier" : "device id <string>" }, "agentInfo": { ],"agentManifest": { "buildInfo": { "class_namecompiler": "org::apache::nifi::minifi::processors::AppendHostInfo<string>", "flags": "<string>", "supportsDynamicPropertiesrevision": "false<string>", "timestamp": "<int>", }, "version": "<string>" {}, "bundles": [ { "properties": [ "componentManifest": { "propertyprocessors": [ ],{ "class_namepropertyDescriptors": "org::apache::nifi::minifi::processors::AppendHostInfo",{ "supportsDynamicPropertiesProperty1": "false"{ } ], "BuildInformation": { "defaultValue": "<value>", "build_date": "date", "build_revdescription": "string<string>", "build_version "expressionLanguageScope": "string<string>", "compiler "name": "string<string>", "compiler_flags "required": "string<true/false>", "device_id "validator": "string" }, "Extensions" : [ "extension1", "extension2", "extensionn" ] }, "metrics" : { "ProcessMetrics" : { "CpuMetrics" : { "involcs" : "string" }, "MemoryMetrics" : { "maxrss" : "string" } }, "QueueMetrics" : { "Connection" : { "datasize" : "string", "datasizemax" : "string", "queued" : "string", "queuedmax" : "string" } }, "RepositoryMetrics" : { "flowfile" : { "full" : "1/0", "running" : "1/0", "size" : "string" }, "provenance" : { "full" : "1/0", "running" : "1/0", "size" : "string" } } }, "operation" : "heartbeat", "state" : { "running" : "true/false", "uptime" : "string" } } |
Responses to the heartbeats have the following structure
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "string",
"operationid": "string",
"name": "string",
"content" : [
{ "string" : "string" }
]
}
]
} |
Operation schemas
The following are the schema definitions for each operation that is contained within the requested operations of a heartbeat response. It is expected that C2 agents adhere to this structure
Clear
The clear operation uses name of connection or repositories to clear either the connections or the repositories. In the case of a connection the content contains the operation arguments, in which the value defines the connection name to clear.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "connection",
"content" : [
{ "unique_map_key" : "connection_name" }
]
}
]
} |
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "repositories",
}
]
} |
Update
Update allows the C2 server to update either the c2 agent or provide a URI from which we download the new flow configuration through a GET request.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "configuration",
"content" : [
{ "location" : "HTTP or HTTPS URL" }
]
}
]
} |
The following activity diagram depicts the flow of updating an agent from failure to success.
Gliffy Diagram | ||||
---|---|---|---|---|
|
Advanced Tables - JSON Table | ||||
---|---|---|---|---|
| ||||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "c2",
"content" : [
{ "option_name" : "option_value" }
]
}
]
} |
Start
Start starts a previously stopped command. If a start is called on a component that is already started, nothing should occur other than an acknowledgement. Name defines the component to start.
Advanced Tables - JSON Table | ||||||
---|---|---|---|---|---|---|
| ||||||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "start",
"operationid" : "string",
"name": "component name",
}
]
} |
...
Stop stops a component that is started. Components can be the FlowController, processors, or RPGs
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "stop",
"operationid" : "string",
"name": "component name",
}
]
} |
Transfer
Transfer will download an object from the C2 designator ( server or other location ). The location URI will be provided by the update JSON. This transfer will not be a JSON
response but will instead be the binary object. The hash of the object will be the acknowledgement ID for the transfer.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation": "acknowledge",
"operationid" : "hash of object"
} |
Restart
Attempts to restart the component defined within name
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "restart",
"operationid" : "string",
"name": "component name",
}
]
} |
Acknowledgements.
Acknowledgements occur through a separate URL. This URL will receive a POST that contains the following payload, which acknowledges that the operation ID was received and executed.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation": "acknowledge",
"operationid" : "string"
} |
...
MQTT can be used as a connecting protocol in lieu of a RESTFul Service. Additionally, MQTT can be used within an enclave and then as conversion to RESTFul to support MQTT → HTTP comms.
Operations and versioning
Apache NIFi MiNiFI C++ automatically supports versions 4, 3, and 2. Version one is not supported by the agent. Agents are backwards compatible and forward compatible.
C2 role delegation
Since C2 defines operations for agents and a conversion protocol, agents can be used for role delegation. This means that Agents can act as a C2 server. This design is intentional, and supports the eventual automation of capabilities. Version four
begins implementing facilities that will be used for autonomous control. Agents can control each other and execute commands based on defined policies. Policies are controller services and configurations that enable activities to be that allow
autonomous and near-autonomous control of subnets of agents.
C2 Policies
This sections defines the sub pages for C2 policies defined as controller services.
Operations and their operands for agents (Version 4* Not released)
...
ACKNOWLEDGE
...
CLEAR
...
connection1=<connection name>, connection2=<connection 2> ...
Will also accept a list
<connection name1>,<connection name2>, ...
...
DESCRIBE
...
UPDATE
...
configkey1=configvalue1, configkey2=configvalue2 ...
*configkey1 is a configuration option that is updated and its new value
...
location=<URL to agent binary or diff>
partial=true/false ( optional)
...
RESTART
...
START
...
Subscripts a C2 server to internal respondables ( Metrics , configuration, and policy/audit events ) .
These will be placed into the heartbeat
...
Operations and their operands for agents (Version 3)
...
ACKNOWLEDGE
...
CLEAR
...
connection1=<connection name>, connection2=<connection 2> ...
Will also accept a list
<connection name1>,<connection name2>, ...
}
},
"supportedRelationships": [
{
"description": "success",
"name": "success"
},
{
"description": "failure",
"name": "failure"
}
],
"inputRequirement": "<INPUT_FORBIDDEN/INPUT_ALLOWED/INPUT_REQUIRED>",
"isSingleThreaded": "<true/false>",
"supportsDynamicProperties": "<true/false>",
"supportsDynamicRelationships": "<true/false>",
"type": "Processor name <string>",
"typeDescription": "<string>"
}
],
"controllerServices": [
{
"propertyDescriptors": {
"Property1": {
"defaultValue": "<value>",
"description": "<string>",
"expressionLanguageScope": "<string>",
"name": "<string>",
"required": "<true/false>",
"validator": "string"
}
},
"supportsDynamicProperties": "<true/false>",
"supportsDynamicRelationships": "<true/false>",
"type": "ControllerService name <string>",
"typeDescription": "<string>"
}
]
},
"artifact": "artifact name <string>",
"group": "<string>",
"version": "<string>"
}
]
},
"status": {
"repositories": {
"ff": {
"size": "<int>",
"running": "<true/false>",
"full": "<true/false>"
},
"repo_name": {
"size": "<int>",
"running": "<true/false>",
"full": "<true/false>"
}
},
"components": {
"componentN": {
"running": "<true/false>",
"uuid": "<uuid string>"
}
},
"resourceConsumption": {
"cpuUtilization": "agent cpu usage since last heartbeat <float>",
"memoryUsage": "agent memory usage in bytes <int>"
},
"uptime": "<int>"
},
"agentClass": "<string>",
"agentManifestHash": "<string>",
"identifier": "agent id <string>"
},
"metrics": {
"QueueMetrics": {
"Connection": {
"datasize": "<string>",
"datasizemax": "<string>",
"queued": "<string>",
"queuedmax": "<string>"
}
},
"RepositoryMetrics": {
"flowfile": {
"full": "<true/false>",
"running": "<true/false>",
"size": "<string>"
},
"provenance": {
"full": "<true/false>",
"running": "<true/false>",
"size": "<string>"
}
}
}
}
|
Responses to the heartbeats have the following structure
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "string",
"operationid": "string",
"name": "string",
"content" : [
{ "string" : "string" }
]
}
]
} |
Operation schemas
The following are the schema definitions for each operation that is contained within the requested operations of a heartbeat response. It is expected that C2 agents adhere to this structure
Clear
The clear operation uses name of connection or repositories to clear either the connections or the repositories. In the case of a connection the content contains the operation arguments, in which the value defines the connection name to clear.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "connection",
"content" : [
{ "unique_map_key" : "connection_name" }
]
}
]
} |
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "repositories",
}
]
} |
Update
Update allows the C2 server to update either the c2 agent or provide a URI from which we download the new flow configuration through a GET request.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "configuration",
"content" : [
{ "location" : "HTTP or HTTPS URL" }
]
}
]
} |
The following activity diagram depicts the flow of updating an agent from failure to success.
Gliffy Diagram | ||||
---|---|---|---|---|
|
Advanced Tables - JSON Table | ||||
---|---|---|---|---|
| ||||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "c2",
"content" : [
{ "option_name" : "option_value" }
]
}
]
} |
Start
Start starts a previously stopped command. If a start is called on a component that is already started, nothing should occur other than an acknowledgement. Name defines the component to start.
Advanced Tables - JSON Table | ||||||
---|---|---|---|---|---|---|
| ||||||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "start",
"operationid" : "string",
"name": "component name",
}
]
} |
Stop
Stop stops a component that is started. Components can be the FlowController, processors, or RPGs
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "stop",
"operationid" : "string",
"name": "component name",
}
]
} |
Transfer
Transfer will download an object from the C2 designator ( server or other location ). The location URI will be provided by the update JSON. This transfer will not be a JSON
response but will instead be the binary object. The hash of the object will be the acknowledgement ID for the transfer.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation": "acknowledge",
"operationid" : "hash of object"
} |
Restart
Attempts to restart the component defined within name
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "restart",
"operationid" : "string",
"name": "component name",
}
]
} |
Pause
Pauses the execution of flows on the C2 agent (if the agent is running and is not in paused state), while the agent keeps running and heartbeating.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "pause",
"operationid" : "string",
"name": "component name",
}
]
} |
Resume
Resumes the execution of flows on the C2 agent if the agent is in paused state.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "resume",
"operationid" : "string",
"name": "component name",
}
]
} |
Acknowledgements.
Acknowledgements occur through a separate URL. This URL will receive a POST that contains the following payload, which acknowledges that the operation ID was received and executed.
Advanced Tables - JSON Table | ||
---|---|---|
| ||
{"operation": "acknowledge",
"operationid" : "string"
} |
MQTT Protocol
MQTT can be used as a connecting protocol in lieu of a RESTFul Service. Additionally, MQTT can be used within an enclave and then as conversion to RESTFul to support MQTT → HTTP comms.
Operations and versioning
Apache NIFi MiNiFI C++ automatically supports versions 4, 3, and 2. Version one is not supported by the agent. Agents are backwards compatible and forward compatible.
C2 role delegation
Since C2 defines operations for agents and a conversion protocol, agents can be used for role delegation. This means that Agents can act as a C2 server. This design is intentional, and supports the eventual automation of capabilities. Version four
begins implementing facilities that will be used for autonomous control. Agents can control each other and execute commands based on defined policies. Policies are controller services and configurations that enable activities to be that allow
autonomous and near-autonomous control of subnets of agents.
C2 Policies
This sections defines the sub pages for C2 policies defined as controller services.
Operations and their operands for agents
...
DESCRIBE
...
UPDATE
...
configkey1=configvalue1, configkey2=configvalue2 ...
*configkey1 is a configuration option that is updated and its new value
...
RESTART
...
START
...
Operation Name | Description | operand/name | content/args | ||
---|---|---|---|---|---|
ACKNOWLEDGE | Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation | N/A | |||
CLEAR | Clear repositories | repositories | N/A | ||
CLEAR | Clears the connection queues | connection | connection1=<connection name>, connection2=<connection 2> ... Will also accept a list <connection name1>,<connection name2>, ... | ||
CLEAR | Clears component state | corecomponentstate | corecomponent1=<component name>, corecomponent2=<component 2> ... | ||
DESCRIBE | Return metrics | metrics | metricsClass=<metric class to obtain> | ||
DESCRIBE | Return configuration options | configuration | N/A | ||
DESCRIBE | Return agent manifest | manifest | N/A | ||
DESCRIBE | Return backtraces from the state monitor | jstack | DESCRIBE | configuration | N/A |
DESCRIBE | Return all core component states | corecomponentstate | N/A | ||
UPDATE | Update flow | configuration | location=<URL to updated flow file> | ||
UPDATE | Update c2 properties | c2 | configkey1=configvalue1, configkey2=configvalue2 ... *configkey1 is a configuration option that is updated and its new value | ||
UPDATE | Update agent | agent | location=<URL to agent binary or diff> | ||
HEARTBEAT | heartbeat operation – may contain embedded heartbeats. | N/A | N/A | ||
PAUSE | Pauses C2 agents | N/A | N/A | ||
RESTART | Restarts C2 agents | N/A | N/A | ||
RESUME | Resumes | RESTART | Restarts C2 agents | N/A | N/A |
START | Starts components within the C2 agents | C2 FlowController <name of component to start> | N/A | ||
STOP | Stops components within the C2 agent | C2 FlowController <name of component to stop> | N/A | ||
TRANSFER | Transfers an object between Stops components within the C2 agent and C2 designator. | debug<name of component to stop> | N/A |
Operations and their operands for agents (Version 1)
UPDATE | Update flow | configuration | location=<URL to updated flow file> or configuration_data=<flow file yaml content> |
UPDATE | Update configuration property | properties | propertykey1=propertyvalue1, propertykey2=propertyvalue2 ... |
UPDATE | Download an asset | asset | file="filename.txt", url="/c2/asset/6c8052a7-93ec-42d2-aa78-69217e3385a7", forceDownload=false forceDownload: If true, existing files with the same name are overwritten. If false, existing files are not downloaded again. |
Operation Name | Description | operand/name | content/args |
---|---|---|---|
UPDATE | Update flow | configuration | location=<URL to updated flow file> |
Future Work
Future architecture of C2 should be open to the discussion of distributed architectures and multiple heads ( i.e. in a client server multiple client/servers in the case where we can talk to geographically distributed agents ).
...