-

C2 Protocol Introduction

Command and control, herein C2, consists of a C2 server and C2 agents. MiNIFi agents must adhere to the C2 protocols to have successful communications. C2 communications occur over a variety of protocols. Currently an HTTP/HTTPS RESTFul paradigm exists to support C2 capabilities to MiNiFi C2 agents. In the future additional protocols may become available for use. Note that when the phrasing "C2 designator" is used, this implies the C2 server, server, or agent that is designated as a responder to the C2 agent(s). All protocols must support the following operations:

...

C2 Requirements

The requirements are an evolving list that have grown organically from an implementation. Any other portions of a heartbeat are considered optional.

C2 Messages 

Heartbeats

   Primary communications are carried over a C2 heartbeat. The heartbeat contains operational information about the C2 agent and can occur at a configurable frequency. The heartbeat provides status information to the C2 server. The response from the heartbeat contains requested operations from the C2 server. These operations are then acknowledged if/when they are completed. This means that the heartbeat is the only operation initiated by the C2 agent and the C2 server responds directly to these heartbeats. Version four of the heartbeat will allow a subscription model to be used for heartbeats to avoid sending unnecessary information. Though Describe can provide parts of the same information the aggregate produced for a heartbeat allows the agent to flush messaging queues to ensure subscribed heartbeats have the most up to date information. Heartbeats are intended to be lightweight structures with minimal information; however, the subscription model supports differing wire protocols and deployment scenarios that support larger payloads.

Protocols

HTTP/S Protocol

  The HTTP/S protocol supports a url for heartbeating and acknowledging operations. These endpoints support the JSON structures defined below. C2 agents must send a heartbeat, defined above, to update the C2 server of its status and to receive operations. The frequency of these calls are up to the C2 agent to define. 

CoAP Protocol

  The CoAP protocol (https://coap.technology/) supports a constrained protocol for smaller devices. In the case of CoAP, the base requirements, as listed above, are fulfilled in each message. While the heartbeat structure, below, contains optional elements, the CoAP protocol implemented in Apache NiFi MiNiFi C++ contains minimal information per heartbeat.

Heartbeat structure

Heartbeats consist of a POST of the following Schema to the C2 heartbeat url. Metrics is a configurable list of metrics that can be returned, so the entirety of that object is optional. AvailableClasses in the structure below is optional.

...

{
"Components" : {
"FlowControlleroperation" : "enabledheartbeat",
"ProcessorName" : "enabled/disabled"
},
"AgentInformation" deviceInfo": {
    "NetworkInfosystemInfo" : {
"deviceid" :     "stringcpuUtilization",
"hostname" : "string: "device cpu usage since last heartbeat <float>",
"ip" : "string",
"flowid" : "string"
},
"SystemInformation" : {
"machinearch" : "stringmachine arch <string>",
"physicalmem"       "memoryUsage": "stringdevice memory usage in bytes <int>",
"vcores"       "operatingSystem": "string<string>"
},
"AvailableClasses      "physicalMem": [
"device memory in bytes {<int>",
      "vCores": "device vCores <int>"
    },
    "propertiesnetworkInfo": [{
      "hostname": "<string>",
      "ipAddress": "<string>"
    },
    "propertyidentifier"
: "device id <string>"
  },
  "agentInfo": {
        ],"agentManifest": {
      "buildInfo": {
         "class_namecompiler": "org::apache::nifi::minifi::processors::AppendHostInfo<string>",
        "flags": "<string>",
        "supportsDynamicPropertiesrevision": "false<string>",
        "timestamp": "<int>",
   },
     "version": "<string>"
      {},
      "bundles": [
        {
 "properties": [
        "componentManifest": {
            "propertyprocessors": [
                ],{
                "class_namepropertyDescriptors": "org::apache::nifi::minifi::processors::AppendHostInfo",{
                  "supportsDynamicPropertiesProperty1": "false"{
             }
  ],
"BuildInformation": {
    "defaultValue": "<value>",
       "build_date": "date",
            "build_revdescription": "string<string>",
             "build_version       "expressionLanguageScope": "string<string>",
            "compiler        "name": "string<string>",
            "compiler_flags        "required": "string<true/false>",
               "device_id     "validator": "string"
        },
"Extensions" : [
"extension1",
"extension2",
"extensionn"
]
},
"metrics" : {
"ProcessMetrics" : {
"CpuMetrics" : {
"involcs" : "string"
},
"MemoryMetrics" : {
"maxrss" : "string"
}
},
"QueueMetrics" : {
"Connection" : {
"datasize" : "string",
"datasizemax" : "string",
"queued" : "string",
"queuedmax" : "string"
}
},
"RepositoryMetrics" : {
"flowfile" : {
"full" : "1/0",
"running" : "1/0",
"size" : "string"
},
"provenance" : {
"full" : "1/0",
"running" : "1/0",
"size" : "string"
}
}
},
"operation" : "heartbeat",
"state" : {
"running" : "true/false",
"uptime" : "string"
}
}

Responses to the heartbeats have the following structure

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "string",
"operationid": "string",
"name": "string",
"content" : [
{ "string" : "string" }
]
}

]

}

Operation schemas

The following are the schema definitions for each operation that is contained within the requested operations of a heartbeat response. It is expected that C2 agents adhere to this structure

Clear

 The clear operation uses name of connection or repositories to clear either the connections or the repositories. In the case of a connection the content contains the operation arguments, in which the value defines the connection name to clear.

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "connection",
"content" : [
{ "unique_map_key" : "connection_name" }
]
}

]

}

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "repositories",
}

]

}

Update

Update allows the C2 server to update either the c2 agent or provide a URI from which we download the new flow configuration through a GET request. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "configuration",
"content" : [
{ "location" : "HTTP or HTTPS URL" }
]
}

]

}

The following activity diagram depicts the flow of updating an agent from failure to success. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "c2",
"content" : [
{ "option_name" : "option_value" }
]
}
]
}

Start

 Start starts a previously stopped command. If a start is called on a component that is already started, nothing should occur other than an acknowledgement. Name defines the component to start. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "start",
"operationid" : "string",
"name": "component name",
}

]

} 

...

 Stop stops a component that is started. Components can be the FlowController, processors, or RPGs

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "stop",
"operationid" : "string",
"name": "component name",
}

]

}

Transfer

 Transfer will download an object from the C2 designator ( server or other location ). The location URI will be provided by the update JSON. This transfer will not be a JSON

response but will instead be the binary object. The hash of the object will be the acknowledgement ID for the transfer. 

{"operation": "acknowledge",
"operationid" : "hash of object" 
}

Restart

  Attempts to restart the component defined within name

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "restart",
"operationid" : "string",
"name": "component name",
}

]

}

Acknowledgements.

 Acknowledgements occur through a separate URL. This URL will receive a POST that contains the following payload, which acknowledges that the operation ID was received and executed.

{"operation": "acknowledge",
"operationid" : "string" 
}

...

MQTT can be used as a connecting protocol in lieu of a RESTFul Service. Additionally, MQTT can be used within an enclave and then as conversion to RESTFul to support MQTT → HTTP comms. 

Operations and versioning

Apache NIFi MiNiFI C++ automatically supports versions 4, 3, and 2. Version one is not supported by the agent. Agents are backwards compatible and forward compatible. 

C2 role delegation

Since C2  defines operations for agents and a conversion protocol, agents can be used for role delegation. This means that Agents can act as a C2 server. This design is intentional, and supports the eventual automation of capabilities. Version four

begins implementing facilities that will be used for autonomous control. Agents can control each other and execute commands based on defined policies. Policies are controller services and configurations that enable activities to be that allow

autonomous and near-autonomous control of subnets of agents. 

C2 Policies

This sections defines the sub pages for C2 policies defined as controller services. 

C2 Failure Policies

C2 Update Policies

Operations and their operands for agents (Version 4* Not released)

...

ACKNOWLEDGE

...

CLEAR

...

connection1=<connection name>, connection2=<connection 2>  ...

Will also accept a list

<connection name1>,<connection name2>, ...

...

DESCRIBE

...

UPDATE

...

configkey1=configvalue1, configkey2=configvalue2 ...

*configkey1 is a configuration option that is updated and its new value

...

location=<URL to agent binary or diff>

partial=true/false ( optional)

...

RESTART

...

START

...

Subscripts a C2 server to internal respondables ( Metrics , configuration, and policy/audit events ) .

These will be placed into the heartbeat

...

Operations and their operands for agents (Version 3)

...

ACKNOWLEDGE

...

CLEAR

...

connection1=<connection name>, connection2=<connection 2>  ...

Will also accept a list

<connection name1>,<connection name2>, ...

          }
                },
                "supportedRelationships": [
                  {
                    "description": "success",
                    "name": "success"
                  },
                  {
                    "description": "failure",
                    "name": "failure"
                  }
                ],
                "inputRequirement": "<INPUT_FORBIDDEN/INPUT_ALLOWED/INPUT_REQUIRED>",
                "isSingleThreaded": "<true/false>",
                "supportsDynamicProperties": "<true/false>",
                "supportsDynamicRelationships": "<true/false>",
                "type": "Processor name <string>",
                "typeDescription": "<string>"
              }
            ],
            "controllerServices": [
              {
                "propertyDescriptors": {
                  "Property1": {
                    "defaultValue": "<value>",
                    "description": "<string>",
                    "expressionLanguageScope": "<string>",
                    "name": "<string>",
                    "required": "<true/false>",
                    "validator": "string"
                  }
                },
                "supportsDynamicProperties": "<true/false>",
                "supportsDynamicRelationships": "<true/false>",
                "type": "ControllerService name <string>",
                "typeDescription": "<string>"
              }
            ]
          },
          "artifact": "artifact name <string>",
          "group": "<string>",
          "version": "<string>"
        }
      ]
    },
    "status": {
      "repositories": {
        "ff": {
          "size": "<int>",
          "running": "<true/false>",
          "full": "<true/false>"
        },
        "repo_name": {
          "size": "<int>",
          "running": "<true/false>",
          "full": "<true/false>"
        }
      },
      "components": {
        "componentN": {
          "running": "<true/false>",
          "uuid": "<uuid string>"
        }
      },
      "resourceConsumption": {
        "cpuUtilization": "agent cpu usage since last heartbeat <float>",
        "memoryUsage": "agent memory usage in bytes <int>"
      },
      "uptime": "<int>"
    },
    "agentClass": "<string>",
    "agentManifestHash": "<string>",
    "identifier": "agent id <string>"
  },
  "metrics": {
    "QueueMetrics": {
      "Connection": {
        "datasize": "<string>",
        "datasizemax": "<string>",
        "queued": "<string>",
        "queuedmax": "<string>"
      }
    },
    "RepositoryMetrics": {
      "flowfile": {
        "full": "<true/false>",
        "running": "<true/false>",
        "size": "<string>"
      },
      "provenance": {
        "full": "<true/false>",
        "running": "<true/false>",
        "size": "<string>"
      }
    }
  }
}

Responses to the heartbeats have the following structure

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "string",
"operationid": "string",
"name": "string",
"content" : [
{ "string" : "string" }
]
}

]

}

Operation schemas

The following are the schema definitions for each operation that is contained within the requested operations of a heartbeat response. It is expected that C2 agents adhere to this structure

Clear

 The clear operation uses name of connection or repositories to clear either the connections or the repositories. In the case of a connection the content contains the operation arguments, in which the value defines the connection name to clear.

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "connection",
"content" : [
{ "unique_map_key" : "connection_name" }
]
}

]

}

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "repositories",
}

]

}

Update

Update allows the C2 server to update either the c2 agent or provide a URI from which we download the new flow configuration through a GET request. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "configuration",
"content" : [
{ "location" : "HTTP or HTTPS URL" }
]
}

]

}

The following activity diagram depicts the flow of updating an agent from failure to success. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "c2",
"content" : [
{ "option_name" : "option_value" }
]
}
]
}

Start

 Start starts a previously stopped command. If a start is called on a component that is already started, nothing should occur other than an acknowledgement. Name defines the component to start. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "start",
"operationid" : "string",
"name": "component name",
}

]

} 

Stop

 Stop stops a component that is started. Components can be the FlowController, processors, or RPGs

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "stop",
"operationid" : "string",
"name": "component name",
}

]

}

Transfer

 Transfer will download an object from the C2 designator ( server or other location ). The location URI will be provided by the update JSON. This transfer will not be a JSON

response but will instead be the binary object. The hash of the object will be the acknowledgement ID for the transfer. 

{"operation": "acknowledge",
"operationid" : "hash of object" 
}

Restart

  Attempts to restart the component defined within name

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "restart",
"operationid" : "string",
"name": "component name",
}

]

}

Pause

  Pauses the execution of flows on the C2 agent (if the agent is running and is not in paused state), while the agent keeps running and heartbeating.

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "pause",
"operationid" : "string",
"name": "component name",
}

]

}

Resume

  Resumes the execution of flows on the C2 agent if the agent is in paused state.

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "resume",
"operationid" : "string",
"name": "component name",
}

]

}

Acknowledgements.

 Acknowledgements occur through a separate URL. This URL will receive a POST that contains the following payload, which acknowledges that the operation ID was received and executed.

{"operation": "acknowledge",
"operationid" : "string" 
}

MQTT Protocol

MQTT can be used as a connecting protocol in lieu of a RESTFul Service. Additionally, MQTT can be used within an enclave and then as conversion to RESTFul to support MQTT → HTTP comms. 

Operations and versioning

Apache NIFi MiNiFI C++ automatically supports versions 4, 3, and 2. Version one is not supported by the agent. Agents are backwards compatible and forward compatible. 

C2 role delegation

Since C2  defines operations for agents and a conversion protocol, agents can be used for role delegation. This means that Agents can act as a C2 server. This design is intentional, and supports the eventual automation of capabilities. Version four

begins implementing facilities that will be used for autonomous control. Agents can control each other and execute commands based on defined policies. Policies are controller services and configurations that enable activities to be that allow

autonomous and near-autonomous control of subnets of agents. 

C2 Policies

This sections defines the sub pages for C2 policies defined as controller services. 

C2 Failure Policies

C2 Update Policies

Operations and their operands for agents

...

DESCRIBE

...

UPDATE

...

configkey1=configvalue1, configkey2=configvalue2 ...

*configkey1 is a configuration option that is updated and its new value

...

RESTART

...

START

...

Operation Name	Description	operand/name	content/args
ACKNOWLEDGE	Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation		N/A
CLEAR	Clear repositories	repositories	N/A
CLEAR	Clears the connection queues	connection	connection1=<connection name>, connection2=<connection 2>  ... Will also accept a list <connection name1>,<connection name2>, ...
CLEAR	Clears component state	corecomponentstate	corecomponent1=<component name>, corecomponent2=<component 2>  ...
DESCRIBE	Return metrics	metrics	metricsClass=<metric class to obtain>
DESCRIBE	Return configuration options	configuration	N/A
DESCRIBE	Return agent manifest	manifest	N/A
DESCRIBE	Return backtraces from the state monitor	jstack	DESCRIBE	configuration	N/A
DESCRIBE	Return all core component states	corecomponentstate	N/A
UPDATE	Update flow	configuration	location=<URL to updated flow file>
UPDATE	Update c2 properties	c2	configkey1=configvalue1, configkey2=configvalue2 ... *configkey1 is a configuration option that is updated and its new value
UPDATE	Update agent	agent	location=<URL to agent binary or diff>
HEARTBEAT	heartbeat operation – may contain embedded heartbeats.	N/A	N/A
PAUSE	Pauses C2 agents	N/A	N/A
RESTART	Restarts C2 agents	N/A	N/A
RESUME	Resumes	RESTART	Restarts C2 agents	N/A	N/A
START	Starts components within the C2 agents	C2 FlowController <name of component to start>	N/A
STOP	Stops components within the C2 agent	C2 FlowController <name of component to stop>	N/A
TRANSFER	Transfers an object between Stops components within the C2 agent and C2 designator.	debug<name of component to stop>	N/A

Operations and their operands for agents (Version 1)

Future Work

Future architecture of C2 should be open to the discussion of distributed architectures and multiple heads ( i.e. in a client server multiple client/servers in the case where we can talk to geographically distributed agents ). 

...