-

C2 Protocol Introduction

...

C2 Requirements

The requirements are an evolving list that have grown organically from an implementation. Any other portions of a heartbeat are considered optional.

...

  The HTTP/S protocol supports a url for heartbeating and acknowledging operations. These endpoints support the JSON structures defined below. C2 agents must send a heartbeat, defined above, to update the C2 server of its status and to receive operations. The frequency of these calls are up to the C2 agent to define. 

CoAP Protocol

  The CoAP protocol (https://coap.technology/) supports a constrained protocol for smaller devices. In the case of CoAP, the base requirements, as listed above, are fulfilled in each message. While the heartbeat structure, below, contains optional elements, the CoAP protocol implemented in Apache NiFi MiNiFi C++ contains minimal information per heartbeat.

...

{
  "operation"{
"Components" : {
"FlowController" : "enabledheartbeat",
"ProcessorName" : "enabled/disabled"
},
"AgentInformation" deviceInfo": {
    "NetworkInfosystemInfo" : {
"deviceid"       "cpuUtilization": "stringdevice cpu usage since last heartbeat <float>",
"hostname"       "machinearch": "stringmachine arch <string>",
"ip"       "memoryUsage": "stringdevice memory usage in bytes <int>",
"flowid"       "operatingSystem": "string<string>"
},
"SystemInformation"       "physicalMem": {
"machinearch" : "string",
"physicalmem" : "string",
"vcores" : "string"
},
"AvailableClasses": [
   "device memory in bytes <int>",
      "vCores": "device vCores <int>"
    },
    "networkInfo": {
      "hostname": "<string>",
         "propertiesipAddress": ["<string>"
    },
    "identifier": "device id <string>"
  },
  "agentInfo": {
    "propertyagentManifest": {
      "buildInfo": {
         ]"compiler": "<string>",
        "flags": "<string>",
        "class_namerevision": "org::apache::nifi::minifi::processors::AppendHostInfo<string>",
        "timestamp": "<int>",
        "supportsDynamicPropertiesversion": "false<string>"
            },
      "bundles": [
     {
    {
            "propertiescomponentManifest": [{
                    "property""processors": [
                ],{
                "class_namepropertyDescriptors": "org::apache::nifi::minifi::processors::AppendHostInfo",{
                  "supportsDynamicPropertiesProperty1": "false"{
             }
  ],
"BuildInformation     "defaultValue": {"<value>",
            "build_date        "description": "date<string>",
                    "build_revexpressionLanguageScope": "string<string>",
               "build_version     "name": "string<string>",
            "compiler        "required": "string<true/false>",
            "compiler_flags        "validator": "string",
            "device_id": "string"
      }
                },
"Extensions" : [
"extension1",
"extension2",
"extensionn"
]
},
"metrics" : {
"ProcessMetrics" : {
"CpuMetrics" : {
"involcs" : "string"
},
"MemoryMetrics" : {
"maxrss" : "string"
}
},
"QueueMetrics" : {
"Connection" : {
"datasize" : "string",
"datasizemax" : "string",
"queued" : "string",
"queuedmax" : "string"
}
},
"RepositoryMetrics" : {
"flowfile" : {
"full" : "1/0",
"running" : "1/0",
"size" : "string"
},
"provenance" : {
"full" : "1/0",
"running" : "1/0",
"size" : "string"
}
}
},
"operation" : "heartbeat",
"state" : {
"running" : "true/false",
"uptime" : "string"
}
}

Responses to the heartbeats have the following structure

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "string",
"operationid": "string",
"name": "string",
"content" : [
{ "string" : "string" }
]
}

]

}

Operation schemas

The following are the schema definitions for each operation that is contained within the requested operations of a heartbeat response. It is expected that C2 agents adhere to this structure

Clear

 The clear operation uses name of connection or repositories to clear either the connections or the repositories. In the case of a connection the content contains the operation arguments, in which the value defines the connection name to clear.

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "connection",
"content" : [
{ "unique_map_key" : "connection_name" }
]
}

]

}

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "repositories",
}

]

}

Update

Update allows the C2 server to update either the c2 agent or provide a URI from which we download the new flow configuration through a GET request. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "configuration",
"content" : [
{ "location" : "HTTP or HTTPS URL" }
]
}

]

}

The following activity diagram depicts the flow of updating an agent from failure to success. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "c2",
"content" : [
{ "option_name" : "option_value" }
]
}
]
}

Start

 Start starts a previously stopped command. If a start is called on a component that is already started, nothing should occur other than an acknowledgement. Name defines the component to start. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "start",
"operationid" : "string",
"name": "component name",
}

]

} 

...

 Stop stops a component that is started. Components can be the FlowController, processors, or RPGs

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "stop",
"operationid" : "string",
"name": "component name",
}

]

}

Transfer

 Transfer will download an object from the C2 designator ( server or other location ). The location URI will be provided by the update JSON. This transfer will not be a JSON

response but will instead be the binary object. The hash of the object will be the acknowledgement ID for the transfer. 

{"operation": "acknowledge",
"operationid" : "hash of object" 
}

Restart

  Attempts to restart the component defined within name

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "restart",
"operationid" : "string",
"name": "component name",
}

]

}

Acknowledgements.

 Acknowledgements occur through a separate URL. This URL will receive a POST that contains the following payload, which acknowledges that the operation ID was received and executed.

{"operation": "acknowledge",
"operationid" : "string" 
}

...

MQTT can be used as a connecting protocol in lieu of a RESTFul Service. Additionally, MQTT can be used within an enclave and then as conversion to RESTFul to support MQTT → HTTP comms. 

Operations and versioning

Apache NIFi MiNiFI C++ automatically supports versions 4, 3, and 2. Version one is not supported by the agent. Agents are backwards compatible and forward compatible. 

C2 role delegation

Since C2  defines operations for agents and a conversion protocol, agents can be used for role delegation. This means that Agents can act as a C2 server. This design is intentional, and supports the eventual automation of capabilities. Version four

begins implementing facilities that will be used for autonomous control. Agents can control each other and execute commands based on defined policies. Policies are controller services and configurations that enable activities to be that allow

autonomous and near-autonomous control of subnets of agents. 

C2 Policies

This sections defines the sub pages for C2 policies defined as controller services. 

C2 Failure Policies

C2 Update Policies

Operations and their operands for agents (Version 4* Not released)

...

ACKNOWLEDGE

...

CLEAR

...

connection1=<connection name>, connection2=<connection 2>  ...

Will also accept a list

<connection name1>,<connection name2>, ...

...

DESCRIBE

...

UPDATE

...

configkey1=configvalue1, configkey2=configvalue2 ...

*configkey1 is a configuration option that is updated and its new value

...

location=<URL to agent binary or diff>

partial=true/false ( optional)

...

RESTART

...

START

...

Subscripts a C2 server to internal respondables ( Metrics , configuration, and policy/audit events ) .

These will be placed into the heartbeat

...

Operations and their operands for agents (Version 3)

...

ACKNOWLEDGE

...

CLEAR

...

connection1=<connection name>, connection2=<connection 2>  ...

Will also accept a list

<connection name1>,<connection name2>, ...

                "supportedRelationships": [
                  {
                    "description": "success",
                    "name": "success"
                  },
                  {
                    "description": "failure",
                    "name": "failure"
                  }
                ],
                "inputRequirement": "<INPUT_FORBIDDEN/INPUT_ALLOWED/INPUT_REQUIRED>",
                "isSingleThreaded": "<true/false>",
                "supportsDynamicProperties": "<true/false>",
                "supportsDynamicRelationships": "<true/false>",
                "type": "Processor name <string>",
                "typeDescription": "<string>"
              }
            ],
            "controllerServices": [
              {
                "propertyDescriptors": {
                  "Property1": {
                    "defaultValue": "<value>",
                    "description": "<string>",
                    "expressionLanguageScope": "<string>",
                    "name": "<string>",
                    "required": "<true/false>",
                    "validator": "string"
                  }
                },
                "supportsDynamicProperties": "<true/false>",
                "supportsDynamicRelationships": "<true/false>",
                "type": "ControllerService name <string>",
                "typeDescription": "<string>"
              }
            ]
          },
          "artifact": "artifact name <string>",
          "group": "<string>",
          "version": "<string>"
        }
      ]
    },
    "status": {
      "repositories": {
        "ff": {
          "size": "<int>",
          "running": "<true/false>",
          "full": "<true/false>"
        },
        "repo_name": {
          "size": "<int>",
          "running": "<true/false>",
          "full": "<true/false>"
        }
      },
      "components": {
        "componentN": {
          "running": "<true/false>",
          "uuid": "<uuid string>"
        }
      },
      "resourceConsumption": {
        "cpuUtilization": "agent cpu usage since last heartbeat <float>",
        "memoryUsage": "agent memory usage in bytes <int>"
      },
      "uptime": "<int>"
    },
    "agentClass": "<string>",
    "agentManifestHash": "<string>",
    "identifier": "agent id <string>"
  },
  "metrics": {
    "QueueMetrics": {
      "Connection": {
        "datasize": "<string>",
        "datasizemax": "<string>",
        "queued": "<string>",
        "queuedmax": "<string>"
      }
    },
    "RepositoryMetrics": {
      "flowfile": {
        "full": "<true/false>",
        "running": "<true/false>",
        "size": "<string>"
      },
      "provenance": {
        "full": "<true/false>",
        "running": "<true/false>",
        "size": "<string>"
      }
    }
  }
}

Responses to the heartbeats have the following structure

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "string",
"operationid": "string",
"name": "string",
"content" : [
{ "string" : "string" }
]
}

]

}

Operation schemas

The following are the schema definitions for each operation that is contained within the requested operations of a heartbeat response. It is expected that C2 agents adhere to this structure

Clear

 The clear operation uses name of connection or repositories to clear either the connections or the repositories. In the case of a connection the content contains the operation arguments, in which the value defines the connection name to clear.

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "connection",
"content" : [
{ "unique_map_key" : "connection_name" }
]
}

]

}

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "clear",
"operationid" : "string",
"name": "repositories",
}

]

}

Update

Update allows the C2 server to update either the c2 agent or provide a URI from which we download the new flow configuration through a GET request. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "configuration",
"content" : [
{ "location" : "HTTP or HTTPS URL" }
]
}

]

}

The following activity diagram depicts the flow of updating an agent from failure to success. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "update",
"operationid" : "string",
"name": "c2",
"content" : [
{ "option_name" : "option_value" }
]
}
]
}

Start

 Start starts a previously stopped command. If a start is called on a component that is already started, nothing should occur other than an acknowledgement. Name defines the component to start. 

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "start",
"operationid" : "string",
"name": "component name",
}

]

} 

Stop

 Stop stops a component that is started. Components can be the FlowController, processors, or RPGs

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "stop",
"operationid" : "string",
"name": "component name",
}

]

}

Transfer

 Transfer will download an object from the C2 designator ( server or other location ). The location URI will be provided by the update JSON. This transfer will not be a JSON

response but will instead be the binary object. The hash of the object will be the acknowledgement ID for the transfer. 

{"operation": "acknowledge",
"operationid" : "hash of object" 
}

Restart

  Attempts to restart the component defined within name

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "restart",
"operationid" : "string",
"name": "component name",
}

]

}

Pause

  Pauses the execution of flows on the C2 agent (if the agent is running and is not in paused state), while the agent keeps running and heartbeating.

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "pause",
"operationid" : "string",
"name": "component name",
}

]

}

Resume

  Resumes the execution of flows on the C2 agent if the agent is in paused state.

{"operation" : "heartbeat",
"requested_operations": [ {
"operation" : "resume",
"operationid" : "string",
"name": "component name",
}

]

}

Acknowledgements.

 Acknowledgements occur through a separate URL. This URL will receive a POST that contains the following payload, which acknowledges that the operation ID was received and executed.

{"operation": "acknowledge",
"operationid" : "string" 
}

MQTT Protocol

MQTT can be used as a connecting protocol in lieu of a RESTFul Service. Additionally, MQTT can be used within an enclave and then as conversion to RESTFul to support MQTT → HTTP comms. 

Operations and versioning

Apache NIFi MiNiFI C++ automatically supports versions 4, 3, and 2. Version one is not supported by the agent. Agents are backwards compatible and forward compatible. 

C2 role delegation

Since C2  defines operations for agents and a conversion protocol, agents can be used for role delegation. This means that Agents can act as a C2 server. This design is intentional, and supports the eventual automation of capabilities. Version four

begins implementing facilities that will be used for autonomous control. Agents can control each other and execute commands based on defined policies. Policies are controller services and configurations that enable activities to be that allow

autonomous and near-autonomous control of subnets of agents. 

C2 Policies

This sections defines the sub pages for C2 policies defined as controller services. 

C2 Failure Policies

C2 Update Policies

Operations and their operands for agents

...

DESCRIBE

...

UPDATE

...

configkey1=configvalue1, configkey2=configvalue2 ...

*configkey1 is a configuration option that is updated and its new value

...

RESTART

...

START

...

Operation Name	Description	operand/name	content/args
ACKNOWLEDGE	Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation		N/A
CLEAR	Clear repositories	repositories	N/A
CLEAR	Clears the connection queues	connection	connection1=<connection name>, connection2=<connection 2>  ... Will also accept a list <connection name1>,<connection name2>, ...
CLEAR	Clears component state	corecomponentstate	corecomponent1=<component name>, corecomponent2=<component 2>  ...
DESCRIBE	Return metrics	metrics	metricsClass=<metric class to obtain>
DESCRIBE	Return configuration options	configuration	N/A
DESCRIBE	Return agent manifest	manifest	N/A
DESCRIBE	Return backtraces from the state monitor	jstack	DESCRIBE	configuration	N/A
DESCRIBE	Return all core component states	corecomponentstate	N/A
UPDATE	Update flow	configuration	location=<URL to updated flow file>
UPDATE	Update c2 properties	c2	configkey1=configvalue1, configkey2=configvalue2 ... *configkey1 is a configuration option that is updated and its new value
UPDATE	Update agent	agent	location=<URL to agent binary or diff>
HEARTBEAT	heartbeat operation – may contain embedded heartbeats.	N/A	N/A
PAUSE	Pauses C2 agents	N/A	N/A
RESTART	Restarts C2 agents	N/A	N/A
RESUME	Resumes	RESTART	Restarts C2 agents	N/A	N/A
START	Starts components within the C2 agents	C2 FlowController <name of component to start>	N/A
STOP	Stops components within the C2 agent	C2 FlowController <name of component to stop>	N/A
TRANSFER	Transfers an object between Stops components within the C2 agent and C2 designator.	debug<name of component to stop>	N/A

Operations and their operands for agents (Version 1)

Future Work

Future architecture of C2 should be open to the discussion of distributed architectures and multiple heads ( i.e. in a client server multiple client/servers in the case where we can talk to geographically distributed agents ). 

...