THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Wire protocol of the headers bytes (if above mentioned attributes bit flag is true)
The below is for the headers wire protocol.
- The headers to be ordered by the int key (ascending)
- Benefits are more prevalent on the server side as it allows faster access to lower numbered keys which can be reserved for server side set/accessed headers.
- Examples of such server side cases that would benefit:
- clusterId tagging the message since KIP-78 introduced clusterId this would be useful in message tracing and routing.
- Server side auditing (http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-meetup-january-27th-2015)
- Examples of such server side cases that would benefit:
- Benefits are more prevalent on the server side as it allows faster access to lower numbered keys which can be reserved for server side set/accessed headers.
| Code Block | ||
|---|---|---|
| ||
Headers (bytes) => Count Set(Key, ValueLength, Value) Count => int32 <---------------------- NEW Number of headers Set => Key => int32 <---------------------- NEW int key of the header ValueLength => int32 <-------------- NEW size of the byte[] of the serialised header value Value => bytes <-------------------- NEW serialised form of the header value |
Example Key Allocation
As mentioned above ranges of keys would will be reserved for different usages, below is an example way this could be managed.
Kafka open source (0x000)
...
Testing (0x003)
- 0x00300000-0x0030FFFF Testing - Infrastructure
- 0x00310000-0x0031FFFF Testing - Client
- 0x00320000-0x0032FFFF Testing - Stack/Middleware
- 0x00330000-0x0033FFFF Testing - Application
- 0x00340000-0x0034FFFF Testing - User
- 0x00350000-0x003FFFFF RESERVED
Reserved
- 0x00400000-0xFFFFFFFF RESERVED
From this perspective we would could have the following:
- A kafka header used in the kafka open source distribution could have the value of 0x00000010 (10).
- A header used internally at organisation X would fall under Local. If it was something used by infrastructure, like tracing, that could use 0x00100010 (1048592)
- A header used by an open source project somebody is working on, that they just want to put out there (without coordination) could start using a value of 0x00220010 (2228240) if it was a plugin for annotating the location on a message.
- An application that was testing whether it can use headers for a new feature it's developing could pick a header with key 0x0034010 (3407888).
- The Kafka open source number space is coordinated via the Apache Kafka opensource project. A class would list all possible headers, their numbers and their string equivalents (for output/logging/etc).
- A Local organisation is in charge of coordinating it's local number space. It would be in charge of writing a text file, a class or a service to coordinate who gets what numbers when.
- In the open internet you can grab any number in the Open space but should expect no guarantees that other people may or may not be using that number.
- When you're doing testing you can safely grab a number in the testing space and be assured that you won't collide with an official system header. It's still possible to collide with other testing headers, but no production system should depend on these headers.
Sample register that would end up having in the open source space.
| key | name | description | by | url |
|---|---|---|---|---|
| 1 | client.id | producers client id | Apache Kafka | some url to a document about it |
| 2 | cluster.id | cluster id of where the message first originated | Apache Kafka | some url to a document about it |
| 3 | correlation.id | correlation id for where a message is for mutex response from a request | Apache Kafka | some url to a document about it |
| 2100001 | new.relic | stores the transaction linking guid for transaction sticking by new relic | New Relic | some url to a document about it |
| 2100002 | appdynamics | stores the transaction linking guid for transaction stiching by app dynamics | AppDynamics | some url to a document about it |
...
| Code Block |
|---|
package org.apache.kafka.common.config;
public class KafkaHeaderKeys
{
public static final int CLIENT_ID_KEY = 1;
public static final int CLUSTER_ID_KEY = 2;
public static final int CORRELATION_ID_KEY = 3;
} |
Sample register that would end up having local (In-house) custom per organisation
| key | name | description | by | notes |
|---|---|---|---|---|
| 1114000 | app.name | ig's unique app name for the producer | IG | some internal document about it |
| 1050000 | charge.tag | tag to make per message chargebacks for | IG | some internal document about it |
...