THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Table of Contents |
|---|
Status
Current state: Under Accepted
Discussion thread: here
Discussion Vote thread: here
JIRA: here
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
...
Kafka users have reported that their security scanners have identified some missing HTTP headers from Connect REST API response. Specific headers that customers have asked about include:
...
...
...
...
We should provide a mechanism to return the desired headers. We can't, in general, anticipate what specific headers their scanners will demand now, much less in the future, so we must make this mechanism configurable. Also we don't foresee the requirement to set different headers for different paths or mime types (since Connect API only return application/json).
Public Interfaces
This proposal adds a new configuration property to customize HTTP response headers. The following section has detailed description.
response.http.headers.config
Type: Comma separated string
Valid Values: Values must use same format as headerConfig defined in Jetty's HeaderFilter documentation:
| Code Block |
|---|
[action] [header name]: [header value] |
[action] can be one of
Public Interfaces
There are no changes to existing public interfaces, but we will add new configuration options. We follow same pattern like configuring different Kafka listeners. We define a new property response.http.headers which defines the names of the header filters that will be configured. The default value for response.http.headers is empty string which means there are no header filters configured. We define a new prefix "response.http.headers.{name}.", then followed by a set of properties which define rules for header. The {name} will be one defined in response.http.headers. The following section has detailed description.
Proposed Changes
Adding Properties
We will add a set of new properties in the org.apache.kafka.connect.runtime.WorkerConfig class. It will allow the REST server administrator to configure headers based on their security policies. We borrow and take advantage of the Jetty HeaderFilter class and use the same format of headerConfig, includedPaths, excludedPaths, includedMimeTypes, excludedMimeTypes, includedHttpMethods, and excludedHttpMethods init parameters as HeaderFilter. Please see two references for Jetty HeaderFilter.
https://www.eclipse.org/jetty/documentation/current/header-filter.html
https://www.eclipse.org/jetty/javadoc/9.4.24.v20191120/org/eclipse/jetty/servlets/HeaderFilter.html
Description of Properties
...
Defines names of header filters which will be separated by comma.
The name could be alphanumeric string which uniquely identify header.
Valid Values: alphanumeric string
...
Define a set of HTTP headers for the header filter defined by {name} which will be one of names defined in property response.http.headers.
Valid Values: See Detailed Explanation section.
...
It is a comma separated values of included path specs applied to HTTP headers.
Valid Values: See path spec rules section.
...
/connectors/connector1/topics/*...
It is a comma separated values of excluded path specs applied to HTTP headers.
Valid Values: See path spec rules section.
...
It is a comma separated values of included mime types applied to HTTP headers
Valid Values: see https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types
...
It is a comma separated values of excluded mime types applied to HTTP headers.
Valid Values: see https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types
...
Detailed Explanation for response.http.headers.{name}.header.config
The format will be "[[action] [header]:[header value],..." which is a list of [action] [header]:[value] separated by comma ",". So it is a CSV of actions to perform on headers with the following syntax:
[action] [header name]: [header value],
[action] can be one of "set, add, setDate, or addDate" which which specify an action to will perform on the header.
setaction is the same as the setHeader function in HttpServletResponse, it will set a response header with the given name and value. If the header had already been set, the new value overwrites the previous one.addaction is the same as the addHeader function in HttpServletResponse, it will add a new value to the header. Response headers can Responses headers could have multiple values.setDateaction is the same as thesetDateHeadersame as setDateHeader function in HttpServletResponse. It will set a HTTP header with a need date value. Such as "setDate Expires: 31540000000" which which indicates the header will be expired approximately one year in the future.addDateaction is the same as theaddDateHeaderaddDateHeader function in HttpServletResponse. It will add a response header with the given name and date-value. Such as "addDate Last-Modified: 0" which indicates the Last-Modified date is same as current system date.
[header name] specify name of header (For examples, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers)
[header value] specify value for the header. We need to put double quotes around the value if the value contains commas because due to we use comma as separator for different headers.
Path Spec Rules:
- If the spec starts with
^, the spec is assumed to be a regex based path spec and will match with normal Java regex rules. - If the spec starts with
/, the spec is assumed to be a Servlet url-pattern rules path spec for either an exact match or prefix based match. - If the spec starts with
*., the spec is assumed to be a Servlet url-pattern rules path spec for a suffix based match. - All other syntaxes are unsupported.
Multiple Headers Configuration
We use Jetty HeaderFilter to implement HTTP response header configuration. We need support different rules applied to different response headers based on different resources. For most usage cases, one header should be sufficient. But we do provide flexible configuration for multiple headers usages.
We define a prefix response.http.headers.{name}. to allow configure multiple configurations for different resources. The "{name}", for instance "connector1" in following example, will be used to uniquely identify a set of HTTP response headers applied to one resource under /connectors/connector1
Example:
...
Default Value: ""
Example:
|
...
|
...
|
...
|
...
|
...
|
...
|
Output of Response Header:
|
Proposed Changes
Since we're using Jetty to serve this, we can take advantage of the Jetty HeaderFilter class to allow the configuration of these headers. We add configuration property response.http.headers.config in the org.apache.kafka.connect.runtime.WorkerConfig class and
Implementation
Implementation will use the Jetty HeaderFilter class. We need to update org.apache.kafka.connect.runtime.rest.RestServer class. During During initialization process, the Connect REST server will check property response.http.headers. If the value of response.http.headers is not empty, then Connect REST server will read all headers header configurations from the properties with prefix property response.http.headers.{name}config, and create a list of FilterHolder with HeaderFilter class and add the list of filter holders holder to the Servlet context handler based on the name of the header. Implementation is similar to how we handle the header access.control.allow.origin in the Connect REST server.
Pseudocode
...
. We only need to provide a single HeaderFilter for the entire server, because that HeaderFilter can set as many headers as the customer needs. Both set and add action will add a header and value to the response header if the header is not already in the response. When the header is there, set action overwrites the existing value, whereas add action adds an additional value to the header value. So it is the Kafka user's applications and server administrator responsibility to manage headers configured and existing headers.
The following is flow how this will be implemented in RestServer class:
- loads configuration file
- parses the property
response.http.headers.configin configuration file sing WorkerConfig - creates FilterHolder with HeadFilter class.
- adds the FilterHolder into ServletContextHandler
Pseudocode
private void configureHttpResponsHeaderFilter(ServletContextHandler context) |
...
|
...
Resources
Referenceshttps://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
https://www.eclipse.org/jetty/documentation/current/header-filter.html
https://www.eclipse.org/jetty/javadoc/9.4.24.v20191120/org/eclipse/jetty/servlets/HeaderFilter.html
Compatibility, Deprecation, and Migration Plan
Since we just add a new property and the default value for new property is empty string, existing use cases and behavior will be unaffectedThis is a new feature that add HTTP headers based on configuration property response.http.headers.config. The default value for response.http.headers.config is empty, so it is backward compatible to old version.
Rejected Alternatives
Another implementation would be writing a customized filter extension to intercept and set HTTP response headers and we have to define same configuration property described in this proposal. Ultimately the purpose of this KIP design will allow users to set HTTP response headers, using . Using this alternative approach make implementation much complex and doesn't gain any benefits.
Supporting multiple header filters add complexity to configuration properties and cause site administrator confusions on configuring Connect REST Server. Most important thing is we do not see applicable customer scenarios at this time. One header filter is sufficient for all HTTP response headers based on existing customer application scenarios and concern. We could expand existing implementation easily if customer need it in the future. For instance, we could add multiple header filters by adding name of header filter between prefix response.http.headers and config such as response.http.headers.{name}.config. We still keep response.http.headers.config as default header filter and internally set name of header filter to default if there is nothing between the response.http.headers and config.
Supporting other optional parameters, that are response.http.headers.included.paths, response.http.headers.included.mime.types, response.http.headers.included.methods, response.http.headers.excluded.paths, response.http.headers.excluded.mime.types, response.http.headers.excluded.methods, don't add value based on existing customer complain about missing header in HTTP response headers. Customer will be happy as long as the headers they want in HTTP response header. This will simply configuration properties lot. We could easily add these optional parameters if customer need it in the future.
Another option would be to try to determine what the right headers are in all cases, and always send them. This is appealing because it would not require any end-user customization. However new security headers are regularly added by the web development community, and it would be difficult for us to anticipate all of our users' needs. By providing a configurable option, users can implement the headers that make sense in their own security environments.