You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »




Apache Knox is designed to be a unified authentication and access control solution for cluster services. It authenticates users at the cluster's edge and can integrate with various identity management systems for authentication. It also offers service-level authorization at the perimeter. 

Knox is typically set up on a virtual machine to secure access to cluster services. With the shift towards Kubernetes, this proposal suggests using Apache Knox as an external authorizer for services running in the Kubernetes cluster.

Problem Statement

Authentication and authorization in Kubernetes production deployments present several challenges and pain points, primarily due to the complexity and dynamic nature of Kubernetes environments.

Following are some of the pain points:

  • Kubernetes RBAC can be quite complex to set up and manage, especially in large or dynamic environments. It requires precise configuration of roles, role bindings, service accounts, and namespace-specific rules.
  • Integrating Kubernetes with external authentication systems (like LDAP, SAML, OAuth, etc.) can be complex and error-prone.
  • Efficiently monitoring and auditing "who did what" and "when" in the cluster is essential for security and compliance.
  • Kubernetes environments are often dynamic and scalable, which makes static access control strategies less effective.
  • Adapting authentication and authorization mechanisms to handle the dynamic nature of containerized applications and services is challenging.
  • Keeping track of user roles and permissions and regularly reviewing and updating them to match current requirements is tedious and error-prone.
  • Ensuring that former employees or inactive users do not retain access is a constant challenge.
  • In scenarios where multiple Kubernetes clusters are used, managing consistent authentication and authorization across clusters is complicated.

Proposed Solution

Apache Knox can be used to address these issues by acting as a centralized gateway for accessing services, which can simplify the management of authentication and authorization. It can serve as a single point of enforcement for security policies, making it easier to manage access in complex Kubernetes environments. 

Apache Knox supports various authentication mechanisms such as LDAP, Kerberos, SAML, and OAuth. This flexibility can make it easier to integrate Kubernetes with existing authentication systems, providing a consistent and secure authentication experience.

Knox can simplify the setup of RBAC in Kubernetes. Knox allows for more granular control over user access, helping enforce the principle of least privilege by restricting access to only what is necessary.

Assumption

                  Examples:

 Design Overview

Knox can be added as an external authorizer by following istio’s External Authorizer guide. The diagram above shows a high level design.   

  • User makes a request access a k8s service 
  • K8s ingress is configured to intercept requests outside the cluster 
  • Ingress controller will forward all unauthenticated requests to Knox for authentication/authorization
  • In case of failure, Knox will respond with 401 (Authentication failure)  or 403 (Authorization failure)
  • In case of success, Knox will forward the request to the intended k8s service
    • Optionally, Knox can add custom headers [TODO File JIRA]
    • Knox can forward configured headers to the downstream service (configured at ingress, this is a white list of headers) 

API Design

This section talks about API that are needed in Knox to support the described design.

auth/api/v1/pre

For Knox to work as an external authorizer, Knox needs to support Envoy ext_authz API. This is done by the endpoint `auth/api/v1/pre` as part of  Knox Auth Service. Knox Auth Service will forward authenticated and authorized requests to the intended k8s service. Based on the configured options (described here) Knox will populate the forwarded request with following headers:

  • Authenticated user principal 
  • Groups authenticated user is associated with
  • Filtering groups based on regex 


auth/api/v1/bearer

This REST API endpoint, when configured, populates the HTTP “Authorization” header with the Bearer Token in the forwarded request. Bearer Token is obtained from an environment variable configured in topology. The current implementation assumes that the token is not rotated as it never gets exposed to the end-user. By default, the BEARER_AUTH_TOKEN environment variable is expected to hold the Bearer token. This can be customized by configuring the `auth.bearer.token.env` service parameter to the desired value. This api is part of  Knox Auth Service. 


Path based authorization

This is an authorization configuration unlike an api endpoint. This configuration will be part of a new authorization provider.

Here, authorization to the downstream services will be done based on the request path. Just like AclsAuthz, authorization will be based on username, groups or ipaddress. This will allow Knox to enforce authorization on different endpoints in k8s cluster allowing for a more fine grained service level authorization.  

Example:

Path

user

group

ipaddress

/foo

user

*

*

/foo/**,/bar/**

user1

group1

*

/foo/*/bar

*

*

164.25.25.16


For authentication we can use the same flow used in previous usecases. 

Currently, Knox does not have a way to enforce path based authorization. We will need a new acls extension (similar to AclsAuthz) to support this functionality.  Following, is an example of how this might look.

 

<param>
       <name>path.KNOX-AUTH-SERVICE.acl</name>
       <value>/foo/* [, *|path...];username[,*|username...];group[,*|group...];ipaddr[,*|ipaddr...]</value>
</param>


This new extension (`path` in the above example) will work with CompositeAuthz and follow the same pattern as AclsAuthz provider. 

JIRA: https://issues.apache.org/jira/browse/KNOX-2998


Refer to the Implementation section for details on how to configure Knox as an external authorizer.


Use Cases

UC1: Authenticate and authorize JWT

This usecase allows external actors to make api calls with JWT as a bearer token. Knox will authenticate the JWT token. Knox can also be configured to look up groups that the authenticated user is part of. Authorization can be performed based on these groups. This allows dynamic authorization (group membership changes are picked up dynamically by Knox by talking to IdP) as opposed to static authorization where groups and users are enforced using static authorization policies. 

Knox will pass groups and principal as headers to the downstream services for more fine grained authorization. 

Authentication

Knox will Authenticate the JWT (issued by a trusted IdP or by Knox) based on configured JWKS url or by using the provided token verification PEM property (using authentication provider JWTProvider). If authentication is successful then authorization will be attempted (if configured) else 401 will be returned.

Hadoop Group Lookup Provider can optionally be configured to look up authenticated user’s group membership using any of the following options

  1. JniBasedUnixGroupsMappingWithFallback - Default implementation, uses shell to resolve a list of groups for a user.
  2. JniBasedUnixGroupsNetgroupMappingWithFallback
  3. ShellBasedUnixGroupsMapping
  4. ShellBasedUnixGroupsNetgroupMapping
  5. LdapGroupsMapping - Use LDAP to resolve the list of groups
  6. CompositeGroupsMapping

Authorization

Based on the principal and groups derived, authorization providers such as AclsAuthz can be used to authorize the request.

Knox Auth Service

Knox Auth Service implements the Envoy ext_authz check API and needs to be configured for Knox to act as an external authorizer. Once configured, Knox Auth Service will forward all authenticated and authorized requests to the intended services. This service will also populate authenticated subject and groups in request headers for downstream services to act on in case of fine grained authorization decisions.

Topology example is included in Appendix (UC1 Topology example)


UC3: SSO

For interactive browser access to k8s services KnoxSSO can be used. To enable SSO, ingress needs to check for knox-issued cookie (hadoop-jwt) if the cookie is not present then ingress will redirect the request to Knox SSO topology. This topology asserts the user's identity by issuing a redirect to a configured SAML provider. Upon success, Knox adds a signed JWT token and redirects the browser back to the original requested endpoint.




UC4: Websocket Applications

Initial POC testing with Tornado - Demo Websockets App suggests that UIs with websocket work with Knox as an external authorizer. This is needed for applications such as Zeppelin. 


UC5: gRPC

gRPC protocol is not yet supported.


UC6: Intracluster / Intercluster

Apache Knox can also be used for service to service or cluster to communication.

Intercluster (service to service)

For service to service authentication and authorization, services can rely on the headers that are added as part of initial Knox authentication. Headers include authenticated subject and groups. Services can use Istio’s Authorization Policy to check request headers for group info to enforce service level authorization. 

Intracluster (cluster to cluster)

For cluster to cluster communication Knox tokens can be used. For this to work both the clusters should have Knox configured and should either use the same signing keys or should expose JWT verification certs exposed as JWKS url to validate JWTs. 

UC6: Basic Authentication using userid and password

This usecase is very similar to UC1, instead of JWT for authentication userid and password will be used for authentication.


Comparison of services using Istio External Authorizer

Examples of services that can currently be configured as Istio external authorizers include:

  • Open Policy Agent (OPA): OPA is a general-purpose policy engine that unifies policy enforcement across the stack. It can be integrated with Istio for external authorization​​. OPA does not provide authentication and authorization capabilities like Knox but is a policy enforcement engine. 
  • oauth2-proxy: This is a reverse proxy that provides authentication with Google, Github, and other identity providers. It can be configured as an Istio external authorizer​​. Like the name suggests oauth2-proxy supports only OAuth and lacks other integrations such as LDAP.
  • Sample External Authorizers (sample-ext-authz-grpc and sample-ext-authz-http): These are sample authorizers provided in the Istio documentation. They demonstrate how to set up external authorization with Istio using either gRPC or HTTP APIs as defined by the Envoy ext_authz filter​​.


JIRAs


Implementation

Apache Knox will be installed in the same k8s cluster under a different namespace (recommendation only, not a requirement). 

Knox handles authentication and authorization by talking to an external identity provider / LDAP.

As there are too many configuration parameters a templated helm chart needs to be used for ease of deployment. This helm chart should have all the topologies needed. I am not sure if the helm chart should be part of the Knox repo or should be in the public helm repo. Keeping it alongside code will ensure that the chart will be in sync with the repo [TODO: file a jira].  


Istio configuration

We use istio external authorization model to configure Knox as an external authorizer. To achieve this we need Knox configured and running.

Label Knox pod

Knox pod needs to be labeled accordingly. The following example labels the namespace where Knox is installed (knox in this case).

kubectl label ns knox istio-injection=enabled


Edit config map

Next, update the istio config map to add the Knox as an external authorizer.

kubectl edit configmap istio -n istio-system


Add extensionProviders snippet

Update the config map with Knox specific settings. Things to note:

  • Add appropriate Knox service address (e.g. knox.knox.svc.cluster.local) 
  • Extension provider name should match the name in the policy file mentioned later (policy-knox-ext-authz)
  • Make sure Knox port is 8443 (unless changed)


trustDomain: cluster.local
### Add following .... remove comment
extensionProviders:
- name: ext-authz-knox
envoyExtAuthzHttp:
service: knox.knox.svc.cluster.local
port: 8443
includeRequestHeadersInCheck: ["Authorization", "authorization"]
pathPrefix: /gateway/cdpauth/auth/api/v1/extauthz
#### Until here .. remove comment
meshNetworks: 'networks: {}'
kind: ConfigMap


Add policy

Add knox authorization policy on `istio-ingressgateway` 

See appendix for details on Policy-knox-ext-authz.yaml

kubectl apply -f policy-knox-ext-authz.yaml


Appendix

Policy-knox-ext-authz.yaml


apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz-knox
  namespace: istio-system
spec:
  selector:
	matchLabels:
  	app: istio-ingressgateway
  	#app: knox
  action: CUSTOM
  provider:
	# The provider name must match the extension provider defined in the mesh config.
	# You can also replace this with sample-ext-authz-http to test the other external authorizer definition.
	name: ext-authz-knox
  rules:
  # The rules specify when to trigger the external authorizer.
  - to:
	- operation:
    	paths: ["/*"]


UC1 Topology example

<topology>
	<gateway>
    	<provider>
        	<role>federation</role>
        	<name>JWTProvider</name>
        	<enabled>true</enabled>
        	<param>
            	<name>knox.token.use.cookie</name>
            	<value>false</value>
        	</param>
        	<param>
            	<name>jwt.expected.sigalg</name>
            	<value>RS256</value>
        	</param>
        	<param>
            	<name>knox.token.jwks.url</name>
            	<value>https://example.com/jwks</value>
        	</param>
        	<param>
            	<name>jwt.expected.audience</name>
            	<value>AUD123</value>
        	</param>
        	<param>
            	<name>jwt.expected.issuer</name>
            	<value>ISSUER123</value>
        	</param>
    	</provider>
    	<provider>
        	<role>identity-assertion</role>
        	<name>HadoopGroupProvider</name>
        	<enabled>true</enabled>
        	<param>
            	<name>CENTRAL_GROUP_CONFIG_PREFIX</name>
            	<value>gateway.group.config.</value>
        	</param>
    	</provider>
    	<provider>
        	<role>authorization</role>
        	<name>AclsAuthz</name>
        	<enabled>true</enabled>
        	<param>
            	<!-- The "preauth" prefix must match the service role below (case insensitive) -->
            	<name>knox-auth-service.acl.mode</name>
            	<value>AND</value>
        	</param>
        	<param>
            	<!-- The "preauth" prefix must match the service role below (case insensitive) -->
            	<name>knox-auth-service.acl</name>
            	<value>*;group1;*</value>
        	</param>
    	</provider>
	</gateway>
     <service>
    		<role>KNOX-AUTH-SERVICE</role>
    	<param>
        	<name>preauth.auth.header.actor.id.name</name>
        	<value>x-knox-actor-username</value>
    	</param>
    	<param>
        	<name>preauth.auth.header.actor.groups.prefix</name>
        	<value>x-knox-actor-groups</value>
    	</param>
    	<param>
        	<name>preauth.group.filter.pattern</name>
        	<value>[^\s]+</value>
    	</param>
	</service>
</topology>
  • No labels