You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Current »

Overview

In Apache Knox v1.6.0 the team added two new UIs that are directly accessible from the Knox Home page:

  • Token Generation
  • Token Management

By default, the homepage topology comes with the KNOXTOKEN service enabled with the following attributes:

  • token TTL is set to 120 days
  • token service is enabled (default to keystore-based token state service)
  • the admin user is allowed to renew/revoke tokens

In this topology, homepage, two new applications were added in order to display the above-listed UIs:

  • tokengen: this is an old-style JSP UI, with a relatively simple JS code included. The source is located in the gateway-applications Maven sub-module.
  • token-management: this is an Angular UI. The source is located in its own knox-token-management-ui Maven sub-module.

On the Knox Home page, you will see a new town in the General Proxy Information table like this:

However, the Integration Token links are disabled by default, because token integration requires a gateway-level alias - called knox.token.hash.key - being created and without that alias, it does not make sense to show those links.

Creating the token hash key

As explained, if you would like to use Knox's token generation features, you will have to create a gateway-level alias with a 256, 384, or 512-bit length JWK. You can do it in - at least - two different ways:

  1. You generate your own MAC (using this online tool for instance) and save it as an alias using Knox CLI.
  2. You do it running the following Knox CLI command:
    generate-jwk --saveAlias knox.token.hash.key

The second option involves a newly created Knox CLI command called generate-jwk:


Generates a JSON Web Key using the supplied algorithm name and prints the generated key value on the screen.
As an alternative to displaying this possibly sensitive information on the screen you may want to save it as an alias.

Options are as follows:
--jwkAlg (optional) defines the name of the desired JSON Web Signature algorithm name; defaults to HS256. Other accepted values are HS384 and HS512
--saveAlias (optional) if this is set, the given alias name is used to save the generated JWK instead of printing it on the screen
--topology (optional) the name of the topology (aka. cluster) to be used when saving the JWK as an alias. If none specified, the alias is going to be saved for the Gateway

Token state service implementations

There was an important step the Knox team made to provide more flexibility for our end-users: there are some internal service implementations in Knox that were hard-coded in the Java source code. One of those services is the Token State service implementation which you can change in gateway-site.xml going forward by setting the gateway.service.tokenstate.impl property to any of:

  1. org.apache.knox.gateway.services.token.impl.DefaultTokenStateService - keeps all token information in memory, therefore all of this information is lost when Knox is shut down
  2. org.apache.knox.gateway.services.token.impl.AliasBasedTokenStateService - token information is stored in the gateway credential store. This is a durable option, but not suitable for HA deployments
  3. org.apache.knox.gateway.services.token.impl.JournalBasedTokenStateService - token information is stored in plain files within $KNOX_DATA_DIR/security/token-state folder. This option also provides a durable persistence layer for tokens and it might be good for HA scenarios too (in case of KNOX_DATA_DIR is on a shared drive), but the token data is written out in plain text (i.e. not encrypted) so it's less secure.
  4. org.apache.knox.gateway.services.token.impl.ZookeeperTokenStateService - this is an extension of the keystore-based approach. In this case, token information is stored in Zookeeper using Knox aliases. The token's alias name equals to its generated token ID.
  5. org.apache.knox.gateway.services.token.impl.JDBCTokenStateService - stores token information in relational databases. It's not only durable, but it's perfectly fine with HA deployments. Currently, PostgreSQL and MySQL databases are supported.

By default, the AliasBasedTokenStateService implementation is used.

Configuring the JDBC token state service

If you want to use the newly implemented database token management, you’ve to set gateway.service.tokenstate.impl in gateway-site.xml to org.apache.knox.gateway.services.token.impl.JDBCTokenStateService.

Now, that you have configured your token state backend, you need to configure a valid database in gateway-site.xml. There are two ways to do that:

  1. You either declare database connection properties one-by-one:
    gateway.database.type - should be set to postgresql or mysql
    gateway.database.host - the host where your DB server is running
    gateway.database.port - the port that your DB server is listening on
    gateway.database.name - the name of the database you are connecting to
  2. Or you declare an all-in-one JDBC connection string called gateway.database.connection.url. The following value will show you how to connect to an SSL enabled PostgreSQL server:
    jdbc:postgresql://$myPostgresServerHost:5432/postgres?user=postgres&ssl=true&sslmode=verify-full&sslrootcert=/usr/local/var/postgresql@10/data/root.crt
jdbc:postgresql://$myPostgresServerHost:5432/postgres?user=postgres&ssl=true&sslmode=verify-full&sslrootcert=/usr/local/var/postgresql@10/data/root.crt

If your database requires user/password authentication, the following aliases must be saved into the Knox Gateway’s credential store (__gateway-credentials.jceks):

  • gateway_database_user - the username
  • gateway_database_password - the password

Database design

As you can see, there are only 2 tables:

  • KNOX_TOKENS contains basic information about the generated token
  • KNOX_TOKEN_METADATA contains an arbitrary number of metadata information for the generated token. At the time of this document being written the following metadata exist:
    • passcode - this is the BASE-64 encoded value of the generated passcode token MAC. That is, the BASE-64 decoded value is a generated MAC.
    • userName - the logged-in user who generated the token
    • enabled - this is a boolean flag indicating that the given token is enabled or not (a disabled token cannot be used for authentication purposes)
    • comment - this is optional metadata, saved only if the user enters something in the Comment input field on the Token Generation page (see below)

Generating a token

Once you configured the knox.token.hash.key alias and optionally customized your token state service, you are all set to generate Knox tokens using the new Token Generation UI:

The following sections are displayed on the page:

  • status bar: here you can see an informative message on the configured Token State backend. There are 3 different statuses:
    • ERROR: shown in red. This indicates a problem with the service backend which makes the feature not work. Usually, this is visible when end-users configure JDBC token state service, but they make a mistake in their DB settings
    • WARN: displayed in yellow (see above picture). This indicates that the feature is enabled and working, but there are some limitations
    • INFO: displayed in green. This indicates when the token management backend is properly configured for HA and production deployments
  • there is an information label explaining the purpose of the token generation page
  • comment: this is an optional input field that allows end-users to add meaningful comments (mnemonics) to their generated tokens. The maximum length is 255 characters.
  • the Configured maximum lifetime informs the clients about the knox.token.ttl property set in the homepage topology (defaults to 120 days). If that property is not set (e.g. someone removes it from he homepage topology), Knox uses a hard-coded value of 30 seconds (aka. default Knox token TTL)
  • Custom token lifetime can be set by adjusting the days/hours/minutes spinners. The default configuration will yield one hour.
  • Clicking the Generate Token button will try to create a token for you.

About the generated token TTL

Out of the box, Knox will display the custom lifetime spinners on the Token Generation page. However, they can be hidden by setting the knox.token.lifespan.input.enabled property to false in the homepage topology. Given that possibility and the configured maximum lifetime the generated token can have the following TTL value:

  • there is no configured token TTL and lifespan inputs are disabled -> the default TTL is used (30 seconds)
  • there is configured TTL and lifespan inputs are disabled -> the configured TTL is used
  • there is configured TTL and lifespan inputs are enabled and lifespan inputs result in a value that is less than or equal to the configured TTL -> the lifespan query param is used
  • there is configured TTL and lifespan inputs are enabled and lifespan inputs result in a value that is greater than the configured TTL -> the configured TTL is used

Successful token generation



On the resulting page there is two sensitive information that you can use in Knox to authenticate your request:

  1. JWT token - this is the serialized JWT and is fully compatible with the old-style Bearer authorization method. Clicking the JWT Token label on the page will copy the value into the clipboard. You might want to use it as the ‘Token’ user:

    $ curl -ku Token:eyJqa3UiOiJodHRwczpcL1wvbG9jYWxob3N0Ojg0NDNcL2dhdGV3YXlcL2hvbWVwYWdlXC9rbm94dG9rZW5cL2FwaVwvdjFcL2p3a3MuanNvbiIsImtpZCI6IkdsOTZfYTM2MTJCZWFsS2tURFRaOTZfVkVsLVhNRVRFRmZuNTRMQ1A2UDQiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImprdSI6Imh0dHBzOlwvXC9sb2NhbGhvc3Q6ODQ0M1wvZ2F0ZXdheVwvaG9tZXBhZ2VcL2tub3h0b2tlblwvYXBpXC92MVwvandrcy5qc29uIiwia2lkIjoiR2w5Nl9hMzYxMkJlYWxLa1REVFo5Nl9WRWwtWE1FVEVGZm41NExDUDZQNCIsImlzcyI6IktOT1hTU08iLCJleHAiOjE2MzY2MjU3MTAsIm1hbmFnZWQudG9rZW4iOiJ0cnVlIiwia25veC5pZCI6ImQxNjFjYWMxLWY5M2UtNDIyOS1hMGRkLTNhNzdhYjkxNDg3MSJ9.e_BNPf_G1iBrU0m3hul5VmmSbpw0w1pUAXl3czOcuxFOQ0Tki-Gq76fCBFUNdKt4QwLpNXxM321cH1TeMG4IhL-92QORSIZgRxY4OUtUgERzcU7-27VNYOzJbaRCjrx-Vb4bSriRJJDwbbXyAoEw_bjiP8EzFFJTPmGcctEzrOLWFk57cLO-2QLd2nbrNd4qmrRR6sEfP81Jg8UL-Ptp66vH_xalJJWuoyoNgGRmH8IMdLVwBgeLeVHiI7NmokuhO-vbctoEwV3Rt4pMpA0VSWGFN0MI4WtU0crjXXHg8U9xSZyOeyT3fMZBXctvBomhGlWaAvuT5AxQGyMMP3VLGw https:/localhost:8443/gateway/sandbox/webhdfs/v1?op=LISTSTATUS

{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16386,"group":"supergroup","length":0,"modificationTime":1621238405734,"owner":"hdfs","pathSuffix":"tmp","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1621238326078,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}

2. Passcode token - this is the serialized passcode token, which you can use as the ‘Passcode’ user (Clicking the Passcode Token label on the page will copy the value into the clipboard):

$ curl -ku Passcode:WkRFMk1XTmhZekV0WmprelpTMDBNakk1TFdFd1pHUXRNMkUzTjJGaU9URTBPRGN4OjpPVEV5Tm1KbFltUXROVEUyWkMwME9HSTBMVGd4TTJZdE1HRmxaalJrWlRVNFpXRTA= https://localhost:8443/gateway/sandbox/webhdfs/v1?op=LISTSTATUS

{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16386,"group":"supergroup","length":0,"modificationTime":1621238405734,"owner":"hdfs","pathSuffix":"tmp","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1621238326078,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}


The reason, we needed to support the shorter Passcode token, is that there are 3rd party tools where the long JWT exceeds input fields limitations so we need to address this issue with shorter token values.

The rest of the fields are complementary information such as the expiration date/time of the generated token or the user who created it.

Token generation failed

If there was an error during token generation, you will see a failure right under the input field boxes (above the Generate Token button):

The above error message indicates a failure that the admin user already generated more tokens than they are allowed to. This limitation is configurable in the gateway-site.xml:

gateway.knox.token.limit.per.user - indicates the maximum number of tokens a user can manage at the same time. -1 means that users are allowed to create/manage as many tokens as they want. This configuration only applies when the server-managed token state is enabled either in gateway-site or at the topology level. Defaults to 10.

Token Management

In addition to the token generation UI, Knox comes with a Token Management UI where logged-in users can see all the active tokens that they generated before. That is, if a token got expired and was removed from the underlying token store, it won't be displayed here.

On this page, you will see basic information about your generated token(s) and you can execute the following actions:

  1. Enable/Disable - based on the current status, you can temporarily enable/disable a token. Please note that disabled tokens are not allowed to use for authentication purposes.
  2. Revoke - you can remove the token from the persistent store. Please note this action cannot be undone, once you revoked a token Knox will delete it from the in-memory cache as well as the underlying persistent token storage

In order to refresh the table, you can use the Refresh icon above the table (if you generated tokens on another tab for instance).


  • No labels