You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

(This document is still Work In Progress!)

Performance test tool design

The general goal is to have an extendable performance test framework that will drive the Knox Gateway configured by different use cases.

Some initial requirements of this framework:

  • easy to extend with new use-cases
  • has the ability to execute long-running jobs
  • has the ability to connect to it and query different metrics
  • generate meaningful and easy-to-read reports of those metrics automatically (in configurable time periods)

In the first phase, the task was to conduct and codify performance testing for scalability and performance benchmarking with concurrent clients, with long-running jobs that stress the backend of the token state server store:

  • use different token state server implementations
  • turn on/off the token state service mechanism
  • use concurrent clients who dealing with tokens (make sure they actually use the tokens and periodically renew them)

The tool itself is fed by its own configuration file located in $YOUR_KNOX_PROJECT_ROOT/gateway-performance-test/src/test/resources/performance.test.configuration.properties:

# Gateway connection related properties
perf.test.gateway.url.protocol=https
perf.test.gateway.url.host=localhost
perf.test.gateway.url.port=8443
perf.test.gateway.jmx.port=8888

# report generation related properties
perf.test.report.generation.periodInSecs=30
perf.test.report.generation.json.enabled=true
perf.test.report.generation.yaml.enabled=true

# Knox Token use case related properties
perf.test.usecase.knoxtoken.enabled=true
perf.test.usecase.knoxtoken.topology.gateway=sandbox
perf.test.usecase.knoxtoken.topology.tokenbased=tokenbased
perf.test.usecase.knoxtoken.numOfThreads=3
perf.test.usecase.knoxtoken.testDurationInSecs=60
perf.test.usecase.knoxtoken.requestDelayLowerBoundInSecs=5
perf.test.usecase.knoxtoken.requestDelayUpperBoundInSecs=10

As of today (17 Aug 2020), there is only one use-case implementation exists to address the above-written acquire/renew/use Knox Delegation token case. The related resources (Java classes, properties files) are located in the gateway-performance-test Maven module. Here is the list of the most relevant resources:

  • src/main/java
    • o.a.k.g.p.t.PerformanceTestRunner - this is the entry point of the tool. This class comes with a main method that reads the given configuration file and executes all enabled use-case runners
    • o.a.k.g.p.t.ResponseTimeCache - this class acts as a holder of response times and shared between the worker threads (which write into the cache) and the report generation threads (reading data from it)
    • o.a.k.g.p.t.reporting.GatewayMetricsReporter - this class generates the human-readable reports in JSON and YAML format in a fixed schedule marked by perf.test.report.generation.periodInSecs 
    • o.a.k.g.p.t.knoxtoken.KnoxTokenUseCaseRunner - this class is responsible for
      • start N worker threads that are acquiring Knox DTs parallel (marked by perf.test.usecase.knoxtoken.numOfThreads)
      • and 2 more threads to
        • renew an already acquired Knox DT
        • do an HDFS ls command using an already acquired Knox DT

    • o.a.k.g.p.t.knoxtoken.KnoxTokenWorkerThread - this represents the job which actually acquires/renews/uses Knox DTs

    • o.a.k.g.p.t.knoxtoken.KnoxTokenCache - stores the already acquired Knox DTs (if the number of DTs reaches 500 the cache is cleaned automatically)
  • src/test/resources
    • performance.test.configuration.properties - contains the above-described configuration file
    • performanceTest-log4j.properties - the Log4j configuration of the tool. By default, it prints log messages on the STDOUT as well as writes them into target/logs/performanceTest.log


Knox gateway requisites

The performance test tool tries to connect to a Knox instance to

  • acquire/renew/use Knox delegation tokens using its token API (/knoxtoken/api/v1/token)
  • fetch useful metrics via JMX

To be able to execute the last item, before running the Knox gateway you are testing against, the following configuration should be done:

  1. Set KNOX_GATEWAY_DBG_OPTS environment variable as follows: 
export KNOX_GATEWAY_DBG_OPTS="$KNOX_GATEWAY_DBG_OPTS -Dcom.sun.management.jmxremote.port=8888 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

      2. Enable JMX reporting in gateway-site.xml:

    <property>
        <name>gateway.metrics.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>gateway.jmx.metrics.reporting.enabled</name>
        <value>true</value>
    </property>

Once you opened the necessary JMX port, you also need to make sure you have at least one topology with the KNOXTOKEN service. During my tests, I extended the sandbox topology with the following service configuration:

   <service>
      <role>KNOXTOKEN</role>
      <param>
         <name>knox.token.ttl</name>
         <value>36000000</value>
      </param>
      <param>
         <name>knox.token.audiences</name>
         <value>tokenbased</value>
      </param>
      <param>
         <name>knox.token.target.url</name>
         <value>https://localhost:8443/gateway/tokenbased</value>
      </param>
      <param>
         <name>knox.token.exp.server-managed</name>
         <value>true</value>
      </param>
      <param>
         <name>knox.token.renewer.whitelist</name>
         <value>guest</value>
      </param>
   </service>

If you plan to create a new topology for this purpose, please change the perf.test.usecase.knoxtoken.topology.gateway configuration accordingly.

As you can see, the newly added service references another topology called tokenbased. As its name suggests, that particular topology uses JWT authentication and is configured as follows:

<?xml version="1.0" encoding="UTF-8"?>
<topology>
   <name>tokenbased</name>
   <gateway>
      <provider>
         <role>federation</role>
         <name>JWTProvider</name>
         <enabled>true</enabled>
         <param>
            <name>knox.token.audiences</name>
            <value>tokenbased</value>
         </param>
      </provider>
   </gateway>
   <service>
      <role>WEBHDFS</role>
      <url>http://YOUR_HDFS_SERVICE_HOST:20101/webhdfs</url>
   </service>
</topology>

Since the KnoxToken Use Case tries to use an already acquired Knox DT to run an action I chose to do this as simple as possible: using KnoxShell's HDFS


How to run

Running the performance tool is as simple as running the following Maven command in the project root:

mvn -DskipTests -Dcheckstyle.skip=true -Dfindbugs.skip=true -Dpmd.skip=true -Drat.skip -Pgateway-performance-test package -am -pl gateway-performance-test

The tool will pick up the above-mentioned configuration file and execute all enabled use-case runners (currently there is only one implementation). You can make the desired changes in that properties file before executing your performance test rounds as your requirements needs. For instance, increasing the number of parallel threads to 10 and the test duration to 6 hours you need to update

perf.test.usecase.knoxtoken.numOfThreads=10
perf.test.usecase.knoxtoken.testDurationInSecs=21600
  • No labels