Impala FE is written in Java. Impalad and Catalogd processes both have JVM inside them. When encounter performance issues in the FE side, you may need to do JVM CPU profiling. Enabling remote JMX access and profiling by tools like jvisualvm is a common choice.

Enable Remote JMX

Normally, you just need to set environment variable JAVA_TOOL_OPTIONS:

JAVA_TOOL_OPTIONS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${SERVER_IP}"

Then use profiling tools like jvisualvm to connect to the process.

Take profiling GET_TABLES() performance as an example:

  1. Pick the coordinator used by HUE. Restart it with additional env var JAVA_TOOL_OPTIONS above.
  2. Launch jvisualvm in a machine with GUI and connect to the process (port=9010).
  3. Start CPU profiling and trigger a Get_Tables() request to the coordinator (e.g. "Clear Cache" in HUE for a db)
  4. Stop CPU profiling after the Get_Tables() request finish
  5. Find the thread executing execHiveServer2MetadataOp() and expand it to get drill-down results

1596pxYou should be able to get results like this:

Enable Remote JMX when SSL is enabled

In a cluster with SSL enabled, you may encounter the following error when restarting Impala process with the above JAVA_TOOL_OPTIONS env var.


W1111 23:31:24.844229 14519 messenger.cc:219] could not get RPC password from configured command: Runtime error: failed to run private key password command: Runtime error: /run/cloudera-scm-agent/process/76-impala-IMPALAD/altscript.sh: process exited with non-zero status 1: Picked up JAVA_TOOL_OPTIONS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.31.114.150
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9010; nested exception is:
        java.net.BindException: Address already in use (Bind failed)
......
I1111 23:31:24.847038 14519 status.cc:125] Could not build messenger: Runtime error: failed to load private key file: unable to load data from file '/etc/cdep-ssl-conf/CA_STANDARD/impala-enc_key.pem': error:0906A068:PEM routines:PEM_do_header:bad password read:pem_lib.c:457
    @           0x9810da
    @           0xb3c262
    @           0xb60312
    @           0xc236e9
    @           0x8fe724
    @     0x7f8c5a8bb830
    @           0x946819
F1111 23:31:24.847081 14519 impalad-main.cc:81] Could not build messenger: Runtime error: failed to load private key file: unable to load data from file '/etc/cdep-ssl-conf/CA_STANDARD/impala-enc_key.pem': error:0906A068:PEM routines:PEM_do_header:bad password read:pem_lib.c:457
. Impalad exiting.
Wrote minidump to /var/log/impala-minidumps/impalad/f79b9001-a4b9-4c69-6c2a80bf-14390d04.dmp

The cause is that the process fail to get the keystore password to read the keystore. When SSL is enabled and --ssl_private_key_password_cmd is set, impalad has to run the given command to retrive keystore password. In CDH cluster, the command launches a Java process and inherit the same JAVA_TOOL_OPTIONS as impala process. So this java process crash in binding the JMI port 9010. This port is already binded by impala process. So impala process crashes with the above error.

One workarround is to use random port in JMX by setting -Dcom.sun.management.jmxremote.port=0. To figure out which port is used finally, create a logging configuration for JMX. Let's say it's /tmp/logging.properties:

handlers= java.util.logging.ConsoleHandler
.level= INFO
java.util.logging.ConsoleHandler.level = CONFIG
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
sun.management.jmxremote.level = CONFIG

Set JAVA_TOOL_OPTIONS as

JAVA_TOOL_OPTIONS="-Djava.util.logging.config.file=/tmp/logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=0 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=${SERVER_IP}"

Then after impalad launch, you can find ports in impalad.ERROR. E.g.

Log file created at: 2019/11/14 00:52:20
Running on machine: quanlong-2.quanlong.root.hwx.site
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1114 00:52:20.905951 161869 logging.cc:121] stderr will be logged to this file.
Nov 14, 2019 12:52:21 AM ConnectorBootstrap startRemoteConnectorServer
CONFIG: JMX Connector ready at: service:jmx:rmi:///jndi/rmi://quanlong-2.quanlong.root.hwx.site:36787/jmxrmi  <-- Here!
Picked up JAVA_TOOL_OPTIONS: -Djava.util.logging.config.file=/tmp/logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=0 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.27.114.65
Nov 14, 2019 12:52:24 AM ConnectorBootstrap startRemoteConnectorServer
CONFIG: JMX Connector ready at: service:jmx:rmi:///jndi/rmi://quanlong-2.quanlong.root.hwx.site:42407/jmxrmi
Picked up JAVA_TOOL_OPTIONS: -Djava.util.logging.config.file=/tmp/logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=0 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.27.114.65
Nov 14, 2019 12:52:25 AM ConnectorBootstrap startRemoteConnectorServer
CONFIG: JMX Connector ready at: service:jmx:rmi:///jndi/rmi://quanlong-2.quanlong.root.hwx.site:41063/jmxrmi
Picked up JAVA_TOOL_OPTIONS: -Djava.util.logging.config.file=/tmp/logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=0 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.27.114.65
Nov 14, 2019 12:52:28 AM ConnectorBootstrap startRemoteConnectorServer
CONFIG: JMX Connector ready at: service:jmx:rmi:///jndi/rmi://quanlong-2.quanlong.root.hwx.site:43275/jmxrmi
Picked up JAVA_TOOL_OPTIONS: -Djava.util.logging.config.file=/tmp/logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=0 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.27.114.65
Nov 14, 2019 12:52:29 AM ConnectorBootstrap startRemoteConnectorServer
CONFIG: JMX Connector ready at: service:jmx:rmi:///jndi/rmi://quanlong-2.quanlong.root.hwx.site:34107/jmxrmi
Picked up JAVA_TOOL_OPTIONS: -Djava.util.logging.config.file=/tmp/logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=0 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.27.114.65
Nov 14, 2019 12:52:30 AM ConnectorBootstrap startRemoteConnectorServer
CONFIG: JMX Connector ready at: service:jmx:rmi:///jndi/rmi://quanlong-2.quanlong.root.hwx.site:39969/jmxrmi

The first port, 36787 in this example, is the JMX port used by impalad. You can use jvisualvm as usual to connect to it.