Status
Current state: Under Discussion
Discussion thread: here (<- link to https://mail-archives.apache.org/mod_mbox/flink-dev/)
JIRA:
Released: <Flink Version>
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
As requested by many Flink users in the community, it will be very helpful to have a feature to change the level of certain loggers dynamically at runtime without restart, so that the issue and bug that is not able to be easily reproduced locally can be detected with the help of additional logs.
The level is able to be changed currently if the cluster is running on an environment where log4j2 is used and log4j.properties can be modified, for example, running on Kubernetes while the log4j.properties file is backed by a ConfigMap. However, there are still cases where the log configuration file cannot be changed. Yarn deployment is quite a good example of this.
Public Interfaces
REST APIs
Change the logging level of a logger
URL: /logconfig
Verb: POST
Request Payload Example:
Set the level of the logger "org.apache.flink.runtime"
to DEBUG
.
{ "loggerName": "org.apache.flink.runtime", "level": "DEBUG" }
Or reset the level to its original one.
{ "loggerName": "org.apache.flink.runtime", "level": null }
Response Code: 200 OK
Response Payload: empty
Proposed Changes
In general, the aforementioned public REST API will be introduced. Besides that, the following two RPC methods are also introduced to support this feature.
RPCs
ResourceManagerGateway
public interface ResourceManagerGateway { /** * Changes the level of the logger at runtime. * * <p>By providing a {@code null} LogLevel, the previously-changed level is reverted to its * original value. * * @param loggerName the name of the logger * @param level the log level * @return future which is completed exceptionally if the operation fails */ CompletableFuture<Void> changeLogLevel(String loggerName, @Nullable LogLevel level); }
This RPC method will be called by the handler that serves the /logconfig
request. When it is called, it does two thing,
- It changes the logging level on this job manager.
- It broadcasts this change to all the task managers that are currently registered on this job manager, by calling the RPC method
TaskExecutorGateway.changeLogLevel
as describe below.
TaskExecutorGateway
public interface TaskExecutorGateway { /** * Change the level of the logger at runtime. * * <p>By providing a {@code null} LogLevel, the previously-changed level is reverted to its * original value. * * @param loggerName the name of the logger * @param level the log level * @return future which is completed exceptionally if the operation fails */ CompletableFuture<Void> changeLogLevel(String loggerName, @Nullable LogLevel level); }
When this method is called by a ResourceManager, it does only one thing: changes the logging level on this task manager.
Logging Abstraction
The only thing left is how the logging level is changed on a job manager or task managers. Although log4j2 is the default logging implementation that is included in the distribution libraries, other logging frameworks, including log4j1 and logback, are also recommended by the documentation "How to use logging". We need to have some kind of abstraction to not directly depend on the logging implementation. Provide interfaces that not only suit the currently supported ones but also logging frameworks in the future.
TBD
Limitations
TBD
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Test Plan
Describe in few sentences how the FLIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.