Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Kafka Connect currently defines a default REST API request timeout of 90 seconds which isn't configurable. If a REST API request takes longer than this, a 500 Internal Server Error  response is returned with the message "Request timed out". In exceptional scenarios, a longer timeout may be required for operations such as connector config validation / or connector creation / updation (both of which internally does do a config validation first) to complete successfully. Consider a database / data warehouse connector which has elaborate validation logic involving querying information schema to get a list of tables/views to validate the user's connector configuration. If the database / data warehouse has a very high number of tables / views and the database / data warehouse is under a heavy load in terms of query volume, such information schema queries can end up taking longer than 90 seconds which will cause connector config validation / creation REST API calls to timeout. 

...

The value of the new worker config rest.api.request.timeout.ms will be read in the RestServer class and will be used to configure the request timeout of each of its resources (each resource essentially represents a group of related Connect REST APIs under a common top level path) via ConnectResource::requestTimeout. Note that this doesn't change how long requests actually run in the herder - currently, if a request exceeds the default timeout of 90 seconds we simply return with the 500 response but the request isn't interrupted or cancelled and is allowed to continue to completion. Furthermore, each connector config validation is anyway done on its own thread via a cached thread pool executor in the herder (create / update connector calls are done asynchronously by simply writing a record to the Connect cluster's config topic, so config validations are the only relevant operation here).

...