Versions Compared

Old Version 28

changes.mady.by.user Konstantin Kolinko

Saved on

compared with

New Version Current

changes.mady.by.user Konstantin Kolinko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Add binary search to the Common Troubleshooting Scenario

...

  1. Look into Tomcat access log (the log file generated by AccessLogValve).
    1. If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall).
    2. You will see what IP address your client is using, and whether it is using an IPv4 (127.0.0.1) or IPv6 address (0:0:0:0:0:0:0:1).
      Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4.
  2. Take a thread dump. This way you will find out what Tomcat is actually doing.
    1. If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress.
  3. Try debugging.
    1. A good place for a breakpoint is org.apache.catalina.connector.CoyoteAdapter.service() method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts.
  4. If you did a long-awaited upgrade, jumping over several years worth of Tomcat releases, and something broke, and you have no clue,
    1. Reading Migration guides may help.
    2. It may help to do a binary search (aka bisecting) to locate the version of Tomcat that triggered the change. If your issue is easy to reproduce, it may be pretty fast. Just 7-8 tries may cover a range of 100 versions. Once you know the version and its release date, the following resources are available:
      1. The release announcement.
        See "former announcements" link at the bottom of the front page of the Apache Tomcat site.
        An announcement mail message can also be found in the archives of the "announce@" mailing list.
      2. The changelog. A release announcement usually has a link to it.
      3. Archives of the "users@" mailing list. You may look for discussions that happened a month or two after the release.

Troubleshooting unexpected Response state problems

If you encounter problems that manifest themselves as accessing a request or response that is an inconsistent state.

E.g. 61289, 58457

The , the main suspect is your own web application (or a library that it uses) keeping a reference to Request / or Response objects outside of their life cycle. Examples: BZ 61289, BZ 58457.

The lifetime of the Response object is documented in the Servlet specification. Quoting from section "5.8 Lifetime of the Response Object" of Servlet 4.0 specification:

...

To troubleshoot the issue:

  1. Set the following system property in Tomcat configuration:
    org.apache.catalina.connector.RECYCLE_FACADES=true
    When the above flag is set, Tomcat recycles Make sure that your Tomcat is configured to discard facades to its internal objects when request processing completes. This makes it easier to spot illegal access when it happens, instead of waiting until side effects of such access become visible. Essentially, it protects Tomcat internals from misbehaving web applications.

    This feature is always on when you are running Tomcat with a Java Security Manager being enabled. Starting with Tomcat 10.0 this feature is enabled by default. It is disabled by default in earlier versions of Tomcat. The way this feature is configured differs between versions: it is controlled by an attribute on Connector element or by a system property.

    If you are running Tomcat 9.0 or earlier, do both of the following:

    - Set the following system property in Tomcat configuration:
    org. This flag apache.catalina.connector.RECYCLE_FACADES=true

    - Add the following attribute to all Connector elements:
    discardFacades="true"

    The Connector attribute was added in Tomcat 10.0.0-M1, 9.0.31, 8.5.51 and 7.0.100. The system property is an older way to configure this feature. In case of a doubt, or if you are switching back and forth between versions while troubleshooting the issue, it is safer to configure both of them.

    This feature is also mentioned on the Security Considerations page . The flag is true when Tomcat runs with enabled Java Security Manager.
    in Tomcat documentation. You can also search the archives of the Tomcat users' mailing lists for previous discussions mentioning the RECYCLE_FACADES flag.
    Read about Java ImageIO issue.

Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat, if the problem manifests as a security issue, see how to report it.

There are some known examples of broken libraries / APIs:

  1. Read about Java ImageIO issue — an issue with javax.imageio.ImageIO API. It may have already been fixed as it is an old issue, but there are no clear records of it.
  2. Read about an issue in PD4ML, a library that is used to generate PDF files, — fixed in their version 3.8.0, earlier versions may be affected.

Troubleshooting "Too many open file descriptors"

The code that opens the descriptors can be identified using a tool such as http://file-leak-detector.kohsuke.org/