Update: We haven't seen this happening even once during the last 10 days, so I suspect that this was due to very rare circumstances and might not be worth investigating any further at this point. Updated priority to trivial, you may choose to dismiss this altogether for the time being.

–

Today I noticed an active request in FusionReactor that was running for more than 16 hours. The stack trace shows it hanging in a fairly ordinary cfhttp call:

<cfhttp url="#variables.strRequestUrl#" result="variables.result" getasbinary="yes" throwOnError="no" timeout="30"></cfhttp>

There is really nothing at all fancy going on in this job, Application.cfc is mostly empty and judging from the stack trace there is no involvement of anything but that cfhttp code. FusionReactor's profiler shows 100% processing time spent in that tag.

RequestTimeout in Lucee Administrator is set to 50 seconds, this.applicationTimeout in the job's Application.cfc is set to 2 hours.

Target URL of that CFHTTP is just a plain http.// URL, not even https.

A first attempt to kill the thread via FusionReactor was not succesful, message was

"We were unable to kill the request transaction. This might be due to unreleasable locks on the request thread" - though that might be a FusionReactor issue. Transit Locks shows
"Transaction Kill Not Attempted: Transit did not attempt to kill this thread because it owns synchronizers which would not automatically be released.
Lock 0: java.util.concurrent.ThreadPoolExecutor$Worker@6182a26"

Forcing the Death of the thread after that didn't work either - result says "The Thread has been stopped.", but it's still showing up as Running in FusionReactor with a "Manual Abort Failed" label tacked on to the status Code.

I have attached the stacktrace and a screenshot of the profiler from FusionReactor from before I attempted to kill the request.

P.S.: Occasionally is perhaps an overstatement, we've only encountered this once so far, but we're only had this in production for some two weeks.

PPS: It's still happening occasionally, i.e. once or twice a week at most (with about 6-8 million web requests served each day). We're running CFHTTP calls to an Amazon webservice inside a named cflock that encloses a larger bit of code; as the primary request hangs, the lock is never released and concurrent requests to that same lock end up in the timeout. I have a gut feeling that there might be some kind of race condition leading to a deadlock situation, but I really cannot make heads or tails from the stack trace alone.

PPPS (2018-09-18): Still happens occasionally, but not just with the CFHTTP to Amazon, but other CFHTTP calls as well - and it seems to be unrelated to a cflock inside the code, but, and that's just a guess, rather some sort of built in locking going on.

In this example we have got a very simple cfhttp call inside a scheduled task, Application.cfc for this app is empty and there is no cflock or anything else suspicious around this call:
<cfhttp timeout="5" method="get" url="http://#variables.strURL#/" result="request.loggedInOutputAll"></cfhttp>

This request running for over three days now. Profiling started 14-Sep-18 20:50:04.242, according to FusionReactor, and the State is "Finished" - though the request is still listed as active. I

have attacked another StackTrace, it's a linear flow until the lucee.runtime.tag.Http._doEndTag(Http.java), where there is a second branch that ends with the final Waiting on lock <0x6349b70d> (a java.lang.Object).

There is a second active request that started some five minutes before that and also seems to hang after a CFHTTP call, the last bit of the StackTrace is
100,0% - 60,169s lucee.runtime.tag.Http.doEndTag(Http.java)
100,0% - 60,169s lucee.runtime.tag.Http._doEndTag(Http.java)

• 74,6% - 44,880s lucee.commons.io.IOUtil.toString(IOUtil.java)
  74,6% - 44,880s lucee.commons.io.IOUtil.toString(IOUtil.java) (hide)
  74,6% - 44,880s lucee.commons.io.IOUtil.copy(IOUtil.java)
  74,6% - 44,880s lucee.commons.io.IOUtil.copy(IOUtil.java)
  74,6% - 44,880s java.lang.Object.wait(Native Method)
  74,6% - 44,880s Waiting on lock <0x744bdd4b> (a java.lang.Object)

• 25,4% - 15,289s java.lang.Object.wait(Native Method)
  25,4% - 15,289s Waiting on lock <0x37b9199d> (a lucee.runtime.tag.Http)

Again, I had to restart Tomcat to get rid of these processes.

Update 2019-09-24: Today we're presented with a wonderful opportunity for reproducing the problem described here, as our hoster seems to be bombarded with DDOS, which currently causes some 30-90% packet loss in our connection to the outside world. In this scenario I would simply expect a cfhttp with a timeout-attribute set to 30 to die after more or less exactly 30 seconds. What I am seeing right now though are five fairly ancient requests, all of which hang in cfhttp calls for more than two hours. Request Timeout in Lucee Admin is set to 50 seconds and there is no cfsetting-override defined anywhere in the code, so there is no reason that these requests should run over a minute, let alone several hours. Thus CFHTTP may cause requests to enter a zombie state, which may only be terminated by a Tomcat restart, as far as I can tell.