Requests hang waiting for queries which dont timeout

Description

I'm seeing requests that never timeout, which appear to be waiting for query to be returned. The query does not timeout even though the jdbc connection is set to timeout after 1 minute in the server admin. The request timeout is set to 50 seconds inside application.cfc: `this.timeout = 50;`. This results in hanging threads that end up bringing down my server on a daily basis. Tomcat basically stops responding.

One interesting note is that it appears that the request is going through the tomcat websocket filter (I see this in the stacktrace and from Fusion Reactor transaction name, which is "Tomcat WebSocket (JSR356) Filter"). The websocket extension is installed on this server, but the application is not making any websocket calls either from the client or the server.

Here's a stack trace of a thread that has been running for over an hour:

Environment

Windows 2012 R2 x64

Activity

Show:
Brad Wood
December 20, 2017, 2:11 AM

Oh perfect, Igal just answered! I hope you're not using queryExecute(). This seems like a good argument for a data source-level default query timeout like I suggested before.

Did you find out why the queries are running long in the first place?

JP
December 20, 2017, 2:34 AM

It’s likely an issue with my RDS instance size not being able to handle the load. I’ve determined with the help of query that the queries become suspended, which is caused in mssql for a number of reasons, but essentially it’s waiting for a resource to become available. I think it might be a locking issue, but I have to dig more.

JP
December 21, 2017, 12:19 AM

FusionReactor support got back to me about the TXN label in the FusionReactor slow requests report where the label reads: “Tomcat Websocket (JSR356) Filter.”. They said: “the tomcat server is using web sockets to process requests internally”. Is this true?

Igal Sapir
December 21, 2017, 5:36 AM

Tell them that I said that that's nonsense.

Janneke Wijnands
March 15, 2019, 10:04 AM

Since this is an old unresolved issue that still happens to people (it did for us) and pops up high when searching for it here's a possible solution: add a socketTimeout={time in ms} to your connection string. The mssql driver at least supports it.

Assignee

Unassigned

Reporter

JP

Priority

Major

Labels

None

Fix versions

None

Affects versions