Query of Query performance is very bad and single threaded for complex SQL

Description

Query of Query support in Lucee does not perform well. The native Java implementation of QoQ only supports the most basic of SQL. All complex queries throw an exception and fall back to a second attempt which runs the select against an embedded HyperSQL database. This has a few issues:

  • The HSQLDB implementation is single threaded due to a synchronized block which makes QoQ untenable under any sort of load

  • There is a base overhead of the JDBC connection and parameter handling that makes any page with a large number of of QofQs to slow down significantly

  • HSQLDBs parser is not very good and throws exceptions for many valid SQL statements such as count(1)

Improve the native Java implementation of QofQ to support

  • group by clause

  • having clause

  • aggregate functions

  • Allow aggregates to reference nested operations including scalar functions

    • ceiling( max( floor( yearsEmployed ) )+count(1) )

  • faster distinct support

  • Fix buginess in unions were results aren't distinct by default (union distinct)

  • Improve count() to support

    • count( all col )

    • count( 1 )

    • count( * )

    • count( 'literal' )

    • count( distinct col )

    • count( distinct scalarFunc( col ) )

  • Fix bugs in SQL parser that incorrectly requires only a single space between multi-word clauses

    • is null

    • is not null

    • not in

    • not like

    • order by

    • group by

  • Remove single-threaded limitations

  • Performance tune speed of queries

I am submitting a pull request that includes all of this. I have done rigorous testing and added a new suite of tests for all the functionality. I have also done side-by-side performance tests with Adobe CF and the new implementation is faster than Lucee's old implementation in every single test and is faster than Adobe CF's implementation in most of the tests. All code in the pull request is fully commented.

I have also added support for the following system props/env vars

  • lucee.qoq.hsqldb.disable=true – Throw exception if native QoQ logic fails

  • lucee.qoq.hsqldb.debug=true – Log message to WEB context's datasource log any time a QofQ "falls back" to HyperSQL. This could include just bad SQL syntax.

Attachments

2
  • 11 Sept 2020, 09:08 pm
  • 10 Sept 2020, 04:59 pm

Activity

Show:

Pothys - MitrahSoft 11 December 2020 at 11:27

, I've checked this ticket. Now, it works fine with lucee fixed version 5.3.8.74-SNAPSHOT and latest version 5.3.8.119-SNAPSHOT. Seems, the performance wise very well using QoQ and we can see the differences in performance between old and latest lucee version as per brad said.

Michael Offner 21 September 2020 at 12:24

Brad has pointed out the following

Please give all this tickets a try and if necessary create test cases for them. Please report in this tickets if hey got solved by this change.

Michael Offner 21 September 2020 at 12:22

you was right with your concern about concurrentmodifucationException, there was actually one place causing problems.

https://github.com/lucee/Lucee/commit/d84b92e28ab0e571bdda996b1fa2af9afb69b56e

Zac Spitzer 11 September 2020 at 21:10

I was playing with a threaded test against Lucee with this patch, given QoQ was previously all synchronised

attached is thread.cfm, sometimes the intermediatory qoq sum values are all over the shop

behaviour is rather different with the lock around the QoQ in the threaded function

Fixed

Details

Assignee

Reporter

New Issue warning screen

Before you create a new Issue, please post to the mailing list first https://dev.lucee.org

Once the issue has been verified, one of the Lucee team will ask you to file an issue

Sprint

Fix versions

Priority

Created 7 September 2020 at 06:02
Updated 7 January 2022 at 14:48
Resolved 18 December 2020 at 16:02