Memory issues because queryobject contains a very large ArrayInt

Description

I’m analyzing a heapdump because of a memory problem on one of my servers. The memory is largely used by a query in the session scope, although the query is just a few record. What strikes me is every query contains a very large ArrayInt (1.048.576 items).

When investigating further it seems every query in memory has the same ArrayInt with exactly the same number of items. What are your opions on this? Is it a bug?

https://dev.lucee.org/t/memory-issues-because-queryobject-contains-a-very-large-arrayint/4029

Environment

None

Activity

Show:
Bas van der Graaf
August 2, 2018, 9:36 AM

Just to add some more information. Downgrading to Lucee 5.1.3.18 resolves this issue.

Michael Offner
August 2, 2018, 1:25 PM

your problem is not related to this, i have checked again your heap dump, the ArrayInt object of your query consumes only 0.1% of the memory and the query itself 73%.

Michael Offner
August 2, 2018, 1:25 PM

the analyze from @joe.gooch is very accurate.

what is important to point out that Lucee has a pool for PCs, so the size is not growing for every request, only in case you have multiple request at the same time (including cfthreads what get it's own PC).

When this implemenation was written (a long time ago) the id was not a static id, is was more like the stack size of the current running threads, a value of course not growing over time. so when the implementation of PC.id changed we have forgotten that query is depending on a low number. a typical example of using a unindential side effect on a system (always expecting low number).

*So why is an array used in the first place and not a map?*
Speed, we needed a solution that is as fast as possible and a native array was the fastest one

*How can we fix this?*

  • make sure the id is always low again, unless you have hunderts of threads open at the same time.

  • find a different solution

Michael Offner
August 2, 2018, 2:13 PM
Michael Offner
August 2, 2018, 2:14 PM

for the moment used a HashMapPro, what is optimised for fast read, maybe we will improve that implementation in the near future.

Fixed

Assignee

Michael Offner

Reporter

Zac Spitzer

Priority

Critical

Labels

Fix versions

Sprint

None
Configure