arrayEach has too much memory overhead from duplicating pageContext

Description

The following code will use no more than 20 threads at a time process 50k items in an array, but the used heap on a 64 bit Windows machine jumps from 54 Megs to 2.9 Gigs! A GC will clear out the heap, but that is just way too much memory.

Not using parallel runs in 312 ms but adding the threading slows the request down to 15 seconds. The FusionReactor profiler shows that nearly 80% of the request was spent cloning the page context.

A heap dump of the server's memory also shows that there are a huge number of page context objects in the heap which were expensive to create.

  • Lucee appears to create 50,000 new page contexts which is very wasteful. Since there are no more than 20 threads running at a time, no more than 20 page contexts should have been created and pooled.

  • I actually don't think Lucee should be cloning the pc at all! Unlike a cfthread which can be a daemon, an arrayEach() or structEach() thread by design will never outlive the parent request and runs "inside" of the parent request. Therefore, why are we cloning the pc at all? This is just wasted resources and time. The threads should be able to share the same pc in this instance.

I've run into this and had it cause out of memory errors due to wasted memory and I also have clients running into this consuming high amounts of memory on their servers while trying to process large batches of items using arrayEach() or queryEach().

Environment

None

Status

Assignee

Unassigned

Reporter

Brad Wood

Labels

None

Affects versions

5.3.4.61

Priority

New
Configure