Lucee 4 and 5 consumes too much heap memory with large uploads due to bug in HTTPServletRequestWrap.java
Description
Environment
None
causes
relates to
Activity
Show:
Zac Spitzer 5 February 2021 at 11:06
Michael Offner 8 December 2020 at 14:44
Zac Spitzer 19 June 2020 at 14:13
Is this also complicated by any threads each reprocessing the pageContext? i.e. LDEV-2903
Michael Offner 19 June 2020 at 13:56Edited
@Bruce Kirkpatrick i cannot simply apply your fix, it trade one issue with an other, this code allows to request the input stream more than once, what for example is necessary when you call getHTTPRequestData(), that change would brake that function.
What we could do, is to not store the content in memory in case it reaches a certain size.
Sure we already have a limitation of 50mb, but i would say only to keep objects in memory that are much smaller. question also is, why does the method isToBig not work as expected (could be a multi threading issue)
Michael Forell 14 June 2019 at 12:58
I’d like to confirm, there is an error.
Here is a stacktrace:
resin-port-8081-3092
at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
at java.util.Arrays.copyOf([BI)[B (Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(I)V (ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(I)V (ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write([BII)V (ByteArrayOutputStream.java:153)
at lucee.commons.io.IOUtil.copy(Ljava/io/InputStream;Ljava/io/OutputStream;I)V (IOUtil.java:287)
at lucee.commons.io.IOUtil.copy(Ljava/io/InputStream;Ljava/io/OutputStream;ZZ)V (IOUtil.java:75)
at lucee.commons.io.IOUtil.toBytes(Ljava/io/InputStream;Z)[B (IOUtil.java:946)
at lucee.commons.io.IOUtil.toBytes(Ljava/io/InputStream;)[B (IOUtil.java:940)
at lucee.runtime.net.http.HTTPServletRequestWrap.getInputStream()Ljavax/servlet/ServletInputStream; (HTTPServletRequestWrap.java:268)
at org.apache.commons.fileupload.servlet.ServletRequestContext.getInputStream()Ljava/io/InputStream; (ServletRequestContext.java:112)
at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(Lorg/apache/commons/fileupload/FileUploadBase;Lorg/apache/commons/fileupload/RequestContext;)V (FileUploadBase.java:952)
at org.apache.commons.fileupload.FileUploadBase.getItemIterator(Lorg/apache/commons/fileupload/RequestContext;)Lorg/apache/commons/fileupload/FileItemIterator; (FileUploadBase.java:310)
at lucee.runtime.type.scope.FormImpl.initializeMultiPart(Llucee/runtime/PageContext;Z)V (FormImpl.java:183)
at lucee.runtime.type.scope.FormImpl.initialize(Llucee/runtime/PageContext;)V (FormImpl.java:128)
at lucee.runtime.PageContextImpl.formScope()Llucee/runtime/type/scope/Form; (PageContextImpl.java:1210)
at lucee.runtime.type.scope.UndefinedImpl.reinitialize(Llucee/runtime/PageContext;)V (UndefinedImpl.java:644)
at lucee.runtime.type.scope.UndefinedImpl.initialize(Llucee/runtime/PageContext;)V (UndefinedImpl.java:621)
at lucee.runtime.PageContextImpl.initialize(Ljavax/servlet/http/HttpServlet;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;ZIZZZ)Llucee/runtime/PageContextImpl; (PageContextImpl.java:496)
at lucee.runtime.CFMLFactoryImpl.getPageContextImpl(Ljavax/servlet/http/HttpServlet;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;ZIZZZJZZZ)Llucee/runtime/PageContextImpl; (CFMLFactoryImpl.java:171)
at lucee.runtime.engine.CFMLEngineImpl._service(Ljavax/servlet/http/HttpServlet;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;S)V (CFMLEngineImpl.java:1061)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(Ljavax/servlet/http/HttpServlet;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (CFMLEngineImpl.java:1039)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(Ljavax/servlet/http/HttpServlet;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (CFMLEngineWrapper.java:102)
at lucee.loader.servlet.CFMLServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (CFMLServlet.java:51)
at javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (HttpServlet.java:97)
at com.caucho.server.dispatch.ServletFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (ServletFilterChain.java:109)
at com.caucho.server.webapp.WebAppFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (WebAppFilterChain.java:156)
at com.caucho.server.webapp.AccessLogFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (AccessLogFilterChain.java:95)
at com.caucho.server.dispatch.ServletInvocation.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (ServletInvocation.java:290)
at com.caucho.server.http.HttpRequest.handleRequest()Z (HttpRequest.java:840)
at com.caucho.network.listen.TcpSocketLink.dispatchRequest()V (TcpSocketLink.java:1362)
Fixed
Details
Assignee
UnassignedUnassignedReporter
Bruce KirkpatrickBruce KirkpatrickPriority
CriticalNew Issue warning screen
Before you create a new Issue, please post to the mailing list first https://dev.lucee.org
Once the issue has been verified, one of the Lucee team will ask you to file an issue
Sprint
None
Details
Details
Assignee
Unassigned
UnassignedReporter
Bruce Kirkpatrick
Bruce KirkpatrickPriority
CriticalNew Issue warning screen
Before you create a new Issue, please post to the mailing list first https://dev.lucee.org
Once the issue has been verified, one of the Lucee team will ask you to file an issue
Sprint
None
Created 20 November 2018 at 22:27
Updated 22 June 2022 at 14:35
Resolved 22 June 2020 at 07:23
I'm pretty sure I just isolated and fixed a problem that Lucee has had for over 3 years. It may have even been there in Railo.
It may even be the solution for the complaint made in this old ticket:
https://luceeserver.atlassian.net/browse/LDEV-548
It appears the code written for the isToBig() is not reliable or not accurate somehow. It attempts to guess if there is enough heap memory left to process a huge post request using the heap instead of reverting to the alternative behavior, which is a very efficient disk based method. The disk based approach only consumes a trivial amount of heap (like up to 1mb), whereas the isToBig is able to consume ALL the heap and accidentally throw java heap space through the inaccuracy of trying to guess how much memory is needed.
This can make Lucee very unstable / able to crash. This is a serious flaw in lucee, that was easy to fix.
If we are going to persist in allowing further optimization beyond the 1mb default, I suggest that value become an Lucee admin option on the Request page named "Limit Request Upload Size" with a comment describing the behavior. I'm talking about giving us access to modify this value in FormImpl.java.
DiskFileItemFactory(DiskFileItemFactory.DEFAULT_SIZE_THRESHOLD, (File) tempDir)
Obviously the user must also configure tomcat (or other server's) connector appropriately to allow a significantly larger post then default, which I have also done. I was able to upload a file that is larger then the heap, and I also monitored the heap during processing and saw it didn't use excessive memory anymore, so basically you had the solution there, but you weren't letting it run.
I'll submit my working commit in a moment.