All work

 
1 of 1

memory increase and never give back tho the system

Description

Hello,
i loop over a file and read line by line.
The file have pheraps 6 Mio lines (800MB).
While the loop runs, the memory first increases slow, but when it come to the end, the memory increases to the limit (6GB).
After the routine ist finished, i cleared all querys, but lucee give never the memory back to the sytem.

In the loop i use different mysql-querys and variables, also with the same name in each loop, so that lucee not use new memory.
mySQL has a normal memory usage

After the loop Ends, i must lucee becomes so slowly, that i must Restart the Service

lucee hav the latest release.

Have everybody a solution?

Environment

W2015R2, 12GB

Details

Assignee

Reporter

Priority

Labels

New Issue warning screen

Before you create a new Issue, please post to the mailing list first https://dev.lucee.org

Once the issue has been verified, one of the Lucee team will ask you to file an issue

Affects versions

Created 23 August 2018 at 13:20
Updated 19 May 2021 at 10:52

Activity

Wilfried Wacker 
18 February 2019 at 17:02

the File is a XML-File, but i do not use the xml-parser.

<cfset dataFile = fileOpen( DownloadDatei, "read" ) />
<cfset Anzeigenzaehler = 1>
<cfloop condition="

--- Datei zeilenweise einlesen bis EOF kommt --->
<cfset line = fileReadLine( dataFile ) /> <!--- Zeile zuordnen --->

// different Operation
<cftransaction>
<cfquery with select / look für this data in de db

<cfquery insert or update, dependence the daata are new or in the db
</cftransaction>
</cfif> <!--- /ENDE Anzeige bearbeiten --->
</cfloop> <!--- /ENDE ZeilenSchleife über Importdatei --->
<cfset fileClose(dataFile)>

In the loop the same querys has always the same Name.
i test with differnt names, but no other result.

The daily readed file have ca. 110.00 entrys, with 10.000 new and 100.000 updates.
Database have now 1 Mio entrys and are daily updated
the loop needs ca. 40 min. (VServer 6core Win2012 with IIS, 12GB Memory with 6GB for Lucee/tomcat and mySQL, )

In a PN i can give you the cfml-Code, the table-Alter-Code and a importfile, so you can test it.
My englisch is not so good, so i hope you understand me. In a PN we can talk in german?
Email: info@inteka.de
regards Wilfried

Michael Offner 
8 February 2019 at 14:14

how exactly loop over that file? cfloop type file?

"but lucee give never the memory back to the sytem" what do you mean exactly with this, have in mind that the JVM not necessary gives back memory to the system the moment it no longer needs it, as long there are no memory issues.
the erliest point the memory goes back is the next time the GC is executed and even then the JVM not necessary frees that memory to the system.
So how you messure that exactly?