All work

 
3 of 3

Large amounts of debug output resulting in massive log files in AWS Elastic Beanstalk

Description

I initially brought up this issue on the dev forums: https://dev.lucee.org/t/massive-amounts-of-debug-logging-in-5-3-5-92/6921

To sum it up, we run our lucee application in AWS's Elastic Beanstalk service. After upgrading from 5.2.8.50 to 5.3.5.92, huge log files are being created on the EC2 instance, outside of the docker container. Logging levels are set to ERROR level in the lucee admin console, and none of the tomcat or lucee log files are bloating. Debugging is also turned off in the lucee admin console.

All we do to upgrade Lucee versions is modify the FROM line in our Docker file. This was not an issue with: "FROM lucee/lucee52:5.2.8.50". It only became an issue with "FROM lucee/lucee:5.3.5.92". My last test with "FROM lucee/lucee:5.3.7.11-SNAPSHOT" still contained this logging issue.

The contents of the log files appear to be debug level logging related to the application making external HTTP and/or S3 requests. The reason the files are so large is because they contain the entire file content (images, pdfs, etc) that have been uploaded to the server by users. I have included a sample of a log that has been redacted for senstive info. If you need to see some of the full (very large) logs, let me know and I'll work on cleaning one up.

For now, we have turned the logging off entirely by adding `--log-driver=none` to the `docker run` command in /opt/elasticbeanstalk/hooks/appdeploy/enacts/00run.sh. This is just a temporary solution because we have to keep adding it manually every time Elastic Beanstalk spawns a new EC2.

If I had to guess, there is some sort of console output that Elastic Beanstalk is picking up and logging. We've run into similar issues in a separate Node.js application where every `console.log()` left behind in the application code was being picked up in log files and eventually eating up all of the disk space of the EC2 instance.

Sample log file, named something like /var/log/eb-docker/containers/eb-current-app/eb-063d69495c9f-stdouterr.log:

Environment

OS: Docker running on 64bit Amazon Linux/2.14.3
Java Version: 11.0.6
Tomcat Version: 9.0.33
Lucee Version: 5.3.5.92

Attachments

1

Details

Assignee

Reporter

Priority

Labels

New Issue warning screen

Before you create a new Issue, please post to the mailing list first https://dev.lucee.org

Once the issue has been verified, one of the Lucee team will ask you to file an issue

Affects versions

Created 28 April 2020 at 11:07
Updated 8 January 2021 at 18:59

Activity

Zac Spitzer 
8 January 2021 at 18:59

Deleted

Ricky Potash 
8 January 2021 at 05:26
(edited)

Please delete the alpha-deploy-master.zip attachment from this issue.

Ricky 
8 May 2020 at 19:02

I’ve uploaded the jar and the source to the original post on this ticket.

Michael Offner 
8 May 2020 at 14:42

could you please attacj your jar here for testing.

we should make sure that this never happens in Lucee.

Ricky 
5 May 2020 at 20:34

So, I’m no java expert, but here’s what I’ve gathered from the source of our custom jar. Compiled within the jar is a log4j.properties file. Within that file contains the line:

ConsoleAppender explains why the logs were only being created outside of docker. By default, Elastic Beanstalk collects all console output and logs it.

What I don’t understand is how simply having this jar file in the project (deployed to /usr/local/tomcat/lucee) is causing lucee to inherit the log4j settings. Luckily, we can just remove this jar from the project since we don’t need it anymore. But this crossing of wires is a tiny bit concerning should we ever need to implement logging in a custom jar in the future. Admittedly, that concern is probably rooted in my lack of understanding of the java ecosystem. I imagine there’s a way to encapsulate the logging directives.