optimize the extension update provider
Description
Attachments
- 27 Jan 2023, 03:38 pm
Activity
Zac Spitzer 27 January 2023 at 15:38(edited)
fixed https://github.com/lucee/lucee-data-provider/commit/bd2a026690e1f2a6135ccb6cef8aa1e79323a584
took a while to build the extension cache, I added an updateCache to prime it
https://github.com/lucee/lucee-data-provider/commit/56f6cea7c0670ab6f827202362761d6743ec3cd4
json is now 395k vs 1mb, and it’s about 50% faster
Zac Spitzer 27 January 2023 at 12:56
reverted deploy, got a NPE
{
Extended_Info: "",
Message: "java.lang.NullPointerException",
Detail: "",
additional: {
Cause: "java.lang.NullPointerException"
},
TagContext: [
{
Raw_Trace: "extension5.extensionprovider_cfc$cf.udfCall1(/extension5/ExtensionProvider.cfc:61)",
codePrintPlain: "59: rtn.meta.url="http://"&cgi.HTTP_HOST;
60: rtn.meta.mode="production";
61: rtn.extensions=new S3Ext(variables.s3Root)
62: .list(type:arguments.type,flush:arguments.flush,withLogo:arguments.withLogo);
63:
",
column: 0,
line: 61,
template: "/var/www/sites/extension/extension5/ExtensionProvider.cfc",
id: "??",
type: "cfml",
codePrintHTML: "59: rtn.meta.url="<http://"&cgi.HTTP_HOST;<br>>
60: rtn.meta.mode="production";<br>
<b>61: rtn.extensions=new S3Ext(variables.s3Root)</b><br>
62: .list(type:arguments.type,flush:arguments.flush,withLogo:arguments.withLogo);<br>
63: <br>
"
}
],
ErrorCode: "0",
type: "java.lang.NullPointerException",
StackTrace: "java.lang.NullPointerException
at lucee.transformer.bytecode.expression.var.Variable._writeOutFirstBIF(Variable.java:551):551
at lucee.transformer.bytecode.expression.var.Variable._writeOutFirst(Variable.java:468):468
at lucee.transformer.bytecode.expression.var.Variable._writeOut(Variable.java:263):263
at lucee.transformer.bytecode.expression.var.Variable._writeOut(Variable.java:244):244
at lucee.transformer.bytecode.expression.ExpressionBase.writeOut(ExpressionBase.java:52):52
at lucee.transformer.bytecode.statement.ExpressionAsStatement._writeOut(ExpressionAsStatement.java:52):52
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.util.ExpressionUtil.writeOut(ExpressionUtil.java:148):148
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:188):188
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:140):140
at lucee.transformer.bytecode.BodyBase._writeOut(BodyBase.java:133):133
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.statement.Condition._writeOut(Condition.java:117):117
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.util.ExpressionUtil.writeOut(ExpressionUtil.java:148):148
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:188):188
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:140):140
at lucee.transformer.bytecode.BodyBase._writeOut(BodyBase.java:133):133
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.statement.Condition._writeOut(Condition.java:117):117
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.util.ExpressionUtil.writeOut(ExpressionUtil.java:148):148
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:188):188
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:140):140
at lucee.transformer.bytecode.BodyBase._writeOut(BodyBase.java:133):133
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.statement.TryCatchFinally._writeOut(TryCatchFinally.java:177):177
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.util.ExpressionUtil.writeOut(ExpressionUtil.java:148):148
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:188):188
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:140):140
at lucee.transformer.bytecode.BodyBase._writeOut(BodyBase.java:133):133
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.statement.tag.TagGroupUtil.writeOutTypeQuery(TagGroupUtil.java:369):369
at lucee.transformer.bytecode.statement.tag.TagLoop._writeOut(TagLoop.java:266):266
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.util.ExpressionUtil.writeOut(ExpressionUtil.java:148):148
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:188):188
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:140):140
at lucee.transformer.bytecode.BodyBase._writeOut(BodyBase.java:133):133
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.statement.tag.TagHelper.doTry(TagHelper.java:416):416
at lucee.transformer.bytecode.statement.tag.TagHelper.writeOut(TagHelper.java:334):334
at lucee.transformer.bytecode.statement.tag.TagBase._writeOut(TagBase.java:137):137
at lucee.transformer.bytecode.statement.tag.TagOther._writeOut(TagOther.java:43):43
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.util.ExpressionUtil.writeOut(ExpressionUtil.java:148):148
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:188):188
at lucee.transformer.bytecode.BodyBase.writeOut(BodyBase.java:140):140
at lucee.transformer.bytecode.BodyBase._writeOut(BodyBase.java:133):133
at lucee.transformer.bytecode.FlowControlBody._writeOut(FlowControlBody.java:33):33
at lucee.transformer.bytecode.statement.StatementBase.writeOut(StatementBase.java:72):72
at lucee.transformer.bytecode.statement.NativeSwitch._writeOut(NativeSwitch.java:73):73
at lucee.transformer.bytecode.Page.writeOutUdfCallInner(Page.java:806):806
at lucee.transformer.bytecode.Page.execute(Page.java:615):615
at lucee.runtime.compiler.CFMLCompilerImpl.compile(CFMLCompilerImpl.java:81):81
at lucee.runtime.PageSourceImpl._compile(PageSourceImpl.java:334):334
at lucee.runtime.PageSourceImpl.compile(PageSourceImpl.java:303):303
at lucee.runtime.PageSourceImpl.loadPhysical(PageSourceImpl.java:241):241
at lucee.runtime.PageSourceImpl.loadPage(PageSourceImpl.java:195):195
at lucee.runtime.config.ConfigImpl.getCachedPage(ConfigImpl.java:3010):3010
at lucee.runtime.component.ComponentLoader.load(ComponentLoader.java:119):119
at lucee.runtime.component.ComponentLoader.loadComponent(ComponentLoader.java:59):59
at lucee.runtime.PageContextImpl.loadComponent(PageContextImpl.java:2971):2971
at lucee.runtime.functions.other.CreateObject.doComponent(CreateObject.java:228):228
at lucee.runtime.functions.other._CreateComponent.call(_CreateComponent.java:39):39
at extension5.extensionprovider_cfc$cf.udfCall1(/extension5/ExtensionProvider.cfc:61):61
at extension5.extensionprovider_cfc$cf.udfCall(/extension5/ExtensionProvider.cfc):-1
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112):112
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:329):329
at lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:217):217
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:643):643
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:532):532
at lucee.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1788):1788
at lucee.runtime.ComponentSpecificAccess.callWithNamedValues(ComponentSpecificAccess.java:153):153
at lucee.runtime.ComponentPage._callRest(ComponentPage.java:438):438
at lucee.runtime.ComponentPage.callRest(ComponentPage.java:354):354
at lucee.runtime.ComponentPage.call(ComponentPage.java:169):169
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:959):959
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:911):911
at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:225):225
at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:37):37
at lucee.runtime.PageContextImpl.executeRest(PageContextImpl.java:2206):2206
at lucee.runtime.engine.CFMLEngineImpl.serviceRest(CFMLEngineImpl.java:505):505
at lucee.loader.servlet.RestServlet.service(RestServlet.java:47):47
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727):727
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303):303
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208):208
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52):52
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241):241
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208):208
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:404):404
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241):241
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208):208
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220):220
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122):122
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501):501
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171):171
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102):102
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116):116
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683):683
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408):408
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040):1040
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607):607
at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2379):2379
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149):1149
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624):624
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61):61
at java.lang.Thread.run(Thread.java:748):748
",
ExtendedInfo: ""
}
Zac Spitzer 27 January 2023 at 12:50
Zac Spitzer 27 December 2022 at 18:26
next step is then to move this to github actions and store the json result on s3
Zac Spitzer 27 December 2022 at 12:17
this is the size of the json before it’s served compressed
2,578,896 extensions - resize 8bit.json // resizing and 8 bit color scale
5,897,824 extensions - resize.json // only resizing to 130px
17,035,680 extensions.json // previousDetails
Details
Assignee
Reporter
Labels
New Issue warning screen
Before you create a new Issue, please post to the mailing list first https://dev.lucee.org
Once the issue has been verified, one of the Lucee team will ask you to file an issue
Priority
New
don’t cache extension *.lex files in temp directory (cache survives restart)
cache complete extension meta data as a json file, reuse for filtered methods
resize logos as thumbs, max 130 pixels wide ( that’s what the admin uses )
reduce the color depth for logos to 8 bit (via hack save as gif)
add a lock on refresh, otherwise fallback to cached info