fileExists() doesn't like s3 locations

Description

var s3Path='s3://'&variables.s3Root&'/'&arrayToList(arguments.path,'/')&'/'&arguments.name;
if( not FileExists( s3Path) ){

The message is
Message: could not find a matching region for [EU], valid region names are [us-gov-west-1,us-gov-east-1,us-east-1,us-east-2,us-west-1,us-west-2,eu-west-1,eu-west-2,eu-west-3,eu-central-1,eu-central-2,eu-north-1,eu-south-1,eu-south-2,ap-east-1,ap-south-1,ap-south-2,ap-southeast-1,ap-southeast-2,ap-southeast-3,ap-southeast-4,ap-northeast-1,ap-northeast-2,ap-northeast-3,sa-east-1,cn-north-1,cn-northwest-1,ca-central-1,me-central-1,me-south-1,af-south-1,us-iso-east-1,us-isob-east-1,us-iso-west-1]

I’ve tried adding export AWS_REGION=eu-west-1 to /etc/init.d/lucce_ctl but no change.

This is the stack:

lucee.runtime.exp.NativeException: could not find a matching region for [EU], valid region names are [us-gov-west-1,us-gov-east-1,us-east-1,us-east-2,us-west-1,us-west-2,eu-west-1,eu-west-2,eu-west-3,eu-central-1,eu-central-2,eu-north-1,eu-south-1,eu-south-2,ap-east-1,ap-south-1,ap-south-2,ap-southeast-1,ap-southeast-2,ap-southeast-3,ap-southeast-4,ap-northeast-1,ap-northeast-2,ap-northeast-3,sa-east-1,cn-north-1,cn-northwest-1,ca-central-1,me-central-1,me-south-1,af-south-1,us-iso-east-1,us-isob-east-1,us-iso-west-1] at org.lucee.extension.resource.s3.S3.toRegions(S3.java:1991) at org.lucee.extension.resource.s3.S3.getBucketRegion(S3.java:1929) at org.lucee.extension.resource.s3.S3.toRegions(S3.java:1955) at org.lucee.extension.resource.s3.S3.getAmazonS3(S3.java:1914) at org.lucee.extension.resource.s3.S3.get(S3.java:869) at org.lucee.extension.resource.s3.S3.is(S3.java:834) at org.lucee.extension.resource.s3.S3.exists(S3.java:829) at org.lucee.extension.resource.s3.S3Resource.exists(S3Resource.java:326) at lucee.commons.io.res.util.ResourceUtil.toResourceNotExisting(ResourceUtil.java:331) at lucee.runtime.op.Caster.toResource(Caster.java:4641) at lucee.runtime.functions.file.FileExists.call(FileExists.java:35) at lucee.runtime.functions.file.FileExists.call(FileExists.java:29) at com.ev.business.s3_cfc$cf.udfCall(/com/ev/business/S3.cfc:57) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:350) at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223) at lucee.runtime.type.scope.UndefinedImpl.call(UndefinedImpl.java:786) at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787) at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1775) at com.ev.business.s3_cfc$cf.udfCall(/com/ev/business/S3.cfc:35) at

Environment

None

Attachments

01 Sept 2023, 02:19 pm
01 Sept 2023, 02:19 pm

Activity

Show:

Tom Chiverton 5 October 2023 at 10:14
Edited

Still broken with the latest S3 extension the output from fileExists() is false for a file that does exist

S3 extension 2.0.1.15, Lucee 5.4.3.2
Java OpenJDK 64-Bit Server VM Corretto-17.0.8.7.1 (build 17.0.8+7-LTS, mixed mode, sharing)

Still logs Unable to execute HTTP request: (my bucket name).s3.eu...

I added <cfset this.s3.host = 'http://s3.eu-west-1.amazonaws.com '/> to Application.cfc (this is where the bucket is) and it now logs

"ERROR","http-nio-8888-exec-10","10/05/2023","10:10:40","s3","The authorization header is malformed; the region 'us-
east-1' is wrong; expecting 'eu-west-1' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalfor
med; Request ID: FKNWQ4VAFVGH1XXK; S3 Extended Request ID: 8FJmOldu0sOk6KMyDB7D7gzk3BgOSaBZgKu7cKeCWJkJlFR7LyobKKmUK
PVsDaajqQHACNSLasVKgOiukpYwtQ==; Proxy: null);com.amazonaws.services.s3.model.AmazonS3Exception: The authorization h
eader is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' (Service: Amazon S3; Status Code: 400; Er
ror Code: AuthorizationHeaderMalformed; Request ID: FKNWQ4VAFVGH1XXK; S3 Extended Request ID: 8FJmOldu0sOk6KMyDB7D7g
zk3BgOSaBZgKu7cKeCWJkJlFR7LyobKKmUKPVsDaajqQHACNSLasVKgOiukpYwtQ==; Proxy: null), S3 Extended Request ID: 8FJmOldu0s
Ok6KMyDB7D7gzk3BgOSaBZgKu7cKeCWJkJlFR7LyobKKmUKPVsDaajqQHACNSLasVKgOiukpYwtQ==
…
       at org.lucee.extension.resource.s3.AmazonS3Client.listObjects(AmazonS3Client.java:1484)
       at org.lucee.extension.resource.s3.S3.get(S3.java:1079)
       at org.lucee.extension.resource.s3.S3.is(S3.java:1031)
       at org.lucee.extension.resource.s3.S3.exists(S3.java:1026)
       at org.lucee.extension.resource.s3.S3Resource.exists(S3Resource.java:328)

Pothys - MitrahSoft 12 September 2023 at 06:31

yes @Michael Offner , From my perspective, I believe it's not a bug but rather expected behavior.

Michael Offner 11 September 2023 at 14:37

i think the underlaying issue itself is solved, this is simply expected behaviour, would you agree @Pothys - MitrahSoft ?

Michael Offner 11 September 2023 at 14:37

yeah, in my opinion that is not a bug, if the region is wrong the provider will complain, best is always not to define any region at all with the host name.

Pothys - MitrahSoft 8 September 2023 at 14:25
Edited

@Michael Offner My bucket is located in the us-east-1 region. When I set the access and secret keys in Application.cfc and used the us-east-1 region, it worked fine.

Application.cfc 
  component {
    this.name = "test";
    this.s3.accessKeyId = "accessKeyId";
    this.s3.awsSecretKey = "awsSecretKey";
    this.s3.host = 's3.us-east-1.amazonaws.com';
  }

test.cfm
  <cfscript>
    echo( fileExists( "s3://bucket-name/filename.txt" ) ); // Returns true
  </cfscript>

But when I used an unsupported region 'eu-east-1', I got the same "lucee.runtime.exp.NativeException: Unable to execute HTTP request: bucketname.s3.eu-east-1.amazonaws.com" issue and This issue only occurs in this specific scenario.

Application.cfc 
  component {
    this.name = "test";
    this.s3.accessKeyId = "accessKeyId";
    this.s3.awsSecretKey = "awsSecretKey";
    this.s3.host = 's3.eu-east-1.amazonaws.com';
  }

test.cfm
  <cfscript>
    echo( fileExists( "s3://bucket-name/filename.txt" ) ); // Returns false and got error on 
          application log ("lucee.runtime.exp. NativeException: Unable to execute HTTP request: bucketname.s3.eu-east-1.amazonaws.com" i)
  </cfscript>

Stack Trace:

lucee.runtime.exp.NativeException: Unable to execute HTTP request: bucket-name.s3.eu-east-1.amazonaws.com
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5470)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5417)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5411)
  at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:1048)
  at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:1054)
  at org.lucee.extension.resource.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:861)
  at org.lucee.extension.resource.s3.S3.getBucketRegion(S3.java:2064)
  at org.lucee.extension.resource.s3.S3.toRegion(S3.java:2090)
  at org.lucee.extension.resource.s3.S3.getAmazonS3(S3.java:2047)
  at org.lucee.extension.resource.s3.S3.get(S3.java:928)
  at org.lucee.extension.resource.s3.S3.is(S3.java:891)
  at org.lucee.extension.resource.s3.S3.exists(S3.java:886)
  at org.lucee.extension.resource.s3.S3Resource.exists(S3Resource.java:326)
  at lucee.commons.io.res.util.ResourceUtil.toResourceNotExisting(ResourceUtil.java:331)
  at lucee.runtime.op.Caster.toResource(Caster.java:4641)
  at lucee.runtime.functions.file.FileExists.call(FileExists.java:35)
  at lucee.runtime.functions.file.FileExists.call(FileExists.java:29)
  at test.test_cfm$cf.call(/test/test.cfm:3)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1056)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:948)
  at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:219)
  at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:44)
  at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2493)
  at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2478)
  at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2449)
  at lucee.runtime.engine.Request.exe(Request.java:45)
  at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1216)
  at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1162)
  at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
  at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
  at org.cfmlprojects.regexpathinfofilter.RegexPathInfoFilter.doFilter(RegexPathInfoFilter.java:47)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
  at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
  at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
  at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at runwar.Server$1.handleRequest(Server.java:510)
  at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:280)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
  at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
  at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
  at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
  at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:260)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
  at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
  at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
  at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
  at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
  at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
  at java.base/java.lang.Thread.run(Unknown Source)
 Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket-name.s3.eu-east-1.amazonaws.com
  ... 79 more
 Caused by: java.net.UnknownHostException: bucket-name.s3.eu-east-1.amazonaws.com
  at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source)
  at java.base/java.net.InetAddress.getAllByName0(Unknown Source)
  at java.base/java.net.InetAddress.getAllByName(Unknown Source)
  at java.base/java.net.InetAddress.getAllByName(Unknown Source)
  at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
  at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
  at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
  at jdk.internal.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.base/java.lang.reflect.Method.invoke(Unknown Source)
  at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
  at com.amazonaws.http.conn.$Proxy7.connect(Unknown Source)
  at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
  at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
  ... 77 more

Details
Assignee
Michael Offner
Reporter
Tom Chiverton
Priority
New
Labels
s3
New Issue warning screen
Before you create a new Issue, please post to the mailing list first https://dev.lucee.org
Once the issue has been verified, one of the Lucee team will ask you to file an issue
Parent
LDEV-4636 S3 regions support
Affects versions
5.4.1.8

Created 14 July 2023 at 12:17

Updated 5 October 2023 at 10:22

fileExists() doesn't like s3 locations

Description

Environment

Attachments

Activity

Tom Chiverton 5 October 2023 at 10:14
Edited

Pothys - MitrahSoft 12 September 2023 at 06:31

Michael Offner 11 September 2023 at 14:37

Michael Offner 11 September 2023 at 14:37

Pothys - MitrahSoft 8 September 2023 at 14:25
Edited

Details

Assignee

Reporter

Priority

Labels

New Issue warning screen

Parent

Affects versions

Flag notifications

Something's gone wrong

Something's gone wrong

fileExists() doesn't like s3 locations

Description

Environment

Attachments

Activity

Tom Chiverton 5 October 2023 at 10:14Edited

Pothys - MitrahSoft 12 September 2023 at 06:31

Michael Offner 11 September 2023 at 14:37

Michael Offner 11 September 2023 at 14:37

Pothys - MitrahSoft 8 September 2023 at 14:25Edited

Details

Assignee

Reporter

Priority

Labels

New Issue warning screen

Parent

Affects versions

Flag notifications

Something's gone wrong

Something's gone wrong

Tom Chiverton 5 October 2023 at 10:14
Edited

Pothys - MitrahSoft 8 September 2023 at 14:25
Edited