Finished loading more work items at the top of the list.

100 of

fileExists() doesn't like s3 locations

Description

var s3Path='s3://'&variables.s3Root&'/'&arrayToList(arguments.path,'/')&'/'&arguments.name;
if( not FileExists( s3Path) ){

The message is
Message: could not find a matching region for [EU], valid region names are [us-gov-west-1,us-gov-east-1,us-east-1,us-east-2,us-west-1,us-west-2,eu-west-1,eu-west-2,eu-west-3,eu-central-1,eu-central-2,eu-north-1,eu-south-1,eu-south-2,ap-east-1,ap-south-1,ap-south-2,ap-southeast-1,ap-southeast-2,ap-southeast-3,ap-southeast-4,ap-northeast-1,ap-northeast-2,ap-northeast-3,sa-east-1,cn-north-1,cn-northwest-1,ca-central-1,me-central-1,me-south-1,af-south-1,us-iso-east-1,us-isob-east-1,us-iso-west-1]

I’ve tried adding export AWS_REGION=eu-west-1 to /etc/init.d/lucce_ctl but no change.

This is the stack:

lucee.runtime.exp.NativeException: could not find a matching region for [EU], valid region names are [us-gov-west-1,us-gov-east-1,us-east-1,us-east-2,us-west-1,us-west-2,eu-west-1,eu-west-2,eu-west-3,eu-central-1,eu-central-2,eu-north-1,eu-south-1,eu-south-2,ap-east-1,ap-south-1,ap-south-2,ap-southeast-1,ap-southeast-2,ap-southeast-3,ap-southeast-4,ap-northeast-1,ap-northeast-2,ap-northeast-3,sa-east-1,cn-north-1,cn-northwest-1,ca-central-1,me-central-1,me-south-1,af-south-1,us-iso-east-1,us-isob-east-1,us-iso-west-1] at org.lucee.extension.resource.s3.S3.toRegions(S3.java:1991) at org.lucee.extension.resource.s3.S3.getBucketRegion(S3.java:1929) at org.lucee.extension.resource.s3.S3.toRegions(S3.java:1955) at org.lucee.extension.resource.s3.S3.getAmazonS3(S3.java:1914) at org.lucee.extension.resource.s3.S3.get(S3.java:869) at org.lucee.extension.resource.s3.S3.is(S3.java:834) at org.lucee.extension.resource.s3.S3.exists(S3.java:829) at org.lucee.extension.resource.s3.S3Resource.exists(S3Resource.java:326) at lucee.commons.io.res.util.ResourceUtil.toResourceNotExisting(ResourceUtil.java:331) at lucee.runtime.op.Caster.toResource(Caster.java:4641) at lucee.runtime.functions.file.FileExists.call(FileExists.java:35) at lucee.runtime.functions.file.FileExists.call(FileExists.java:29) at com.ev.business.s3_cfc$cf.udfCall(/com/ev/business/S3.cfc:57) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:350) at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223) at lucee.runtime.type.scope.UndefinedImpl.call(UndefinedImpl.java:786) at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787) at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1775) at com.ev.business.s3_cfc$cf.udfCall(/com/ev/business/S3.cfc:35) at

Environment

None

Attachments

01 Sept 2023, 02:19 pm
01 Sept 2023, 02:19 pm

Details

Assignee

Michael Offner

Reporter

Tom Chiverton

Priority

New

Labels

New Issue warning screen

Before you create a new Issue, please post to the mailing list first https://dev.lucee.org

Once the issue has been verified, one of the Lucee team will ask you to file an issue

Parent

LDEV-4636 S3 regions support

Affects versions

5.4.1.8

Created 14 July 2023 at 12:17

Updated 5 October 2023 at 10:22

Activity

Tom Chiverton
5 October 2023 at 10:14
(edited)

Still broken with the latest S3 extension the output from fileExists() is false for a file that does exist

S3 extension 2.0.1.15, Lucee 5.4.3.2
Java OpenJDK 64-Bit Server VM Corretto-17.0.8.7.1 (build 17.0.8+7-LTS, mixed mode, sharing)

Still logs Unable to execute HTTP request: (my bucket name).s3.eu...

I added <cfset this.s3.host = 'http://s3.eu-west-1.amazonaws.com '/> to Application.cfc (this is where the bucket is) and it now logs

"ERROR","http-nio-8888-exec-10","10/05/2023","10:10:40","s3","The authorization header is malformed; the region 'us-
east-1' is wrong; expecting 'eu-west-1' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalfor
med; Request ID: FKNWQ4VAFVGH1XXK; S3 Extended Request ID: 8FJmOldu0sOk6KMyDB7D7gzk3BgOSaBZgKu7cKeCWJkJlFR7LyobKKmUK
PVsDaajqQHACNSLasVKgOiukpYwtQ==; Proxy: null);com.amazonaws.services.s3.model.AmazonS3Exception: The authorization h
eader is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' (Service: Amazon S3; Status Code: 400; Er
ror Code: AuthorizationHeaderMalformed; Request ID: FKNWQ4VAFVGH1XXK; S3 Extended Request ID: 8FJmOldu0sOk6KMyDB7D7g
zk3BgOSaBZgKu7cKeCWJkJlFR7LyobKKmUKPVsDaajqQHACNSLasVKgOiukpYwtQ==; Proxy: null), S3 Extended Request ID: 8FJmOldu0s
Ok6KMyDB7D7gzk3BgOSaBZgKu7cKeCWJkJlFR7LyobKKmUKPVsDaajqQHACNSLasVKgOiukpYwtQ==
…
       at org.lucee.extension.resource.s3.AmazonS3Client.listObjects(AmazonS3Client.java:1484)
       at org.lucee.extension.resource.s3.S3.get(S3.java:1079)
       at org.lucee.extension.resource.s3.S3.is(S3.java:1031)
       at org.lucee.extension.resource.s3.S3.exists(S3.java:1026)
       at org.lucee.extension.resource.s3.S3Resource.exists(S3Resource.java:328)

Pothys - MitrahSoft
12 September 2023 at 06:31

yes @Michael Offner , From my perspective, I believe it's not a bug but rather expected behavior.

Michael Offner
11 September 2023 at 14:37

i think the underlaying issue itself is solved, this is simply expected behaviour, would you agree @Pothys - MitrahSoft ?

Michael Offner
11 September 2023 at 14:37

yeah, in my opinion that is not a bug, if the region is wrong the provider will complain, best is always not to define any region at all with the host name.

Pothys - MitrahSoft
8 September 2023 at 14:25
(edited)

@Michael Offner My bucket is located in the us-east-1 region. When I set the access and secret keys in Application.cfc and used the us-east-1 region, it worked fine.

Application.cfc 
  component {
    this.name = "test";
    this.s3.accessKeyId = "accessKeyId";
    this.s3.awsSecretKey = "awsSecretKey";
    this.s3.host = 's3.us-east-1.amazonaws.com';
  }

test.cfm
  <cfscript>
    echo( fileExists( "s3://bucket-name/filename.txt" ) ); // Returns true
  </cfscript>

But when I used an unsupported region 'eu-east-1', I got the same "lucee.runtime.exp.NativeException: Unable to execute HTTP request: bucketname.s3.eu-east-1.amazonaws.com" issue and This issue only occurs in this specific scenario.

Application.cfc 
  component {
    this.name = "test";
    this.s3.accessKeyId = "accessKeyId";
    this.s3.awsSecretKey = "awsSecretKey";
    this.s3.host = 's3.eu-east-1.amazonaws.com';
  }

test.cfm
  <cfscript>
    echo( fileExists( "s3://bucket-name/filename.txt" ) ); // Returns false and got error on 
          application log ("lucee.runtime.exp. NativeException: Unable to execute HTTP request: bucketname.s3.eu-east-1.amazonaws.com" i)
  </cfscript>

Stack Trace:

lucee.runtime.exp.NativeException: Unable to execute HTTP request: bucket-name.s3.eu-east-1.amazonaws.com
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5470)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5417)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5411)
  at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:1048)
  at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:1054)
  at org.lucee.extension.resource.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:861)
  at org.lucee.extension.resource.s3.S3.getBucketRegion(S3.java:2064)
  at org.lucee.extension.resource.s3.S3.toRegion(S3.java:2090)
  at org.lucee.extension.resource.s3.S3.getAmazonS3(S3.java:2047)
  at org.lucee.extension.resource.s3.S3.get(S3.java:928)
  at org.lucee.extension.resource.s3.S3.is(S3.java:891)
  at org.lucee.extension.resource.s3.S3.exists(S3.java:886)
  at org.lucee.extension.resource.s3.S3Resource.exists(S3Resource.java:326)
  at lucee.commons.io.res.util.ResourceUtil.toResourceNotExisting(ResourceUtil.java:331)
  at lucee.runtime.op.Caster.toResource(Caster.java:4641)
  at lucee.runtime.functions.file.FileExists.call(FileExists.java:35)
  at lucee.runtime.functions.file.FileExists.call(FileExists.java:29)
  at test.test_cfm$cf.call(/test/test.cfm:3)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1056)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:948)
  at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:219)
  at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:44)
  at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2493)
  at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2478)
  at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2449)
  at lucee.runtime.engine.Request.exe(Request.java:45)
  at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1216)
  at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1162)
  at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
  at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
  at org.cfmlprojects.regexpathinfofilter.RegexPathInfoFilter.doFilter(RegexPathInfoFilter.java:47)
  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
  at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
  at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
  at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at runwar.Server$1.handleRequest(Server.java:510)
  at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:280)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
  at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
  at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
  at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
  at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:260)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
  at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
  at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
  at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
  at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
  at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
  at java.base/java.lang.Thread.run(Unknown Source)
 Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: bucket-name.s3.eu-east-1.amazonaws.com
  ... 79 more
 Caused by: java.net.UnknownHostException: bucket-name.s3.eu-east-1.amazonaws.com
  at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source)
  at java.base/java.net.InetAddress.getAllByName0(Unknown Source)
  at java.base/java.net.InetAddress.getAllByName(Unknown Source)
  at java.base/java.net.InetAddress.getAllByName(Unknown Source)
  at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
  at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
  at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
  at jdk.internal.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.base/java.lang.reflect.Method.invoke(Unknown Source)
  at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
  at com.amazonaws.http.conn.$Proxy7.connect(Unknown Source)
  at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
  at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
  ... 77 more

fileExists() doesn't like s3 locations

Description

Environment

Attachments

Details

Assignee

Reporter

Priority

Labels

New Issue warning screen

Parent

Affects versions

Activity

Tom Chiverton
5 October 2023 at 10:14
(edited)

Pothys - MitrahSoft
12 September 2023 at 06:31

Michael Offner
11 September 2023 at 14:37

Michael Offner
11 September 2023 at 14:37

Pothys - MitrahSoft
8 September 2023 at 14:25
(edited)

Flag notifications

Something's gone wrong

Something's gone wrong

Something's gone wrong

fileExists() doesn't like s3 locations

Description

Environment

Attachments

Details

Assignee

Reporter

Priority

Labels

New Issue warning screen

Parent

Affects versions

Activity

Tom Chiverton 5 October 2023 at 10:14(edited)

Pothys - MitrahSoft 12 September 2023 at 06:31

Michael Offner 11 September 2023 at 14:37

Michael Offner 11 September 2023 at 14:37

Pothys - MitrahSoft 8 September 2023 at 14:25(edited)

Flag notifications

Something's gone wrong

Something's gone wrong

Something's gone wrong

Tom Chiverton
5 October 2023 at 10:14
(edited)

Pothys - MitrahSoft
12 September 2023 at 06:31

Michael Offner
11 September 2023 at 14:37

Michael Offner
11 September 2023 at 14:37

Pothys - MitrahSoft
8 September 2023 at 14:25
(edited)