Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Support S3 CommonPrefixes for ListObjects

Description

I've been working on supporting custom host in s3 resource urls for https://luceeserver.atlassian.net/browse/LDEV-3594

https://github.com/lucee/extension-s3/pull/12

I've been using https://min.io but the resources.cfc test is failing, checking for directoryExists

minio has a nice verbose log facility with it's mc admin util (see PR for screenshots and deets)

localhost:9000 [REQUEST s3.PutObject] [2021-07-22T10:30:52:000] [Client IP: 127.0.0.1] localhost:9000 PUT /lucee-res-c8d5b2cb4244fc2218deca5d33fd245d/test1/ localhost:9000 [RESPONSE] [2021-07-22T10:30:52:000] [ Duration 1.333ms ↑ 84 B ↓ 230 B ] localhost:9000 [REQUEST s3.ListObjectsV1] [2021-07-22T10:30:52:000] [Client IP: 127.0.0.1] localhost:9000 GET /lucee-res-c8d5b2cb4244fc2218deca5d33fd245d/?max-keys=10&prefix=test1%2FfileSub.txt&delimiter=%2C localhost:9000 [RESPONSE] [2021-07-22T10:30:52:000] [ Duration 486µs ↑ 61 B ↓ 528 B ] localhost:9000 <?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>lucee-res-c8d5b2cb4244fc2218deca5d33fd245d</Name><Prefix>test1/fileSub.txt</Prefix> <Marker></Marker><MaxKeys>10</MaxKeys><Delimiter>,</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult> localhost:9000 [REQUEST s3.ListObjectsV1] [2021-07-22T10:30:52:000] [Client IP: 127.0.0.1] localhost:9000 GET /lucee-res-c8d5b2cb4244fc2218deca5d33fd245d/?max-keys=10&prefix=test1&delimiter=%2C localhost:9000 [RESPONSE] [2021-07-22T10:30:52:000] [ Duration 500µs ↑ 61 B ↓ 572 B ] <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>lucee-res-c8d5b2cb4244fc2218deca5d33fd245d</Name> <Prefix>test1</Prefix><Marker></Marker><MaxKeys>10</MaxKeys><Delimiter>,</Delimiter> <IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>test1/</Prefix></CommonPrefixes></ListBucketResult> localhost:9000 [REQUEST s3.ListObjectsV1] [2021-07-22T10:30:52:000] [Client IP: 127.0.0.1] localhost:9000 GET /lucee-res-c8d5b2cb4244fc2218deca5d33fd245d/?max-keys=1000 localhost:9000 [RESPONSE] [2021-07-22T10:30:52:000] [ Duration 1.002ms ↑ 61 B ↓ 1.1 KiB ] localhost:9000 <?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>lucee-res-c8d5b2cb4244fc2218deca5d33fd245d</Name> <Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter></Delimiter><IsTruncated>false</IsTruncated> <Contents> <Key>dirPlaceholder.txt</Key> <LastModified>2021-07-22T10:30:52.316Z</LastModified> <ETag>&#34;5d41402abc4b2a76b9719d911017c592&#34;</ETag><Size>5</Size><Owner> <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID> <DisplayName>minio</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents> <Contents><Key>test1/</Key> <LastModified>2021-07-22T10:30:52.374Z</LastModified> <ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner> <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID> <DisplayName>minio</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult> localhost:9000 [REQUEST s3.ListObjectsV1] [2021-07-22T10:30:52:000] [Client IP: 127.0.0.1] localhost:9000 GET /lucee-res-c8d5b2cb4244fc2218deca5d33fd245d/?max-keys=1000 localhost:9000 [RESPONSE] [2021-07-22T10:30:52:000] [ Duration 1.154ms ↑ 61 B ↓ 1.1 KiB ] <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>lucee-res-c8d5b2cb4244fc2218deca5d33fd245d</Name> <Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter></Delimiter><IsTruncated>false</IsTruncated> <Contents><Key>dirPlaceholder.txt</Key> <LastModified>2021-07-22T10:30:52.316Z</LastModified> <ETag>&#34;5d41402abc4b2a76b9719d911017c592&#34;</ETag><Size>5</Size><Owner> <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID> <DisplayName>minio</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents> <Contents><Key>test1/</Key> <LastModified>2021-07-22T10:30:52.374Z</LastModified> <ETag>&#34;d41d8cd98f00b204e9800998ecf8427e&#34;</ETag><Size>0</Size><Owner> <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID> <DisplayName>minio</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>

the s3 ext does checking using a prefix, minio doesn't directly match the prefix, but it still does return it as in the commonPrefix section (3rd request in the above logs)

<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>lucee-res-c8d5b2cb4244fc2218deca5d33fd245d</Name> <Prefix>test1</Prefix> <Marker></Marker> <MaxKeys>10</MaxKeys> <Delimiter>,</Delimiter> <IsTruncated>false</IsTruncated> <CommonPrefixes><Prefix>test1/</Prefix></CommonPrefixes> </ListBucketResult>

Cyberduck uses this approach https://github.com/minio/minio/issues/8148#issuecomment-526502108

relates to

Activity

Show:

Zac Spitzer 5 August 2021 at 15:37

Zac Spitzer 22 July 2021 at 15:15
Edited

I’ve added in some code to support this here

https://github.com/lucee/extension-s3/commit/911af22b5fac4b6abe172be68c64b49f509e7f24

but then deleting an object uses a different code path and fails

https://github.com/lucee/extension-s3/runs/3135476640?check_suite_focus=true#step:10:153

[java] [script] lucee.runtime.exp.NativeException: can't delete file/directory lucee-res-71031a2f03455ebf2cc5b6304bb99788/test1/test2/, file/directory does not exist [java] [script] [java] [script] at org.lucee.extension.resource.s3.S3.delete(S3.java:696) [java] [script] at org.lucee.extension.resource.s3.S3Resource.remove(S3Resource.java:391)

Details

Assignee

Reporter

Labels

New Issue warning screen

Before you create a new Issue, please post to the mailing list first https://dev.lucee.org

Once the issue has been verified, one of the Lucee team will ask you to file an issue

Priority

Created 22 July 2021 at 15:08
Updated 5 August 2021 at 15:37

Flag notifications