Okay, It's been almost a year and it's time to revisit this. When I left off I never got StoreSetMetaData() to work so we just starting using the AWS SDK. But I really wanted to get Lucee to do this natively so I could stop having to include a bunch of extra jar files in our .war deployment. So, we upgraded to Lucee5 and upgraded to the latest S3 extension. The function now exists meaning I no longer get a reference error when calling StoreSetMetaData(). However, I believe it's still not working correctly. Below is some example code:

/*** assuming we set our S3 credentials in Application.cfc and the bucket has the correct policy for downloads ***/

param name="url.filename" default="test.txt";
myFile = "s3://mybucket/#url.filename#";

fileWrite(myFile,"Sample s3 text");
StoreSetMetadata(myFile,{
"Content-Type":"text/plain",
"Content-Disposition":"download"
});

writeOutput('<a href="https://s3.amazonaws.com/mybucket/#url.filename#">download</a>');
writeDump(StoreGetMetadata(myFile));

So... when I run this code in ACF11 it downloads fine and when I log into S3 I can see permissions are changed. However, when I run this in Lucee5 the dump shows the meta data as being set but in S3 the permissions are not actually changed and when I try to download it just takes me to the page instead of downloading. Again, I'm not getting any errors anymore but it doesn't seem to actually be setting the Metadata on S3.

please reopen with more details when you still have this issue with the most recent version

this functions are supported with the S3 Extension that is installed by default with a fresh install or an update from 4.5, BUT this extensions are not installed when doing an update from an older Lucee 5 Beta installation, in taht case you have to install the S3 extension in the admin.
There are even testcases for this functions:
https://github.com/lucee/Lucee/blob/master/test/functions/StoreMetaData.cfc
that need to pass, otherwise travis would not made a build.

Maybe you have to be more percise with what is happeing when you call this functions...