getObjectMetadata Action for S3 Bucket Policy
Attempting to create a connection via the S3 Connector to a bucket that doesn't allow the
getObjectMetadata action fails with the exception:
WARN [2018-05-04 08:06:58.026] [qtp897697267-14] (DataStore.java:211) - connection fails: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 8863CBFD8AD072D3) awstasks.com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 8863CBFD8AD072D3), S3 Extended Request ID: 2SWqbBj2yI/ie1Mm3KvkBlyd4kArdqfP+oDT/ej+/OAqPyBQBHwdG/uKWaHKlb9uO+QrCxTaijg= at awstasks.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1586) at awstasks.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1254) at awstasks.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)
- As of Datameer 6.4 there are two S3 Connectors are available - S3 and S3 Native. They have some differences in implementation.
- The S3 connector uses
getObjectMetadatato distinguish files and folders. This implementation is more extensive than most others clients as the implementation can be used as a Hadoop file system.
- The S3 Native connector doesn't use
getObjectMetadata. However, at the moment it only allows for exporting files into S3 buckets.
According to the tests against the bucket whose policy doesn't allow
- Attempting to create a connection via the S3 connector fails as
getObjectMetadatais not allowed by the bucket policy.
- Attempting to create a connection via the S3 Native connector is successful.
- Export jobs created on this S3 Native connector run fine.
You could export data to the bucket whose policy doesn't allow
getObjectMetadata via the S3 Native connector. The ability to import via this connector is expected in a future releases.