Question

我正在尝试使用最新的aws-sdk java库从GCS存储区列出对象。

请参阅此处的代码段

ClientConfiguration clientConfiguration = new ClientConfiguration();
  // Solution is update the Signer Version. 
  clientConfiguration.setSignerOverride("S3SignerType"); 

  AWSCredentials awsCredentials = new BasicAWSCredentials("XXX","XXX");
  AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard()
    .withCredentials(new  AWSStaticCredentialsProvider(awsCredentials))
    .withClientConfiguration(clientConfiguration)
    .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("https://storage.googleapis.com","Multi-Regional")).build();

  String bucketName = "bucket_name";

  // List Objects 
  amazonS3Client.listObject(bucketName);

但是接收到无效的参数。请参阅错误和DEBUG日志此外，我可以使用上面的amazonS3Client getObjects和putObjects。有什么想法吗？

2017-11-13 17:54:15,360 [main] DEBUG com.amazonaws.request - Sending Request: GET https://bucket_name.storage.googleapis.com / Parameters: ({"encoding-type":["url"]}Headers: (User-Agent: aws-sdk-java/1.11.158 Linux/4.10.0-38-generic Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11/1.8.0_131, amz-sdk-invocation-id: 121cd76e-1374-4e5d-9e68-be22ee2ad17a, Content-Type: application/octet-stream, ) 
2017-11-13 17:54:16,316 [main] DEBUG com.amazonaws.request - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Invalid argument. (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: null), S3 Extended Request ID: null
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Invalid argument. (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: null), S3 Extended Request ID: null
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
        at

Answer 1

S3有一个名为“encodingtype”的对象列表调用参数，当设置为“url”时，使用URL编码对无法在XML 1.0中本地呈现的字符进行编码。客户端库似乎正在使用该标志。我不相信GCS的XML API支持该参数，因此调用将因InvalidArgument错误而失败。

您可以使用ListObjectRequest并致电setEncodingType(null)来避免这种情况，但我还没有尝试过。

Answer 2

我在同一条船上但是使用了aws-sdk php库。我将问题跟踪到S3Client类的构造函数，其中添加了一堆中间件，其中一个是设置编码类型。注释掉这一行可以让我成功地执行请求，所以我知道我在正确的轨道上。

$stack->appendSign(PutObjectUrlMiddleware::wrap(), 's3.put_object_url');
$stack->appendSign(PermanentRedirectMiddleware::wrap(), 's3.permanent_redirect');
$stack->appendInit(Middleware::sourceFile($this->getApi()), 's3.source_file');
$stack->appendInit($this->getSaveAsParameter(), 's3.save_as');
$stack->appendInit($this->getLocationConstraintMiddleware(), 's3.location');
// $stack->appendInit($this->getEncodingTypeMiddleware(), 's3.auto_encode');
$stack->appendInit($this->getHeadObjectMiddleware(), 's3.head_object');

查看java的aws-sdk是否至少为您提供了一些有条件地应用中间件的选项，但似乎它与php版本没有任何关系。

Answer 3

正如Brandon和Pez所注意到的那样，GCS不喜欢S3Client本机添加的EncodingType标头。

幸运的是，有一种简单的方法可以使用一块中间件来解决。这避免了通常应避免对供应商文件夹的编辑。

let array1 =[
   { "id": 1, "name": "a"},
   { "id": 2, "name": "b"},
   { "id": 3, "name": "c"},
   { "id": 4, "name": "d"},
 ];

let array2 =[
   { "id": 1, "name": "a"},
   { "id": 2, "name": "b"},
   { "id": 3, "name": "c"},
   { "token": 4, "name": "d"}, 
];

result = array1.filter(x => array2.find(y => x.id !== (y.id || y.token 
         )));

另请参阅：https://blog.bandhosting.nl/blog/avoid-listobjects-invalid-query-parameter-s-encoding-type-errors-when-using-s3-with-google-cloud-storage

Answer 4

我遇到了同样的问题，发现Amazon Java API在S3调用期间包含多个挂钩，可用于从HTTP请求中删除编码类型。

{
  return AmazonS3ClientBuilder.standard()//
       .withCredentials(getAwsCredentialsProvider())//
       .withEndpointConfiguration(getEndpointConfiguration(regionName))//
       .withRequestHandlers(new GoogleRequestHandler()) // IMPORTANT PART
       .build();
}

public class GoogleRequestHandler extends RequestHandler2 {

    @Override
    public void beforeRequest(Request<?> request) {
        // google does not support the encoding-type parameter so just remove it from the request
        // This appears to be only true for ListObjects
        if (request.getOriginalRequest() instanceof ListObjectsRequest) {
            Map<String, List<String>> params = request.getParameters();
            params.remove("encoding-type");
        }
    }
}

有关更多文档，请参见RequestHandler2。

无法从GCS存储桶中列出对象

4 个答案: