Question

我有以下方法对amazonS3进行分段上传：

public static void uploadToS3UserPath(String filePath, String bucket, String userPath) {
    ClientConfiguration config = new ClientConfiguration();
    config.setProxyHost("host");
    config.setProxyPort(3128);
    config.setProtocol(Protocol.HTTP);

    TransferManager tm = new TransferManager(new AmazonS3Client(credentials, config));
    TransferManagerConfiguration conf = new TransferManagerConfiguration();
    conf.setMinimumUploadPartSize(50 * 1024 * 1024); //use 50 megabytes parts;
    tm.setConfiguration(conf);

    PutObjectRequest req = new PutObjectRequest(bucket, userPath, new File(filePath));
    req.setCannedAcl(CannedAccessControlList.PublicRead);
    req.setStorageClass(StorageClass.ReducedRedundancy); //we do it to decrease storage costs
    Upload up = tm.upload(req);

    try {
        up.waitForCompletion();
        tm.shutdownNow();
        System.out.println("Upload completed successfully");
    } catch (AmazonClientException ex) {
        Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
        tm.shutdownNow();
    } catch (InterruptedException ex) {
        Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, ex.getLocalizedMessage(), ex);
    }
}

事情有时我会得到：

    com.amazonaws.AmazonClientException: Unable to complete transfer: null
            at
    com.amazonaws.services.s3.transfer.Transfer.unwrapExecutionException(Transfer.java:226)
            at
    com.amazonaws.services.s3.transfer.Transfer.rethrowExecutionException(Transfer.java:210)
            at
    com.amazonaws.services.s3.transfer.Transfer.waitForCompletion(Transfer.java:116)
            at
   xx.xxx.xxx.xxx.agent.Utilities.uploadToS3UserPath(Utilities.java:373)
            at
    xx.xxx.xxx.xxx.agent.AbstractAgent.uploadOutputToS3(AbstractAgent.java:312)
            at
    uk.org.infectogenomics.agent.AbstractAgent.uploadOutputToS3(AbstractAgent.java:305)
            at xx.xxx.xxx.xxx.hostEl.HostEl.run(HostEl.java:598)
            at xx.xxx.xxx.xxx.agent.Agent.main(Agent.java:98)
    Caused by: java.util.concurrent.RejectedExecutionException
            at
    java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1759)
            at
    java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:216)
            at
    java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:378)
            at
    com.amazonaws.services.s3.transfer.internal.UploadMonitor.reschedule(UploadMonitor.java:210)
            at
    com.amazonaws.services.s3.transfer.internal.UploadMonitor.upload(UploadMonitor.java:197)
            at
    com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:148)
            at
    com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:49)
            at
    java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
            at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
            at java.lang.Thread.run(Thread.java:619)

事情是我不知道为什么这只是随机上传。显然原因是RejectedExecutionException？可能是因为我过早地关闭了TransferManager吗？

Answer 1

可能是因为我过早地关闭了TransferManager吗？

这确实是这种行为的触发器，但根本原因是AWS SDK for Java中的一个错误，请参阅Eric Milas类似的问题Amazon AWS Java SDK TransferManager issue：

TransferManager包含一个UploadMonitor，而UploadMonitor有一个 static ScheduledExecutorService，名为timedThreadPoool。 TransferManager.shutdownNow（）调用静态方法 UploadMonitor.shutdownNow（）调用 timedThreadPoool.shutdownNow（）。下次再打电话给 UploadMonitor.reschedule（）是RejectedExecutionException 抛出。

His own answer引用了AWS团队对TransferManager Thread Issue 的回复，该回复确认了此错误，并且从Release 1.3.2开始实施了相应的修补程序：

Amazon S3 TransferManager class已更新，以解决两个错误：

从InputStream上传时可能出现数据损坏   在上载的某些部分遇到IO错误。干净   在调用shutdownNow（）时关闭计时器线程   TransferManager的实例

使用AWS TransferManager时RejectedExecutionException

1 个答案: