Question

当前，我正在使用Java的S3 SDK中的AWS的TransferManager触发从SFTP服务器到S3的上传。我触发此上传的方式如下：

（伪代码...）

    @Autowired
    TransferManager transferManager;

    @Autowired
    SftpStreamFactory sftpStreamFactory;

    SftpStream sftpStream = sftpStreamFactory.createStream(filePath);
    ObjectMetadata objectMetadata = new ObjectMetadata();
    objectMetadata.setContentLength(sftpStream.getSizeBytes());
    PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, key, sftpStream.getStream(), objectMetadata);
    putObjectRequest.setGeneralProgressListener(new UploadBeginEndNotificationListener(uploadRequest, statusNotifier));

    transferManager.upload(putObjectRequest);

这是SftpStream的定义：

@AllArgsConstructor
public class SftpStreamFactory {

@Getter
@AllArgsConstructor
public static class SftpStream {
    private final long sizeBytes;
    private final InputStream stream;
}

private final SftpRemoteFileTemplate sftpTemplate;
private final SftpProperties sftpProperties;

public SftpStream createStream(Path relativePath) {
    return sftpTemplate.<SftpStream, ChannelSftp>executeWithClient(session -> createStream(session, relativePath));
}

SftpStream createStream(ChannelSftp channelSftp, Path relativePath) {

    String path = sftpProperties.getRoot().resolve(relativePath).toString();

    try {
        SftpATTRS fileAttrs = channelSftp.lstat(path);
        long size = fileAttrs.getSize();
        return new SftpStream(size, channelSftp.get(path));
    }
    catch (SftpException e) {
        throw new UncheckedIOException(new NestedIOException("SFTP Error", e));
    }
}

}

这种上传方法效果很好。但是，如果分段上传在中间暂停/取消/否则中止，我们希望从上次停止的地方继续，而不是重新开始。我们知道采用resumeUpload的TransferManagers PersistableUpload方法。

但是，在PersistableUpload的javadoc中，期望在构造函数中传递file路径，并稍后尝试从中创建File对象： https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/PersistableUpload.html

我们想知道的是，是否仍然可以在没有此文件对象的情况下继续上传，而这是我们无法从ChannelSftp中获得的？也就是说，我们可以从流而不是文件中恢复上传吗？还是我们不得不切换到使用低级s3 API来执行这样的简历。任何建议表示赞赏。

Answer 1

答案是，不，您不能在没有文件的情况下继续上传，但是有一种类似的解决方法：

＃终止之前成功暂停

当且仅当在发生任何中断之前尝试暂停连接

boolean forceCancel = true;
PauseResult<PersistableUpload> pauseResult = myUpload.tryPause(forceCancel);

将info to resume数据保存到文件

PersistableUpload persistableUpload = pauseResult.getInfoToResume();

File f = new File("UNIQUE-ID-FOR-UPLOADED-FILE"); //blob
if (!f.exists())
    f.createNewFile();
FileOutputStream fos = new FileOutputStream(f);

// Serialize the persistable upload to the file.
persistableUpload.serialize(fos);
fos.close();

稍后再继续

TransferManager tm = new TransferManager();
FileInputStream fis = new FileInputStream(new File("UNIQUE-ID-FOR-UPLOADED-FILE"));

// Deserialize PersistableUpload information from disk.
PersistableUpload persistableUpload = PersistableTransfer.deserializeFrom(fis);

// Call resumeUpload with PersistableUpload.
tm.resumeUpload(persistableUpload);

fis.close();

＃在某些情况下（例如JVM崩溃）保存流以维护文件

使用S3SyncProgressListener至TransferManager#upload保留所有更改并将数据序列化到磁盘。

transferManager.upload(putObjectRequest, new S3SyncProgressListener() {

    ExecutorService executor = Executors.newFixedThreadPool(1);

    @Override
    public void onPersistableTransfer(final PersistableTransfer persistableTransfer) {

       executor.submit(new Runnable() {
          @Override
          public void run() {
              try {
                  File f = new File("UNIQUE-ID-FOR-UPLOADED-FILE");
                  if (!f.exists()) {
                      f.createNewFile();
                  }
                  FileOutputStream fos = new FileOutputStream(f);
                  persistableTransfer.serialize(fos);
                  fos.close();
              } catch (IOException e) {
                  throw new RuntimeException("Unable to persist transfer to disk.", e);
              }
          }
       });
    }
});

希望有帮助。

使用AWS Java S3 SDK TransferManager从SFTP流恢复上传

1 个答案: