Question

我的情况是，只要文件被修改，就只能将一个文件从AWS S3存储桶传输到EC2实例一次。我使用以下配置，并在服务器启动时手动启动了适配器。

问题是服务器启动时重复执行5到6次。看起来正在执行不同的线程。我能够在日志中看到不同的任务执行器，不知道它是轮询程序问题还是适配器问题。

我正在使用服务激活器根据S3位置中的文件更改执行其他操作。

注意：此问题在启动时仅发生一次。可以进一步进行文件修改。

配置：

<bean id="s3SessionFactory" 
            class="org.springframework.integration.aws.support.S3SessionFactory"></bean>
        <bean id="acceptOnceFilter"
            class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />
        <task:executor id="s3PollingExecutor" pool-size="1" queue-capacity="10" />
        <integration:channel id="s3FilesChannel"/>
        <int-aws:s3-inbound-channel-adapter id="s3FileInbound"
                     channel="s3FilesChannel" 
                     session-factory="s3SessionFactory" 
                     auto-create-local-directory="false"
                     delete-remote-files="false" 
                     preserve-timestamp="true"
                     filter="acceptOnceFilter"
                     local-directory="local_directory"
                     auto-startup="false" 
                     remote-directory="s3_bucket">
            <integration:poller id="s3FilesChannelPoller" 
                                fixed-rate="1000" 
                                max-messages-per-poll="1" time-unit="MILLISECONDS" 
                                task-executor="s3PollingExecutor">
            </integration:poller>
        </int-aws:s3-inbound-channel-adapter>
        <integration:service-activator id="s3FilesChannelWatcher" 
                                       input-channel="s3FilesChannel" 
                                       output-channel="nullChannel"
                                       ref="configurationFileWatcher" 
                                       method="getConfigurationFileWatcher">
        </integration:service-activator>

如您所建议，我已经尝试了以下方法。

<bean id="acceptOnceFilterRegion"
      class="cS3FileFilterOnLastModifiedTime">
    <constructor-arg index="0" ref="metaDataStoreRegion"/>
    <constructor-arg index="1" value="*"/>
</bean>

添加了用于检查上次修改时间的逻辑

import org.springframework.integration.aws.support.filters.S3PersistentAcceptOnceFileListFilter;
import org.springframework.integration.metadata.ConcurrentMetadataStore;
import com.amazonaws.services.s3.model.S3ObjectSummary;
public class S3FileFilterOnLastModifiedTime extends S3PersistentAcceptOnceFileListFilter {

    Long delayTime = 1000L;

    public S3FileFilterOnLastModifiedTime(ConcurrentMetadataStore store, String prefix) {
        super(store, prefix);
    }

    @Override
    public boolean accept(S3ObjectSummary file) {
        long lastModified = modified(file);
        long currentTime = System.currentTimeMillis();
        long timeDifference = currentTime - lastModified;
        return timeDifference > delayTime;
    }   
}

仍然没有希望.log像这样.......

[INFO ] 2018-10-11 11:22:10,888 [s3PollingExecutor-1] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,892 [s3PollingExecutor-2] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,892 [s3PollingExecutor-3] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,893 [s3PollingExecutor-4] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,893 [s3PollingExecutor-5] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,894 [s3PollingExecutor-6] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully

Answer 1

我认为您丢失了s3_bucket在启动时不是空的事实，而<int-aws:s3-inbound-channel-adapter>在启动时会拾取所有文件的事实。这就是您看到的几项任务的方式：每个文件。

如果您真的只担心该存储桶中的新文件，则需要考虑不要使用内存AcceptOnceFileListFilter，而应基于共享的MetadataStore实现切换到某些持久性实现。为此，S3PersistentAcceptOnceFileListFilter和spring-integration-aws中有一个DynamoDbMetadataStore，可将过滤结果持久保存到AWS上的DynamoDb：https://github.com/spring-projects/spring-integration-aws#metadata-store-for-amazon-dynamodb

s3-inbound-channel-adapter中的轮询器/适配器无法正常工作

1 个答案: