如何提高弹簧批处理的速度?

时间:2018-07-05 09:30:57

标签: spring-batch

我的代码应该从表A中获取where子句,并将它们逐个应用于表REPORTS的记录,并将每个结果存储在表OUTPUTREPORT中。表A具有427条记录,而REPORTS具有约3900000条记录,大约需要4个小时。

这是我的工作,步骤和阅读器代码:

@Bean
    public Job QueryJob() throws ClassNotFoundException, SQLException {
        Connection dbConnection = dataSource.getConnection();
        String selectSQL = "select WHERE_CONDITION from A";
        PreparedStatement preparedStatement = dbConnection.prepareStatement(selectSQL);
        ResultSet rs = preparedStatement.executeQuery();
        List<Step> steps = new ArrayList<>();
        while (rs.next()) {
            Step step = createStep(rs.getString("WHERE_CONDITION"));
            steps.add(step);
        }
        rs.close();
        preparedStatement.close();
        dbConnection.close();

        return jobBuilderFactory.get("QueryJob")
                .start(createParallelFlow(steps)).end().build();
    }

    int i = 1;

    private Step createStep(String query) {

        return stepBuilderFactory.get("convertStepFor" + i++)
                .<Report, Report> chunk(5000).reader(createQueryReader(query))
                .writer(writer()).build();
    }

    private Flow createParallelFlow(List<Step> steps) {
        SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
        taskExecutor.setConcurrencyLimit(maxThreads);

        List<Flow> flows = new ArrayList<>();

        Iterator<Step> i = steps.iterator();
        while (i.hasNext()) {
            Step x = i.next();
            Flow flow = new FlowBuilder<Flow>("flow_" + x.getName()).start(x)
                    .build();
            if (flow != null)
                flows.add(flow);
        }

        return new FlowBuilder<SimpleFlow>("parallelStepsFlow")
                .split(taskExecutor).add(flows.toArray(new Flow[flows.size()]))
                .build();
    }

    public JdbcCursorItemReader<Report> createQueryReader(String query) {
        JdbcCursorItemReader<Report> reader = new JdbcCursorItemReader<>();
        reader.setDataSource(dataSource);
        reader.setSql("select * from reports" + query);
        reader.setRowMapper(new ReportRowMapper());
        return reader;
    }

我是Spring Batch的新手。如何提高性能?

0 个答案:

没有答案