Can AWS step function executes more than 25000 times?

时间:2019-01-18 18:43:09

标签: amazon-web-services aws-lambda aws-sdk aws-step-functions

I am currently evaluating AWS state machine that can process single document. The state machine would take 5-10 mins to process a single document.

{
  "Comment":"Process document",
  "StartAt": "InitialState",
  "States": {
          //the document goes through multiple states here
  }
}

The C# code invokes the state machine by passing some json for each document. Something like

      // max 100 documents
      public Task Process(IEnumerable<Document> documents)
      {   
          var amazonStepFunctionsConfig = new AmazonStepFunctionsConfig { RegionEndpoint = RegionEndpoint.USWest2 };
          using (var amazonStepFunctionsClient = new AmazonStepFunctionsClient(awsAccessKeyId, awsSecretAccessKey, amazonStepFunctionsConfig))
          {
            foreach(var document in documents)
            {
                var jsonData1 = JsonConvert.SerializeObject(document);
                var startExecutionRequest = new StartExecutionRequest
                {
                  Input = jsonData1,
                  Name = document.Id, 
                  StateMachineArn = "arn:aws:states:us-west-2:<SomeNumber>:stateMachine:ProcessDocument"
                };
                var taskStartExecutionResponse = await amazonStepFunctionsClient.StartExecutionAsync(startExecutionRequest);                
            }
          }
      }

We process the documents in batch of 100. So in above loop the max number of documents will be 100. However we process thousands of documents weekly (25000+).

As per the AWS documentation Maximum execution history size is 25,000 events. If the execution history reaches this limit the execution will fail.

Does that mean we can not execute a single state machine more than 25000 times? Why execution of state machine should depend on its history, why cant AWS just purge history?

I know there is a way to continue as new execution but I am just trying to understand the history limit and its relation to state machine execution, and is my understanding is correct?

Update 1
I don't think this is duplicate question. I am trying find if my understanding of history limit is correct? Why history has anything to do with number of times state machine can execute? When state machine executes, it creates history record, if history records goes more 25000+, then purge them or archive them. Why would AWS stop execution of state machine. That does not make sense.

So question, Can single state machine (unique arn) execute more than 25000+ times in loop? if i have to create new state machine (after 25000 executions) wouldn't that state machine will have different arn?

Also if i had to follow linked SO post where would i get current number of executions? Also he is looping with-in the step function, while i am calling step function with-in the loop

Update 2
So just for testing i created the following state machine

{
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Pass",
      "Result": "Hello World!",
      "End": true
    }
  }
}

and executed it 26000 times with NO failure

    public static async Task Main(string[] args)
    {
        AmazonStepFunctionsClient client = new AmazonStepFunctionsClient("my key", "my secret key", Amazon.RegionEndpoint.USWest2);
        for (int i = 1; i <= 26000; i++)
        {
            var startExecutionRequest = new StartExecutionRequest
            {
                Input = JsonConvert.SerializeObject(new { }),
                Name = i.ToString(),
                StateMachineArn = "arn:aws:states:us-west-2:xxxxx:stateMachine:MySimpleStateMachine"
            };

            var response = await client.StartExecutionAsync(startExecutionRequest);
        }

        Console.WriteLine("Press any key to continue");
        Console.ReadKey();
    }

and on AWS Console i am able to pull the history for all 26000 executions enter image description here

So i am not sure exactly what does it mean by Maximum execution history size is 25,000 events

1 个答案:

答案 0 :(得分:0)

我认为您做对了。状态机执行历史记录的上限为25,000。您已经测试了26,000个状态机执行。 State Machine执行限制为1,000,000个开放执行。

状态机最多可以运行1年,并且在此期间其执行历史记录不应超过25,000。

希望有帮助。