卡夫卡生产商优化

Question

我有大量的数据记录要使用批处理和压缩为Kafka生成。

我已经尝试了许多生产者的属性来修复异常/超时，同时将数据发送给代理。

Update

以下是我的属性

const float timeToCountAsHeldDown = 0.3f;
float pressTimer = 0;

IEnumerator moveChecker()
{
    while (true)
    {
        //Check when the D key is pressed
        if (Input.GetKeyDown(KeyCode.D))
        {
            //Continue to check if it is still heldown and keep counting the how long
            while (Input.GetKey(KeyCode.D))
            {
                //Start incrementing timer
                pressTimer += Time.deltaTime;

                //Check if this counts as being "Held Down"
                if (pressTimer > timeToCountAsHeldDown)
                {
                    //It a "key held down", call the OnKeyHeldDown function and wait for it to return
                    yield return OnKeyHeldDown();
                    //No need to continue checking for Input.GetKey(KeyCode.D). Break out of this whule loop
                    break;
                }

                //Wait for a frame
                yield return null;
            }
        }


        //Check if "D" key is released 
        if (Input.GetKeyUp(KeyCode.D))
        {
            //Check if we have not not reached the timer then it is only a key press
            if (pressTimer < timeToCountAsHeldDown)
            {
                //It just a key press, call the OnKeyPressedOnly function and wait for it to return
                yield return OnKeyPressedOnly();
            }

            //Reset timer to 0 for the next key press
            pressTimer = 0f;
        }

        //Wait for a frame
        yield return null;
    }
}

IEnumerator OnKeyPressedOnly()
{
    Debug.Log("D key was only Pressed");

    //Move 1 unit only
    transform.position += new Vector3(1, 0, 0);
    yield return null;
}


IEnumerator OnKeyHeldDown()
{
    Debug.LogWarning("D key is Held Down");

    //Don't move for 0.5 seconds 
    yield return new WaitForSeconds(0.5f);

    //Move 1 unit every frame until edge detection is reached!
    while (!CheckIsValidPosition())
    {
        transform.position += new Vector3(1, 0, 0);

        //Wait for a frame
        yield return null;
    }
}

肯定会增加超时和其他大值会影响我的制作人的吞吐量。

我怀疑哪一个是好的。 1.通过增加batch.size，其他超时来优化batch.size。 2.使用重试计数+超时属性的最佳值。

例外

My one record size is 5486 Bytes(total count is around 20000).