Question

我使用以下函数将多个记录插入数据库。此函数使用单个查询将记录添加到mysql数据库中。

function addHostBulk($value, $label, $group, $type, $user)
{
    $res = array("type" => intval($type), "value" => $value, "label" => $label, "group_id" => $group, "owner" => intval($user['id']));

        $hosts = $common->cidrToRange($res['value'], $user); //Returns array with all hosts from this CIDR range /array('hosts' => array('1.2.3.4', '1.2.3.5.', ...)/
        $final_count = count($hosts['hosts']);

        $this->db->beginTransaction();
        $query = ("INSERT INTO hosts (`id`, `type`, `value`, `label`, `group_id`, `owner`) VALUES ");
        $qPart = array_fill(0, count($hosts['hosts']), "(?, ?, ?, ?, ?, ?)");
        $query .=  implode(",",$qPart);

        $statement = $this->db->prepare($query);

        $i = 1;
        foreach($hosts['hosts'] as $host){
                $statement->bindParam($i++, $common->uuid(), PDO::PARAM_STR);
                $statement->bindParam($i++, $res['type'], PDO::PARAM_STR);
                $statement->bindParam($i++, $host, PDO::PARAM_STR);
                $statement->bindParam($i++, $res['label'], PDO::PARAM_STR);
                $statement->bindParam($i++, $res['group_id'], PDO::PARAM_STR);
                $statement->bindParam($i++, $res['owner'], PDO::PARAM_STR);
                unset($host);
        }

        $statement->execute();
        $this->db->commit();

    return true;
}

代码工作正常，但是如果我尝试一次插入大约30000条记录，我就会遇到mysql限制并引发异常：＆＃34;一般错误：1390准备语句包含太多占位符＆＃34;

据我所知，我可以通过将查询分成小批量来解决这个问题。将它分成1000个批次就可以了（一个查询有6个占位符x 1000个记录= 6k个占位符.Mysql限制是~65k占位符）。

任何人都可以帮我拆分吗？

Answer 1

function addHostBulk($value, $label, $group, $type, $user)
{
    $res = array("type" => intval($type), "value" => $value, "label" => $label, "group_id" => $group, "owner" => intval($user['id']));

    $hosts = $common->cidrToRange($res['value'], $user); //Returns array with all hosts from this CIDR range /array('hosts' => array('1.2.3.4', '1.2.3.5.', ...)/
    $totalHosts = count($hosts['hosts']);
    $batchSize = 1000;

    for ($idx=0; $idx*$batchSize < $totalHosts; $idx++) { 
        $hostsPartial = array_slice($hosts['hosts'], $idx*$batchSize, $batchSize);

        $this->db->beginTransaction();
        $query = ("INSERT INTO hosts (`id`, `type`, `value`, `label`, `group_id`, `owner`) VALUES ");
        $qPart = array_fill(0, count($hostsPartial), "(?, ?, ?, ?, ?, ?)");
        $query .=  implode(",",$qPart);

        $statement = $this->db->prepare($query);

        $i = 1;
        foreach($hostsPartial as $host){
                $statement->bindParam($i++, $common->uuid(), PDO::PARAM_STR);
                $statement->bindParam($i++, $res['type'], PDO::PARAM_STR);
                $statement->bindParam($i++, $host, PDO::PARAM_STR);
                $statement->bindParam($i++, $res['label'], PDO::PARAM_STR);
                $statement->bindParam($i++, $res['group_id'], PDO::PARAM_STR);
                $statement->bindParam($i++, $res['owner'], PDO::PARAM_STR);
                unset($host);
        }

        $statement->execute();
        $this->db->commit();
    }

    return true;
}

Answer 2

不要尝试创建一个巨大的SQL语句并绑定数千个变量！这在许多方面都是错误的：绑定变量的数量有限，SQL语句大小也限制在1MB等等......

相反，将代码更改为使用单个静态INSERT语句，如下所示：

$query = "INSERT INTO hosts (`id`, `type`, `value`, `label`, `group_id`, `owner`)"
       . "VALUES (?, ?, ?, ?, ?, ?)";

然后做这样的事情：

$this->db->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);  # important!
$this->db->beginTransaction();
$statement = $this->db->prepare($query);
foreach($hosts['hosts'] as $host) {
     # bind 6 variables...
     $statement->execute();
}
$this->db->commit();

此代码适用于一个事务中的数百万行，并且应该可以非常快速地工作。

Answer 3

你可以做这样的事情，把条件放在foreach中检查是否有超过1000条记录然后插入第一个1000并循环其他像这样

    $i = 1;
    $totalCount = 0;
    foreach($hosts['hosts'] as $host){

            $statement->bindParam($i++, $common->uuid(), PDO::PARAM_STR);
            $statement->bindParam($i++, $res['type'], PDO::PARAM_STR);
            $statement->bindParam($i++, $host, PDO::PARAM_STR);
            $statement->bindParam($i++, $res['label'], PDO::PARAM_STR);
            $statement->bindParam($i++, $res['group_id'], PDO::PARAM_STR);
            $statement->bindParam($i++, $res['owner'], PDO::PARAM_STR);
            unset($host);

            if($i==1000)
            {
               $statement->execute();
               $this->db->commit();
               $i=0;
            }

    }

    //check for last time if any records left then you have to insert those records also

    if($i<1000)
    {
         $statement->execute();
         $this->db->commit();

    }

使用带有预准备语句的php pdo批量拆分大插入查询

3 个答案: