覆盖计数查询为何缓慢？

时间：2017-05-19 12:10:08

标签： mongodb mongodb-query

我正在运行mongodb v3.2.12分片群集。分片键是_id，它是md5哈希。 问题是，覆盖计数查询需要很长时间。

每个mongodb节点上使用的索引大约为5 GB。所有索引的总大小为32 GB，完全适合RAM，因为每个节点都有128 GB RAM。

查询为：db.offer.count({ "shopId": 275419, "missingSince": null})

已使用的索引已创建为：db.offer.createIndex({shopId:1, missingSince:1, merchantId:1, _id:1}, {background:true})

如您所见，索引不稀疏，因此索引中甚至存在空值。

在查询运行时运行db.currentOp()表明查询正在使用正确的索引，但是，它已经运行超过2814秒：

    {
        "desc" : "conn56062",
        "threadId" : "140131556767488",
        "connectionId" : 56062,
        "client_s" : "x.x.x.x:39177",
        "active" : true,
        "opid" : "offerStoreIT02:1075309911",
        "secs_running" : 2814,
        "microsecs_running" : NumberLong("2814791918"),
        "op" : "command",
        "ns" : "offerStore.offer",
        "query" : {
                "query" : {
                        "count" : "offer",
                        "query" : {
                                "missingSince" : null,
                                "shopId" : 275419
                        }
                },
                "$readPreference" : {
                        "mode" : "primaryPreferred"
                }
        },
        "planSummary" : "IXSCAN { shopId: 1.0, missingSince: 1.0, merchantId: 1.0, _id: 1.0 }",
        "numYields" : 249244,
        "locks" : {
                "Global" : "r",
                "Database" : "r",
                "Collection" : "r"
        },
        "waitingForLock" : false,
        "lockStats" : {
                "Global" : {
                        "acquireCount" : {
                                "r" : NumberLong(498490)
                        }
                },
                "Database" : {
                        "acquireCount" : {
                                "r" : NumberLong(249245)
                        }
                },
                "Collection" : {
                        "acquireCount" : {
                                "r" : NumberLong(249245)
                        }
                }
        }
}

迭代5 GB的内存中索引永远不会花费太多时间。在查询运行时，每个mongodb主节点从磁盘上不断读取75-100 MB /秒。当查询没有运行时，从磁盘读取只有5-10 MB /秒，所以我的假设是mongodb将文件从SSD提取到内存中以便对它们进行计数。

但为什么会这样呢？索引应该包含查询，因为索引中存在包括shardkey在内的所有字段，这应该足以根据mongodb文档覆盖查询： https://docs.mongodb.com/manual/core/query-optimization/#covered-queries

随访：

我将问题分解为简约，无损的设置。我插入了以下类型的文件：

  • a）没有字段shopId和missingSince
    • 的3个文档
  • b）包含字段shopId:1且没有字段missingSince
    • 的5个文档
  • c）包含字段shopId:1和missingSince:null
    • 的7个文档
  • d）包含字段shopId:1和missingSince:ISODate("2017-05-22T07:52:40.831Z")
    • 的13个文档

我创建了索引{shopId:1, missingSince:1}。 查询count({"shopId":1, "missingSince":null})的执行计划表明"totalDocsExamined" : 12，这意味着必须提取12个文档。这些必须是b）的5份文件加上c）的7份文件。所有这12个文档都应该在shopId:1, missingSince:null的索引中，从而满足查询。

但为什么mongodb仍然需要获取并检查这12个文档？

这是我的测试集：

rs1:PRIMARY> db.offer.find()
{ "_id" : 1, "v" : 1 }
{ "_id" : 2, "v" : 1 }
{ "_id" : 3, "v" : 1 }
{ "_id" : 4, "shopId" : 1, "v" : 1 }
{ "_id" : 5, "shopId" : 1, "v" : 1 }
{ "_id" : 6, "shopId" : 1, "v" : 1 }
{ "_id" : 7, "shopId" : 1, "v" : 1 }
{ "_id" : 8, "shopId" : 1, "v" : 1 }
{ "_id" : 9, "shopId" : 1, "missingSince" : null, "v" : 1 }
{ "_id" : 10, "shopId" : 1, "missingSince" : null, "v" : 1 }
{ "_id" : 11, "shopId" : 1, "missingSince" : null, "v" : 1 }
{ "_id" : 12, "shopId" : 1, "missingSince" : null, "v" : 1 }
{ "_id" : 13, "shopId" : 1, "missingSince" : null, "v" : 1 }
{ "_id" : 14, "shopId" : 1, "missingSince" : null, "v" : 1 }
{ "_id" : 15, "shopId" : 1, "missingSince" : null, "v" : 1 }
{ "_id" : 16, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 17, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 18, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 19, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 20, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 21, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 22, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 23, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 24, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 25, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 26, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 27, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }
{ "_id" : 28, "shopId" : 1, "missingSince" : ISODate("2017-05-22T07:52:40.831Z"), "v" : 1 }

这是explain（）的输出：

rs1:PRIMARY> db.offer.explain(true).count({"shopId":1, "missingSince":null})
{
    "queryPlanner" : {
        "plannerVersion" : 1,
        "namespace" : "test.offer",
        "indexFilterSet" : false,
        "parsedQuery" : {
            "$and" : [
                {
                    "missingSince" : {
                        "$eq" : null
                    }
                },
                {
                    "shopId" : {
                        "$eq" : 1
                    }
                }
            ]
        },
        "winningPlan" : {
            "stage" : "COUNT",
            "inputStage" : {
                "stage" : "FETCH",
                "filter" : {
                    "missingSince" : {
                        "$eq" : null
                    }
                },
                "inputStage" : {
                    "stage" : "IXSCAN",
                    "keyPattern" : {
                        "shopId" : 1,
                        "missingSince" : 1
                    },
                    "indexName" : "shopId_1_missingSince_1",
                    "isMultiKey" : false,
                    "isUnique" : false,
                    "isSparse" : false,
                    "isPartial" : false,
                    "indexVersion" : 1,
                    "direction" : "forward",
                    "indexBounds" : {
                        "shopId" : [
                            "[1.0, 1.0]"
                        ],
                        "missingSince" : [
                            "[null, null]"
                        ]
                    }
                }
            }
        },
        "rejectedPlans" : [ ]
    },
    "executionStats" : {
        "executionSuccess" : true,
        "nReturned" : 0,
        "executionTimeMillis" : 0,
        "totalKeysExamined" : 12,
        "totalDocsExamined" : 12,
        "executionStages" : {
            "stage" : "COUNT",
            "nReturned" : 0,
            "executionTimeMillisEstimate" : 0,
            "works" : 13,
            "advanced" : 0,
            "needTime" : 12,
            "needYield" : 0,
            "saveState" : 0,
            "restoreState" : 0,
            "isEOF" : 1,
            "invalidates" : 0,
            "nCounted" : 12,
            "nSkipped" : 0,
            "inputStage" : {
                "stage" : "FETCH",
                "filter" : {
                    "missingSince" : {
                        "$eq" : null
                    }
                },
                "nReturned" : 12,
                "executionTimeMillisEstimate" : 0,
                "works" : 13,
                "advanced" : 12,
                "needTime" : 0,
                "needYield" : 0,
                "saveState" : 0,
                "restoreState" : 0,
                "isEOF" : 1,
                "invalidates" : 0,
                "docsExamined" : 12,
                "alreadyHasObj" : 0,
                "inputStage" : {
                    "stage" : "IXSCAN",
                    "nReturned" : 12,
                    "executionTimeMillisEstimate" : 0,
                    "works" : 13,
                    "advanced" : 12,
                    "needTime" : 0,
                    "needYield" : 0,
                    "saveState" : 0,
                    "restoreState" : 0,
                    "isEOF" : 1,
                    "invalidates" : 0,
                    "keyPattern" : {
                        "shopId" : 1,
                        "missingSince" : 1
                    },
                    "indexName" : "shopId_1_missingSince_1",
                    "isMultiKey" : false,
                    "isUnique" : false,
                    "isSparse" : false,
                    "isPartial" : false,
                    "indexVersion" : 1,
                    "direction" : "forward",
                    "indexBounds" : {
                        "shopId" : [
                            "[1.0, 1.0]"
                        ],
                        "missingSince" : [
                            "[null, null]"
                        ]
                    },
                    "keysExamined" : 12,
                    "dupsTested" : 0,
                    "dupsDropped" : 0,
                    "seenInvalidated" : 0
                }
            }
        },
        "allPlansExecution" : [ ]
    },
    "serverInfo" : {
        "host" : "Kays MacBook Pro",
        "port" : 27017,
        "version" : "3.2.6",
        "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25"
    },
    "ok" : 1
}

2 个答案:

答案 0 :(得分：1)

由于没有人能找到这个问题的正当理由，我昨天开了一个mongodb错误报告：https://jira.mongodb.org/browse/SERVER-29326

Mongodb工程师证实这是一个错误。不幸的是，在mongodb的文档中没有提到它可以节省我们很多时间来追踪问题并从一开始就部署另一个模式设计。

答案 1 :(得分：0)

试试这个索引：

如果您的现有索引db.offer.createIndex({shopId:1, missingSince:1, merchantId:1, _id:1}, {background:true}) 用于其他目的，尝试在商品集合上创建以下索引：

db.offer.createIndex({missingSince:1,shopId:1}, {background:true})

这将优化查询本身，从而计算。

相关问题
最新问题