存储和查询极大量的JSON数据

时间:2016-11-29 21:50:53

标签: sql json database storage nosql

上下文

游戏的远程公共API大约每五秒生成300000行JSON数据。此数据在下次更新后立即消失,永远不会再次访问。我希望能够保存这些数据的压缩历史记录,这些记录可以可靠,高效地编制索引和搜索。

问题:

考虑到可能的压缩方法以及最重要的查询效率,在这个规模上存储JSON数据的最佳数据库系统是什么?

编辑添加示例解压缩的JSON(大约30000个中的一个对象):

{
  "accountName": "Kabancheg11",
  "lastCharacterName": "KabanNeedBuff",
  "id": "427d66d977b5a63af08b3c39173a1a567eb783fde6d6f97df505dbe6c1fa5988",
  "stash": "~b/o 30 chaos",
  "stashType": "PremiumStash",
  "items": [
    {
      "verified": false,
      "w": 1,
      "h": 1,
      "ilvl": 71,
      "icon": "http://web.poecdn.com/image/Art/2DItems/Amulets/Amulet37.png?scale=1&w=1&h=1&v=25bebeac11abfd9f7931d574765521093",
      "league": "Hardcore",
      "id": "a7bb36a28e55865b9d766dcebd5bc0c6893d5ecb6cd9870dc0bcd3c9f1164f29",
      "sockets": [],
      "name": "",
      "typeLine": "Onyx Amulet",
      "identified": false,
      "corrupted": false,
      "lockedToCharacter": false,
      "implicitMods": [
        "+15 to all Attributes"
      ],
      "frameType": 3,
      "x": 11,
      "y": 0,
      "inventoryId": "Stash1",
      "socketedItems": []
    },
    {
      "verified": false,
      "w": 1,
      "h": 3,
      "ilvl": 75,
      "icon": "http://web.poecdn.com/image/Art/2DItems/Weapons/OneHandWeapons/Daggers/Heartbreaker.png?scale=1&w=1&h=3&v=b4f18d5c0602a17381df6a45b4c57be33",
      "league": "Hardcore",
      "id": "c955f054ffae9439e5d257c4a444b02423b2dbe304d1fa581227d21a0f203aa9",
      "sockets": [
        {
          "group": 0,
          "attr": "D"
        },
        {
          "group": 0,
          "attr": "D"
        }
      ],
      "name": "<<set:MS>><<set:M>><<set:S>>Heartbreaker",
      "typeLine": "Royal Skean",
      "identified": true,
      "corrupted": false,
      "lockedToCharacter": false,
      "properties": [
        {
          "name": "Dagger",
          "values": [],
          "displayMode": 0
        },
        {
          "name": "Physical Damage",
          "values": [
            [
              "15-59",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Critical Strike Chance",
          "values": [
            [
              "6.60%",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Attacks per Second",
          "values": [
            [
              "1.45",
              0
            ]
          ],
          "displayMode": 0
        }
      ],
      "requirements": [
        {
          "name": "Level",
          "values": [
            [
              "50",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Dex",
          "values": [
            [
              "71",
              0
            ]
          ],
          "displayMode": 1
        },
        {
          "name": "Int",
          "values": [
            [
              "102",
              0
            ]
          ],
          "displayMode": 1
        }
      ],
      "implicitMods": [
        "40% increased Global Critical Strike Chance"
      ],
      "explicitMods": [
        "45% increased Spell Damage",
        "+50 to maximum Mana",
        "+50 to maximum Energy Shield",
        "10% faster start of Energy Shield Recharge",
        "Your Spells have Culling Strike"
      ],
      "flavourText": [
        "A heart can be stabbed by thoughts, \r",
        "If your mind is sharp enough."
      ],
      "frameType": 3,
      "x": 0,
      "y": 9,
      "inventoryId": "Stash2",
      "socketedItems": []
    },
    {
      "verified": false,
      "w": 2,
      "h": 3,
      "ilvl": 75,
      "icon": "http://web.poecdn.com/image/Art/2DItems/Weapons/OneHandWeapons/Scepters/MontregulsGrasp.png?scale=1&w=2&h=3&v=fed985831c32e8e25b7663608a2b7a5c3",
      "league": "Hardcore",
      "id": "a78e6bfff62145c78b92714cc7de10e4484c42745d48b585ef905b3972c2a319",
      "sockets": [
        {
          "group": 0,
          "attr": "S"
        },
        {
          "group": 0,
          "attr": "S"
        }
      ],
      "name": "",
      "typeLine": "Void Sceptre",
      "identified": false,
      "corrupted": false,
      "lockedToCharacter": false,
      "properties": [
        {
          "name": "One Handed Mace",
          "values": [],
          "displayMode": 0
        },
        {
          "name": "Physical Damage",
          "values": [
            [
              "42-63",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Critical Strike Chance",
          "values": [
            [
              "6.50%",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Attacks per Second",
          "values": [
            [
              "1.25",
              0
            ]
          ],
          "displayMode": 0
        }
      ],
      "requirements": [
        {
          "name": "Str",
          "values": [
            [
              "104",
              0
            ]
          ],
          "displayMode": 1
        },
        {
          "name": "Int",
          "values": [
            [
              "122",
              0
            ]
          ],
          "displayMode": 1
        }
      ],
      "implicitMods": [
        "15% increased Elemental Damage"
      ],
      "frameType": 3,
      "x": 0,
      "y": 6,
      "inventoryId": "Stash3",
      "socketedItems": []
    },
    {
      "verified": false,
      "w": 2,
      "h": 4,
      "ilvl": 74,
      "icon": "http://web.poecdn.com/image/Art/2DItems/Weapons/TwoHandWeapons/TwoHandAxes/TwoHandAxe11Unique.png?scale=1&w=2&h=4&v=49f4f7aff220d356fc6300d7454df63a3",
      "league": "Hardcore",
      "id": "c8d722c909939d1fda4431722c732e7af4c17781d9d9c58da613bf776ea1e993",
      "sockets": [
        {
          "group": 0,
          "attr": "D"
        }
      ],
      "name": "",
      "typeLine": "Superior Karui Chopper",
      "identified": false,
      "corrupted": false,
      "lockedToCharacter": false,
      "properties": [
        {
          "name": "Two Handed Axe",
          "values": [],
          "displayMode": 0
        },
        {
          "name": "Quality",
          "values": [
            [
              "+10%",
              1
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Physical Damage",
          "values": [
            [
              "88-138",
              1
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Critical Strike Chance",
          "values": [
            [
              "5.00%",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Attacks per Second",
          "values": [
            [
              "1.15",
              0
            ]
          ],
          "displayMode": 0
        }
      ],
      "requirements": [
        {
          "name": "Str",
          "values": [
            [
              "151",
              0
            ]
          ],
          "displayMode": 1
        },
        {
          "name": "Dex",
          "values": [
            [
              "43",
              0
            ]
          ],
          "displayMode": 1
        }
      ],
      "frameType": 3,
      "x": 2,
      "y": 0,
      "inventoryId": "Stash4",
      "socketedItems": []
    },
    {
      "verified": false,
      "w": 2,
      "h": 3,
      "ilvl": 51,
      "icon": "http://web.poecdn.com/image/Art/2DItems/Armours/BodyArmours/StrInt3AUnique2.png?scale=1&w=2&h=3&v=3ef1aa6bc50b635ca334653ca95485a63",
      "league": "Hardcore",
      "id": "26c55af00fb31f63bec7229b622c3c79fb6efcfa03d3d2595672a92778d04d99",
      "sockets": [
        {
          "group": 0,
          "attr": "I"
        },
        {
          "group": 1,
          "attr": "I"
        }
      ],
      "name": "<<set:MS>><<set:M>><<set:S>>Ambu's Charge",
      "typeLine": "Crusader Chainmail",
      "identified": true,
      "corrupted": false,
      "lockedToCharacter": false,
      "properties": [
        {
          "name": "Armour",
          "values": [
            [
              "477",
              1
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Energy Shield",
          "values": [
            [
              "141",
              1
            ]
          ],
          "displayMode": 0
        }
      ],
      "requirements": [
        {
          "name": "Level",
          "values": [
            [
              "43",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Str",
          "values": [
            [
              "64",
              0
            ]
          ],
          "displayMode": 1
        },
        {
          "name": "Int",
          "values": [
            [
              "64",
              0
            ]
          ],
          "displayMode": 1
        }
      ],
      "explicitMods": [
        "182% increased Armour and Energy Shield",
        "+15% to all Elemental Resistances",
        "Gain an Endurance Charge when you take a Critical Strike",
        "2% of Life Regenerated per second while on Low Life",
        "Share Endurance Charges with nearby party members"
      ],
      "flavourText": [
        "Nothing stops the pain like a courageous rush into battle."
      ],
      "frameType": 3,
      "x": 0,
      "y": 0,
      "inventoryId": "Stash5",
      "socketedItems": []
    },
    {
      "verified": false,
      "w": 1,
      "h": 1,
      "ilvl": 64,
      "icon": "http://web.poecdn.com/image/Art/2DItems/Jewels/basicint.png?scale=1&w=1&h=1&v=cd579ea22c05f1c6ad2fd015d7a710bd3",
      "league": "Hardcore",
      "id": "34103c553f0ae3e36c695664a7540dd6e547338596c33554ca3a8443dd5349c1",
      "sockets": [],
      "name": "<<set:MS>><<set:M>><<set:S>>Cataclysm Heart",
      "typeLine": "Cobalt Jewel",
      "identified": true,
      "corrupted": false,
      "lockedToCharacter": false,
      "explicitMods": [
        "2% increased Cast Speed",
        "7% increased maximum Energy Shield",
        "+13% to Lightning Resistance"
      ],
      "descrText": "Place into an allocated Jewel Socket on the Passive Skill Tree. Right click to remove from the Socket.",
      "frameType": 2,
      "x": 0,
      "y": 3,
      "inventoryId": "Stash6",
      "socketedItems": []
    },
    {
      "verified": false,
      "w": 1,
      "h": 1,
      "ilvl": 0,
      "icon": "http://web.poecdn.com/image/Art/2DItems/Gems/VaalGems/VaalArc.png?scale=1&w=1&h=1&v=b4f32328e279496ebb227521e8dce6793",
      "support": false,
      "league": "Hardcore",
      "id": "be9a23cfe0a53942b19c22da32b7d461b464838749110d04689e15498dcad86b",
      "sockets": [],
      "name": "",
      "typeLine": "Vaal Arc",
      "identified": true,
      "corrupted": true,
      "lockedToCharacter": false,
      "properties": [
        {
          "name": "Vaal, Spell, Chaining, Lightning",
          "values": [],
          "displayMode": 0
        },
        {
          "name": "Level",
          "values": [
            [
              "20 (Max)",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Mana Cost",
          "values": [
            [
              "0",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Souls Per Use",
          "values": [
            [
              "32",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Can Store %0 Use",
          "values": [
            [
              "1",
              0
            ]
          ],
          "displayMode": 3
        },
        {
          "name": "Cast Time",
          "values": [
            [
              "0.80 sec",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Critical Strike Chance",
          "values": [
            [
              "5.00%",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Damage Effectiveness",
          "values": [
            [
              "80%",
              0
            ]
          ],
          "displayMode": 0
        }
      ],
      "requirements": [
        {
          "name": "Level",
          "values": [
            [
              "70",
              0
            ]
          ],
          "displayMode": 0
        },
        {
          "name": "Int",
          "values": [
            [
              "155",
              0
            ]
          ],
          "displayMode": 1
        }
      ],
      "secDescrText": "An arc of lightning stretches from the caster to a targeted nearby enemy and chains on to many additional targets.",
      "explicitMods": [
        "Deals 49 to 938 Lightning Damage",
        "Chain +40 Times",
        "100% chance to Shock enemies"
      ],
      "descrText": "Place into an item socket of the right colour to gain this skill. Right click to remove from a socket.",
      "frameType": 4,
      "x": 0,
      "y": 4,
      "inventoryId": "Stash7",
      "socketedItems": []
    }
  ],
  "public": true
}

示例查询可能会询问嵌套在每个对象中的项目数量(如上所示)有ilvl字段大于或等于整数。

0 个答案:

没有答案