用于Mongo DB更新的Grails批量更新耗尽内存

时间:2015-04-27 18:39:17

标签: mongodb grails gorm batch-processing

我使用Grails与MongoDB进行交互,主要用于域验证。在进行数十万条记录的批量更新时,我的内存不足(超出GC开销限制)。我已经解决了在域对象上使用findBy的问题。如果我用等效的GMongo.DB.collection.findOne替换那个DomainClass.findByProperty,我没有内存问题。

批量更新是在一个事务中(静态事务=' mongo')但是用完了一个运行脚本。

我经常尝试刷新并清除会话,但无济于事。我一点一点地耗尽了记忆。这仅在使用GORM时发生。

// This is will fail after a while
// def cust = Customer.findByCustomerNo (custno, [ cache: false ]) ?: new Customer ()

// This will work but if I follow it by a get it will fail after a while
def found = mongodbService.DB.customers.findOne (customerNo: custno)
def cust =  (found) ? Customer.get ("${found._id}") : new Customer ()

这是我经历的一种退化,我使用的是Grails 2.5.0和mongodb-3.0.3。

processing [1] (0%) errors (0) freeMem: 259M 0...
processing [2187] (1%) errors (0) freeMem: 314M 7.478 seconds...
processing [4373] (2%) errors (0) freeMem: 248M 4.446 seconds...
processing [6560] (3%) errors (0) freeMem: 285M 3.875 seconds...
processing [8746] (4%) errors (0) freeMem: 237M 3.847 seconds...
processing [10933] (5%) errors (0) freeMem: 358M 4.572 seconds...
processing [13119] (6%) errors (0) freeMem: 340M 3.203 seconds...
processing [15306] (7%) errors (0) freeMem: 322M 3.172 seconds...
processing [17492] (8%) errors (0) freeMem: 307M 3.185 seconds...
processing [19679] (9%) errors (0) freeMem: 292M 3.638 seconds...
processing [21865] (10%) errors (0) freeMem: 274M 3.933 seconds...
processing [24052] (11%) errors (0) freeMem: 393M 5.666 seconds...
processing [26238] (12%) errors (0) freeMem: 373M 3.851 seconds...
processing [28425] (13%) errors (0) freeMem: 356M 3.843 seconds...
processing [30611] (14%) errors (0) freeMem: 313M 3.866 seconds...
processing [32798] (15%) errors (0) freeMem: 314M 3.987 seconds...
processing [34984] (16%) errors (0) freeMem: 276M 4.097 seconds...
processing [37171] (17%) errors (0) freeMem: 257M 3.824 seconds...
processing [39357] (18%) errors (0) freeMem: 427M 7.126 seconds...
processing [41544] (19%) errors (0) freeMem: 406M 4.479 seconds...
processing [43730] (20%) errors (0) freeMem: 388M 4.523 seconds...
processing [45917] (21%) errors (0) freeMem: 368M 4.589 seconds...
processing [48103] (22%) errors (0) freeMem: 348M 5.034 seconds...
processing [50290] (23%) errors (0) freeMem: 327M 5.259 seconds...
processing [52476] (24%) errors (0) freeMem: 310M 6.497 seconds...
processing [54662] (25%) errors (0) freeMem: 290M 5.533 seconds...
processing [56849] (26%) errors (0) freeMem: 270M 5.380 seconds...
processing [59035] (27%) errors (0) freeMem: 319M 8.608 seconds...
processing [61222] (28%) errors (0) freeMem: 280M 5.736 seconds...
processing [63408] (29%) errors (0) freeMem: 262M 6.227 seconds...
processing [65595] (30%) errors (0) freeMem: 242M 9.254 seconds...
processing [67781] (31%) errors (0) freeMem: 223M 8.851 seconds...
processing [69968] (32%) errors (0) freeMem: 204M 8.922 seconds...
processing [72154] (33%) errors (0) freeMem: 185M 9.584 seconds...
processing [74341] (34%) errors (0) freeMem: 168M 10.954 seconds...
processing [76527] (35%) errors (0) freeMem: 152M 10.292 seconds...
processing [78714] (36%) errors (0) freeMem: 153M 9.801 seconds...
processing [80900] (37%) errors (0) freeMem: 147M 9.493 seconds...
processing [83087] (38%) errors (0) freeMem: 131M 11.280 seconds...
processing [85273] (39%) errors (0) freeMem: 115M 11.728 seconds...
processing [87460] (40%) errors (0) freeMem: 99M 13.177 seconds...
processing [89646] (41%) errors (0) freeMem: 83M 16.407 seconds...
processing [91833] (42%) errors (0) freeMem: 66M 16.940 seconds...
processing [94019] (43%) errors (0) freeMem: 50M 23.063 seconds...
processing [96206] (44%) errors (0) freeMem: 34M 35.799 seconds...

0 个答案:

没有答案