使用Neo4j 2.0 RC1批量提交节点时如何提高性能？
我很想知道是否可以改进程序（如下）以更快地执行。似乎，当我们有约束时（ 90秒约束， 1秒没有），在Transaction的close（）中花费了大量时间
以下是该计划的输出：
2013-12-05 14:30:20,399 [main] INFO net.ahm.graph.Lab - ############# WAVE 1 #############
2013-12-05 14:30:20,399 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 14:30:45,296 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 24897
2013-12-05 14:30:45,296 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 14:32:20,372 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 95076
2013-12-05 14:32:20,372 [main] INFO net.ahm.graph.Lab - ############# WAVE 2 #############
2013-12-05 14:32:20,372 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 14:32:47,168 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 26796
2013-12-05 14:32:47,168 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 14:34:16,040 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 88872
2013-12-05 14:34:16,040 [main] INFO net.ahm.graph.Lab - ############# WAVE 3 #############
2013-12-05 14:34:16,040 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 14:34:44,423 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 28383
2013-12-05 14:34:44,423 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 14:36:16,964 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 92541
2013-12-05 14:36:16,964 [main] INFO net.ahm.graph.Lab - ############# WAVE 4 #############
2013-12-05 14:36:16,964 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 14:36:43,376 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 26412
2013-12-05 14:36:43,376 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 14:38:13,156 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 89780
2013-12-05 14:38:13,156 [main] INFO net.ahm.graph.Lab - ############# WAVE 5 #############
2013-12-05 14:38:13,156 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 14:38:39,459 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 26303
2013-12-05 14:38:39,459 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 14:40:10,581 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 91122
2013-12-05 14:40:10,582 [Thread-5] INFO net.ahm.graph.Lab - ### GRAPHDB SHUTDOWNHOOK INVOKED !!!
以下是测试程序：
import java.io.File;
import java.util.Iterator;
import org.apache.log4j.Logger;
import org.neo4j.graphdb.DynamicLabel;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.Transaction;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;
import org.neo4j.graphdb.factory.GraphDatabaseSettings;
import org.neo4j.graphdb.schema.ConstraintDefinition;
import org.neo4j.graphdb.schema.Schema;
import org.neo4j.kernel.impl.util.FileUtils;
public class Lab {
private static final Logger LOG = Logger.getLogger(Lab.class);
public static void main(String[] args) throws Exception {
FileUtils.deleteRecursively(new File("graphdb"));
GraphDatabaseService graphDb = new GraphDatabaseFactory().newEmbeddedDatabaseBuilder("graphdb")
.setConfig(GraphDatabaseSettings.use_memory_mapped_buffers, "true")
.setConfig(GraphDatabaseSettings.keep_logical_logs, "false").setConfig(GraphDatabaseSettings.query_cache_size, "1000")
.setConfig(GraphDatabaseSettings.relationship_grab_size, "1000").newGraphDatabase();
registerShutdownHook(graphDb);
try (Transaction tx = graphDb.beginTx()) {
Schema schema = graphDb.schema();
createConstraint(schema, "TEST", "COUNT");
tx.success();
}
int count = 0;
long time;
for (int i = 1; i <= 5; i++) {
LOG.info("############# WAVE " + i + " #############");
time = System.currentTimeMillis();
Transaction tx = graphDb.beginTx();
LOG.info("TIME TO BEGIN TX (m-sec): " + (System.currentTimeMillis() - time));
try {
time = System.currentTimeMillis();
for (int j = 1; j <= 10000; j++) {
count++;
Node n = graphDb.createNode(DynamicLabel.label("TEST"));
n.setProperty("COUNT", count);
}
LOG.info("TIME TO CREATE 10000 nodes (m-sec): " + (System.currentTimeMillis() - time));
time = System.currentTimeMillis();
tx.success();
LOG.info("TIME TO MARK SUCCESS (m-sec): " + (System.currentTimeMillis() - time));
} finally {
time = System.currentTimeMillis();
tx.close();
LOG.info("TIME TO COMMIT (m-sec): " + (System.currentTimeMillis() - time));
}
}
}
private static void createConstraint(Schema schema, String label, String propertyName) {
Iterator<ConstraintDefinition> constraints = schema.getConstraints(DynamicLabel.label(label)).iterator();
if (constraints == null || !constraints.hasNext()) {
try {
schema.constraintFor(DynamicLabel.label(label)).assertPropertyIsUnique(propertyName).create();
} catch (org.neo4j.graphdb.ConstraintViolationException ex) {
LOG.error("CONSTRAINT ALREADY DEFINED ON: " + label);
}
}
}
private static void registerShutdownHook(final GraphDatabaseService graphDb) {
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
LOG.info("### GRAPHDB SHUTDOWNHOOK INVOKED !!!");
graphDb.shutdown();
}
});
}
}
我评论了对createConstraint（）的调用，以下是该程序的输出：
2013-12-05 15:34:34,686 [main] INFO net.ahm.graph.Lab - ############# WAVE 1 #############
2013-12-05 15:34:34,686 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 15:34:35,529 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 843
2013-12-05 15:34:35,529 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 15:34:37,635 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 2106
2013-12-05 15:34:37,635 [main] INFO net.ahm.graph.Lab - ############# WAVE 2 #############
2013-12-05 15:34:37,635 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 15:34:37,729 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 94
2013-12-05 15:34:37,729 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 15:34:38,915 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 1186
2013-12-05 15:34:38,915 [main] INFO net.ahm.graph.Lab - ############# WAVE 3 #############
2013-12-05 15:34:38,915 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 15:34:38,963 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 48
2013-12-05 15:34:38,963 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 15:34:39,684 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 721
2013-12-05 15:34:39,684 [main] INFO net.ahm.graph.Lab - ############# WAVE 4 #############
2013-12-05 15:34:39,684 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 15:34:39,715 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 31
2013-12-05 15:34:39,715 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 15:34:40,464 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 749
2013-12-05 15:34:40,464 [main] INFO net.ahm.graph.Lab - ############# WAVE 5 #############
2013-12-05 15:34:40,464 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-05 15:34:40,527 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 63
2013-12-05 15:34:40,527 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-05 15:34:41,416 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 889
2013-12-05 15:34:41,432 [Thread-4] INFO net.ahm.graph.Lab - ### GRAPHDB SHUTDOWNHOOK INVOKED !!!
2 个答案:
答案 0 :(得分：2)
我发现了问题，只是索引编写器经常刷新。请参阅提交https://github.com/neo4j/neo4j/commit/dcc72fa83e0f3742c9ddebaad625254b9f3c9404
答案 1 :(得分：0)
使用悲观锁定替换约束并执行我们自己的查找会将节点创建时间增加到50秒。但是提交时间不到2秒。对于约束，这个（50 + 2）仍然比90 + 25秒要好得多。
我应该采用这种方法吗？
输出：
2013-12-06 12:01:39,535 [main] INFO net.ahm.graph.Lab - ############# WAVE 1 #############
2013-12-06 12:01:39,544 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-06 12:02:30,505 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 50961
2013-12-06 12:02:30,505 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-06 12:02:33,020 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 2515
2013-12-06 12:02:33,020 [main] INFO net.ahm.graph.Lab - ############# WAVE 2 #############
2013-12-06 12:02:33,020 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-06 12:03:16,146 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 43126
2013-12-06 12:03:16,146 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-06 12:03:17,241 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 1095
2013-12-06 12:03:17,241 [main] INFO net.ahm.graph.Lab - ############# WAVE 3 #############
2013-12-06 12:03:17,241 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-06 12:04:00,194 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 42953
2013-12-06 12:04:00,194 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-06 12:04:01,019 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 825
2013-12-06 12:04:01,019 [main] INFO net.ahm.graph.Lab - ############# WAVE 4 #############
2013-12-06 12:04:01,019 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-06 12:04:45,073 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 44054
2013-12-06 12:04:45,073 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-06 12:04:45,932 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 859
2013-12-06 12:04:45,932 [main] INFO net.ahm.graph.Lab - ############# WAVE 5 #############
2013-12-06 12:04:45,932 [main] INFO net.ahm.graph.Lab - TIME TO BEGIN TX (m-sec): 0
2013-12-06 12:05:29,401 [main] INFO net.ahm.graph.Lab - TIME TO CREATE 10000 nodes (m-sec): 43468
2013-12-06 12:05:29,401 [main] INFO net.ahm.graph.Lab - TIME TO MARK SUCCESS (m-sec): 0
2013-12-06 12:05:30,182 [main] INFO net.ahm.graph.Lab - TIME TO COMMIT (m-sec): 781
2013-12-06 12:05:30,183 [Thread-4] INFO net.ahm.graph.Lab - ### GRAPHDB SHUTDOWNHOOK INVOKED !!!
源代码：
package net.ahm.graph;
import java.io.File;
import java.util.Iterator;
import java.util.concurrent.TimeUnit;
import org.apache.log4j.Logger;
import org.neo4j.graphdb.DynamicLabel;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Label;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.ResourceIterable;
import org.neo4j.graphdb.ResourceIterator;
import org.neo4j.graphdb.Transaction;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;
import org.neo4j.graphdb.factory.GraphDatabaseSettings;
import org.neo4j.graphdb.schema.ConstraintDefinition;
import org.neo4j.graphdb.schema.IndexDefinition;
import org.neo4j.graphdb.schema.Schema;
import org.neo4j.kernel.impl.util.FileUtils;
public class Lab {
private static final String TEST_PROPERTY = "COUNT";
private static final String TEST_LABEL = "TEST";
private static final Logger LOG = Logger.getLogger(Lab.class);
private static Node lockNode = null;
public static void main(String[] args) throws Exception {
FileUtils.deleteRecursively(new File("graphdb"));
GraphDatabaseService graphDb = new GraphDatabaseFactory().newEmbeddedDatabaseBuilder("graphdb")
.setConfig(GraphDatabaseSettings.use_memory_mapped_buffers, "true")
.setConfig(GraphDatabaseSettings.keep_logical_logs, "false").setConfig(GraphDatabaseSettings.query_cache_size, "1000")
.setConfig(GraphDatabaseSettings.relationship_grab_size, "1000").newGraphDatabase();
registerShutdownHook(graphDb);
/*
* try (Transaction tx = graphDb.beginTx()) {
* Schema schema = graphDb.schema();
* createConstraint(schema, "TEST", "COUNT");
* tx.success();
* }
*/
createIndex(TEST_LABEL, TEST_PROPERTY, graphDb);
int count = 0;
long time;
try (Transaction tx = graphDb.beginTx()) {
lockNode = graphDb.createNode(DynamicLabel.label("LOCKNODE"));
tx.success();
}
for (int i = 1; i <= 5; i++) {
LOG.info("############# WAVE " + i + " #############");
time = System.currentTimeMillis();
Transaction tx = graphDb.beginTx();
LOG.info("TIME TO BEGIN TX (m-sec): " + (System.currentTimeMillis() - time));
try {
time = System.currentTimeMillis();
for (int j = 1; j <= 10000; j++) {
count++;
if (findNode(TEST_LABEL, TEST_PROPERTY, count, graphDb) == null) {
tx.acquireWriteLock(lockNode);
if (findNode(TEST_LABEL, TEST_PROPERTY, count, graphDb) == null) {
Node n = graphDb.createNode(DynamicLabel.label(TEST_LABEL));
n.setProperty(TEST_PROPERTY, count);
} else {
LOG.error("NODE WITH PROPERTY ALREADY EXISTS: " + count);
}
} else {
LOG.error("NODE WITH PROPERTY ALREADY EXISTS: " + count);
}
}
LOG.info("TIME TO CREATE 10000 nodes (m-sec): " + (System.currentTimeMillis() - time));
time = System.currentTimeMillis();
tx.success();
LOG.info("TIME TO MARK SUCCESS (m-sec): " + (System.currentTimeMillis() - time));
} finally {
time = System.currentTimeMillis();
tx.close();
LOG.info("TIME TO COMMIT (m-sec): " + (System.currentTimeMillis() - time));
}
}
}
private static void createConstraint(Schema schema, String label, String propertyName) {
Iterator<ConstraintDefinition> constraints = schema.getConstraints(DynamicLabel.label(label)).iterator();
if (constraints == null || !constraints.hasNext()) {
try {
schema.constraintFor(DynamicLabel.label(label)).assertPropertyIsUnique(propertyName).create();
} catch (org.neo4j.graphdb.ConstraintViolationException ex) {
LOG.error("CONSTRAINT ALREADY DEFINED ON: " + label);
}
}
}
private static void createIndex(String label, String propertyName, GraphDatabaseService graphDb) {
IndexDefinition indexDefinition;
try (Transaction tx = graphDb.beginTx()) {
Schema schema = graphDb.schema();
indexDefinition = schema.indexFor(DynamicLabel.label(label)).on(propertyName).create();
tx.success();
}
try (Transaction tx = graphDb.beginTx()) {
Schema schema = graphDb.schema();
schema.awaitIndexOnline(indexDefinition, 10, TimeUnit.SECONDS);
tx.success();
}
}
private static Node findNode(String sLabel, String propertyName, Object propertyValue, GraphDatabaseService graphDb) {
if (propertyValue != null) {
Label label = DynamicLabel.label(sLabel);
ResourceIterable<Node> ri = graphDb.findNodesByLabelAndProperty(label, propertyName, propertyValue);
if (ri != null) {
try {
ResourceIterator<Node> iter = ri.iterator();
if (iter != null && iter.hasNext()) {
return iter.next();
}
} catch (Exception e) {
LOG.error("ERROR WHILE FINDING ID: " + propertyValue + " , LABEL: " + sLabel + " , PROPERTY: " + propertyName, e);
}
}
}
return null;
}
private static void registerShutdownHook(final GraphDatabaseService graphDb) {
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
LOG.info("### GRAPHDB SHUTDOWNHOOK INVOKED !!!");
graphDb.shutdown();
}
});
}
}
相关问题
最新问题
- 我写了这段代码，但我无法理解我的错误
- 我无法从一个代码实例的列表中删除 None 值，但我可以在另一个实例中。为什么它适用于一个细分市场而不适用于另一个细分市场？
- 是否有可能使 loadstring 不可能等于打印？卢阿
- java中的random.expovariate()
- Appscript 通过会议在 Google 日历中发送电子邮件和创建活动
- 为什么我的 Onclick 箭头功能在 React 中不起作用？
- 在此代码中是否有使用“this”的替代方法？
- 在 SQL Server 和 PostgreSQL 上查询，我如何从第一个表获得第二个表的可视化
- 每千个数字得到
- 更新了城市边界 KML 文件的来源？