Question

我正在处理一个爬虫程序，它将检索和解析的数据存储到我的MySQL数据库中。存储结果非常有效，但在结束连接时我很困难。

我正在使用request，cheerio，async和mysql：

// DB params
var con = mysql.createConnection({
    host: "localhost",
    user: "user",
    password: "password",
    database: "mydatabase",
});

// Open connection to DB
con.connect(function(err) {
    if (err) {
        console.log('Error connecting to Db');
        return;
    }
    console.log('Connection established');
});

// Array of Cities to crawl
var cities = ["olten", "zurich"];

// Todays Date 
var today = new Date();
...
today = dd + '.' + mm + '.' + yyyy;

// Search every city
async.each(cities, function(item, callback){    
        // The Page to crawl
        var page = "https://www.thepageto.com/search_xhr?fn=" + /*cities[i]*/ item + "&tn=&db=" + today + "&sort=trip_date&order";
        console.log("Visiting page " + page);
        request(page, function(error, response, body) {
            if (error) {console.log("Error: " + error);}
            // Check status code (200 is HTTP OK)
            console.log("Status code: " + response.statusCode);
            if (response.statusCode === 200) {
                // Parse JSON
                var data = JSON.parse(body);
                // Use only result > html code
                var content = data.html.results;                
                // Parse html
                var $ = cheerio.load(content);
                // Iterate through offers & Extract information & Store in a Object
                $('.trip.relative').each(function() {
                    // Empty Object, has the same structure as the Db table
                    var json = {id: "", driver: "", rating: "", date: "", time: "", start: "", stops: "", end: "", price: "", url: "", query: ""};
                    // Assigning values
                    json.driver = $(this).find('.username').text();
                        ...  
                    // Save filled object in Db
                    save(json, callback);
                });         
            }
        });
  }, function(err){ 
        if (err) {
            console.log('could not save');
        } else {
            console.log('lets end connection');
            con.end(function(err) {if (err) throw err;});
        }
  }
);

function save(item, callback) {
   con.query('INSERT INTO offers SET ?', item, callback); 
}

问题是，在将任何内容存储到数据库之前，数据库连接已经结束。 con.end()始终在实际查询之前或查询完成之前触发。我有一个意思是save函数在完成时必须返回一些东西。如果我在没有con.end()的情况下运行脚本，数据将完美地存储到数据库中。我是Node的新手，所以我还在学习，希望你可以帮助我。

修改我现在得到一个：

ReferenceError: callback is not defined
    at save (/home/juki/crawler/crawler.js:143:46)

以及：

Error: Callback was already called.
    at Query._callback (/home/juki/crawler/node_modules/async/dist/async.js:839:                                                                                                                     36)

Answer 1

您似乎没有从抓取工具中拨打callback。

您似乎也没等待保存完成。尝试重做save函数，所以它也需要回调。类似的东西：

function save(item, callback) {
    con.query('INSERT INTO offers SET ?', item, callback); 
}

然后称之为：

save(json, callback);

其中callback是您从async.each获得的。

NodeJs将Json从多个请求存储到MySQL DB

1 个答案: