Question

我在macOS Sierra上，我试图用scrapy更新我的MySQL记录，最近我已经得到了2006年MySQL已经消失了，所以我实施了解决方案跟随this guide，但是在爬行的中途。终端将崩溃，具体如下：

崩溃报告：

python(85034,0x70000b397000) malloc: *** error for object 0x7fa52317c400: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
python(85034,0x70000a78e000) malloc: *** error for object 0x7fa52317c400: double free
*** set a breakpoint in malloc_error_break to debug
python(85034,0x70000ab91000) malloc: *** error for object 0x7fa52317c400: double free
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

代码：

def connect(self):
    self.conn = MySQLdb.connect(host='127.0.0.1',
               user='root',
               passwd='<my password>',
               db='jobs',
               charset='utf8'
                )
def checkRecord(self, query, params, msg):
    try:
        cursor = self.conn.cursor()
        cursor.execute(query, params)
        fetch = cursor.fetchone()
        return fetch
    except (MySQLdb.OperationalError):
        logging.info("Attempting to reconnect for select query") 
        self.connect()
        self.checkRecord(query, params, msg)
    finally:
        logging.info(msg)
        cursor.close()

def checkCategory(self, query, params, msg):
    try:
        cursor = self.conn.cursor()
        cursor.execute(query, params)
        fetch = cursor.fetchone()
        return fetch[0]
    except (MySQLdb.OperationalError):
        logging.info("Attempting to reconnect for select query") 
        self.connect()
        self.checkRecord(query, params, msg)
    finally:
        logging.info(msg)
        cursor.close()

def insertRecord(self, query, params, msg):
    try:
        cursor = self.conn.cursor()
        cursor.execute(query, params)
    except (MySQLdb.OperationalError):
        logging.info("Attempting to reconnect for inserting record") 
        self.connect()
        self.insertRecord(query, params, msg)
    finally:
        logging.info(msg)
        cursor.close()

def updateCategory(self, query, params, msg):
    try:
        cursor = self.conn.cursor()
        cursor.execute(query, params)
    except (MySQLdb.OperationalError):
        logging.info("Attempting to reconnect for updating record") 
        self.connect()
        self.updateCategory(query, params, msg)            
    finally:
        self.conn.commit()
        logging.info(msg)
        cursor.close()

def _conditional_insert(self,tx,item):

    selectquery = "SELECT count(*) FROM dbJobs WHERE jobdetailsurl LIKE %s AND company_name LIKE %s AND title LIKE %s"
    selectparams = (item['jobdetailsurl'], item['company_name'], item['title'])
    msg = "Checking for whether record exists in Database"

    fetch = self.checkRecord(selectquery, selectparams, msg)
    ....
    ....

我还在Windows 10计算机上测试了此代码，命令提示符也崩溃了。解决这个问题的方法是什么？

编辑:(会在每个光标足够后打开一个新连接并关闭连接吗？）

def checkRecord(self, query, params, msg):
    try:
        self.connect()
        cursor = self.conn.cursor()
        cursor.execute(query, params)
        fetch = cursor.fetchone()
        return fetch
    except (MySQLdb.OperationalError):
        logging.info("Attempting to reconnect for select query") 
        self.connect()
        self.checkRecord(query, params)
    finally:
        cursor.close()
        self.conn.close()

    logging.info(msg)

Answer 1

我已经设法通过为每个查询/更新/插入创建一个新连接来解决此问题，然后在完成之后关闭该连接，而不是为所有查询留下1个连接。通过在整个晚上在2台机器上运行来测试它，并且没有更多的崩溃。

示例：

def checkRecord(self, query, params, msg):
    try:
        self.connect()
        cursor = self.conn.cursor()
        cursor.execute(query, params)
        fetch = cursor.fetchone()
        return fetch
    except (MySQLdb.OperationalError):
        logging.info("Attempting to reconnect for select query") 
        self.connect()
        self.checkRecord(query, params)
    finally:
        cursor.close()
        self.conn.close()

    logging.info(msg)

尝试重新连接时MySQL / Python终端崩溃

1 个答案: