Python/MySQL avoid duplicate insert

Question

I have one table where fields are :-

1. ID (auto increantal)
2. Heading (actual text data)
3. Flag (yes or no, based on some other business logic)

Now, i want to keep inserting data using python in this table. I have a URL crawler which captures URL from a given web page. The issue is, every 10 mins i read this page to see any new links. In the current setup, the crawler gives URL and i am inserting but i want to check if the links are already avilable then it should not get inserted. For example:-

1am - 10 links found. 2am - 10 links found (out of which 2 are new). So python should only insert 2 new links and not the rest 8 links.

The size of table rite now is 10k records. So i need a logic to solve this problem. Please help.

Answer 1

在链接列上创建unique key。

并使用INSERT IGNORE语句来避免重复。