Question

def get_connection_and_channel(self, connection_parameters):
    connection = pika.BlockingConnection(connection_parameters)
    channel = connection.channel()
    return (connection, channel)  


connection_parameters = pika.ConnectionParameters( server, port, virtual_host, credentials=pika.PlainCredentials(user_name, password))

connection,channel = self.get_connection_and_channel(connection_parameters)

channel.confirm_delivery()
count=0
for json_string in open(json_file, 'r'):
    result_json = json.loads(json_string)
    message_body = json.dumps(result_json['body'])
    routing_key = result_json['RoutingKey']
    channel.basic_publish(exchange=self.output_exchange_name,routing_key=routing_key,body=message_body.strip())
    count += 1
self.logger.info('Sent %d messages' % count)
connection.close()

我正在使用此代码将消息发送到RabbitMQ服务器。但偶尔这不会将所有消息发送到相应的队列。它每次运行时都会丢失随机数量的消息。

我无法理解这里的问题是什么。

Answer 1

您的邮件可能会被返回，因为它无法将邮件路由到任何现有队列。尝试在channel.confirm_delivery中添加回调：

channel.confirm_delivery(on_delivery_confirmation)

def on_delivery_confirmation(self, method_frame):
        confirmation_type = method_frame.method.NAME.split('.')[1].lower()            
        if confirmation_type == 'ack':
            self.logger.info('message published')
        elif confirmation_type == 'nack':
            self.logger.info('message not routed')

如果是这种情况，那么在发布消息之前，首先尝试使用交换和路由密钥绑定使用者队列。

Answer 2

简单的（不太可靠）方式

首先，启用持久队列添加：

channel.queue_declare(queue='your_queue', durable=True)

发布商和消费者 （在进行发布/消费之前）。

然后，即使RabbitMQ服务器死机并重新启动，您也可以确保您的队列不会丢失。

出版商

在发布商处，将properties=pika.BasicProperties(delivery_mode=2)添加到您的basic_publish来电，以确保您的消息是持久的。

channel.basic_publish(exchange=self.output_exchange_name,
                      routing_key=routing_key,
                      body=message_body.strip(),
                      properties=pika.BasicProperties(delivery_mode=2))

这应该可以避免丢失_published消息。

消费者

从消费者的角度来看，official RabbitMQ tutorial for python说：

为了确保消息永不丢失，RabbitMQ支持消息确认。从消费者发回 ack（nowledgement），告诉RabbitMQ已收到，处理了特定消息，RabbitMQ可以自由删除它。 [...]默认情况下会启用消息确认。

构建消费者时，请确保发送 ack 正确地，让RabbitMQ将其从队列中删除。

def callback(ch, method, properties, body):
    print "Received %r" % (body,)
    ch.basic_ack(delivery_tag = method.delivery_tag)

channel.basic_consume(callback, queue='your_queue')

真正安全的方式

如果您需要更强大，更可靠的方法来完全确定在RabbitMQ上发布确认中继，您应该使用AMQP协议的plublish confirm功能。

来自pika documentation：

import pika

# Open a connection to RabbitMQ on localhost using all default parameters
connection = pika.BlockingConnection()

# Open the channel
channel = connection.channel()

# Declare the queue
channel.queue_declare(queue="test", durable=True, exclusive=False, auto_delete=False)

# Turn on delivery confirmations
channel.confirm_delivery()

# Send a message
if channel.basic_publish(exchange='test',
                         routing_key='test',
                         body='Hello World!',
                         properties=pika.BasicProperties(content_type='text/plain',
                                                         delivery_mode=1)):
    print 'Message publish was confirmed'
else:
    print 'Message could not be confirmed'

所以根据你的代码，我会使用类似的东西：

count=0
for json_string in open(json_file, 'r'):
    result_json = json.loads(json_string)
    message_body = json.dumps(result_json['body'])
    routing_key = result_json['RoutingKey']
    if channel.basic_publish(exchange=self.output_exchange_name,routing_key=routing_key,body=message_body.strip(),
                             properties=pika.BasicProperties(delivery_mode=2)):  # Make it persistent
        count += 1
    else:
        # Do something with your undelivered message
self.logger.info('Sent %d messages' % count)
connection.close()

或者作为暴力方法，您可以使用while循环代替if来确保发送所有邮件：

count = 0
for json_string in open(json_file, 'r'):
    result_json = json.loads(json_string)
    message_body = json.dumps(result_json['body'])
    routing_key = result_json['RoutingKey']
    while not channel.basic_publish(exchange=self.output_exchange_name,
                                    routing_key=routing_key,
                                    body=message_body.strip(),
                                    properties=pika.BasicProperties(delivery_mode=2)):
        pass # Do nothing or even you can count retries
    count += 1
self.logger.info('Sent %d messages' % count)

Answer 3

尝试使用您的命令只接收一条消息：

#!/usr/bin/env python
import pika
import ujson as json


def receive():
    parameters = pika.ConnectionParameters(host='localhost')
    connection = pika.BlockingConnection(parameters)
    channel = connection.channel()
    channel.queue_declare(queue='raw_post', durable=True)

    method_frame, header_frame, body = channel.basic_get(queue='raw_post')

    if method_frame.NAME == 'Basic.GetEmpty':
        connection.close()
        return ''
    else:
        channel.basic_ack(delivery_tag=method_frame.delivery_tag)
        connection.close()
        return json.loads(body), method_frame.message_count


a = ''
while a not in ['quit', 'sair', 'exit', 'bye']:
    a = input("whats up?")
    print(receive())

只是一个有5000条消息要排队的发件人：

#!/usr/bin/env python
import pika
import ujson as json

connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='raw_post', durable=True)

for i in range(5000):
    info = {"info": "test", "value": i}

    channel.basic_publish(exchange='',
                          routing_key='raw_post',
                          body=json.dumps(info),
                          properties=pika.BasicProperties(
                              delivery_mode=2,  # make message persistent
                          ))

    print(" [x] Sent 'Hello World!' {}".format(i))
connection.close()

RabbitMq - pika - python - 发布时删除消息

3 个答案:

简单的（不太可靠）方式

出版商

消费者

真正安全的方式