亚马逊SQS死信队列:它真的是死信还是毒药?

时间:2020-12-29 17:37:38

I'm trying to get clarification on what exactly Amazon's SQS Dead Letter Queue is doing.

我正在努力澄清亚马逊的SQS死信队列究竟在做什么。

According to http://aws.typepad.com/aws/2014/01/amazon-sqs-new-dead-letter-queue.html

根据http://aws.typepad.com/aws/2014/01/amazon-sqs-new-dead-letter-queue.html

Dead Letter Queue - The ARN (Amazon Resource Name) of an SQS queue that will receive the messages which were not successfully processed after maximum number of receives by consumers.

死信队列 - SQS队列的ARN(亚马逊资源名称),它将接收消费者在接收到最大数量后未成功处理的消息。

Doesn't that sound more like a Poision Queue? The key distinction being that the consumers did receive the message. A dead letter would be when the message is potentially fine, but can't be delivered, probably due to a service outage. http://www.eaipatterns.com/DeadLetterChannel.html

这听起来不像Poision Queue吗?关键区别在于消费者确实收到了消息。当信息可能正常,但无法传递时,可能是由于服务中断,这是一封死信。 http://www.eaipatterns.com/DeadLetterChannel.html

Where as this sounds like the message is being successfully received multiple times, but processing the message fails, which I understand to be the meaning of a Poison Message Queue.

这听起来好像消息被多次成功接收,但处理消息失败,我理解这是毒药消息队列的含义。

Message Bus vs Queue

消息总线与队列

Does the Dead Letter Pattern have different meaning in the context of a plain old queue? Since SQS is just a queue, not a message bus, it isn't responsible for delivering messages. Instead it waits for messages to be picked up (requested). So the traditional Dead Letter pattern doesn't really apply since there isn't a message bus attempting to deliver a message and not being able to find a recipient.

死信模式在普通旧队列的上下文中有不同的含义吗?由于SQS只是一个队列,而不是消息总线,因此它不负责传递消息。相反,它等待拾取(请求)消息。因此传统的死信模式并不真正适用,因为没有消息总线尝试传递消息而无法找到收件人。

Can SQS behave like a message bus?

SQS可以像消息总线一样吗?

Is there a way through SQS to set up channels and listeners instead of explicitly polling for messages from the queue?

是否有办法通过SQS设置通道和侦听器,而不是显式轮询队列中的消息?

3 个解决方案

#1


13  

Good question.

好问题。

Based on the definition from the canonical source, which you quoted (citations removed for clarity):

基于您引用的规范来源的定义(为清晰起见,删除了引文):

The specific way a Dead Letter Channel works depends on the specific messaging system’s implementation, if it provides one at all. The channel may be called a “dead message queue” or “dead letter queue.” Typically, each machine the messaging system is installed on has its own local Dead Letter Channel so that whatever machine a message dies on, it can be moved from one local queue to another without any networking uncertainties. This also records what machine the message died on. When the messaging system moves the message, it may also record the original channel the message was supposed to be delivered on.

Dead Letter Channel工作的具体方式取决于特定消息传递系统的实现,如果它提供了一个。通道可以被称为“死信息队列”或“死信队列”。通常,安装消息系统的每台机器都有自己的本地死信通道,这样无论机器上的消息是什么,它都可以从一个机器移动本地队列到另一个没有任何网络不确定性。这也记录了消息死亡的机器。当消息传递系统移动消息时,它还可以记录消息应该被传递的原始信道。

...it's not clear if there's really a difference. I understand what you mean by "poison queue," and your understanding of how SQS works is sound. Semantically, the difference between a DLQ and a PQ -- "undeliverable" in the style of email versus "poison" -- isn't clear to me. Perhaps a PQ is a flavor of a DLQ.

......目前尚不清楚是否真的存在差异。我明白你的意思是“毒药队列”,你对SQS如何运作的理解是合理的。从语义上讲,DLQ和PQ之间的区别 - 电子邮件与“毒药”风格“无法传递” - 对我来说并不清楚。也许PQ是DLQ的味道。

FWIW, ActiveMQ's redelivery policy uses the same definition of DLQ -- a hybrid DLQ / PQ -- as SQS does.

FWIW,ActiveMQ的重新传递策略使用与SQS相同的DLQ定义 - 混合DLQ / PQ。

Can SQS behave like a message bus?

SQS可以像消息总线一样吗?

SQS can't, but there are similar products that can.

SQS不能,但有类似的产品可以。

  1. Amazon SNS

    亚马逊SNS

    SNS (Simple Notification Service) is a generalized publish-subscribe topic system. SNS allows you to create topics, and then register subscribers that receive push notifications. Currently, push notifications can come in the form of HTTP/S, email, SMS, SQS, and mobile device push notifications.

    SNS(简单通知服务)是一种通用的发布 - 订阅主题系统。 SNS允许您创建主题,然后注册接收推送通知的订阅者。目前,推送通知可以以HTTP / S,电子邮件,SMS,SQS和移动设备推送通知的形式出现。

    SNS has a pretty sane retry policy for HTTP/S, but does not support a DLQ or PQ AFAIK.

    SNS对HTTP / S有一个非常合理的重试策略,但不支持DLQ或PQ AFAIK。

  2. IronMQ's Push Queues

    IronMQ的推送队列

    IronMQ is another REST-ful message queueing service that is a little more fully-featured than SQS. (True FIFO message ordering, longer delays, and so on, but sadly smaller message sizes.) Push queues allow you to set up push "subscribers," which then receive an HTTP POST any time a new message is put onto the queue.

    IronMQ是另一种REST-ful消息排队服务,它比SQS功能更强大。 (真正的FIFO消息排序,更长的延迟等等,但遗憾的是消息大小更小。)推送队列允许您设置推送“订户”,然后在将新消息放入队列时接收HTTP POST。

    If IronMQ fails to deliver a message -- the HTTP POST times out, or your endpoint returns anything but a 2xx -- then it will retry the delivery. If it runs out of retries, then it will put the message onto an error queue -- a combination DLQ and PQ in this case.

    如果IronMQ无法传递消息 - HTTP POST超时,或者您的端点返回除2xx之外的任何内容 - 那么它将重试传递。如果它没有重试,那么它会将消息放入错误队列 - 在这种情况下组合DLQ和PQ。

    This is probably as close as you're going to get to a true "ESB" in a managed service.

    这可能与您在托管服务中获得真正的“ESB”一样接近。

Of course, then there are true open-source ESBs and SOA frameworks -- MULE, ServiceMix, and so on -- but I don't know nearly enough about what you're trying to do to make any kind of recommendation there. :)

当然,那里有真正的开源ESB和SOA框架 - MULE,ServiceMix等等 - 但我不太了解你要做什么来在那里做任何推荐。 :)

#2


2  

I'm not sure in most cases that a distinction between DLQ and PQ is necessary. In fact I find this definition to be rather arbitrary. For most transactional messaging implementations if the message isn't successfully consumed off the queue within the specified number of retries it goes to the DLQ. Having a separate queue for malformed messages means that you now just have two places to look for your messages that aren't being successfully processed, two exception queues to monitor or operational considerations, and some percentage of messages that seem like they might belong in either queue(batch processing scenarios come to mind).

在大多数情况下,我不确定DLQ和PQ之间的区别是必要的。事实上,我发现这个定义是相当武断的。对于大多数事务性消息传递实现,如果消息未在指定的重试次数内从队列中成功消耗,则它将转到DLQ。对于格式错误的消息具有单独的队列意味着您现在只有两个位置来查找未成功处理的消息,两个要监视的异常队列或操作注意事项,以及一些看起来可能属于这两者的消息的百分比队列(批处理场景)。

#3


0  

No, it will not behave like an active ESB. Simple Queue Service is simple by definition. There is an "at-least-once" delivery guarantee, but beyond that it makes very few promises.

不,它不会像活跃的ESB那样行事。根据定义,简单队列服务很简单。有一个“至少一次”的交付保证,但除此之外它几乎没有承诺。

It's designed only for polling/long polling. You can have multiple queues each serving a different purpose, but a single queue is very simple and not intended to service multiple functions or provide advanced logic. SWF may provide what you want, but chances are you'll need to implement an ESB.

它仅适用于轮询/长轮询。您可以拥有多个队列,每个队列服务于不同的目的,但单个队列非常简单,并非旨在为多个功能提供服务或提供高级逻辑。 SWF可能提供您想要的,但您可能需要实施ESB。

http://aws.amazon.com/swf/

http://aws.amazon.com/swf/

#1


13  

Good question.

好问题。

Based on the definition from the canonical source, which you quoted (citations removed for clarity):

基于您引用的规范来源的定义(为清晰起见,删除了引文):

The specific way a Dead Letter Channel works depends on the specific messaging system’s implementation, if it provides one at all. The channel may be called a “dead message queue” or “dead letter queue.” Typically, each machine the messaging system is installed on has its own local Dead Letter Channel so that whatever machine a message dies on, it can be moved from one local queue to another without any networking uncertainties. This also records what machine the message died on. When the messaging system moves the message, it may also record the original channel the message was supposed to be delivered on.

Dead Letter Channel工作的具体方式取决于特定消息传递系统的实现,如果它提供了一个。通道可以被称为“死信息队列”或“死信队列”。通常,安装消息系统的每台机器都有自己的本地死信通道,这样无论机器上的消息是什么,它都可以从一个机器移动本地队列到另一个没有任何网络不确定性。这也记录了消息死亡的机器。当消息传递系统移动消息时,它还可以记录消息应该被传递的原始信道。

...it's not clear if there's really a difference. I understand what you mean by "poison queue," and your understanding of how SQS works is sound. Semantically, the difference between a DLQ and a PQ -- "undeliverable" in the style of email versus "poison" -- isn't clear to me. Perhaps a PQ is a flavor of a DLQ.

......目前尚不清楚是否真的存在差异。我明白你的意思是“毒药队列”,你对SQS如何运作的理解是合理的。从语义上讲,DLQ和PQ之间的区别 - 电子邮件与“毒药”风格“无法传递” - 对我来说并不清楚。也许PQ是DLQ的味道。

FWIW, ActiveMQ's redelivery policy uses the same definition of DLQ -- a hybrid DLQ / PQ -- as SQS does.

FWIW,ActiveMQ的重新传递策略使用与SQS相同的DLQ定义 - 混合DLQ / PQ。

Can SQS behave like a message bus?

SQS可以像消息总线一样吗?

SQS can't, but there are similar products that can.

SQS不能,但有类似的产品可以。

  1. Amazon SNS

    亚马逊SNS

    SNS (Simple Notification Service) is a generalized publish-subscribe topic system. SNS allows you to create topics, and then register subscribers that receive push notifications. Currently, push notifications can come in the form of HTTP/S, email, SMS, SQS, and mobile device push notifications.

    SNS(简单通知服务)是一种通用的发布 - 订阅主题系统。 SNS允许您创建主题,然后注册接收推送通知的订阅者。目前,推送通知可以以HTTP / S,电子邮件,SMS,SQS和移动设备推送通知的形式出现。

    SNS has a pretty sane retry policy for HTTP/S, but does not support a DLQ or PQ AFAIK.

    SNS对HTTP / S有一个非常合理的重试策略,但不支持DLQ或PQ AFAIK。

  2. IronMQ's Push Queues

    IronMQ的推送队列

    IronMQ is another REST-ful message queueing service that is a little more fully-featured than SQS. (True FIFO message ordering, longer delays, and so on, but sadly smaller message sizes.) Push queues allow you to set up push "subscribers," which then receive an HTTP POST any time a new message is put onto the queue.

    IronMQ是另一种REST-ful消息排队服务,它比SQS功能更强大。 (真正的FIFO消息排序,更长的延迟等等,但遗憾的是消息大小更小。)推送队列允许您设置推送“订户”,然后在将新消息放入队列时接收HTTP POST。

    If IronMQ fails to deliver a message -- the HTTP POST times out, or your endpoint returns anything but a 2xx -- then it will retry the delivery. If it runs out of retries, then it will put the message onto an error queue -- a combination DLQ and PQ in this case.

    如果IronMQ无法传递消息 - HTTP POST超时,或者您的端点返回除2xx之外的任何内容 - 那么它将重试传递。如果它没有重试,那么它会将消息放入错误队列 - 在这种情况下组合DLQ和PQ。

    This is probably as close as you're going to get to a true "ESB" in a managed service.

    这可能与您在托管服务中获得真正的“ESB”一样接近。

Of course, then there are true open-source ESBs and SOA frameworks -- MULE, ServiceMix, and so on -- but I don't know nearly enough about what you're trying to do to make any kind of recommendation there. :)

当然,那里有真正的开源ESB和SOA框架 - MULE,ServiceMix等等 - 但我不太了解你要做什么来在那里做任何推荐。 :)

#2


2  

I'm not sure in most cases that a distinction between DLQ and PQ is necessary. In fact I find this definition to be rather arbitrary. For most transactional messaging implementations if the message isn't successfully consumed off the queue within the specified number of retries it goes to the DLQ. Having a separate queue for malformed messages means that you now just have two places to look for your messages that aren't being successfully processed, two exception queues to monitor or operational considerations, and some percentage of messages that seem like they might belong in either queue(batch processing scenarios come to mind).

在大多数情况下,我不确定DLQ和PQ之间的区别是必要的。事实上,我发现这个定义是相当武断的。对于大多数事务性消息传递实现,如果消息未在指定的重试次数内从队列中成功消耗,则它将转到DLQ。对于格式错误的消息具有单独的队列意味着您现在只有两个位置来查找未成功处理的消息,两个要监视的异常队列或操作注意事项,以及一些看起来可能属于这两者的消息的百分比队列(批处理场景)。

#3


0  

No, it will not behave like an active ESB. Simple Queue Service is simple by definition. There is an "at-least-once" delivery guarantee, but beyond that it makes very few promises.

不,它不会像活跃的ESB那样行事。根据定义,简单队列服务很简单。有一个“至少一次”的交付保证,但除此之外它几乎没有承诺。

It's designed only for polling/long polling. You can have multiple queues each serving a different purpose, but a single queue is very simple and not intended to service multiple functions or provide advanced logic. SWF may provide what you want, but chances are you'll need to implement an ESB.

它仅适用于轮询/长轮询。您可以拥有多个队列,每个队列服务于不同的目的,但单个队列非常简单,并非旨在为多个功能提供服务或提供高级逻辑。 SWF可能提供您想要的,但您可能需要实施ESB。

http://aws.amazon.com/swf/

http://aws.amazon.com/swf/