Skip to content

[Messenger] Resend failed messages to failure transport using a delay from retry strategy #57672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: 6.4
Choose a base branch
from
Open

[Messenger] Resend failed messages to failure transport using a delay from retry strategy #57672

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c79d4cb
[Messenger] Resend failed messages to failure transport
tunterreitmeier Jul 8, 2024
6acfab5
[FrameworkBundle] Inject default retry strategy locator into messenge…
tunterreitmeier Jul 7, 2024
ea61fcb
[Messenger][Amqp] Do not use redelivery routing key when sending to f…
tunterreitmeier Jul 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[Messenger] Resend failed messages to failure transport 
Use retry strategy with an increasing delay to prevent handling failed messages too fast/often
  • Loading branch information
@tunterreitmeier
tunterreitmeier committed Jul 8, 2024
commit c79d4cbd1c97dad238eb12fa23cc45f12df4c11a
35 changes: 25 additions & 10 deletions ...Symfony/Component/Messenger/EventListener/SendFailedMessageToFailureTransportListener.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@
use Psr\Log\LoggerInterface;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;
use Symfony\Component\Messenger\Event\WorkerMessageFailedEvent;
use Symfony\Component\Messenger\Retry\RetryStrategyInterface;
use Symfony\Component\Messenger\Stamp\DelayStamp;
use Symfony\Component\Messenger\Stamp\ReceivedStamp;
use Symfony\Component\Messenger\Stamp\RedeliveryStamp;
use Symfony\Component\Messenger\Stamp\SentToFailureTransportStamp;

Expand All @@ -28,11 +30,13 @@ class SendFailedMessageToFailureTransportListener implements EventSubscriberInte
{
private ContainerInterface $failureSenders;
private ?LoggerInterface $logger;
private ?ContainerInterface $retryStrategyLocator;

public function __construct(ContainerInterface $failureSenders, ?LoggerInterface $logger = null)
public function __construct(ContainerInterface $failureSenders, ?LoggerInterface $logger = null, ?ContainerInterface $retryStrategyLocator = null)
{
$this->failureSenders = $failureSenders;
$this->logger = $logger;
$this->retryStrategyLocator = $retryStrategyLocator;
}

/**
Expand All @@ -44,28 +48,30 @@ public function onMessageFailed(WorkerMessageFailedEvent $event)
return;
}

if (!$this->failureSenders->has($event->getReceiverName())) {
$originalTransportName = $event->getEnvelope()->last(ReceivedStamp::class)
?->getTransportName() ?? $event->getReceiverName();

if (!$this->failureSenders->has($originalTransportName)) {
return;
}

$failureSender = $this->failureSenders->get($event->getReceiverName());
$failureSender = $this->failureSenders->get($originalTransportName);

$envelope = $event->getEnvelope();

// avoid re-sending to the failed sender
if (null !== $envelope->last(SentToFailureTransportStamp::class)) {
return;
}
$delay = $this->getRetryStrategyForTransport($event->getReceiverName())
?->getWaitingTime($envelope, $event->getThrowable()) ?? 0;

$envelope = $envelope->with(
new SentToFailureTransportStamp($event->getReceiverName()),
new DelayStamp(0),
new SentToFailureTransportStamp($originalTransportName),
new DelayStamp($delay),
new RedeliveryStamp(0)
);

$this->logger?->info('Rejected message {class} will be sent to the failure transport {transport}.', [
$this->logger?->info('Rejected message {class} will be sent to the failure transport {transport} using {delay} ms delay.', [
'class' => $envelope->getMessage()::class,
'transport' => $failureSender::class,
'delay' => $delay,
]);

$failureSender->send($envelope);
Expand All @@ -77,4 +83,13 @@ public static function getSubscribedEvents(): array
WorkerMessageFailedEvent::class => ['onMessageFailed', -100],
];
}

private function getRetryStrategyForTransport(string $transportName): ?RetryStrategyInterface
{
if (null === $this->retryStrategyLocator || !$this->retryStrategyLocator->has($transportName)) {
return null;
}

return $this->retryStrategyLocator->get($transportName);
}
}
2 changes: 1 addition & 1 deletion src/Symfony/Component/Messenger/Retry/MultiplierRetryStrategy.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ public function isRetryable(Envelope $message, ?\Throwable $throwable = null): b
*/
public function getWaitingTime(Envelope $message, ?\Throwable $throwable = null): int
{
$retries = RedeliveryStamp::getRetryCountFromEnvelope($message);
$retries = \count($message->all(RedeliveryStamp::class));

$delay = $this->delayMilliseconds * $this->multiplier ** $retries;

Expand Down
71 changes: 65 additions & 6 deletions ...mponent/Messenger/Tests/EventListener/SendFailedMessageToFailureTransportListenerTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
use Symfony\Component\Messenger\Envelope;
use Symfony\Component\Messenger\Event\WorkerMessageFailedEvent;
use Symfony\Component\Messenger\EventListener\SendFailedMessageToFailureTransportListener;
use Symfony\Component\Messenger\Retry\RetryStrategyInterface;
use Symfony\Component\Messenger\Stamp\DelayStamp;
use Symfony\Component\Messenger\Stamp\ReceivedStamp;
use Symfony\Component\Messenger\Stamp\SentToFailureTransportStamp;
use Symfony\Component\Messenger\Transport\Sender\SenderInterface;

Expand All @@ -29,6 +32,10 @@ public function testItSendsToTheFailureTransportWithSenderLocator()
/* @var Envelope $envelope */
$this->assertInstanceOf(Envelope::class, $envelope);

$delayStamp = $envelope->last(DelayStamp::class);
$this->assertNotNull($delayStamp);
$this->assertSame(5000, $delayStamp->getDelay());

/** @var SentToFailureTransportStamp $sentToFailureTransportStamp */
$sentToFailureTransportStamp = $envelope->last(SentToFailureTransportStamp::class);
$this->assertNotNull($sentToFailureTransportStamp);
Expand All @@ -40,6 +47,36 @@ public function testItSendsToTheFailureTransportWithSenderLocator()
$serviceLocator = $this->createMock(ServiceLocator::class);
$serviceLocator->expects($this->once())->method('has')->willReturn(true);
$serviceLocator->expects($this->once())->method('get')->with($receiverName)->willReturn($sender);

$retryStrategy = $this->createMock(RetryStrategyInterface::class);
$retryStrategy->expects($this->once())->method('getWaitingTime')->willReturn(5000);

$retryStrategyLocator = $this->createMock(ServiceLocator::class);
$retryStrategyLocator->expects($this->once())->method('has')->with($receiverName)->willReturn(true);
$retryStrategyLocator->expects($this->once())->method('get')->with($receiverName)->willReturn($retryStrategy);

$listener = new SendFailedMessageToFailureTransportListener($serviceLocator, null, $retryStrategyLocator);

$exception = new \Exception('no!');
$envelope = new Envelope(new \stdClass());
$event = new WorkerMessageFailedEvent($envelope, 'my_receiver', $exception);

$listener->onMessageFailed($event);
}

public function testItSendsToTheFailureTransportWithoutRetryStrategyLocator()
{
$sender = $this->createMock(SenderInterface::class);
$sender->expects($this->once())->method('send')->with($this->callback(function (Envelope $envelope) {
$this->assertSame(0, $envelope->last(DelayStamp::class)->getDelay());

return true;
}))->willReturnArgument(0);

$serviceLocator = $this->createStub(ServiceLocator::class);
$serviceLocator->method('has')->willReturn(true);
$serviceLocator->method('get')->willReturn($sender);

$listener = new SendFailedMessageToFailureTransportListener($serviceLocator);

$exception = new \Exception('no!');
Expand All @@ -55,7 +92,8 @@ public function testDoNothingOnRetryWithServiceLocator()
$sender->expects($this->never())->method('send');

$serviceLocator = $this->createMock(ServiceLocator::class);
$listener = new SendFailedMessageToFailureTransportListener($serviceLocator);
$retryStrategyLocator = $this->createStub(ServiceLocator::class);
$listener = new SendFailedMessageToFailureTransportListener($serviceLocator, null, $retryStrategyLocator);

$envelope = new Envelope(new \stdClass());
$event = new WorkerMessageFailedEvent($envelope, 'my_receiver', new \Exception());
Expand All @@ -64,19 +102,40 @@ public function testDoNothingOnRetryWithServiceLocator()
$listener->onMessageFailed($event);
}

public function testDoNotRedeliverToFailedWithServiceLocator()
public function testDoRedeliverToFailedWithServiceLocator()
{
$receiverName = 'my_receiver';
$failedReceiver = 'failed_receiver';

$sender = $this->createMock(SenderInterface::class);
$sender->expects($this->never())->method('send');
$sender->expects($this->once())->method('send')->with($this->callback(function (Envelope $envelope) use ($receiverName) {
$delayStamp = $envelope->last(DelayStamp::class);
$this->assertNotNull($delayStamp);
$this->assertSame(1000, $delayStamp->getDelay());

$sentToFailureTransportStamp = $envelope->last(SentToFailureTransportStamp::class);
$this->assertNotNull($sentToFailureTransportStamp);
$this->assertSame($receiverName, $sentToFailureTransportStamp->getOriginalReceiverName());

return true;
}))->willReturnArgument(0);
$serviceLocator = $this->createMock(ServiceLocator::class);
$serviceLocator->expects($this->once())->method('has')->willReturn(true);
$serviceLocator->expects($this->once())->method('get')->with($receiverName)->willReturn($sender);

$listener = new SendFailedMessageToFailureTransportListener($serviceLocator);
$retryStrategy = $this->createStub(RetryStrategyInterface::class);
$retryStrategy->method('getWaitingTime')->willReturn(1000);
$retryStrategyLocator = $this->createMock(ServiceLocator::class);
$retryStrategyLocator->expects($this->once())->method('has')->willReturn(true);
$retryStrategyLocator->expects($this->once())->method('get')->with($failedReceiver)->willReturn($retryStrategy);

$listener = new SendFailedMessageToFailureTransportListener($serviceLocator, null, $retryStrategyLocator);
$envelope = new Envelope(new \stdClass(), [
// the received stamp is assumed to be added by the FailedMessageProcessingMiddleware
new ReceivedStamp($receiverName),
new SentToFailureTransportStamp($receiverName),
]);
$event = new WorkerMessageFailedEvent($envelope, $receiverName, new \Exception());
$event = new WorkerMessageFailedEvent($envelope, $failedReceiver, new \Exception());

$listener->onMessageFailed($event);
}
Expand All @@ -87,7 +146,7 @@ public function testDoNothingIfFailureTransportIsNotDefined()
$sender->expects($this->never())->method('send');

$serviceLocator = $this->createMock(ServiceLocator::class);
$listener = new SendFailedMessageToFailureTransportListener($serviceLocator, null);
$listener = new SendFailedMessageToFailureTransportListener($serviceLocator);

$exception = new \Exception('no!');
$envelope = new Envelope(new \stdClass());
Expand Down
40 changes: 27 additions & 13 deletions src/Symfony/Component/Messenger/Tests/FailureIntegrationTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@

use PHPUnit\Framework\TestCase;
use Psr\Container\ContainerInterface;
use Psr\Log\NullLogger;
use Symfony\Component\DependencyInjection\ServiceLocator;
use Symfony\Component\EventDispatcher\EventDispatcher;
use Symfony\Component\Messenger\Envelope;
Expand Down Expand Up @@ -115,7 +114,11 @@ public function testRequeueMechanism()
$dispatcher->addSubscriber(new AddErrorDetailsStampListener());
$dispatcher->addSubscriber(new SendFailedMessageForRetryListener($locator, $retryStrategyLocator));

$dispatcher->addSubscriber(new SendFailedMessageToFailureTransportListener($sendersLocatorFailureTransport));
$dispatcher->addSubscriber(new SendFailedMessageToFailureTransportListener(
$sendersLocatorFailureTransport,
null,
$retryStrategyLocator
));
$dispatcher->addSubscriber(new StopWorkerOnMessageLimitListener(1));

$runWorker = function (string $transportName) use ($transports, $bus, $dispatcher): ?\Throwable {
Expand Down Expand Up @@ -195,14 +198,14 @@ public function testRequeueMechanism()
$this->assertCount(1, $transport2->getMessagesWaitingToBeReceived());

/*
* Message is retried on failure transport then discarded
* Message is retried on failure transport then re-queued
*/
$runWorker('the_failure_transport');
// only the "failed" handler is called a 4th time
$this->assertSame(4, $transport1HandlerThatFails->getTimesCalled());
$this->assertSame(1, $allTransportHandlerThatWorks->getTimesCalled());
// handling fails again, message is discarded
$this->assertCount(0, $failureTransport->getMessagesWaitingToBeReceived());
// handling fails again, message is re-queued with a delay
$this->assertCount(1, $failureTransport->getMessagesWaitingToBeReceived());

/*
* Execute handlers on transport2
Expand All @@ -214,10 +217,10 @@ public function testRequeueMechanism()
$this->assertSame(2, $allTransportHandlerThatWorks->getTimesCalled());
// transport1 handler called for the first time
$this->assertSame(1, $transport2HandlerThatWorks->getTimesCalled());
// all transport should be empty
// all original transports should be empty - failed queue still holds the message
$this->assertEmpty($transport1->getMessagesWaitingToBeReceived());
$this->assertEmpty($transport2->getMessagesWaitingToBeReceived());
$this->assertEmpty($failureTransport->getMessagesWaitingToBeReceived());
$this->assertCount(1, $failureTransport->getMessagesWaitingToBeReceived());

/*
* Dispatch the original message again
Expand All @@ -226,9 +229,18 @@ public function testRequeueMechanism()
// handle the failing message so it goes into the failure transport
$runWorker('transport1');
$runWorker('transport1');
$this->assertCount(2, $failureTransport->getMessagesWaitingToBeReceived());
// now make the handler work!
$transport1HandlerThatFails->setShouldThrow(false);
$runWorker('the_failure_transport');
$runWorker('the_failure_transport');

// the message is now handled and the failure transport is empty
// transport1Handler is called 4 more times - 2 retries and twice successfully from failure transport
$this->assertSame(8, $transport1HandlerThatFails->getTimesCalled());
$this->assertSame(3, $allTransportHandlerThatWorks->getTimesCalled());
$this->assertSame(1, $transport2HandlerThatWorks->getTimesCalled());

// the failure transport is empty because it worked
$this->assertEmpty($failureTransport->getMessagesWaitingToBeReceived());
}
Expand Down Expand Up @@ -297,7 +309,8 @@ public function testMultipleFailedTransportsWithoutGlobalFailureTransport()
$dispatcher->addSubscriber(new SendFailedMessageForRetryListener($locator, $retryStrategyLocator));
$dispatcher->addSubscriber(new SendFailedMessageToFailureTransportListener(
$sendersLocatorFailureTransport,
new NullLogger()
null,
$retryStrategyLocator,
));
$dispatcher->addSubscriber(new StopWorkerOnMessageLimitListener(1));

Expand Down Expand Up @@ -342,7 +355,8 @@ public function testMultipleFailedTransportsWithoutGlobalFailureTransport()
// "transport1" handler is called again from the "the_failed_transport1" and it fails
$this->assertSame(2, $transport1HandlerThatFails->getTimesCalled());
$this->assertSame(0, $transport2HandlerThatFails->getTimesCalled());
$this->assertCount(0, $failureTransport1->getMessagesWaitingToBeReceived());
// message is not discarded but remains in failed transport
$this->assertCount(1, $failureTransport1->getMessagesWaitingToBeReceived());
$this->assertCount(0, $failureTransport2->getMessagesWaitingToBeReceived());

// Receive the message from "transport2"
Expand All @@ -351,7 +365,7 @@ public function testMultipleFailedTransportsWithoutGlobalFailureTransport()
$this->assertSame(2, $transport1HandlerThatFails->getTimesCalled());
// handler for "transport2" is called
$this->assertSame(1, $transport2HandlerThatFails->getTimesCalled());
$this->assertCount(0, $failureTransport1->getMessagesWaitingToBeReceived());
$this->assertCount(1, $failureTransport1->getMessagesWaitingToBeReceived());
// the failure transport "the_failure_transport2" has 1 new message failed from "transport2"
$this->assertCount(1, $failureTransport2->getMessagesWaitingToBeReceived());

Expand All @@ -360,9 +374,9 @@ public function testMultipleFailedTransportsWithoutGlobalFailureTransport()
$this->assertSame(2, $transport1HandlerThatFails->getTimesCalled());
// "transport2" handler is called again from the "the_failed_transport2" and it fails
$this->assertSame(2, $transport2HandlerThatFails->getTimesCalled());
$this->assertCount(0, $failureTransport1->getMessagesWaitingToBeReceived());
// After the message fails again, the message is discarded from the "the_failure_transport2"
$this->assertCount(0, $failureTransport2->getMessagesWaitingToBeReceived());
$this->assertCount(1, $failureTransport1->getMessagesWaitingToBeReceived());
// After the message fails again, the message is re-queued to the "the_failure_transport2"
$this->assertCount(1, $failureTransport2->getMessagesWaitingToBeReceived());
}

public function testStampsAddedByMiddlewaresDontDisappearWhenDelayedMessageFails()
Expand Down
5 changes: 4 additions & 1 deletion src/Symfony/Component/Messenger/Tests/Retry/MultiplierRetryStrategyTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,10 @@ public function testIsRetryableWithNoStamp()
public function testGetWaitTime(int $delay, float $multiplier, int $maxDelay, int $previousRetries, int $expectedDelay)
{
$strategy = new MultiplierRetryStrategy(10, $delay, $multiplier, $maxDelay);
$envelope = new Envelope(new \stdClass(), [new RedeliveryStamp($previousRetries)]);
$envelope = Envelope::wrap(new \stdClass());
for ($i = 0; $i < $previousRetries; ++$i) {
$envelope = $envelope->with(new RedeliveryStamp($i));
}

$this->assertSame($expectedDelay, $strategy->getWaitingTime($envelope));
}
Expand Down
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy