Hi all.

We have been testing the gcpubsub broker and we have seen a quite consistent case where some tasks got delayed for 4 minutes before being executed. This 4 minutes matches the default acknowledgement deadline that was set on the kombu-celery subscription. The behaviour happened quite frequently on task that performs self.retry (Note: we use both Celery canvas and standalone task). We also tried to change the deadline and the delay also matched the updated value so it looked like to us that the message was unavailable until it was redelivered after the acknowledgement deadline has past.

Upon further investigation, we can see that when a retry was performed, the task was pushed to the topic, but the subsequent pulls in Celery were unsuccessful and timed out until the 4 minutes period has past. We believe that at the 4 minutes mark, the pub/sub message was marked available for redelivery so it was delivered eventually. The kombu-celery subscription metrics also indicated that the task was successfully delivered and was waiting for acknowledgement during this window. Thus, our hypothesis is that, the timeout kicked in after the message was sent from the Pub/Sub servers but just before it was received by the client, causing the message to be marked as delivered and became unavailable until the ack deadline has past.

Did anyone experience the same behaviour and managed to find a solution? We are suspecting networking and potential Celery threading might play a role in this but cannot confirm at this point. We are still working with Google to confirm this.