Quorum queue not remove member that not exist anymore on cluster. #11213
-
|
I have a RabbitMQ cluster with three nodes, using quorum queues distributed among the three nodes. This cluster is set up to utilize an auto-scaling group. When an EC2 instance dies and another instance joins the RabbitMQ cluster, the quorum queues, where the node that died was the leader, still reference it as if it remained active. In the queue details, we see three nodes online, but in the members section, there are four nodes listed. The problem occurs with the quorum queues when another node, which was a pair of the node that died, also fails. I have tried removing this reference using the following commands: rabbitmq-queues shrink I also tried running some commands from the other nodes like: rabbitmq-queues grow rabbit@hostname all However, these attempts have not resolved the issue. In conf the rabbitmq is set cluster_partition_handling = autoheal RabbitMQ 3.13.1Erlang 26.2.3 If anyone can help me |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 4 replies
-
|
I think @SimonUnge can you submit the docs for this feature? Thanks |
Beta Was this translation helpful? Give feedback.
-
|
For quorum queue is a good practice work with Autoscaling Group ? Or to quorum queue is bettler work with fixed instance ? |
Beta Was this translation helpful? Give feedback.
-
|
Static membership is definitely preferred. However, there are RabbitMQ users who rely on autoscaling groups, which is why AWS contributed improvements around support for autoscaling groups. |
Beta Was this translation helpful? Give feedback.
-
|
I've a question in rabbitmq version 3.13.2 for this scenario is necessary set "quorum_queue.continuous_membership_reconciliation.auto_remove = true" in rabbitmq.config ? or has it already been adjusted ? |
Beta Was this translation helpful? Give feedback.
-
|
no, it's |
Beta Was this translation helpful? Give feedback.
-
Yes, let me add a section in the QQ chapter! |
Beta Was this translation helpful? Give feedback.
-
|
Is this issue solved? It looks like we have the same problem: a 3 node cluster in AWS with autoscaling. Terminating one instance, then we wait untill the new instance comes back with all the required software and config. The quorum queues still thinks the 'old' leader is down.
the nodes are up
Our config is ` /etc/rabbitmq/rabbitmq.confloopback_users.guest = true ssl_options.cacertfile = /opt/certs/trusted.pem log.file.level = debug default_user = rmqadmin cluster_formation.peer_discovery_backend = aws management.ssl.port = 15672 quorum_queue.continuous_membership_reconciliation.auto_remove = true cluster_partition_handling = autoheal |
Beta Was this translation helpful? Give feedback.
-
|
@mkelderm our team does not appreciate hijacking of existing issues. We won't comment on "reports" that do not even mention RabbitMQ versions and provide no logs either. |
Beta Was this translation helpful? Give feedback.
I think
quorum_queue.continuous_membership_reconciliation.auto_remove = trueis what you're looking for, see #8218. Seems like we forgot to add these setting to the docs.@SimonUnge can you submit the docs for this feature? Thanks