NSQ: Requeue vs Requeue Without Backoff

NSQ enables us to requeue a message with a delay duration. There are two functions to requeue messages, that is Requeue and RequeueWithoutBackoff. This article will show you the differences and help you choose the most suitable function for your use case.

I am using go-nsq (https://github.com/nsqio/go-nsq) for the client library of the NSQ. This is the official client library for Go. Below is a sample of a consumer function.

    func myConsumer(msg *nsq.Message) error {
        msgStr := string(msg.Body)
        log.Printf("consuming %s\n", msgStr)
        msq.Finish()
        return nil
    }

In the sample, we call the function Finish of the message. This will tell the nsqd that the message is already consumed. We can also requeue the message by calling function Requeue or RequeueWithoutBackoff instead of function Finish. But what is the difference between Requeue and RequeueWithoutBackoff? And which one should we use?

Requeue

The Requeue function put the message back to the queue with a specified delay. This message will be consumed again after the delay time elapsed. The Requeue function also tells the consumer to back off for a certain time. It means that the consumer won’t consume any message until back off time elapsed. The sample usage of Requeue function:

    func myConsumer(msg *nsq.Message) error {
        msgStr := string(msg.Body)
        log.Printf("[myConsumer] consuming: %s\n", msgStr)
        if msgStr == "message1" {
            msg.Requeue(10 * time.Second)
        }

        return nil
    }

The consumer will requeue messages with 10 seconds delay if it consumes message1. Let’s see what would happen to the messages and the consumer. We publish some messages to the topic at the same time to test the consumer with this command.

myuser :: ~ » curl -X POST http://localhost:4151/mpub\?topic\=mytopic -d 'message1
message2
message3'

Let’s see the result log

2020/09/17 14:45:40 INF    1 [mytopic/mychannel] querying nsqlookupd http://localhost:4161/lookup?topic=mytopic
2020/09/17 14:45:40 INF    1 [mytopic/mychannel] (192.168.100.17:4150) connecting to nsqd
2020/09/17 14:45:40 [myConsumer] consuming: message1
2020/09/17 14:45:40 WRN    1 [mytopic/mychannel] backing off for 2s (backoff level 1), setting all to RDY 0
2020/09/17 14:45:42 WRN    1 [mytopic/mychannel] (192.168.100.17:4150) backoff timeout expired, sending RDY 1
2020/09/17 14:45:42 [myConsumer] consuming: message2
2020/09/17 14:45:42 WRN    1 [mytopic/mychannel] exiting backoff, returning all to RDY 1
2020/09/17 14:45:42 [myConsumer] consuming: message3
2020/09/17 14:45:50 [myConsumer] consuming: message1
2020/09/17 14:45:50 WRN    1 [mytopic/mychannel] backing off for 2s (backoff level 1), setting all to RDY 0

The consumer consumed message1 at 14:45:40. We requeued this message with delay of 10 seconds. You can see in the next line that it’s backing off for 2 seconds, so the consumer wouldn’t consume any message for the next 2 seconds. This is useful if you requeue because of system error and want to give some rest to the consumer. It can also be used as a circuit breaker. But it can be bad if you only want to requeue a specific message because it will continuously back off the consumer and pile the messages up.

RequeueWithoutBackoff

So what is RequeueWithoutBackoff? Just like the name says. It requeues the message without backing off the consumer. Let’s see it in action. This is the code for the consumer that we use.

    func myConsumer(msg *nsq.Message) error {
        msgStr := string(msg.Body)
        log.Printf("[myConsumer] consuming: %s\n", msgStr)
        if msgStr == "message1" {
            msg.RequeueWithoutBackoff(10 * time.Second)
        }

        return nil
    }

We publish some messages at the same time to test it like before and see the logs.

2020/09/17 15:42:12 INF    1 [mytopic/mychannel] querying nsqlookupd http://localhost:4161/lookup?topic=mytopic
2020/09/17 15:42:12 INF    1 [mytopic/mychannel] (192.168.100.17:4150) connecting to nsqd
2020/09/17 15:42:12 [myConsumer] consuming: message1
2020/09/17 15:42:12 [myConsumer] consuming: message2
2020/09/17 15:42:12 [myConsumer] consuming: message3
2020/09/17 15:42:22 [myConsumer] consuming: message1

See that when we requeue message1, the consumer is still consuming other messages. You can use this to requeue a message and let the consumer continue its work.

Conclusion

You can use Requeue if you think there is some system error and want to break the circuit to prevent more error to the messages. For other than that, use RequeueWithoutBackoff to prevent messages piling up. I hope that you know the difference between Requeue and RequeueWithoutBackoff. And that you can choose which one is the most suitable function for your case. Please comment on questions or corrections.

nsq go

Requeue

RequeueWithoutBackoff

Conclusion

See also