Jobs can sometimes be processed after a queue is paused locally.
Created by: josephwarrick
Jobs can sometimes be processed after a queue is paused locally.
This happens because we wait up to five seconds for a job to become available when we try to get the next job.
This is the sequence of events when this happens:
The queue does queue.bclient.brpoplpush
to get the next job, but none are available yet.
We locally pause the queue; queue.bclient.brpoplpush
is still pending. Since some jobs are still actively processing, we wait for them to finish before resolving the pause.
A new job is pushed to the queue.
queue.bclient.blpoprpush
returns with the new job, and our queue starts processing it.
The original set of active jobs all complete and queue.pause(true)
resolves. The new job is still processing.
So we think our queue is paused, but we're actually still processing a job.
If we kill the queue, the active job may be terminated before it can complete.