Jobs get stuck after Redis reconnect
Description
When the Redis gets disconnected and connected again, the queue doesn't pick up any jobs. (Easy To Reproduce)
Minimal, Working Test code to reproduce the issue.
const Queue = require('bull')
const Redis = require('ioredis')
const client = new Redis({
host: 'localhost',
port: 6379
})
const queue = new Queue('test', client)
client.on('ready', () => {
console.log('client ready')
})
client.on('connect', () => {
console.log('client connect')
})
client.on('close', () => {
console.log('client close')
})
client.on('end', error => {
console.log('client end', error)
})
client.on('reconnecting', error => {
console.log('client reconnecting', error)
})
client.on('error', error => {
console.log('client error', error.message)
})
queue
.on('completed', (job, results) => {
console.log('queue completed a job')
})
.on('error', async (error) => {
console.error('queue error', error.message)
})
.on('failed', async (job, error) => {
console.error('queue error', error.message)
})
queue.process(async (job) => {
console.log('running task')
})
setInterval(function () {
queue.add({}).then(() => console.log('pushed task')).catch((e) => console.log('failed to push task'))
}, 10000)
Bull version
Bull 3.18.0 Node 12 and 14 Redis 3.2.8
Steps to reproduce.
- Run the above script
- Bull JS start accepting and processing Jobs
- Stop Redis
- Bull JS will keep retrying
- Start Redis
- Bull JS will connect in a few milliseconds
- At this stage, the queue can push the jobs and but it won't process any jobs. :-(
Things we tried
- Tried explicitly the reconnect option of
ioredis
. https://github.com/luin/ioredis#auto-reconnect and https://github.com/luin/ioredis#reconnect-on-error But issue don't seem to be coming fromioredis
.