updateDelaySet may cause very high redis cpu usage
Created by: amit-gshe
We have a very large queue which has about several millions of delay jobs. And our queue has tens of workers(pods) in k8s cluster and even double or triple more pods when the business peak time. We have encountered an issue that bull can lead high redis cpu usage and the redis gets slow then stop prosessing jobs. It's a pain last for several months. Finally we find that the updateDelaySet get triggered so frequently on each worker and thus caused the high redis cpu usage and then consumer get so slow even stop processing waiting jobs and the waiting queue get stacked and finally the entire bull queue stop working.
I found that the updateDelaySet was expensive when the queue was huge so we should stop doing updateDelaySet
when the waiting queue has too many jobs to process. We can add an option say maxWaitingJobs
which default set Number.MAX. The user can set this value according their needs.