perf: Speed up performance of queue.clean when called with a limit

Created by: pxpeterxu

Motivation

I was trying to clean a large queue with millions of entries. Originally, I ran:

import Queue from 'bull';
const queue = new Queue(...);
queue.clean(0, 'completed');

But this blocked Redis from saving any other records with the error message:

BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE

I switched this to something more like:

import Queue from 'bull';
const queue = new Queue(...);

async function run() {
  let count = null;
  while (count == null || count >= 100) {
    count = (await queue.clean(0, 'completed', 100)).length;
  }
}

But each loop of 100 still took a really long time. Digging into the script, regardless of how many we try to delete, we're listing all the keys on every run of .clean()

https://github.com/OptimalBits/bull/blob/019d6125bea8a195d7690d52707b4e7089a92f16/lib/commands/cleanJobsInSet-3.lua#L23

Changes

• Change the cleanJobsInSet script so that:
  • If we're trying to clean n entries, only fetch n keys at a time
  • (Note that we might have to loop through this if the first n entries don't have all the timestamps we need, but this should generally still be an improvement)
• Cleanup: make cleanJobsInSet's variables be all named for slightly easier readability

Testing

Tested in production with my queue -- this sped it up a lot! Also ran it on empty queues to make sure that we ended properly

Also added an automated test for some pagination edge cases

Note for reviewers

It's a lot easier to see changes with "Hide whitespace changes" turned on: https://github.com/OptimalBits/bull/pull/2205/files?diff=split&w=1