Error processing job: SyntaxError: Unexpected token u in JSON at position

Created by: jahraphael

Hi there,

I am using Bull 2.6.5 (Redis 3.2.4) and Node 6.9.4 across about 30 containers each running a single queue consumer process with concurrency = 2.

Problem I'm Running Into Over time my queues start to fill up with invalid jobs that Bull does not remove, and I start to see many Error processing job: SyntaxError: Unexpected token u in JSON at position errors in the logs:

Error processing job: SyntaxError: Unexpected token u in JSON at position 0
    at Object.parse (native)
    at Function.Job.fromData (node_modules/bull/lib/job.js:503:33)
    at node_modules/bull/lib/job.js:65:18
    at tryCatcher (node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (node_modules/bluebird/js/release/promise.js:691:18)
    at Async._drainQueue (node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:649:20)
    at tryOnImmediate (timers.js:622:5)
    at processImmediate [as _immediateCallback] (timers.js:594:5)

When these errors occur the process is not killed. Instead these jobs just repeatedly get re-attempted and never removed. I think because the error is happening at the point where the Job object would actually be created, so it is not able to handle this like a standard job failure.

When I just checked I had about ~6000 jobs in this state.

What I've Done to Debug In node:

• When I print out the contents of jobData inside of Job.fromId I see that in these failure cases it is always either {} or { delay: '0' }.

In redis-cli:

> hgetall bull:rio-bnet-queue-111-us:1986600
1) "delay"
2) "0"

How to reproduce I do not have a solid code-based reproduction process. I'm not sure why I end up with jobs with the empty data as I mentioned above. I'm not doing any manual redis queue manipulation in my actual code, but I believe you would be able to reproduce the problem by setting one of your jobs in your queue to be a hash map with just a single entry. Something like:

hmset bull:rio-bnet-queue-111-us:1986600 delay 0

Workaround My current workaround here, which I just got setup, is to proactively check for this invalid job state in Job.fromId, craft a Job with the same Id (and no data), and then return a rejected promise to remove this job from the queue (since it is invalid).

Job.fromId = function(queue, jobId){
  // jobId can be undefined if moveJob returns undefined
  if(!jobId) {
    return Promise.resolve();
  }
  return queue.client.hgetall(queue.toKey(jobId)).then(function(jobData){
    if(jobData.data === undefined || jobData.opts === undefined) {
      var job = new Job(queue, {}, {});
      job.jobId = jobId;
      return Promise.reject(job.remove());
    }
    return Job.fromData(queue, jobId, jobData);
  });
};

This appears to properly clean up these bugged jobs when they're encountered, but I understand it may not be an appropriate solution for the library. I'd be happy to help contribute something more appropriate, but I'd be curious to get @manast opinions on this whole situation.

Is there additional useful information could I provide to help narrow this down?