This PR aims to re-implement the job fetching mechanism by relaying on pub/sub and some minimal polling in order to obtain a more robust and simpler codebase (and probably faster too).
This apparently simple change requires a lot of changes all over the place. I am taking the opportunity to refactor and clean up the code as much as I can. Several commits with more polishing will come after the initial PR.