We implemented this with additional DB checks. For example: we put only one job to queue at a time per customer, the rest are in database until one that in progress is not processed.
Actually most of the prioritization could be implemented through additional DB. With SQS in most cases you need persistent reflection of job to keep its status, process times, results. You can put to queue only few items that are highest priority to guarantee that workers are busy next 10-30 minutes.
Thanks for the suggestion, ill reflect on that; we were also considering an "overflow" queue that would receive jobs if there was a recent high insert rate of a customer's jobs to the main queue, but that didn't solve the "cost" problem of a job potentially being large.
Was hoping to avoid having "side car" infrastructure for this, but I don't think I can escape it :)
Actually most of the prioritization could be implemented through additional DB. With SQS in most cases you need persistent reflection of job to keep its status, process times, results. You can put to queue only few items that are highest priority to guarantee that workers are busy next 10-30 minutes.