Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I explicitly want single-threaded execution within a tenant. After writing the original post, I figured out how to parallelize across tenants. The trick is to limit the query to only look at the next pending job for each tenant, which then allows for using SKIP LOCKED to process tenants in parallel:

    SELECT job.job_id, rm.data
    FROM qbo.transmit_job job
      JOIN resource_mutation rm USING (tenant_id, resource_mutation_id)
    WHERE job.state = 'pending'
      AND job.job_id = (
         SELECT qj2.job_id
         FROM qbo.transmit_job qj2
         WHERE job.tenant_id = qj2.tenant_id
           AND qj2.state = 'pending'
         ORDER BY qj2.create_time
         LIMIT 1
      )
    ORDER BY job.create_time
    LIMIT 1 FOR NO KEY UPDATE OF job SKIP LOCKED

> you should just use Temporal

Do you mean the Airflow-esque DAG runner? I prefer Postgres because it's one less moving part, I like transactional guarantees, my volume is tiny, and I can tweak the queue logic like above with a simple predicate change.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: