Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

People considering this project should also probably consider Graphile Worker[1] I've scaled Graphile Worker to 10m daily jobs just fine

The behavior of this library is a bit different and in some ways a bit lower level. If you are using something like this, expect to get very intimate with it as you scale- a lot of times your custom workload would really benefit from a custom index and it's handy to understand how the underlying system works.

[1] https://worker.graphile.org/



have also used/introduced this to several places I've worked and it's been great each time. My only qualm is it's not particularly easy to modify the exponential back off timing without hacky solutions. Have you ever found a good way to do that?


Do you add jobs inside a transaction? Or with a root connection? There doesn't seem to be an out of the box way to add jobs in a transaction.


yes you can add jobs with sql in a transaction, e.g.

SELECT graphile_worker.add_job( 'send_email', json_build_object( 'to', 'someone@example.com', 'subject', 'graphile-worker test' ) );

(can also use the PERFORM statement instead of SELECT)


Is there something like that in the jvm world?


You don't need anything specific. SELECT ... FROM queue FOR UPDATE SKIP LOCKED is the secret sauce.


Would be nice to get some goodies for free, like overview, pausing, state, statistics etc. :-)


This is exactly how pgmq is implemented, + the usage of VT.


https://www.jobrunr.io//en/ seems to be popular at the moment.


That looks pretty good! Too bad that the OSS version limits the number of jobs quite a bit.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: