Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If you like xargs, but want more flexibility, I'd highly suggest GNU parallel. Such flexibility includes running jobs on multiple computers, running intensive command using all available CPU's (like xargs -P), and creating unique scripts to handle multiple parameters.

http://www.gnu.org/software/parallel/man.html



Parallel also lets you transparently run stuff on remote servers, automatically handling stuff like copying files back and forth. When I have some heavy ad-hoc data processing to do my new favorite trick is spinning up 50-100 ec2 spot instances, point GNU parallel at them and just fire and forget.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: