I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it.
Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.
> However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on.
Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense.
The consequence of this is that many of my projects which require these overlapping domains (think stuff like build automation) ends up being the two interleaved— either an outer shell script that calls small Python programs (for parsing, URL fetching, and basically any error prone stuff where sane recovery/cleanup may be necessary), or it's a Python program that renders little templates of bash into strings and passes them to bash stdin.
Or in some cases, it's CMake that's the outer orchestration piece, using configure_file to fill templates and then invoke them either at configure time (with execute_process) or at build time (with add_custom_command).
> Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense.
This hits the nail on the head for me. What I'd like is something bash-like to handle running programs, pipes, tab-complete for file names, and so on, and something like Python syntax for control flow and strings (quoting / escaping), and access to something powerful like Python's standard math libraries (and ability to import other stuff like requests). I don't know how you'd roll all that into a single shell, but as close as you can get is what I'd like to see in a new shell.
(I've briefly looked at Oil before, but it has seemed a bit complicated to merit trying to switch full time at this point. I'm definitely following it for future developments though.)
Edit: just found Xonsh from another user's comment, which seems almost exactly what I'm looking for, if a bit hacked together at first glance. Going to try that out.
> Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense.
This is true, but when I am writing a script in Python I generally don't need to pipe as often as I do in bash. Python offer many replacements for things that in Bash you need to call another process.
That is, of course, if performance isn't necessary. When it is bash is still unbeatable because of the pipes.
> Python offer many replacements for things that in Bash you need to call another process.
True, you're never going to pipe something to grep in Python when you can just capture the output and use the inbuilt regex capability.
That said, it can be surprising how much worse performance-wise Python can be than shelling out to external utilities. Consider the simple case of downloading and extracting a tarball. I found that check_call("curl path/to/thing.tar | tar -x", shell=True) was significantly faster than anything I could figure out how to do with requests streaming, httpx (async), and the inbuilt tarfile module.
Depending on your use case there are other options e.g., fabric http://fabfile.org plumbum
Use shell for what is really good for: concise DSL for running commands (one-liners). Leave complex logic (branches, explicit loops) for sane general purpose languages such as Python.
Right, and I do that, but it's an obvious hack; it causes a separate sh process to spawn, you open yourself to shell injection issues if you're not careful, teeing the pipe is trickier than it should be, etc etc.
you might have known this but the practice shows that It has to be repeated: there is no shell injection issues with literal string in your own code.
YMMV but for those rare cases when I care about the shell injection, I just don't use shell and run the command directly in Python. Combination of shell one-liners and the main logic in Python works well in practice.
Oh yeah, for sure. I think it's just frustrating having that one more piece of mental overhead. Like, it should be that there's one sane way to do all this stuff in Python. Instead it's literal blobs of shell in some cases, and subprocess.STDXX pipes linked up in other cases, and maybe sometimes you use shlex to sanitize your arguments, or you give up on streaming and just use communicate() to get the whole result in memory at once. Blah.
I don’t see it as an overhead, I see using shell syntax as just another DSL like regex.
There are many different use cases related to running processes—it is natural that different solutions may be preferred for different cases (check_{call,output}, with/out shell, run, Popen, pty/PIPE, threads/Asunción—all may be useful. And it is just stdlib ).
That’s not a very convincing post because it’s a bit too shallow:
> I encountered a nice blog post, Replacing Shell Scripts with Python, which, in my opinion, inadvertently proves the opposite point. The Python version is longer and has more dependencies. In other words, it's more difficult to write and maintain.
The Python version uses only the standard library, so it doesn’t make sense to worry about dependencies if you aren’t counting every program your shell scripts call similarly. Having just one thing providing a consistent baseline beats learning that your script needs to upgrade RHEL to get a feature which would make something safer or easier.
Similarly, the Python script is longer because it does better work and consistent, useful error handling. That adds lines of code but it’s usability and correctness, not overhead.
It’s similarly incorrect to treat line count as a proxy for difficulty, as anyone who’s ever had to deal with quoting or data structures in shell scripts knows. The use of pathlib or os.path sometimes gets grumbles for extra characters but I’ve found it nearly inevitable that the grumblers will have something fail or destroy data because they hit a filename or argument with a space or special character in it. On one notable occasion, that resulted in an `rm -rf something something /`. (If you’re writing a shell script, shellcheck is mandatory)
The best criticism I’d make is that it could be easier to replace a shell pipeline. Python’s subprocess makes that pretty easy but you still need a minimum of 3 calls per command plus whatever you need to actually do with the output. Since subprocess.{run,check_output} handles most of my needs this isn’t a big deal and there are excellent modules if you don’t mind dependencies.
The Bash script actually has more dependencies, it relies on a number of external programs (ps, kill, mkdir, sort, ls, cp, echo). What versions are on your system? What versions on the systems of the people running the script? Do they support the same features? If you're running on a Mac are you using the Mac-shipped programs or the GNU coreutils from homebrew?
Also, that Bash script that the author is defending has a subtle bug in it, because Bash is subtle. The bug is here:
$APP $filename >"${output_dir}/summary_${name}.txt"
if [ $? != 0 ]; then
echo "Error $? in app"
fi
The return value of [ ... ] overwrites the $?, so the inner $? is the result of the test, not $APP.
Which is exactly the kind of crap that everyone's talking about when saying that we should avoid Bash scripts. It's full of landmines like this.
Bash is great for interactive work on the terminal. It is not great for writing correct, maintainable programs.
Good catch: I’ve written many thousands of lines of shell scripts over the years and that’s exactly the kind of rake in the grass which is too easy to miss. Using set -eu -o PIPEFAIL helps as does shellcheck but it’s still easier than it should be to have unexpected behavior from something you thought was well tested.
About 15 years ago I switched to Python for anything which doesn’t fit on a single screen or uses any advanced feature and have had zero reasons to regret that decision. It’s especially good for anything you use with other people since you spend your time talking about features rather than how to trick the shell into working correctly.
> However, Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. But Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows).
Ruby has pretty good thing wrappers around running shell commands (e.g. using backticks to run a command and get the output as a string or calling system() on a string to execute as a command and get back the error code), and my understanding is that it doesn't really have great Windows support. My impression here is that the author of this FAQ is probably not super familiar with Ruby based on the way they seem to equate the abstractions in both languages.
There is a better option and its name is Perl. Perl mixes the ins of shell scripts like easy argument and output passing with complex and easy to use control structures.
You could use Python, but having been forced to use Perl extensively Perl is the superior choice for a complex shellscript-like workload. There's less boilerplate in Perl. There's a couple Python projects that come close like Fabric, but Perl was literally made for this type of workload.
There is nothing in the (IT) world I fear more then... Perl projects. The shear madness it will leash upon you when the dependencies fail is maddening. Never will I voluntarily touch anything written in Perl ever again in my live.
I've seen the power of Perl, the beauty and elegance of it, but everything was ruined by the absolute shit show when it comes to its packet managers.
I have 20 year old Perl code I run in production. I had to re-install it our move to Amazon Linux 2. My 10 year old install process for the cpanm setup still worked. My 10 year old Python had to be moved to python 3; but the 20 year old Perl still runs
I’m a big time python fan (80% of the code I write is python) but growing up I loved Perl (after starting out as a PHP dev) as a scripting language and I hate how much flak people give it. Though I do understand since there’s a million ways to do everything in Perl if you aren’t familiar with the language it can look daunting. But I’d argue the same can be said about JavaScript and I still love that language as well.
It is not that hard either (unless perl is completely foreign to you).
Ironically just today I wrote a python script (using concurrent.futures) to batch re-encode 10gb of mp3 podcasts into 3gb of opus files to save space. Before I deleted the old mp3's, I did a quick file count of mp3 and opus files only to have 5 extra opus files... I beat my head really good trying to glue a bunch of bash stuff together but I could not find where the 5 extra opus files were coming from.
I defected back to perl. All I needed was an alphabetical list of all the files (there were many sub-directories), perfect for a little perl oneliner:
I did the same thing for * .mp3 files and then diffed the two files. I quickly noticed the extra 5 opus files were from me doing test encodes a couple of weeks ago that I forgot about.
Sure it looks hideous, but the explanation is easy: for 'perl -nle', the 'l' means process each line from stdin individually, the 'n' means no auto printing of the line, and the 'e' is basically the perl 'one liner' mode. Either way, each line comes in and is stored in '$_', I use a capturing regex to grab the file name without parent directory names, basically everything after the last '/' in a path that ends with a three letter extension (\w\w\w). The "captured" part of the regex ends up in '$1'. Also see 'perldoc run' for all the runtime one liner options.
If I've not wildly misunderstood what you're doing with that line (and apologies if I have), could you not just stick `-printf '%f\n'` on the end of the `find` command?
(By definition, since you're looking for `.ogg` files, everything `find` finds will have a path that ends in a three letter extension.)
It's one of those things that's immensely helpful but difficult to discover organically, I think, since a lot of people will go "find outputs this, now I pipeline to transform that, this is The Unix Way" and never realise that `find` can do a bunch of transforms for you for free.
Space was because HN uses it as italics and I don't know the escape. And yes there are cleaner ways, but the point was a real world example of having something not work out in pure shell but yet quickly barfing out something working with perl. And yes, looks like I didn't escape the dot, funny!
I think the reason people badmouth Perl is because it is safe. People claim PHP improved but all I can see is language without design. JavaScript has lots of bad parts and we should not talk about them. I like AWK matchers though not its functions. I am not familiar with Perl but it is designed.
Why am I not surprised that the PERL part could be solved with bash in the same space. Why are you so eager to use lang on top of lang, in such simple cases?
If you have that in a variable, which you presumably would if you're reading them one at a time, you can use "remove longest matching prefix" substitution.
$ i="/podcast/with/nasty'--c h a r$/and/stuff/track1.mp3"
$ echo "${i##*/}"
track1.mp3
[Edit: 'thezilch beat me too it by a couple of minutes]
I hate everything there is to hate about CPAN. I also hate how Perl libraries are handled. I have to have like 5 lines of code related to it in ~/.bashrc or ~/.bash_profile. Bleh. Perl libraries are always a problem when I want my program to run on other machines. No thank you.
I find this to be a weird statement. I've never really fought with "cpan" but have had huge fights with node and python packages.
The old school "cpan" command has not been recommended to be used for nearly a decade now. Maybe that is your issue?
Everyone just uses cpan-minus: http://cpanmin.us/ or "cpanm". It will install anything, if you have write access to the installed perl location, it will install globally, otherwise it installs packages into "~/perl5" then just do something like "export PERL5LIB=~/perl5/lib/perl5/" in your bashrc.
Or maybe you are talking about having to compile code in certain packages? Alot of stuff using ssl has C code which links against openssl which is always a pain.
Not sure what other issues you might be alluding to... There are some "packaging" tools to try to bundle and pin libraries, kind of like venv for python. I've never messed with them though. We just compile the latest perl binary and install all our needed cpan modules globally. Never worry about pinning since nothing is ever updated on cpan anymore...
> The old school "cpan" command has not been recommended to be used for nearly a decade now. Maybe that is your issue?
I did not know that it was not supposed to be used, so that might explain my issues, but at the same time it did work for me on my machine, but had a hard time running my Perl script on a VPS, because I had to set everything up again and for some reason I ran into some errors.
I often ran into packages that failed to compile, too, yeah.
> Never worry about pinning since nothing is ever updated on cpan anymore...
A lot of people have a knee-jerk negative reaction to the name perl without understanding that perl chewed up and spit out this particular problem space. You probably don't want to develop your next webapp with a team of perl coders, but the language has its niche.
Golang managed to persuade people to write tests, by making it easy and self-contained. I'd suggest with Test::More that Perl did something similar. Almost all the modules you'd find on CPAN are full of test-cases.
There might well be thinks we can argue about with Perl, but it is definitely true that it has/had one of the biggest and most consistent set of extensions out there. CPAN has held up pretty well, compared to some of the later alternatives (such as node modules, ruby gems, etc). One reason that I was able to work with Perl so easily was because I could find Stripe integrations, and similar with ease.
I mean, I worked in the .com era with perl coders who made some pretty compelling products out of Perl, for the time, with some discipline, and TBH... it wasn't terrible, and when they "Did the right thing" and rewrote the whole thing in the new hotness (J2EE) it _was_ terrible.
Perl will let you hang yourself, and I hated it at the time, but it also got the job done.
I recently bought a copy of the Camel book and noticed the exact same thing that Steve Yegge pointed out here:
Perl's references are basically pointers. As in, C-style pointers. You know. Addresses. Machine addresses. What in the flip-flop are machine addresses doing in a "very high-level language (VHLL)", might you ask? Well, gosh, what they're doing is taking up about 30% of the space in all Perl documentation worldwide.
This is really a huge, unnecessary wart in the language.
The other wart I pointed out in the post is that Perl 5 does dynamic parsing of its own code (parsing that depends on the value of variables), despite the book complaining about the same issue in shell. In contrast, Raku and Oil are statically parsed.
----
I have a lot of respect for Perl, but there's a reason that Raku exists (and again from the FAQ: Raku and Python 3 are both worse shell-like languages than their predecessors.)
> Perl's references are basically pointers. ... This is really a huge, unnecessary wart in the language.
Call me crazy :), but I count this as a pro for perl! It makes thinking about data structures in perl very "regular" for me.. there's a few rules to learn, but then I find it simple to construct (or de-construct) nested data structures while applying those rules. As opposed to other scripting languages, where the difference between "value" and "reference" frequently seems to be more hidden from sight...
> In contrast, Raku and Oil are statically parsed.
I don't know about Oil, and if I understand your use of "statically parsed" correctly, Raku is not statically parsed. A "use" statement can affect how Raku parses source code from there on, see e.g. the OO::Monitors module that adds a "monitor" keyword.
But even simpler, adding an operator changes the grammar. For instance, adding a postfix ! operator for faculty, can be as simple as:
sub postfix:<!>(\value) { [*] 1 .. value }
say 5! # 120
OK interesting, I guess the question is if "use" requires running (not just parsing) arbitrary code? I guess if it's like Python's "import", it does.
Changing the grammar could still be considered static parsing, as long as the change doesn't depend on the values of variables at runtime, e.g. your argv array or something.
I recall Larry Wall saying that Perl 5 was at times confused about the language it was parsing, and the goal was to fix that in Perl 6. I don't have a lot of experience with it, but yeah that claim could be wrong, or at least un-nuanced.
The process of exporting symbols is done in an EXPORT subroutine that can be provided by the module developer. This subroutne is supposed to return a Map of symbol names and what they refer to. This Map can be constructed depending on external factors such as an argv array or an environment variable, although I have yet to see this in the wild.
So I guess one could say that Raku parsing is usually static, but it does not need to be.
i find that ruby combines the best features of perl and python in that respect. makes shell-like scripts really convenient and easy while still providing the means to add structure.
Performance is not a factor in shell scripts where the majority of the work is done by C-programs and the script merely shuttles parameters to the desired places.
It's not as if bash were fast, mind you. Lisps already exist.
Nim's not Lispy either, and macros alone are not enough. To make macro use feel natural because they're not much different than using the core language, the language needs to be homoiconic, which to my knowledge Nim is not.
In any case, macros are not what draw me to Lisp, but its unparalleled combination of ease of use, readability, and power... along with 70 years of development and tooling.
So I could just use a Lisp or Scheme, which I do. They can be plenty fast too (see Chicken, for instance, which -- like Nim -- compiles down to C).
I agree. Generally I find the approach to have a few tools that work well in their specific niches to be superior to having one tool for all jobs.
I use a shell for shell stuff, I use a scripting language for most other stuff, and I might use a systems level compiled language (or a scripting language that calls into a compiled library) for more performance specific needs. If you're already within a specific area and only need to venture into the other for a minimal aspect of the current project, it can be useful to stick with what you're in, but you quickly reach the point where it's better to choose a better tool because of diminishing returns from using a tool for something it's not good for.
Maybe the hammer in the hand is fine for prying the single board off fence or wall, but if you're going to be doing it to ten or twenty boards, maybe walking over to the shed to get the crowbar will save a lot of time and effort in the end.
If your shell scripts are self-contained pieces of logic, you can use any language to write it. If you want to leverage external programs available in the shell, shell scripting is still the best glue language.
Python has a REPL but it isn't sufficient to be a shell. Therefore, I can't write a program in the Python REPL just as I'm doing stuff normally and copy it into a file for further editing and polishing. I have to port whatever I did in the shell to whatever other language I'm using, which is more porting than I want to do for most shell scripts, especially ones which are mostly pipelines.
Heck, Python isn't even a good language for one-liners. Not saying it should be, but it further demonstrates that I can't use Python live the way I can use zsh live.
I'm often asked this about Oil [1]: Why do you want to write programs in shell?
That's not the idea of shell. The idea is that I write programs in Python, JavaScript, R, and C++ regularly, and about 10 different DSLs (SQL, HTML, etc.) And I work on systems written by others, consisting of even more languages.
I need a language to glue them together. A language to express build automation and describe deployed systems. Most big systems consist of more than one language.
Shell is the best language for that, but it's also old and crufty, with poor implementations.
When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".
-----
If you want some concrete examples, look in the Oil repo. There are dozens of shell scripts that invoke custom tools in Python, R, and C++.
>When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".
I don't think this is a good characterization at all... Shell does not have git functionality (for example) built in. That is a dependency just like it would be for a python or rust project.
What I mean is that shell speaks paths, pipes, and processes natively, and you can creatively use tools that are "just there". Example: http://www.oilshell.org/blog/2017/09/19.html
It's an analogy, not a precise statement. It could be made more precise by using Oil as the center of a container-based "semi-distro", i.e. a distro that does everything that's not hardware related.
I guess a little like the complement to what CoreOS was doing (or is?). (This is a project I've been thinking about for awhile; anyone should feel free to contact me if they've done something like that or have ideas. It's related to the dev tools problems described in the blog post.)
> I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it.
I've seen, and still see, plenty of Python scripts launched by a shell script.
9 out of 10 times, Python just causes more problems than it solves, and problems that are trivial to solve with plain old shell scripts.
The only reason Python "won" is because we see a constant inflow of rookie developers who so far did practically only used Python ever, and that's pretty much the only thing they know and the only tool of their trade. They see all problems as nails, and thus insist in using their little hammer all the time without even looking at the current infrastructure or even the toolbox.
I have seen shell scripts launch Python scripts. I have seen powershell scripts launch Python scripts. I have seen npm launch Python scripts. Hell, I have seen Python scripts launch Python scripts by firing up a Python interpreter as a child process.
Python did not won. Python outnumbered. It did so because people who don't know better happened to know Python.
Don't let that be the reason why you make terrible technical decisions.
This is ridiculous. Python won scripting because it offers a sane way to do sequential, shell-ish things, without having to wade through "man bash" or searching stack overflow for the umpteenth time about the syntax to do something that should be trivial but is anything but.
Saying that younger devs only know python is like a FORTRAN engineer in the 90s saying young devs only know java. No one needs to apologize for growing up learning better mature readable shit.
The number of gotchas and tricky nonsense in bash could (and probably does) fill books (array indexing, string comparison, quoting, toggling 'set -e', many more). I don't doubt that there are clever grey beards that are wizards that know the arcana. That doesn't mean arcane should be what you build an engineering culture around.
> This is ridiculous. Python won scripting because it offers a sane way to do sequential, shell-ish things, without having to wade through "man bash" or searching stack overflow for the umpteenth time about the syntax to do something that should be trivial but is anything but.
You've inadvertently supported exactly my point.
You just advocate for the lazy way out. You know Python, so that's what you use and nothing more. God forbid you check out a reference. I mean, your lazyness leads you to believe that having docs.python.com on speed dial for a dozen different major or backwards incompatible releases is ok, but oh God forbid you check up a single man page of a tool that exists since the beginning of time.
Ridiculous, and all of this just because you believe you know Python, and that's all you have to offer.
Shell scripts are standard, omnipresent, reliable, and readily available infrastructure. There's no way around it. There's no excuse, let alone laziness and refusal to learn, which in this field is outright incompetence. The only reason someone refuses to use shell scripts for this sort of job is dereliction of duty and outright incompetence, and that is not winning in anyone's books.
So you say we should choose the hard way, because the easy way is lazy?
Also, your point can be turned around - you could use python which produces readable and maintanable code, but instead you choose the easy, lazy way of using bash, which you already know really well.
I'm curious what scripting tasks you do with python? I've found bash to be more than enough for everything I'd like to script, except for maybe stuff with heavy json processing (still doable with `jq`)
Anything with a non-trivial program flow, e.g. "do thing A, B, or C, and optionally mixin arguments D and E, with caching in between the shared steps."
I've actually started to transition my shell scripts to eLisp for better integration in to Emacs and eshell.
As a Lisp, eLisp is not the greatest, but I'd still much rather use it than Python.
I also don't want to sit and twiddle my thumbs while a Python script takes its sweet time in loading. Slow startup time is the kiss of death for most shell scripts.
Python startup time is slow for you?? It’s basically instantaneously on every machine I’ve ever used, unless there’s some seriously poorly written imports.
I'm not the original poster, but judging from the answer it looks as though they'd already have an Emacs session open and can execute it without any extra startup-time; in which case the answer is yes, because there's no startup time.
Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.