Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense.
The consequence of this is that many of my projects which require these overlapping domains (think stuff like build automation) ends up being the two interleaved— either an outer shell script that calls small Python programs (for parsing, URL fetching, and basically any error prone stuff where sane recovery/cleanup may be necessary), or it's a Python program that renders little templates of bash into strings and passes them to bash stdin.
Or in some cases, it's CMake that's the outer orchestration piece, using configure_file to fill templates and then invoke them either at configure time (with execute_process) or at build time (with add_custom_command).
> Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense.
This hits the nail on the head for me. What I'd like is something bash-like to handle running programs, pipes, tab-complete for file names, and so on, and something like Python syntax for control flow and strings (quoting / escaping), and access to something powerful like Python's standard math libraries (and ability to import other stuff like requests). I don't know how you'd roll all that into a single shell, but as close as you can get is what I'd like to see in a new shell.
(I've briefly looked at Oil before, but it has seemed a bit complicated to merit trying to switch full time at this point. I'm definitely following it for future developments though.)
Edit: just found Xonsh from another user's comment, which seems almost exactly what I'm looking for, if a bit hacked together at first glance. Going to try that out.
> Even something that should be as simple as piping output from one process to another in Python is a nightmare, and fraught with opportunities to shoot yourself in the foot with buffering deadlocks and other nonsense.
This is true, but when I am writing a script in Python I generally don't need to pipe as often as I do in bash. Python offer many replacements for things that in Bash you need to call another process.
That is, of course, if performance isn't necessary. When it is bash is still unbeatable because of the pipes.
> Python offer many replacements for things that in Bash you need to call another process.
True, you're never going to pipe something to grep in Python when you can just capture the output and use the inbuilt regex capability.
That said, it can be surprising how much worse performance-wise Python can be than shelling out to external utilities. Consider the simple case of downloading and extracting a tarball. I found that check_call("curl path/to/thing.tar | tar -x", shell=True) was significantly faster than anything I could figure out how to do with requests streaming, httpx (async), and the inbuilt tarfile module.
Depending on your use case there are other options e.g., fabric http://fabfile.org plumbum
Use shell for what is really good for: concise DSL for running commands (one-liners). Leave complex logic (branches, explicit loops) for sane general purpose languages such as Python.
Right, and I do that, but it's an obvious hack; it causes a separate sh process to spawn, you open yourself to shell injection issues if you're not careful, teeing the pipe is trickier than it should be, etc etc.
you might have known this but the practice shows that It has to be repeated: there is no shell injection issues with literal string in your own code.
YMMV but for those rare cases when I care about the shell injection, I just don't use shell and run the command directly in Python. Combination of shell one-liners and the main logic in Python works well in practice.
Oh yeah, for sure. I think it's just frustrating having that one more piece of mental overhead. Like, it should be that there's one sane way to do all this stuff in Python. Instead it's literal blobs of shell in some cases, and subprocess.STDXX pipes linked up in other cases, and maybe sometimes you use shlex to sanitize your arguments, or you give up on streaming and just use communicate() to get the whole result in memory at once. Blah.
I don’t see it as an overhead, I see using shell syntax as just another DSL like regex.
There are many different use cases related to running processes—it is natural that different solutions may be preferred for different cases (check_{call,output}, with/out shell, run, Popen, pty/PIPE, threads/Asunción—all may be useful. And it is just stdlib ).
The consequence of this is that many of my projects which require these overlapping domains (think stuff like build automation) ends up being the two interleaved— either an outer shell script that calls small Python programs (for parsing, URL fetching, and basically any error prone stuff where sane recovery/cleanup may be necessary), or it's a Python program that renders little templates of bash into strings and passes them to bash stdin.
Or in some cases, it's CMake that's the outer orchestration piece, using configure_file to fill templates and then invoke them either at configure time (with execute_process) or at build time (with add_custom_command).