> strange outcome that writing really fast Python code is all about writing your code in anything but Python
I really have to wonder why people are so insistent on keeping up the “Python charade”, like, surely at some point we recognise that all the effort trying to back-port performance into something that really isn’t having it, isn’t really worth it, and we’re better off just porting the bits of Python that work out to other languages that are fast/ergonomic/correct/etc and everyone involved can get some time back.
This is a slow process which takes many years, or even decades.
And what language is really the alternative? There is no clear answer. Many languages might somewhat be ok but I'm not sure. Python has grown so much because:
- It's very simple.
- Its syntax almost reflects algorithmic pseudo code.
- It's a managed language with garbage collection, you will usually not see segfaults or other memory corruption, and also not the simple kind of memleaks.
- Debugging is easy.
Those are features which are really nice for newcomers from other scientific fields, but also for professionals when Python is just the gluing language which orchestrates some heavy workers, e.g. TensorFlow or PyTorch. Or when it's just used as a better shell scripting language.
I'm myself using Python mostly for machine learning (TensorFlow, PyTorch), and as a shell script language alternative, and for other small utilities. And I really don't know what alternative language there is, which would be a better fit for this, even ignoring the state of the tooling and library support.
Some people might say Julia. But Julia is more complicated and more difficult to understand, esp for newcomers. And Julia has slow startup times. Slow startup times already disqualifies Julia for these applications, or at least it's a very huge disadvantage.
It appears simple on the surface, it is as complex as Common Lisp or C++, when one starts exploring the language surface, runtime capabilities, and standard library.
As an experienced Python programmer who has dabbled in C++ (and Rust), I find this claim very dubious, though I can't say either way about Common Lisp. The difficult parts of Python-the-language (e.g. metaclasses) are both fairly easily to explain and also so rarely used even by library authors that explanation isn't usually necessary.
I suppose the standard library can be considered complex, but is hardly worse than C++.
I've seen plenty of non-programmers become proficient enough in Python to be productive, but maybe one guy -- a statistician maintaining R libraries -- learn enough C++ to be dangerous.
How much would you score on a Python pub quiz, including all versions of Python between 1.6 and 3.10?
Questions would include changes across minor versions, meta programming, old and new classes, ABC, slots, magical methods, async, standard library, famous libraries like stackless, ...
Outside of tiny niches the world is on Python 3 now, so I don't think older versions are particularly relevant when evaluating how complex Python is. For that reason old vs new style classes don't cause issues. Nor do the existence of niche implementations like Stackless, unless we want to also claim such implementations as sources of complexity for C++ or (especially) Common Lisp, at which point it's a wash.
Having trained many newly-graduated data scientists and engineers to a level of being reasonably productive in Python, my assessment is that it is simply not that difficult to get a handle on. But if I'm expected to take non-career-programmers to comparable C++ fluency, that would be a considerably longer process.
Things they'd have to worry about in C++ that they don't have to worry about in Python: memory leaks, memory debugging, complex templates, libraries written in the lowest-level way possible, move semantics, ...
The thing is, what works really well for python is not just easily portable features.
Having a REPL and things like notebooks is great. Having pip install. Having a syntax that is very beginner friendly, especially for reading. Just making it easy to glue code together without worrying about types.
Half of these things are hard to impossible to port, and many others are actively undesirable to port.
If you consider python as a sort of bash-scripting language, that is explicitly meant to call 'real' code and very easily combine it into something new that is more usable, that might make more sense. Sadly the extra ergonomics of python vs bash mean that making code that can easily be called by python is a lot more work.
pip (and it's predecessor, easy_install) were improvements over CPAN, PEAR or the complete absence of solutions at the time in languages like Java or C, but I think it's fair to state that a package manager is table stakes for a language now and newer iterations on the concept like cargo have surpassed pip.
Because Python is the new BASIC, the big difference is that at least BASIC not only has compilers for decades, it was originally designed (Dartmouth BASIC) as JIT based language.
The interpreters got famous on the 8-bit machines because there wasn't space for either JIT or AOT compilers.
The latest talk I saw regarding Python performance actually addresses this charade, with a profiler that helps locate which parts of Python code should be rewritten in C, some people really love their "Python".
Well, there are a bunch of reasons of maintaining the "charade".
Firstly, faster languages don't have the ergonomics.
Secondly, there's a tonne of Python code out there already that isn't going away any time soon, and any effort to make the runtime faster benefits all that code.
Thirdly, there's little reason Python can't be as fast as the likes of JavaScript beyond the issues imposed by the C extension API. The reality is that not a lot of effort has historically been put into making the CPython runtime particularly fast. It's only recently that anyone's put any concerted effort into it.
That's a whole other issue. PyPy and CPython have conflicting aims, which is partly why it never replaced CPython as the defacto default Python runtime. Also, PyPy is much slower for stuff that's only going to run for a short time. Still, many of the lessons learned in PyPy can be used to improve CPython performance.
They only have conflicting aims due to politics, otherwise a way to work towards a common goal would have happened.
CPython devs will only accept proper JIT when the language really starts getting a betting from the likes of Julia, Chapel, Go, JVM and .NET bindings to the same underylying C/C++/Fortran libs.
Rust has a good shot at taking over Python's role in a couple years: it is fast, cargo is even simpler to use than pip, multithreading is mostly idiot-proof, and there's a growing ecosystem of data science and ML libraries that bring the benefits of Rust's more expressive type system to the table. But it will take years to grow that ecosystem to a size that makes it really competitive with Python.
Rust will never overtake Python's role as long as data scientists are the rank-and-file using machine learning libraries. Rust is simply too complex and too difficult to learn for non-CS types. I'm not casting aspersions at data scientists, who largely could adapt to Rust, but in my experience they wouldn't (in general!) unless they were forced to by the job market. And employers aren't going to see the need to make that change (again, in general), because a lengthy re-education process would be expensive.
My gut feeling is that Rust won't compete with Python in the ML space, but contrary to the common assertion that Rust is too hard for non-CS types, it has actually seen notable adoption from Python-users. While the strictness of the compiler can be seen as a deterrent to some, it is also a huge benefit for people who are new to systems programming and worried about shooting themselves in the foot.
Julia is probably the most likely competitor for Python in the data science space. It's designed for such use cases and it's very good at it.
Still, Python does have so much momentum that it's almost certainly going to continue its rise. But other languages can still succeed, the ecosystem doesn't need to end up as winner-take-all.
I really have to wonder why people are so insistent on keeping up the “Python charade”, like, surely at some point we recognise that all the effort trying to back-port performance into something that really isn’t having it, isn’t really worth it, and we’re better off just porting the bits of Python that work out to other languages that are fast/ergonomic/correct/etc and everyone involved can get some time back.