I don’t want to be too negative, and I realize this isn’t an option for everyone ... but seriously, just use conda if you can.
Judging from comments here and in the past, we all know packaging is one of Python’s Achilles’ heels. I spent a couple years bouncing around with pyenv/pipenv, then poetry, then back. I lovingly followed all the flamewars here and on github.
Ultimately, I realize now, I laboured under the conceit that it was desirable to have a “native” solution that did not depend on a suite like conda. I now think I was totally wrong.
I’m so much happier and more productive now that I’ve switched. For any data science use case conda is hands-down better, and for everything else it’s still more useable than any of the alternatives. I hate to say this but it’s true. The only thing I can think of that was harder to work with a conda setup was the keychain in macOS, and I’m not even sure now that that’s an issue anymore.
I find the original virtualenv system works well. There's no need for virtualenvwrapper or pipenv or any of that nonsense. Just make a virtual environment whereever you like ("virtualenv -p python3 ~/envs/myenv" if you want to share between projects, or "virtualenv -p python3 path/to/project/env" if you don't). Use "pip install -r requirements.txt" if your project has one. Activating and deactivating is straightforward. Deleting a virtual environment is easiest of all - literally just delete the directory and you're done.
The main problem that Conda was originally meant to solve was that basic Python packaging (virtualenv, pip) required you to build C extensions yourself which is a huge pain (mostly getting all the C dependencies installed). But with the advent of binary wheels and (eventually, in 2016) a binary wheel standard for Linux [1] you can now get binary builds of most packages on most OSs by just doing a "pip install packagename" - typically this is even more likely to work than Conda in recent years.
The only limitation of virtualenv is it requires you to have already installed Python yourself. On Linux I just use the system-installed version (the Python 3 one obviously), but on Windows I use Conda to create the environments and then pip to install the packages, which works fine.
I've had the opposite experience. I used conda for several years, primarily for data science, but regularly ran into problems with packages not working, updates for packages bring severely delayed, and issues with conda itself. Switched to pyenv+virtualenv 18 months ago and haven't had a single issue.
I believe both of our stories. I just always wonder how this happens that two people have such opposite experiences.
Just to note: this kind of difference is part of why it's so difficult to improve packaging. There are lots of different tools, and they all work well for some people in some situations. There's no realistic prospect of persuading them all to use a single tool - but the fragmentation is a big part of what makes packaging so daunting.
Ha!! Likewise, I believe you 100%. It’s fascinating isn’t it? Just goes to show what a diverse (and frankly wonderful) platform Python is, despite the warts.
If you don’t mind my asking, what kind of stuff we’re you working on? Just curious!
I have definitely found it’s easy to bork a conda install with updates, enough to make it easier to just wipe it out and start over.
I mostly work on financial software and only conda seemed to play nicely with pandas and brokerage APIs. I should probably mention that I do a lot of this is cross-platform and conda seemed to be the only thing that worked without a lot of tweaking on Windows, Mac and Linux consistently. But the Linux experience with pyenv and virtualenv (and pipenv too, for the most part) was great. If I was Linux-only I honestly might never have switched.
Default Python's Venv puts reproducibility of environment and scientific computing results at risk, by not creating a file which includes all the checksums recursively. One needs to take extra steps to achieve that. It is possible though.
Conda envs often get quite big and resolving deps takes time.
Add to that, that you download binaries, which you have to trust.
Another issue is, when there is no binary and it needs to compile stuff. On GNU/Linux that might work fine, but on Windows, like for some of my coworkers, you need Visual Studio (MS bundles it that way and I could not find a way around installing that whole thing) installed, for getting the required compiler stuff and even then the compilation might not work.
Conda in the past also had issues with reproducible builds of envs. I hit a case where on one OS it worked, on the other it did not.
Also I had a case where deps of deps change and when you try to build the env, suddenly it does not work any longer. That's very bad for deployments.
Since then we abandoned Conda entirely and built Python with all requirements ourselves. Envs need checksums.
Conda has problems but generally it is pretty incredible. It was way more useful before python wheels were everywhere tho. It’s still amazing and you can install so many non python related things with it. Mad props to conda-forge too.
Judging from comments here and in the past, we all know packaging is one of Python’s Achilles’ heels. I spent a couple years bouncing around with pyenv/pipenv, then poetry, then back. I lovingly followed all the flamewars here and on github.
Ultimately, I realize now, I laboured under the conceit that it was desirable to have a “native” solution that did not depend on a suite like conda. I now think I was totally wrong.
I’m so much happier and more productive now that I’ve switched. For any data science use case conda is hands-down better, and for everything else it’s still more useable than any of the alternatives. I hate to say this but it’s true. The only thing I can think of that was harder to work with a conda setup was the keychain in macOS, and I’m not even sure now that that’s an issue anymore.
conda really does just work.