Why not tell people to “simply” use pyenv, poetry or anaconda
Comments
jakewins
blame-troi
Every time I get the urge to pick up Python, I find the packaging situation so off putting that I quickly find some other rabbit hole to dive into. I find emacs and vim add on configuration easy and transparent by comparison.
andrewflnr
As OP suggests, you can ignore basically all of it. If you're just messing around, use your system Python. That'll get you a long way. If you're Starting A Project and want to install packages locally, use the virtualenv package that came with your system Python. That's gotten me through basically all my Python programming for 15 years. Anything more complicated than that, you can learn when you come to it.
nerpderp82
> use your system Python.
Maybe you are an accidental accelerationist? Using the system Python is fastest way to a broken system and a broken Python.
kaashif
What? Why? How?
I always use the system Python plus virtualenv and that works fine, it's certainly never broken my system...
I don't even know by what mechanism that breakage would happen.
w0m
'sudo pip install <something huge with a ton of deps>' could potentially make system packages go wonky. Working inside a venv is perfectly safe.
kaashif
Oh, I guess I never even considered doing that for some reason.
If you just run "pip install" doesn't literally tell you to use the --user flag or virtualenv?
I guess it's probably an easy mistake to make.
nerpderp82
If you are using system python only to create venvs you are fine. As soon as you install or start modifying the system python in anyway, you are on the path to ruin.
unmole
Somehow in the 15 years of working with Python on Linux, I've never broken anything using pip.
sigjuice
Likely due to your good taste and judgement of never ever running pip with sudo.
integricho
I'd argue that running pip or any package installation with sudo is the devs fault.
salawat
Never tried to update something eh?
Debian is particularly bad because they'd comingle python 2 and 3 packages in the same filesystem locations if you used pip out of the box.
That was a horrid discovery on my part. I assumed, coming from java where this sort of thing has been a long solved problem, that no one would be daft enough to do something like that.
Filligree
God save people who don't know you're 'supposed' to use something else.
Most projects state something like "run pip install whatever", which people will then do. If you're lucky, it will ask for sudo. If you're even luckier, you stop to think before entering your password.
nerpderp82
I have a default Python env that my shell sources for every login shell. I can deactivate it and activate different envs on a per project basis.
That default Python env means I don't ever modify the system python for any reason. I type `python` and I get my own Python3 from the base.env
If you use homebrew, there are two settings that help keep your old python binaries around so that venvs that reference those old envs don't break.
HOMEBREW_NO_CLEANUP_FORMULAE, this takes a lists of packages to not cleanup
HOMEBREW_NO_INSTALL_CLEANUP, prevents old packages from getting removed
deaddodo
Node has the exact same issue for people who globally install all packages. The only differences are A) python makes environment management the dev's choice/responsibility and B) defaults to system vs environmental.
I actually far prefer the Python method to, say, Go's GOPATH. But, I far prefer Rust/Cargo to either.
unscaled
Sure, GOPATH really sucked, but that's why we got Go Modules. I don't think it's fair to compare Python to Go anymore.
I wish Python got a standard integrated solution for package management that works.
pip doesn't cut it in my book since it doesn't let you specify dependencies reproducibly. It either doesn't support lockfiles or ONLY supports having a lockfile, without dependencies.
venv also doesn't cut it since you have to remember to explicitly activate it and keep track of which venv is activated right now.
If we take a look at the much-maligned Node.js+npm, it's still far better than what built-in tools in Python let you have. Yes, Node.js doesn't provide full isolation from globally-installed node_modules, but at least it supports a local node_modules directory and lets it take preference. Notably Node.js+npm, with all its warts, doesn't suffer from the the two issues I've mentioned above.
noloblo
Golang is far better when it comes to go path and compile speeds
dns_snek
> If you're Starting A Project and want to install packages locally, use the virtualenv package that came with your system Python.
I think this should be emphasized more, I don't think it's just a matter of preference, not separating packages between projects using virtualenvs will land you in a world of hurt as soon as you want to update or uninstall any of them and they're hopelessly entangled with system packages and other projects.
andrewflnr
If you're just starting with the language though, you might only have one project for a long time, or your projects might have disjoint dependencies (or close enough; I'm imagining someone with both a Django and pygame project). And you can learn a lot without any libraries at all. As a teen, I went a couple years and learned a lot before I learned how to use virtualenv. You'll want it sooner, but it absolutely shouldn't be a barrier to getting your feet wet.
dns_snek
That's certainly valid, if someone is just getting started like you say then it's not something they should worry about. I wasn't thinking of complete beginners when I wrote that comment.
hodgesrm
Exactly the same thing happened to me recently. I found refuge in Golang. Not my favorite language but it's tough to beat for utilities that you want to distribute quickly.
nonethewiser
For all the shit node gets for being too dependency heavy, npm, yarn, pnpm all work exactly how I’d expect.
hodgesrm
I love npm. It actually works. ;)
noloblo
Not to mention golang sub second compile time and all the productivity boosts coming from that
pjmlp
Hardly impressive for anyone that has used compiled languages with modules since the 1980's.
What is impressive, is how so many in the industry have forgotten about them.
noloblo
With all these years of improving compile times why is haskell compiled time very slow still?
hodgesrm
Incremental compilation has been a thing since the days of Smalltalk. (At least.) It's a feature of every Java IDE I've ever used. I do appreciate quick Golang builds but being able to change something in an IDE and see instantly if it compiles is a big plus of many lanugage.s
nonethewiser
Install dependency.
Save in requirements.txt. Includes sub dependencies that are system specific.
Install from requirements.txt on different machine and get errors.
Uninstall dependency and save to requirements.txt.
Look at requirements.txt and see that sub decencies are still there.
What is the right way to avoid these issues on Python?
bart_spoon
I use pip-compile in the pip-tools package:
Keep your high-level dependencies in requirements.in.
Create a virtual environment using venv.
Run `pip-compile resolver=backtracking`. It autogenerates a requirements.txt file with all dependencies and sub-dependencies and their versions. This essentially acts as a lock file from other languages/frameworks.
Install from the autogenerated requirements.txt file in the venv virtual environment.
If a dependency changes, change in requirements.in, recompile the .txt file, and reinstall.
MonaroVXR
>Run `pip-compile resolver=backtracking`
This is new for me. I used `pip freeze > requirements.txt`
Fradow
My solution is to:
* use a virtualenv, so that dependencies are installed per project and not globally
* only specify my top-level dependencies in a requirements.in file, and let pip-compile (a dependency, part of pip-tools) compile my requirements.txt
So far it has served me well and I've not encountered any error due to this method.
I cannot claim it is "the right way" though, just something that works for me.
mikepurvis
Specify your actual dependencies only in setup.py/cfg and only ever write your requirements file from pip freeze.
Or junk all this and just use poetry, which manages both the abstract dependencies (pyproject.toml) and concrete ones (poetry.lock).
jakewins
Poetry seems to solve the dependency spec part, both for libraries that need to ship specs for supported version ranges of dependencies and apps that ship lock files.
However: poetry still falls short in managing the python runtime, I am continuously having to divert time to help our data scientists untangle the messes poetry makes with virtualenvs and their local python setups.
Also, they broke backwards compat on the lock file format? So now devs running newer poetry versions break projects for devs on older versions because the lock files aren’t compatible?!
mikepurvis
Yes, I was very upset about the lockfile format change when we hit that; seemed like a very shortsighted thing to have done, but I guess I don't know the actual motivations or what was being achieved with it.
jwestbury
Use pipreqs instead of pip freeze. It resolves the minimal set of dependencies for your package.
MonaroVXR
Going from Windows to Linux wasn't a problem for me
liveoneggs
containers
influx
I use docker and just use pip install requirements.txt. Been working well for me.
bb88
I don't think you can, nor would you want to at this point. It would just be another standard (ala XKCD-927). Virtualenvs, pip, and wheels have been better than eggs and whatever else came before wheels.
Once you set the version of python in the virtualenv and activate it, the right python will always be there. The virtualenv even has an instance of python in it's bin. When that python is run, whatever's installed in the virtualenv comes with it.
Poetry, however, does suck even though it's using virtualenv underneath. It's doing frankly too much and seems to have a lot of regressions. In the end it should only be looking at the contents of the virtualenv and using pip to install the correct packages into it.
Edited to add: With the last version of poetry causing issues in my development env, I'm about ready to ban it from our team.
trallnag
Virtualenv has a Python in its bin? Any source on that?
jeroenhd
Venvs will actually contain a symlink to a python binary in its venv/bin directory. It doesn't copy a Python executable or download one from the internet as far as I can tell.
Still, this means you can install multiple system Python versions and your venv will always contain an executable file of the version the venv was last installed/updated with.
╭─android@localhost ~/tmp/py
╰─$ python3 -m venv testvenv
╭─android@localhost ~/tmp/py
╰─$ file testvenv/bin/python3
testvenv/bin/python3: symbolic link to /data/data/com.termux/files/usr/bin/python3Xophmeister
There’s a command line flag that will make it create an actual copy, rather than a symlink.
bb88
It's a copy on windows too.
deaddodo
Venv creates a /bin in it's environment dir that symlinks the Python version used to initialize it or as specified. That's it's entire basic model of operation and takes a few seconds to verify.
rubenfiszel
For the need of windmill.dev (OSS retool + FaaS), we support deno and python. To provide that same experience that you're speaking of which is "python run myscript.py" without having to pre-install the dependencies, we wrote our own python parser in Rust which infer the PyPi packages from the imports (and then our workers download them if they are not cached). Here is the code from the parser: https://github.com/windmill-labs/windmill/blob/main/backend/... which we could carve out as a separate module.
IshKebab
I doubt that would work. It isn't just the packaging that's broken. The entire import system is broken. I doubt there are more than a handful of developers in the entire world that really understand how Python's `import` works.
So to fix it you would need to change that, but then you would break a load of Python libraries which are most of the reason people use Python in the first place.
Deno doesn't really have that issue so badly because JS imports are ... not completely sane but they're quite a lot saner.
Anyway I think if you were going to break compatibility with Python by fixing the packaging and import system, it's not a much bigger step to fix the rest of Python too and then you have Nim or Lobster.
mixmastamyk
What problem are you having with imports? Believe I had a few issues in the Py2 times, but things improved and haven't in years.
DemocracyFTW2
> > The entire import system is broken
> What problem are you having with imports?
This is a Python function I wrote some years ago to import a module when given a path:
def module_from_path( name, path ):
### thx to https://stackoverflow.com/a/50395128/7568091 ###
### thx to https://stackoverflow.com/a/67692/7568091 ###
import importlib
import importlib.util
spec = importlib.util.spec_from_file_location( name, path )
module = importlib.util.module_from_spec( spec )
sys.modules[ spec.name ] = module
spec.loader.exec_module( module )
return importlib.import_module( name )
mixmastamyk
It is easier in Python 3 than it was. I wrote this and it worked on the first try, although I have some experience with it.
The dynamic module:
» cat sub/test_module.py
def hello(name):
print(f'Hello {name}')
» cat test_script.py
from importlib import import_module
full_module_path = 'sub.test_module' # no ".py"
module = import_module(full_module_path)
# optional, if you want to use across the project:
import sys
sys.modules[full_module_path] = module
print(sys.modules[full_module_path])
# end optional
# try it:
module.hello('there!')
» python3 test_script.py
<module 'sub.test_module' from '/home/foo/sub/test_module.py'>
Hello there!
https://docs.python.org/3/library/importlib.html#importlib.i...
In general, I don't see the intrinsic complexity being reduced much more than this:
from importlib import import_module
module = import_module('sub.test_module')DemocracyFTW2
Yeah this was years ago so if that has improved since good for the Python people.
> In general, I don't see the intrinsic complexity being reduced much more than this
you know in NodeJS I do `const tm = require('sub/test_module');` which I for a number of reasons do find inherently less complex.
mixmastamyk
It's not a builtin in Python-land because importing from a string is exceedingly rare there. I prefer to skip quotes and parens for the 99% case.
IshKebab
Nice idea, unfortunately it will break all static tooling. IDEs, static type checker, linters, etc.
I used to work at a company where they had a custom import system and it was a nightmare.
mixmastamyk
We use it to load classes from the command-line in a pretty big project. That part works perfectly.
EdwardDiego
My one massive hate about Python imports, path resolution aside, is that if I want to do
from foo.bar import Baz
And because of how Django's settings work (import django.settings and all your stuff is magically there), people in my company have gotten into the habit of putting code in settings/__init__.py to do the things they need, like retrieve secrets from AWS.
Which I don't want to have happen when I'm trying to import settings.bla_module to use a type in a unit test.
mixmastamyk
> And there's a foo/__init__.py file it gets executed merely from path traversal.
Yes, and it needs to—to find the variables there, which may/not be other packages.
> how Django's settings work
Right. Django is great and popular, and so I think many folks don't realize how downright clunky it is in a number of places. Many parts of it are over-engineered like this one. It has improved over time but some things are set in stone due to history. I'd like their settings redesigned myself.
> __init__.py to do the things they need, like retrieve secrets from AWS
No, no, no, no... NO! ;-)
Imports should only create variables, define functions, and/or alias a package. They should never, ever run lengthy computations or access the network. That's basic software design 101, unfortunately not taught everywhere.
I don't see this one as a clear flaw of Python. "Dumb" coding can happen anywhere. I suppose it could prohibit this kind of stuff, but that might prohibit other cool things that might be reasonable in limited situations. It is a "consenting adults" design, with those trade offs.
IshKebab
This question should give you an idea of some of the issues:
https://stackoverflow.com/questions/14132789/relative-import...
mixmastamyk
Looks like people thought "." refers to the filesystem CWD instead of the current package. Python never made shell constructs that central. Admittedly, it never occurred to me to confuse these two, and so haven't had any of the problems any of those folks mention.
And you need (maybe used to?) an __init__.py file in each package.
But once you know that, it's straightforward, no? Certainly not, "only a few people in the world know this..." level. Don't think I'm some kinda level-10 genius here.
IshKebab
> Looks like people thought "." refers to the filesystem CWD instead of the current package.
Yes because referring to the filesystem is logical and intuitive and referring to some abstract "package" that changes depending on how you invoke Python is insane.
> But once you know that, it's straightforward, no?
I wish!
mixmastamyk
It's a programming language, not a shell, of course. It has always been strict about syntax as well. I don't expect "&" or ">" to work the same in the shell as I do in Python—why would I expect "." to?
Close to none of the notation of Python works the same as a shell. Even quotes, which are superficially similar but have significant differences.
> abstract "package"
Packages are not abstract, they are simply a folder with .py files inside. Sounds concrete enough to me.
> changes depending on how you invoke Python
I'm not sure what this refers to? I know you can change PYTHONPATH. And the current folder portion changes with it.
qbasic_forever
Develop in a container. It will remove all other issues from your machine/environment and let you declaratively specify the exact environment you need (i.e. packages to install, version of python, etc). Forget all about venv and all the other tools in the python ecosystem--in your container there is _one_ python and you control all of its dependencies.
nawgz
Pragmatic advice, certainly, but maybe it's missing the point - you shouldn't need a clean OS every time you want to use a new language version. That's a ridiculously large failure of packaging.
qbasic_forever
It's core to the design of python right now. Remember when python was designed in the 90s people had big multi-user Unix systems that were bespoke pets. It was assumed that python would be a system-level tool and all packages, etc. would be installed at the system level. Your Python install would be a living thing that was upgraded and administrated like any other software on the machine.
Fast forward 30 years and we really don't use most systems like that anymore, especially in production environments. We run things in VMs or containers and build them up from scratch with ease--it's all just cattle. Python hasn't really adapted to that new reality.
dvdkon
The fact that many popular Python packages rely on system-level native tools also doesn't help. "Solving Python packaging" isn't just about distributing portable Python source code, that's easy. It's also about packaging and distributing unruly non-Python software for a myriad different platforms, all without user intervention. Sure, the solution's not great, but it's a really hard problem.
nonethewiser
Why aren’t virtual environments enough?
scruple
I want to hear the answer to this, too. I'm back around to Python professionally (haven't touched it since 2016) and I'm working with the basic tooling: pip, requirements.txt, and virtualenv. I'd like to know what sort of issues I'm going to run into and when I can expect them.
mixmastamyk
They are enough, just need care and sysad knowledge which are in short supply these days. Docker is an end-run around that, but you'll then have to know it as well.
qbasic_forever
You have to remember to activate them. Node/npm has a similar concept with its scripts execution, but it automatically runs them in the local node_modules environment (i.e. virtualenv).
You might think this is silly but think about something like making a systemd service to run your python script in a venv--how do you do it? You have to activate the script or call its python bin, but it's not obvious how to do that in any python docs.
wswope
It is silly. You put together a three-line bash script to activate the venv, run a pip install, and call your program.
The Python docs do cover this topic, and do so rather well: https://packaging.python.org/en/latest/guides/installing-usi...
jakewins
But the problem isn't my python environment - the problem is distributing packages, particularly libraries, to downstream users. That one over there uses virtualenvs and a bash script she wrote, that one just ran "pip" and has 4 different system python installs entangled into each other; that one ran `pipenv install`, that one `poetry add`.
How do I ensure that at each customer site, my library has the set of dependencies it needs, with versions that it is known to work with? Each installation method has a different constraint solver, a different means of specifying dependencies.
wswope
Did you mean to respond here? This is a nonsequitor from the GP I was responding to. Regardless:
If your question is "how can I ensure technical end-users have the same set of python packages as I do for the code they run using standard pip+venv?", the answer is to pin dependencies in a requirements.txt file.
If your question is "how do I stop end users from installing dependencies for my software cowboy-style?", the answer is to write installation and usage instructions, and/or include an AIO run script.
If your question is "how do I package my library so that end users' package managers know my library's downstream dependencies when they install it?", you build a wheel using `pip wheel`, which again relies on a requirements.txt. If I'm understanding you right, you're mistaken that you have to handle the package managers separately; they all use pip + wheels under the hood. Conda is a bit of an asterisk in that you can package things differently if you desire, but it plays nice with pip + wheel builds too.
https://docs.conda.io/projects/conda-build/en/latest/user-gu...
bb88
You can do that today. You're not forced to run in a virtualenv if you don't want. Install the python as a system package, and then sudo pip install your way to happiness. That's how our production containers are built.
varjag
Other languages manage somehow. Python also could perhaps, if not the cultural dysfunction.
morkalork
I work with a bunch of data scientists and they manage to go about 1.5 years between "I've totally fucked my Python environment and have to burn it down completely and reinstall it from scratch" events. So yes, I would agree with your statement about Python's packaging system being a ridiculous failure.
Gordonjcp
I disagree.
It doesn't matter which language you're using, any non-trivial code you have is going to have dependencies which you will need to pull in.
If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!" problems that used to plague people.
nawgz
> I disagree
You disagree that "you shouldn't need a clean OS every time you want to use a new language version"!? ... I don't even know what to say, that's not something you can disagree with
> If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!"
This is what testing is for. You don't need to cripple your development environment to have confidence your code works in other environments. Talk about the ends not justifying the means.
Gordonjcp
Can you explain why having a reliable, repeatable and well-tested development environment is "crippling" it?
nawgz
Because it's trivial to accomplish these things in other languages without containers?
bick_nyers
When I was first learning python/tensorflow I tried using docker containers, but got a huge headache trying to navigate using a GPU with a container. I know VMs also can't really share GPU resources that effectively (at least consumer cards in pedestrian setups).
I built this resistance to docker about 5 years ago, is it (or was it) still correct? Is there a best practice for sharing GPUs with containers that's not a pain in the ass?
Edit: I should specify that I typically develop on Ubuntu, but occasionally will do so on Windows (without WSL).
monkellipse
This was my solution. It feels like overkill at times but I never have to worry about packaging messes anymore. That being said I can understand the desire to solve it on the metal, it’s a mess.
pdpi
I thoroughly resent that I'm being forced into that workflow. I'm trying to get enough distance to tell if it's actually overkill or just being a curmodgeonly old git.
pharmakom
Agree. We don't need to use containers so often in other language stacks. this is a failure of the python ecosystem.
nerpderp82
Resent it, like resent wearing a seatbelt or eating your vegetables. It is the right thing to do even if it causes you frustration. The worst thing is if people read your statement and copy you. A container is the easiest and most correct solution here.
qbasic_forever
Well your alternative is to use a VM, which is a slower and clunkier container, or fight the myriad of bespoke python environment management tools so you can make one Python install work for X number of projects and all their unique dependencies.
pdpi
I meant in general, not for Python specifically (which I don't use nearly enough for it to be a problem).
On the one hand, it bothers me that software is so brittle that you need a container around it so that everything is just right. On the other hand, you can argue that containerisation is a clever way to make it all more robust.
nerpderp82
The issue is with the OS and how dynamic linking works (or doesnt). The solution isn't for the maintainer of the dependencies to "try harder" or be better at what they do. The underlying system is broken and containerization is way to compartmentalize those flaws so they are less destructive.
If you put everything into an Uber Container, the same problem would surface.
Containers exist to solve the fragile dependency/dynamic linking problem.
darepublic
Try it out you may find it makes you more productive then the old way of doing things
LanternLight83
Guix (and maybe nix?) insists on being the "one package manager to rule them all", so Python packages and Rust crates are all re-packaged (sometimes with automated importers). Well I do still occasionally need to package something myself (or cheat pip/requirements.txt it), it definitely covers this "burn it down" philosophy and keeps environments isolated and reproducible.
bayesian_horse
I have never used poetry... Mostly I just use conda or plain python envs. Never had those problems like you mention. When using anaconda in the stable channel you'll get a straight-forward "conservative" distribution of all the data science/numerical packages (and more). That's why people often say "just use anaconda", it really is quite simple as long as you don't mess up your stable environments with exotic packages you just want to try out. And it works well on Windows. No idea about Mac and don't care.
Python doesn't need "something like deno". All programming languages need package management.
nonethewiser
How many people work on these projects managed by conda?
> And it works well on Windows. No idea about Mac and don't care.
Package management is a cross platform problem. You may not need it but that doesn’t make the current solution good.
bayesian_horse
There are no "projects managed by conda". You use conda to package or deploy Python software and its dependencies. The Python software itself really doesn't care that much.
tetha
Dude, you are dragging up bad thoughts of how finicky it can be to get ansible to work consistently across a number of systems.
This is where I kinda regret switching from and miss chef. Pick up an RPM/APT package from vendor, shove it into your package manager, the environment works. It developed a sufficient amount of weird behaviors later on, but that's besides the point.
Instead we have a really precise documentation of 4-5 steps to get everything installed in a way that should overall work, and each step is annotated with 4-5 ways that look deceptively good but end up being even more quirky. And when you do all of that, and review it twice, it still doesn't even work consistently across 6 workstations. 3 work the same, 2 have interpreter discovery anomalies, 1 has import conflicts with seemingly unrelated other installed stuff.
Whenever I have a working python VM / container build or installation, I kinda feel a need to shut up about my jenga tower because otherwise the universe will find a way specifically to ruin my day.
I hate how I have to feel like this about a language I really like.
noloblo
Just create a folder for your project
And then
Python3 -m venv envdir
source envdir/bin/activate
bin/python3 -m pip install --upgrade pip
And similarly the rest of the packages needed for the project
This does not mess up the system library or other python projects
BeFlatXIII
> source envdir/bin/activate
This step is the least intuitive until you've done it enough times that it's muscle memory.
d0mine
"There are only two kinds of languages: the ones people complain about and the ones nobody uses."
Can it be better--yes. Is it worse than _any_ other language's packaging if it were to scale to the number/kind of cases Python supports/used--no.
"thoroughly broken" is useless. Are there bugs--most definitely yes. Do report them, or even help fixing them if you care.
There are many many different use-cases where different approaches are preferable (no size fits all).
Personally, most most of the time packaging is a non-issue e.g., pinning versions and cache are highly recommended for CI to avoid unnecessary surprises, flakiness.
bheadmaster
"There are only two kinds of languages: the ones people complain about and the ones nobody uses."
- the guy who created C++
I always bring up Go in such arguments - according to the survey [0] done by the Go team, 93% of Go users are satisfied with the language.
d0mine
When if ever Go becomes as widespread as either C++ or Python, then we talk
bheadmaster
C++ being horrible has less to do with its widespreadedness and more with the committee-driven design and being intentionally designed as a C derivative in order to gain maximum backwards compatibility.
lmm
I got fed up enough with Python dependency management that I started using Maven to build/run my Python programs. It worked well enough, but having to package the whole world myself got old, and I realised I preferred Scala anyway.
kjkjadksj
This is literally why conda exists…
cure
> A “burn all the packaging down and start over” path?
In Python land one of those seems to come along every couple of years. They start over and implement a solution for some subset of the problem space. Then they realize the Python packaging mess is much, much bigger than anticipated and progress stalls. At that point there are 15 partial "standard" packaging solutions for Python, where there were 14 before. None of them are feature complete, and they don't really coexist well or at all.
Cue the obligatory https://xkcd.com/927/ and https://xkcd.com/1987/
rcme
Python is only popular because of the packages. I doubt many would use Python if they didn't get to use PyTorch, etc. Another thing that makes Python different than JS is that many of the most widely used Python libraries are really Python wrappers around native code. Of course some Node libraries rely on node-gyp and other native tooling, but many applications are pure-JS. Porting native libraries is bound to be challenging unless a similar native API is provided.
v3ss0n
lets compare apples to apples. 1 - Scientific Libraries are written C python use as glue code. 2 - Almost all of web frameworks are written in Python except Blacksheep where core parts are written in C.
From that presepective , most python libs are written in python only.
deaddodo
This is what pip+venv were built for. The alternatives are from people who wanted a "better" or "preferred" experience. Or to solve a specific problem that the in-built tools haven't.
If you get any python job, you'd get a lot of side-eyes for not knowing of/how to use and initialize basic venv package sets. Even if the shop uses conda.
BiteCode_dev
We used the "burn all" strategy I/O during the 2 to 3 transitions.
People were not impressed.
Hence the previous article (https://bitecode.substack.com/p/relieving-your-python-packag...) advocating to go back to the basics yourself.
See the reddit thread to witness exactly this in action:
https://www.reddit.com/r/Python/comments/124hktv/comment/je5...
nonbirithm
With the sheer brokenness of Python packaging I wish that the Python 2->3 transition had been between two completely different languages, such that the entire thing could have been thrown away. Yet because there was breakage, but not enough breakage to amount to moving towards a better rewrite, nobody's ever going to attempt a true Python 4. They blew their chance.
Now we have a never ending train of hotshot developers that think they can "solve" Python packaging with another tool, without realizing that you can't design a tool to fix a broken 15-year-old system. It's not just a "there are now X competing standards" issue, it's also the fact that anyone who makes an earnest attempt is doomed. They will never be able to get it past the point of universally good enough.
I lament the fact that I'm dragged back into the Python ecosystem every time I need to do something with a PyTorch-based tool someone spent years of effort building. I would never recommend the language to a newcomer. Ruby pollutes the global namespace, and Gemfiles execute arbitrary code, but at least with Ruby there's Gem, and then there's Bundler, and nobody complains.
Will we forever be beholden to the mistaken decisions of the original developers in the early 90's, just because Python is simply too widespread? Because even people like me who detest the packaging experience are forced to go back there because nobody's going to rewrite the hottest ML project in Elixir or something? Even the simple mistakes like putting venv files in Scripts/ instead of bin/ forcing one to work around it in cross-platform CI make no sense in hindsight. Nobody remembers why it was made that way after so long. Now it just floats over you like a spectre.
noloblo
Use haskell ocaml or racket - golang to try a different paradigm that's not a holy mess
Python's environment management is just the tip of the ice berg
Dynamic typing
Gil
Speed is slow at best
Async streams having its set of non tractable issues
Debugging by printing pdb all over and breaking
MonaroVXR
So, wait a minute I get that everybody can choose a laptop. But if this holding you back, you can a Linux vm or a Linux distro.
kjkjadksj
Conda has none of these issues. Its so easy to use and maintain dependencies imo.
rektide
Having encountered poetry recently for the first time, it was "simply" hell. I just wanted to use a single file python project, https://github.com/rumpelsepp/oscclip
I spent about three hours trying to figure out how to setup python keyrings to work, to let me just get started using poetry. On a system I was ssh'ed I to. Gnome-keyring-daemom was up. I spent a while adding random pam rules suggested by archwiki in to inject more gnome-daemon stuff in my envs. Random gnome-keyring-unlock scripts, which quickly start talking about Clevis and tpm and fido 2-factor. Wading through hundreds of responses about Seahorse, a gui tool unsuitable for ssh. Many long miserable sad stories.
In the end I stumbled upon someone who suggested just nulling out & turning off keyring with some config to make it have a null provider. After this the poetry project just worked.
The tiny handful of deps this project has were already installed on my system, but poetry was also a task runner, instrumental for the usage of this single-file script.
There's been so many years of churn in the python world of tools. A fractal nesting doll of virtual-env, mkvirtualenv, & various offshoots. I hope some day there is a mature reasonable option folks generally find agreeable. Poetry eventually worked for me, but what a miserable gauntlet I had to walk, and the cries of so many who'd walked the path & utterly failed echoed out at me at every step.
AndyKluger
FWIW, for the case of installing Python packages as runnable tools (as opposed to importable libraries), pipx is solid.
For example, you can install oscclip to a temp dir with a dedicated venv and run it immediately:
$ pipx run --spec 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help
$ pipx install 'oscclip @ git+https://github.com/rumpelsepp/oscclip'
$ pipz runpkg 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help
$ pipz install 'oscclip @ git+https://github.com/rumpelsepp/oscclip'mixmastamyk
Yeah, that's the problem with all these "helpful" posts.
You don't need poetry for that. My single CLI packages still use a setup.py that I haven't touched in five years and was simple enough to write.
whalesalad
Yikes. One of my favorite features from pip is how easily you can install from a git repo, or even the absolute url to the master.zip
dsr_
In general, don't tell people to "simply" or "just" use anything unless you're willing to provide the precise config that they need or otherwise hand-hold them through the starting phase.
Nothing in computing is "simply".
inconceivable
"trivial" also fits into this category. i've been keyboard jockeying for 20 years and every time i see that word i groan a little, because it's both a shitty flex and probably untrue.
tetha
My documentation skills have improved when I started to be critical about "obvious", or "trivial". And now I routinely end up writing "obviously", stop, and end up with 3 pages of clarification.
m463
Actually the best use of "trivial" is in the negative, like...
"Wellllll, that's non-trivial."
mparnisari
THIIIIS i hate it when people say "oh just do this"!! "JUST" implies it will take me 3 seconds, where in 90% of the cases it takes me 3 hours!!
theptip
I think this applies in some cases, but disagree here. “Just use poetry and pyenv” is my line, and I stand by it because these tools are well documented, and work well for most use-cases. It’s contrasted with a bunch of other more complex options, in tedious yak shave conversations. Often in threads where many people are complaining about how confusing the multitude of options are.
It’s not confusing if you short-circuit the conversation; if you need advice, don’t go down the rabbit hole, just use poetry and pyenv on MacOS, you will make your life easier than the alternatives. There are plenty of docs that will give the hand-holding that you are looking for, so nice you have made the decision.
If you know what you are doing there are arguments for other options, and data scientists have different tool chains. But I think we make it harder for new engineers by having five different ways of doing it, each with people arguing that actually their way is better.
nonbirithm
I agree with you, but the difference between being able to tell someone "if you need to install some dependencies in Rust, just use Cargo" and show them the documentation, and whatever the equivalent documentation is in Python are worlds apart. I think the author's argument would be weaker if he focused on a language other than Python that doesn't have a hellscape of a package ecosystem.
And I remember custom build scripts with Cargo being a major pain at the time I used it, but the Cargo developers were able to sidestep major fundamental problems because they knew what didn't work with packaging in the decades before and managed to think about the design carefully enough from the start.
Rust is different because the choice of package manager is "simply" Cargo, not necessarily that Cargo itself is trivial to use.
kjkjadksj
Check then for conda, which only asks you to copy and paste a shell command they provide for you and eventually say “yes.”
livelielife
except that side of computing for which users are really "input" to be turned into "output"
which is the industry for which users are really the product. user's data in, profits out.
jooz
Ive heard that 'venv' are very problematic, but honestly, Ive never had a problem. And I used them daily. I understand that it can not be enough on some cases... that don't concern me.
I would recommend to 'python -m venv' and thats all.
abrichr
Agreed, never had a problem with this approach.
The only limitation I've encountered is when moving the environment or renaming one of the parent directories. In which case it's easy to create a new one:
# optional: freeze the environment if you don't already have a requirements.txt
source .venv/bin/activate
pip freeze > requirements.txt
deactivate
# remove the old environment
rm -rf .venv
# create a new one
python3.10 -m venv .venv
# activate it
source .venv/bin/activate
# install the requirements
pip install -r requirements.txtnicoburns
I've found that venv's work. They're just very inconvenient (having to constantly manage which venv you're in when you change directory). Consider that with NPM, only two of those commands are ever needed:
# install the requirements
npm install
# remove it
rm -rf node_modules
sureglymop
Python has this too, check out PEP 582. There's a package manager called PDM which will just create a __pyoackages__ folder in this style.
rami3l
As a PDM enthusiast for quite a while, I'm sorry to inform you that PEP 582 has been officially abandoned (which doesn't mean that it won't be revived someday, just not so easily). https://discuss.python.org/t/pep-582-python-local-packages-d...
pepa65
This just illustrates that the most recommended solution is not easy to use, many steps one could forget. And I guess if your package manager happens to update your default python, more/other steps will be necessary...
abrichr
Venvs are self contained python environments, no interaction with system python.
For those already familiar with venvs, the above just condenses to “remove the old venv and create a new one”.
Groxx
I've heard a lot (a lot) of complaints that are wildly misattributed to venv (like "version X of Y broke my project"), but I've essentially never heard of issues with venv itself.
Aside from needing to know to use it. Which is certainly a problem. But python blessing a single venv-system might be worse in the long run...?
BiteCode_dev
That's pretty much the take of the article this one references.
The whole reason I posted this one is to justify the "just use -m pip and -m venv" stance I wrote in the article 4 days ago.
Because you will always have a very vocal minority of people that will fill the threads with counter arguments actually increasing complexity and risks of failure.
tetha
This is one of the jenga towers that seem to hold up most reliably for me as well. (i) Use python installed via the package manager. (ii) If python is updated, wipe all venvs, because there can be strange interactions between venvs and the installed python and rebuilding venvs is cheap, and (iii) Work in venvs as much as possible.
bobx11
This is also my setup. It has the added benefit of being already included on every python install already so there is nothing extra to use.
nagonago
I'm curious what these problems are. The only problem I've had with venv is that sometimes I forget to activate it.
As the article says, I think these days it's pretty safe to just use Python's built-in venv and stay away from everything else.
hobs
Same no problem, different solution - I use pycharm extensively and it manages my venvs 99% of the time with no issues at all, the only time it took me a minute of head scratching was realizing I needed to install a new system python to make a venv with it, but that would be clear if you were doing it via the shell approach you are using as well.
atemerev
How do you manage python versions with venv? E.g. numba still doesn’t run on my system python 3.11, so I want a 3.10 somewhere, but can’t install it through a package manager, etc.
0xDEFACED
If using apt, you can add the deadsnakes repository which allows installing the other versions of python3 not included with your linux distro. Then you can install them with
sudo apt install python3.X
sudo apt install python3.X-venv
python3.X -m venv .venvtibbon
Python as a language I find pretty nice. What I don't find is their environment and packaging system compared to something like Rust.
"There should be one-- and preferably only one --obvious way to do it.", unless it is how to setup your environment.
I only use Python every few months, and it is always a struggle.
In comparison, "cargo build" works 98% of the time just after "git checkout"
belval
Yes, but Python really is in a tough spot when it comes to dependencies because a lot of it are compiled in a lot of different languages using libraries that may or may not be available on your OS. That's the real issue. If you install python packages honestly anything works, it's when you have the ML stack of PyTorch (needs cuda, cudnn, needs to be compiled to match your OS version) + custom CUDA operators (also need to be compiled) with something like a mariadb connection (needs the mariadb OS library).
conda solves it by packaging EVERYTHING, giving you atrocious 30GB environments, pip doesn't solve it at all and none of the challengers really have much to offer (in my opinion).
blactuary
That's Anaconda that packages everything, not conda itself. No one should really use Anaconda imo, instead use Miniconda which only installs the bare minimum needed for conda to work in a base environment, and then you create environments for each project.
2h
100% agree. I use a programming language to get stuff done. and if the day ever comes that someone wants me to show them how I do what I do, I dont want to start that conversation with a sigh "well...", I want to start it with a "OK cool..." and all these Python "tools on top of tools" make me sad.
Personally I like Go. If someone wants to build my stuff, then I just say go here http://go.dev/dl and download Go, then set location to where the code is, and enter "go build". thats it. All languages should be that easy.
bayesian_horse
Go gets around all these problems by being not all that popular (compared to Python and Node), not being as general purpose, and starting from a blank slate with control mostly from a corporate overlord. Static linking also simplifies things.
Go doesn't make a habit of interfacing with lots of legacy C,C++ and Fortran Code. There's probably some of that. But nothing like Numpy, Scipy, Tensorflow...
nicoburns
Rust/Cargo manage to deal with interfacing with lots of legacy C and C++ code (not sure about Fortran) while still "just working" seamlessly. Usually it's easier to get a C library to build via the Rust package (just `cargo add package && cargo build`) than it is to integrate it into a C build system.
bayesian_horse
Not my point. Pip/setuptools can build most C/C++ code just fine. Most, maybe even "almost all an average user would care about". But Rust doesn't extend into the same depth, and once it does you can expect the same issues. And you hardly ever need to compile Python extensions with pip or conda either, because both can distribute binaries for the most used platforms anyways.
Yoric
FWIW, last time I attempted to install a Python dev environment under Windows (for teaching purposes), I eventually gave up because I couldn't get all my dependencies working. That is an important data point for me. Fortunately, I had a macbook at hand for teaching, but that's hardly ideal.
I'm not sure what you mean about Rust not extending into the same depth, could you elaborate?
bayesian_horse
That must have been some strange dependencies. When I install anaconda on Windows, there's almost nothing to do to get Tensorflow, Torch, Jupyter etc going. Same with Blender, it just plain works.
And if not then that's not a packaging issue but rather a problem about portability because yes, some packages are system specific, and Windows is quite specific. One way around even those issues is to use either WSL2 or Docker desktop, maybe using VSC development containers or WSL remote.
There's always a fringe of packages in every package manager that deosn't have the greates compatibility among different dependencies and systems. Python's ecosystem is no exception to that.
Yoric
I was attempting to install PyGame. Hardly an unusual package. But the dependencies were somehow broken on Windows. After 20 minutes of trying, I gave up.
bsder
Not even close.
What is going on in the Python ecosystem is people expect Python to interface to C/C++ modules on Windows. Reliably. And no other ecosystem even really tries.
As an example, Rust interfacing to Skia (C++) used to cause hideous build errors on various Windows installs. I needed to install and upgrade significant chunks of Visual Studio (not VSCode--actual Visual Studio) before the Rust system could compile it correctly. And the next Windows update it all broke again.
Yoric
While this is largely true, the Python situation remains a mess that needs to be resolved.
Also, Rust seems to indicate that being general purpose and interfacing with lots of legacy C (and some C++) aren't the issues.
bayesian_horse
Nope, Rust is not doing anything better. It is doing significantly less. The fewer packages you have the fewer dependency issues you run into. Rust will still run into the same DLL issues on Windows, requiring custom solutions, once the package ecosystem comes anywhere close to what Python offers.
Yoric
I'm not sure how we ended up discussing dll. You were writing about "interfacing with lots of legacy C,C++ and Fortran Code", which is something else entirely. Does this mean that we agree that "interfacing with lots of legacy C,C++ and Fortran Code" is not the issue and that we have moved to another issue?
Dynamic libraries can definitely be a problem, across languages. I've seen plenty of crates that rely on platform-specific .so/.dylib, so I'm not really nervous about that being supported, but it's absolutely possible that there may be Windows-specific issues that I haven't heard about.
edit Complete rewrite, let's hope nobody has responded yet :)
paulddraper
Wait did you just call Go versioning and dependency management "easy"??
jerf
Are you running on 5+ year old data now? I'm seriously asking, because I'm actually a bit mystified as to the people who think Go's dependency management is some sort of disaster nowadays.
Just about the only question I've fielded every so often in the past several months have been people who found their way to an old tutorial based on the old system and them being confused about what happens, and a quick link to the current docs seems to resolve all their problems.
cure
Compared to the dumpster fire aka Python packaging, Go versioning is easy and predictable indeed.
silverwind
Until you hit the obscure rules around v2+ go modules.
2h
I have been coding in Go for years, even professionally sometimes, and I have never had to use "v2" in any of my code. granted my stuff is not popular, but its really just a way for popular repos to handle big changes. Once I got past v1.9.9, I just changed to v1.10.0.
silverwind
SemVer dictates a new major version on every breaking change, but it seems this concept is alien to many go developers and they just release breaking changes in patch releases, at the detriment of the ecosystem.
2h
my biggest project has 13 stars, so I think I can do what I want with it.
paulddraper
Fair
2h
I dont have trouble with it. Link to my code is in my bio, wheres yours? Unless you actually write Go code, I dont think you have a leg to stand on here.
blondin
that was before go modules and now workspaces right?
NuSkooler
Why not "simply" admit that Python and it's ecosystem while broad, are an absolute clusterfuck?
klibertp
Because when you try to manage dependencies with Gradle on Android you realize that it could be worse. Much, much, worse.
These are the languages and package managers I have experience with:
- JavaScript and npm
- Python and pip/conda
- Kotlin and Gradle
- Scala and mill
- Common Lisp and quicklisp
- Racket and raco
- Raku and zef
- Lua and luarocks
- Elixir and mix
- Haxe and haxelib
- Rust and cargo
- Nim and nimble
- OCaml and opam
- Smalltalk and Monti/Meta-cello; also Visual Works parcel manager
- and a few others
Out of all these, cargo, opam, mix, and zef are the only ones that are mostly working as intended. They're still a pain, but they're predictable and honest about their limitations. Everything else is just bad, to the point where it's a waste of time to think about which is the biggest and which are a bit less of a clusterfuck. For some of these I had to resort to building a chroot env with a copy of my system inside to convince them to compile and link against correct libraries. If that's not a clusterfuck, I don't know what is...
Unless you were lucky to see something better than "mainstream state of the art" solutions, you won't realize how bad it actually is. At that point you start inventing yarns or poetries, adding layers on top of layers to paper over simplistic, hardcoded assumptions made by original authors 20-30 years ago, and go into denial when someone tells you that's not going to fix anything.
whalesalad
Because that’s not at all the case and your rhetoric is what perpetuates the false image.
stathibus
Yeah it totally is, and people who pretend it's not are the problem.
Somehow we ended up in a state where step one of doing anything new with python is to fire up an empty docker container. And I'm awfully tired of folks in the "python community" blaming the victims of the mess they made.
nonethewiser
> Somehow we ended up in a state where step one of doing anything new with python is to fire up an empty docker container.
This approach seems insane but correct.
BiteCode_dev
No, use the procedure described in the previous article.
It will be less hard than using docker.
JohnFen
As a user, it kinda is.
My recent example: last week, I was installing a new program that uses some python scripts during its operation. I could not get those scripts to run. I knew it was because of the usual python problem of the scripts not matching the version of python that was executing them, but figuring out how to fix that took me a full day.
Just to make Python work. Not even as a dev.
Wearing my user hat, this is a clusterfuck. There is no other language that I am exposed to that presents this sort of problem, and this is a very common issue with Python.
This is why I start to get sweaty any time that I'm using software that involves Python. It turns into a crapshoot and half the time, it's going to cost me a lot of time and stress.
> perpetuates the false image.
You can believe it's a false image if you like, but there are a whole lot of end user experiences that indicate it's very real.
BiteCode_dev
Use the procedure described in the previous article.
It will help with exactly that.
JohnFen
Right. But the point is that if I have to have special knowledge or do tricky things with Python just to get end-user software to work right, that's a problem. You should just be able to install the software and have it work right. End-users who are not so technically oriented would never even know to look for those instructions, let alone be comfortable following them.
Python is the only language I've encountered that causes this sort of trouble for ordinary end-users. As a dev, I find this frustrating because the python scripts in question are clearly only "glue" in the first place. That's a lot of burden to put on the end user for just glue.
As for my issue, I did make it work in the end. I'm 90% sure I didn't do it in the right way and it will cause me problems at some point in the future, but that will be a problem for future me.
Danjoe4
I am a huge fan of python and have rebutted some of the criticisms in this thread. I would also avoid shipping python to users because of the issue you point out. If it couldn't be avoided I'd probably package the version of python I want them to use, with my software.
tough
python apps as .exe's is all you need to know exist to recognise the cluster fuck from miles afar
pepa65
If those .exe's (or Linux/OSX binaries) would work as intended, than maybe that is a solution..!
BiteCode_dev
True.
Ideally it should be fixed at the installer level.
I will write about this at the end of the week.
DemocracyFTW2
In my experience 'clusterfuck' is not a fair description of the packaging and import system. 'Fractally intertwined half-broken huge ball of macaroni with spaghetti nested inside' is more accurate.
BiteCode_dev
Believe it or not, it improved tremendously in the last 10 years:
- easy_install, distutils and eggs were deprecated
- ensurepip and venv were introduced
- the whole wheel ecosystem flourished
- a new dependency resolver was introduced to python
But:
- a lot of noise has accumulated. Hence the procedure in the previous article, to avoid the traps from all that noise and go back to what usually works.
- we have a long way to go, but it's not something easily fixed (hence the conclusion of the current article explaining why it's going to take a long time to get better).
AtlasBarfed
"we have a long way to go"
Hopefully, the direction of that long way is relegate python (and javascript!) to legacy languages.
Something, hopefully, will come out of webassembly. Otherwise, the last decade only saw the rise of two fundamentally flawed languages as the most popular ones.
Ruby losing to both was not a good thing. And I'm not even a Ruby programmer! I hated the language back in the Rails vs Java days. But it is impossible to deny that great software came out of Ruby.
Javascript and Python has produced no great software, just ponderous stacks of crappy code and poorly organized libraries. They are popular because they are used for throwaway web applications and throwaway scientific / non-programmer code, and it shows.
The fact that golang is what is producing new "good" software: a language that is intentionally bad, as opposed to python and javascript being worse because of cruft accumulation or Eich writing it a weekend, is also not a great sign but it's not the step backward that javascript and python are.
belval
> Hopefully, the direction of that long way is relegate python (and javascript!) to legacy languages.
People writing in JS are usually techies who could/would migrate. People writing in Python are on a wide spectrum that goes from "one-step-above-matlab" to "know-it-all-c-wizard". You won't actually get all these people to move on any reasonable timeframe.
HN is biased because we tend to love tech and a lot of us know several languages, but Python is much more democratized. My friends in physics, biotech, mechanical engineering, basically everyone in STEM is using Python and to them it's a tool that took a lot of time to learn and they won't drop it all because dependencies are hard.
Danjoe4
"good software" is not a goal. My goal, especially if I'm a solo dev, is to make something cool, efficiently. Instagram is built on Django and probably lots of JavaScript too. Maybe their software is a slow, ponderous stack of crappy libraries, but if those languages can function at that scale they're probably good enough for 99% of people.
Besides, code at a large scale in any language is always messy. There are many ways to manage a growing codebase but if your only approach is relying a "good" language with static typing, verbose/explicit syntax, and a better built-in package manager, you are doomed to fail anyways.
Perhaps this is a case of correlation=/=causation? Python is more accessible so less adept programmers are more likely to use it.
BiteCode_dev
If youtube doesn't count has great software, I don't know what does.
NuSkooler
All of these wonky tools are written for and by Python developers who themselves trip over the insanity. I think I have enough evidence after watching this go down for over a decade.
throwawaaarrgh
Stockholm syndrome?
melody_calling
If you're just using python as a local scripting language, and not pushing production code, the other option is to simply not bother with any of this.
When there's a new python version I'm interested in, I install it via Homebrew and update my zshrc to clobber everything else via $PATH. All my scripts and tools are broken? Just reinstall the packages globally. Whatever.
Since the big 3.x transition, it's pretty rare for forwards-compatibility to break (IME), and if something does, I can just try running prior python3x binaries until I find the last version that worked.
It's hideous, but honestly the least stressful way I've found to date.
kjkjadksj
Or you can just do everything in conda and maintain compatibility forever.
zaptheimpaler
Python packaging is sooo fun it has given me permanent brain damage and PTSD. Now I have a docker devbox with all my language toolchains and fun unix tools installed. I could install 3 different versions of cuda, 8 different pyenv pythons all sharing parts (but not all) of each others modules with torch compiled for a 4th different version of cuda that is NOT installed, then replace the core system python with a duck. pipx has somehow installed a version of borg backup that depends on a secret hidden 9th python. Then I will simply `docker rm` and `docker compose up -d` and I'm back. Yesterday I ran a random academic paper ML model in python in 5 minutes on my docker machine. HAHAHAHAHAÀÄÄÄÄÄ i am invincible!!!!!!
jamesralph8555
Docker is the only way to do ML. One thing I didn’t think to do before for a while was make separate docker files for each project so all of the deps are installed automatically and that has since helped tremendously.
throwawaaarrgh
Newbs add abstractions to avoid complexity. Veterans avoid complexity by removing abstractions.
regularjack
My personal experience is the complete opposite of this.
nonethewiser
Back to assembly I guess
bick_nyers
That is way too high of an abstraction. Use breadboards.
tpoacher
I often find the reason for all this hell is, ironically, an effort to help people who dont know "how computers work", by offering "useful automations". And then these automations clash and fail because of the complexity involved.
If you "simply" (yes yes I know) download the python version you want directly and compile/install in a local folder, and use that with venv as a way to manage individual project dependencies, all problems go away.
jakewins
Until you need to distribute that project as a library, or you’ve shipped it as an app and now you need all users to upgrade library X or add library Y.
Python packaging is fine for small local dev; problems arise in distribution, rollouts of upgrades and ensuring your apps work in the zoo of local setups your users have
davb
Python runtime deployment is a major pain point for us (CS department at a university).
On the most tightly managed lab machines, which are all in lockstep on a fixed configuration (latest Ubuntu LTS with a updated image pushed annually), we can provide a consistent Python setup (e.g. Python 3.10 and a fixed set of C-based modules like psycopg2). However, our staff and PhD desktops and laptops are more diverse - with the OS often only being upgraded when the distro is going out of support, they could be running n, n-1 or n-2. That, most likely, means three different Python versions.
We could use pyenv to let people install their own preferred version. Installing with pyenv requires building from source (slow on some of our oldest machines). This also means installing the Python build deps, which is fine for our departmental machines but not possible on the HPC cluster (operated by a different business unit) or the Physics shared servers. It's also less than ideal for our servers where students deploy their projects (where we want to minimise the packages installed, like the build-essentials meta package).
It's also a massive stumbling block for less experienced students with their own laptops which could be running any distro, of any age. Many CS101 or engineering/business/humanities students taking a programming class, who have never programmed before, would really struggle.
So, classes might tend towards teaching lowest common denominator Python (i.e. the oldest conceivable version a student might have installed on their machine).
Sure, we have in-person and remote lab machines students can use - but it's not always convenient (especially for the data science / ML students running Jupyter notebooks with their own GPU).
There are workarounds, but they all have serious downsides.
Compared with Node.js and Go, where users can just download the appropriate package and unzip/untar the runtime or compiler version of their choice, deploying the Python runtime has enormous friction (especially for less experienced users). This has the bonus of simplifying deployments elsewhere in our infrastructure (CI/CD, containers, etc).
And while we all complain about node_modules, Python venvs not being trivially relocatable is another huge frustration.
We've used Anaconda, but that comes with its own issues (most recently, finding that it ships with its own gio/gvfs binaries and libraries which fail to mount our CIFS DFS shares - causing confusion for users running `gio mount` from within a conda environment).
aldanor
Unfortunately or not, in some fields conda is the only sane choice because it can manage non-Python binary dependencies that Python packages may depend on. Some of those dependencies may be huge C libraries that are a pain to build, like HDF5, so if you're not using conda you'll be relying on your OS's package manager to serve your particular venv's needs - we all know what usually happens next.
zackees
[dead]
doublepg23
I think I mostly wrapped my head around pyenv and used Anaconda the other day. It was quite the pain, to setup and then it seemingly mangled my fish and bash configs causing a noticeable delay every start up. Not something I was hoping for just for hacking around on some AI project.
Disclaimer: I was using Fedora which has Python 3.11, using Fish which is clearly non-standard and I’m a sysadmin not a Python dev.
nightfly
You don't have to activate venvs, you can just refer to the paths to pip and python inside of them as needed
ElectricalUnion
Second this, just run python/whatever-binary-you-need inside a venv:
`${my-venv-path}/bin/python`
`${my-venv-path}/bin/${whatever-binary-you-need}`
`%my-venv-path%\Scripts\%whatever-binary-you-need%` (because Windows...)
phendrenad2
The problem is, dependency management isn't a solved problem. I was nodding my head in agreement at the part where the author mentions that pyenv compiles Python from source when it installs. If you try to figure out the reason behind this, it becomes obvious that the only one that makes any sense at all is that distributing binaries, in a secure way, is hard for open-source projects to manage. After all, bandwidth costs money, someone has to pay for the build server, someone has to pay for the data transfer bandwidth. And, do you trust that person to not be a rogue actor? It's easier to pull down the source, on the end user's machine, cross your fingers for good luck, and compile it.
We also haven't figured out how to incentivize maintaining backward-compatibility. 99.9% of the time when some library updates, and stops working with language version X, it's using some hot new feature of the language. Usually just because the library author thought it would be cool or more elegant. The entire software world needs to update now, because someone left-padded us for elegance.
kjkjadksj
These comments in this thread were a bit surprising to me. People really like to make things hard on themselves not doing a little do diligence. Yes, I still say to simply use conda. It spells out exactly what is getting installed in the environment, and uses a separate python installation for each env than the system. If you don’t trust it just type which python. I never get these headaches people seem to have, since conda is easy and well documented and supported.
DemocracyFTW2
IMO to fix these issues the first thing to do is to write an alternative to Python's `import`, more like a function call that works more similar to NodeJS' `require()`. That should start life in userland and only later become part of the language.
How to transition a bazillion packages though I do not know.
BiteCode_dev
I have an article planned solely on imports.
It saddens me to say, but imports are indeed a complicated topic on Python. The PYTHONPATH is not something people know about, and once they do, it's not intuitive.
The weird handling of namespaces or relative paths doesn't help.
Havoc
The variety of solutions sure isn't helping.
I'm just sticking all dev projects into a separate LXC and calling it a day. Don't want to deal with all the various separation models the various languages and package managers cooked up
tipsytoad
Aside: I usually use direnv to activate the venv (or poetry) when entering a dictionary
https://gist.github.com/tom-pollak/8326cb9b9989e0326f0d2e19f...
trey-jones
> You should really use docker
>> I think you missed the point.
Maybe I did, but I've been using Docker as version management for pretty much every technology I employ for five or six years. Prior to that I sparsely used things like rbenv and virtualenv and I actually thought it was super dangerous and unreliable. Maybe it's gotten better in recent years, and certainly people who write python and ruby every day are going to know more about this than I do.
I don't install anything on my computer if I can just use Docker for it. OK, I do have go:latest, but I use docker images for various projects that might be on any version of go from 1.8 to 1.20. Your website still runs on PHP5.3? I can help you (I won't, but I could totally run it locally!).
Reasons I like docker better:
1. Any scripts or configs can explicitly refer to the version number. No guessing or assuming.
2. Our whole team uses the same version.
3. Only one dependency: docker.
Granted I'm more of a sysadmin than a developer and I'm sure that biases apply.
whstl
Suggesting Docker in the context of Python dependencies is still missing the point, because you still need proper dependency management in case you have to rebuild your image for some reason.
If you have a Docker image that builds with "pip install whatever" and this library gets updated with a breaking change, you won't be able to rebuild the image without changing the dependency or the code itself, for example.
BiteCode_dev
Author here.
I'm late to the party but AMA :)
vqbd
There's a version of pyenv for windows called pyenv-win. The cli installs as `pyenv` and the commands are the same. So not game over imo.
Also, [even if it did], it installs binaries and doesn't compile.
BiteCode_dev
You are not telling them to use pyenv, you are telling them to use a totally different tool that happen to have a similar name and provide a similar interface.
You have already lost.
oconnor663
How do you see this advice changing in 3-5 years?
BiteCode_dev
This procedure was valid 5 years ago, and all the alternatives have changed a lot in the same time period.
So if I had to chose one way for the next decade, I would bet on this.
But I will not pretend we can be sure about anything on this matter.
In fact, my hope is that we can improve the process tremendously by creating an unified way of bootstrapping Python for all OS. If this ever happens, the procedure would become mostly obsolete.
At the end of the week I will write about this, and the various experiments the community have done so far in that regard.
INTPenis
I don't use any of those, I use direnv with standard python3 -m venv module.
zajio1am
Perhaps people should accept that these are more developer tools (like git) and not end-user distribution tools and just use regular distribution packages (rpm / deb) for that.
CMCDragonkai
I dealt with this years ago by only using Nix to do python Dev. Worked great for the entire ML stack. Had to do a bunch of packaging for nixpkgs early on though.
akasakahakada
Any of these is complex as hell. I wish everything is as simple as downloading exe file on my Windows Desktop and double click, done.
pmarreck
"simply" use Nix, which solves this problem for every language.
n8henrie
I wish this was a "simply" answer! Been getting more into nix lately and really enjoying it, but converting python to nix has been a challenge. It seems like the existing and recommended suggestions are either deprecated / unmaintained or still alpha at best (pip2nix, mach-nix, dream2nix). https://discourse.nixos.org/t/basic-flake-run-existing-pytho...
pmarreck
Well one way of going about it is a hybrid solution. So you set up your python dev directory with a nix file, but then you use pyenv or whatever to encapsulate everything else inside it using python tooling. So it's not strictly Nix-all-the-way-down, but at least as far as being in the directory is concerned, the "right" python version and libraries are available.
not_enoch_wise
One does not simply tell people to use pyenv, poetry or anaconda...
bayesian_horse
In my personal experience, I'll take Python's packaging hell over nuget or npm any day.
And it's often less about the package manager and more about the ecosystem: Can you find what you need? Is what you need stable enough? Does it break every few months with new versions of the runtime, either because of actual incompatibility (npm/node) or versioning shenanigans (nuget)?
nonethewiser
> Is what you need stable enough? Does it break every few months with new versions of the runtime, either because of actual incompatibility (npm/node) or versioning shenanigans
But these are strictly package level issues, not package management.
The downside of node is the over reliance on flaky dependencies. The actual package management is good, especially compared to python.
bayesian_horse
I don't agree. There are multiple package managers for Python which have at least comparable feature sets with npm and yarn. And in the end, pip and conda are biting me a lot less often than npm and yarn.
It's not just "flakiness" of the package maintainers. The basic problem is that there is no single standard javascript package, and no package manager, especially npm can solve this.
inferiorhuman
Let's back up a moment. npm at least works on *BSD. Anaconda is just as toxic as Electron in that regard.
bayesian_horse
Given the target audience of Anaconda, I can see how BSD compatibility doesn't matter to them. I guess it would take some major lifting because much of what they do is wrangling the compiling environment. Supporting Windows is hard enough.
That's not "being toxic", you just can't be everything to all people.
inferiorhuman
Nah it's toxic. It's not just that Anaconda won't provide binaries, it's that you can't even build a project that uses Anaconda on not-Linux/Windows. In terms of why you shouldn't tell someone to simply use Anaconda, that's pretty high up there.
As tedious as javascript package management can be, python has consistently given me more trouble.
bayesian_horse
I think that's an "Am I the Asshole" situation, and you are getting it wrong. BSD is niche, especially in scientific computing.
Saying anaconda is worthless because its maintainers don't expend a ton of effort on making your particular niche more convenient is the actual toxicity...
inferiorhuman
Saying anaconda is worthless because its maintainers
don't expend a ton of effort on making your particular
niche more convenient is the actual toxicity...
It's a ridiculous non-solution given that even in the data science context most stuff is POSIX compatible. If I can get R and Python on their own to work without hassle, there's no reason a mere package manager should throw up unnecessary walls.
bayesian_horse
I think this discussion is pointless. No "project" depends on conda/anaconda, because a package manager, in Python, is a means of deployment at most. If indeed you have some project that comes with a conda environment description (or dockerfile or whatever) then all you need to do to make it run on BSD is figure out a way that works on BSD. Of which there are certainly many. It's really just that conda environments are pretty damn convenient for Windows and Linux in the context of numerical libraries.
sacrosancty
[dead]
Are there any efforts akin to deno for python? A “burn all the packaging down and start over” path?
It’s so thoroughly broken, every day a dev on some team gets their poetry env entangled with some system installed python, or numpy suddenly decides all the CI builds will now compile it from scratch on every build, or.. Today it was segfaults on poetry version X on the M1 Mac’s, that went away in version Y but of course version Y broke pandas for the windows devs..