Compromised LiteLLM 1.107.0 on PyPI Installs a Credential Stealer and Backdoor

595 points
1/21/1970
4 days ago
by abhisek

Comments


crossroadsguy

All the sandbox-exec and all the Lulu rules are not going to protect against something like this.

At one point permission pop-up fatigue sets in we start giving any permission as if it's normal. We also become desensitised to isntalling tools: brew, npm, it becomes second nature.

"Oh, those 367 tools installed by package X? Yeah yeah, must be normal, that's how all these work now".

"Brew shows 59 packages outdated?". "Must upgrade". "Yeah, done. Perfect".

And the advent of agents and harnesses. It's not a matter of whether, it's when! OSes need to do something about it. Help users have complete separate "program space" at ease with better, easier, and lighter VMs if nothing else. I know, I know - Unix and Linux solved all this even before Alexander III of Macedon started wearing his nappies. Of course.

4 days ago

isityettime

> All the sandbox-exec and all the Lulu rules are not going to protect against something like this.

No, you can block this with sandboxing tools. For my LLM agents, I use a tool called Nono, whose macOS implementation is built on the same foundation as sandbox-exec. One of the key features it supports (and it's not the only tool that does this) is proxying and the injection of credentials via the proxy. For example, when I run Codex at work, Codex does not have access to my Azure AI Foundry API credentials. It sends out requests via dummy credentials, and the proxy injects the real ones outside the sandbox.

The other thing that proxy does is let you restrict network access in various ways. So you can run something like LiteLLM so that it doesn't have access to the internet at large; it can only talk to Azure AI Foundry or AWS Bedrock or whoever your inference provider is. You sandbox LiteLLM separately from your agents, and you give it access to much less. The software firewall stuff isn't perfect but it is a significant defense against things like this.

There are also lots of other restrictions that are easy to do which mitigate this. Nono blocks access to the root of $HOME, then selectively allows access to paths within it, and also explicitly blocks access to common paths to secrets. So it's pretty easy to make sure that access to important things are blocked: SSH credentials, GnuPG socket (and local Unix sockets in general), local OS keyrings, etc.

The way I use tools like this is exclusively by writing policy, testing it, and revising where the policy is inadequate. I never get approval fatigue because I don't do approvals.

You're right that attacks like this are hard to defend against, and that we're used to tool sprawl, and also that LLMs can do things like install tools we don't expect them to themselves! But there are ways to anticipate and mitigate attacks like this. Sandboxing tools (including contained and VMs) that support credential injection are genuinely useful for this.

4 days ago

drdaeman

Until this Nono tool gets a supply chain attack and happily ships all your Azure credentials somewhere they aren’t supposed to be, then starts mining Monero or something.

LiteLLM is frequently used in roughly the same scenario - trading virtual LAN-only keys for real ones as it relays requests to the inference providers.

4 days ago

isityettime

> LiteLLM is frequently used in roughly the same scenario - trading virtual LAN-only keys for real ones as it relays requests to the inference providers.

LiteLLM isn't a sandboxing tool so it just can't perform as much risk reduction even when everything is working as intended.

And what nono adds over sandbox-exec is a uniform interface across platforms and some policy management tools. You can impose the same restrictions via the OS built-in mechanisms yourself via Seatbelt or a built-in interface like sandbox-exec. You can also combine an OS-level sandboxing tool like nono, zerobox, Anthropic's srt, Seatbelt, landrun, Bubblewrap, Fence, etc., with a VM-based one like Docker's sbx, smolvm, krunvm, microsandbox, or Shuru. So now you need two simultaneous supply chain attacks in unrelated projects to line up just right at the same time.

"Eventually you have to trust some part of your security stack" is neither particularly interesting nor evidence that there's no way to defend against supply chain attacks. Even if you only have one sandboxing tool and it's OS-level rather than VM-based, that becomes a single part of your stack that you can give special attention to in evaluating updates in a way that LiteLLM can't be.

There's also just the empirical fact that at this point, trusting LiteLLM's maintainers to lock down their release pipelines looks foolhardy. This is the second supply chain attack they've had in 7 months. It's absolutely wiser to choose a sandboxing tool you like, audit it to examine its release pipelines, code signing requirements, attestation strategy, and whatever else, then put LiteLLM inside that tool's sandbox than to just thow your hands up and say "I gotta trust something, so it might as well be LiteLLM!". Maybe nono ain't it either, idk. I'm still experimenting to figure out how I want my setup to work.

But I think neither the defeatism nor the relativism in this thread are very compelling.

4 days ago

fdsjgfklsfd

No no it's really important that I have 5-factor authentication on every website and it requires me to go through the whole login process again every time I break eye contact.

4 days ago

Terr_

And if it ever detects that you briefly used a browser with any kind of privacy-preserving features on any website, you will be logged out from all devices and forced to revalidate your email and gene sequences.

Discord does this with Firefox: It doesn't even tell you that a captcha failed or anything, just poof logged out and you must re reverify the same email you verified 10 minutes ago.

4 days ago

smolder

Discord should be illegal and we should use IRC. It did the same thing better without centralized logging.

3 days ago

smolder

Downvote me if you want but I am dead serious. Attaching profit motives to basic infrastructure is why the USA is a dystopia today. Protocols really are better than platforms. IRC was a better system than discord.

We have been a petri dish for insane megalomaniacs for too long, and now freedom is coming back to bite us, because it gave freedom to the worst imaginable people to oppress the rest of us.

3 days ago

orangedog

IRC didn't and hasn't meaningfully evolved the features Discord has, which is why people use Discord.

You're getting downvoted probably because it isn't a serious suggestion. IRC is missing features and you don't have a meaningful reason to make it illegal.

For the record, I use IRC.

3 days ago

lioeters

> Attaching profit motives to basic infrastructure

Water, food, housing, healthcare, telecommunications, computing (including artificial intelligence). I guess it's "socialist" to believe that these baseline needs should be, and can be, free for all of humanity.

One day we'll look back and laugh at the stupidity of our current state of civilization, that we'd need to pay anybody anything to live on this planet that provides all those things and more for free. Pity us, future people, I hope we grow up soon.

3 days ago

craftkiller

> Water, food, housing, healthcare, telecommunications, computing

> planet that provides all those things and more for free

This planet does not provide healthcare, telecommunications, or computing for free (unless you consider nature's "the sick get eaten" approach "healthcare"). Housing is only free if you either don't value the labor or you are fine living in caves. Water and food could be free if you're only interested in drinking from rivers and foraging for fruit, otherwise we're back to whether or not you value the labor in farming or building the infrastructure for water collection and transport.

(for the record, I think those things should be all funded by taxes. I just think it is silly to claim the planet provides those things for free. We created those things, through millennia of toil.)

3 days ago

Der_Einzige

100000% agree. Discord is one of the most disgusting pieces of software in human history.

3 days ago

sznio

and yet somehow an attacker never needs all these auth methods.

my boyfriend didn't have 2fa enabled, but google would still ask him to verify a code on his phone. recently his password leaked, the hackers got into the account and google never bothered them with an extra verification. amazing. the fact that they signed in from a completely different country? who cares!

(yes i enabled 2fa for him now)

3 days ago

hoppp

Ready to add anal probes?

4 days ago

tensegrist

for what, chess.com?

4 days ago

NonDairyMatt

Yes! I'll be damned if someone improves my elo

4 days ago

supaflybanzai

Hey! I spat toothpaste all over the floor!

4 days ago

yturijea

This is seriously one of the reasons I see Rust as quite suspicious(with its 200-300 dependencies in all builds), unlike golang where you can actually create a full app with no dependencies.

3 days ago

smolder

There are not that many dependencies in Rust if you know how to write it. That's just what vibe coders end up with. It's a great language which is overtaken by people with no idea what they're doing that heard it was a great language.

3 days ago

yturijea

most modern linux toolings have 200-300 dependencies.. and some of these are before the whole AI thing, so not just vibe coders. The problem with rust is that is doesn't ship with any basic functionality, like crypto, or json encoding, or similar. So you end up having to depend a lot, because what should be part of the standard library simply isn't there. Hence my reference to golang which has all those. Of course rust has other benefits, but that one is a huge issue, for the same reason we see so many issues with js and python related to security attacks through pipeline.

2 days ago

decentstates

Sandboxing should be built in to your windows manager like: https://decentstat.es/posts/shai-halud-nix-housing/

3 days ago

happyPersonR

This software should be rewritten until it has no dependencies other than the compiler. Like it should use no libraries at all other than crypto.

Also put it in rust ….

2 days ago

abhisek

What are you talking about? Even dependency cooldown of 24h is enough to protect the consumers from these breaches. The attack did not have any novelty, used a newly registered look-alike domain for C2 and a PTH technique which is now known.

Sandbox comes in for the unknown ones and yes, its a pain to get the profile right.

4 days ago

lrvick

> Even dependency cooldown of 24h is enough to protect the consumers from these breaches.

From some of them perhaps, but plenty hide for -years- undetected.

4 days ago

woodruffw

> From some of them perhaps, but plenty hide for -years- undetected.

Which ones? Not even xz-utils was years (it was weeks), and it’s the poster child for a attacker doing everything right until the very end.

3 days ago

lrvick

Some top of mind examples that -could- have been intentional like Heartbleed, Milksad, the or the terraform entropy flaw I found all come to mind. Bugdoors like those are great because they are subtle, can hide for years, and two of those persist even after patching. All are also things someone can just say were an accident and we kind of have to believe them.

I am aware of at least one evil firm that finds bugs like these months and sometimes years before they are public and sells the knowledge to spooks until someone else independently spots them and tells the public.

Only a matter of time before these types of firms introduce the bugs themselves given the incentives, if they are not already.

3 days ago

woodruffw

“Could” is doing a lot of lifting here. Like `d=4`, the parsimonious explanations are honest mistakes, incompetence, and institutional error, not intentional bug placement.

3 days ago

Eisenstein

When they catch a shipment of drugs being smuggled across the border, do they say 'we got them all'? You have to assume for every number X that you intercept, you miss X * Y.

3 days ago

woodruffw

I think drug smuggling and open source package compromise have pretty different failure modes, which explains differences in perpetrator behavior.

(This is also different from what the GP argues.)

3 days ago

Eisenstein

I think the concept is exactly the same. When you detect something, that means you have found the ones that made a mistake or got unlucky enough to get caught. You must assume that some haven't.

Not every malicious actor is a financially driven crypto-miner or credential stealer. It seems likely that some exploits are going to be sat on or used in a highly targeted manner such that we may never know they exist.

3 days ago

woodruffw

I don’t think the actor’s persona is the most relevant thing here, although I do think that more sophisticated attackers generally don’t prefer open source supply chain attacks because it doesn’t have the right visibility-to-impact ratio.

But again: it’s impossible to argue against a “silent majority” claim. The argument passes the sniff test for drug smuggling because we have separate evidence that most drugs are not interdicted. We have no such separate evidence for open source supply chain compromises.

3 days ago

[deleted]
3 days ago

[deleted]
3 days ago

nvme0n1p1

Don't let perfect be the enemy of good

4 days ago

lrvick

It would be like having a one day quarantine for a deadly and rapidly spreading disease known to have a very long asymptomatic period.

I mean okay, it probably helps in some random cases, but the disease will not care at all. Easy to route around and we just wasted a lot of developer energy treating symptoms and ignoring the actual problems.

People are running around acting like cooldowns fix supply chain attacks. This is completely absurd. Imagine an antivirus that just makes you wait one day to install any new software or updates. Should we do that too? We should question what problem are we actually solving before this type of thinking gets even more out of hand.

What we need is actually reviewing all heavily used OSS code with as many eyes and agents as possible in a coordinated and decentralized way, and then federated reproducible signed releases. If we are going to have a one day cooldown, we must do something actually useful with it or people will see it does no good and stop doing it. Without a review plan, a day might as well be a year.

Like literally fund this with taxes or something like caring for public roads. Add it to our internet bill like a utility safety fee.

4 days ago

brookst

You’re imagining one single threat. Which isn’t the case.

It would be like having a one-day quarantine against 500k viruses, which have a median asymptomatic time of 3 hours and stdev of 1 hour: sure, statistically some get by you. But it’s dramatically better than insisting on no protection except perfect protection.

3 days ago

[deleted]
3 days ago

anonreplier

Yes, that’s not going to happen. At best you’ll get curated commercial repos.

4 days ago

atmosx

Yes it’s messed up. I was thinking about pi agent and the extensions getting updates every three hours.

4 days ago

cptmurphy

Fuchsia solved that

4 days ago

caaqil

> Unix and Linux solved all this even before Alexander III of Macedon started wearing his nappies

I mean, didn't Linux shells invent pop-up fatigue? I thought we (yes, we) all used to -y flag preemptively so the upgrades and installations/setups go through. Linux really did do it first, the agents are just more needy and their popups more verbose now, but not exactly new.

VM isn't a silver bullet.

4 days ago

fc417fc802

Stuff like this is highly context dependent. It is a good thing that apt asks about each step. I voluntarily add more of this by opting for things like listbugs. If I then decide to override such protections with a -y for whatever reason that's a choice on my part to prioritize convenience.

Either way it's got nothing to do with security - I was never going to catch a vulnerability in the central debian repos by reading a list of dependent packages.

4 days ago

isityettime

It's kind of the opposite. The more typical Unix nehavior is to just assume you knew what you were doing and meant it, and not ask for confirmation in the first place. -y is just the reversion of an abberation.

4 days ago

linsomniac

Just going to remind you all that https://canarytokens.org/ exists.

4 days ago

radku

Canary token is great to tell you've been hacked.

There are ways to try to prevent this from happening in a first place. I wrote kow (keys-on-the-wire) to fix this. Apps only see placeholder tokens. The proxy injects real key from the vault into outbound requests for allowed hosts.

Author here. Apache-2.0. [1] https://keysonthewire.com [2] https://github.com/inflightsec/keys-on-the-wire

3 days ago

TacticalCoder

Yeah it's not totally unlike people who consider learning that they've been hacked is worth more than $x and so, even if they're not into cryptocurrencies, they leave the equivalent of $x on their computer in a "software wallet", in an obvious place.

It's not a guaranteed you'll know you've been hacked, but it's still quite a flag to leave. And there are services doing the equivalent of what you posted, but for cryptocurrencies: they contact you through email, SMS, snail mail even, if they detect that "coins at address ..." ever move.

4 days ago

dropbox_miner

TIL! this is cool

4 days ago

myzek

This is awesome, why did I just learn about this! Gonna add these all over my homelab

3 days ago

rsanheim

This is the second significant exploit against LiteLLM in what, a year or two ? Just last night I came across notes and a script from an older claude session to help audit all project dependencies and verify nothing used litellm.

Okay, checked, it was March of this year: https://futuresearch.ai/blog/litellm-pypi-supply-chain-attac.... What even is time anymore in the llm age?

4 days ago

halJordan

And buried under the lede is that one of the compromised people was one of the organization owners. So it seems like the culture of insecurity comes from the top, which is unfortunate especially bc they keep acting like "this could've happened to anyone guys"

4 days ago

brookst

Yeah the excuse that it wasn’t a project vuln, just the project maintainers having a machine taken over… that’s not great.

3 days ago

sebdufbeau

there's another one this summer that didn't get as much attention (or any at all) which allowed exfiltrating master provider keys with any authenticated user/virtual key

https://nvd.nist.gov/vuln/detail/cve-2026-84377

4 days ago

te_chris

I looked at the code a year ago and decided I will always avoid it.

3 days ago

atmosx

Indeed, I don't think I'm upgrading ever again. I'll stuck with 1.04.2 for a year or two. Feels like a Russian rulette at this point.

4 days ago

detente18

Hi atmosx - i would recommend upgrading.

We work hard on improving the stability and security of LiteLLM on each release.

Please let me (krrish@berri.ai) know if you have any issues upgrading!

4 days ago

atmosx

Will absolutely do. Sorry for my remarks, I wasn’t thinking straight. I appreciate your work.

Will upgrade ASAP. Thanks for the amazing product and best of luck with everything.

2 days ago

prepend

Isn't the issue with upgrading the insecurity of litellm and its inability to prevent being pwned?

How did you address this issue?

3 days ago

staticman2

If you stick with older LiteLLM versions you won't get proper support for new LLM releases, correct?

3 days ago

pkulak

How could this happen to a project with 55k commits in three years?

4 days ago

Aurornis

The compromise appears somewhere in the build and deploy system.

> No public commit in BerriAI/litellm contains data = "cook" or a cook/ folder.

> SafeDep assesses, with moderate to high confidence, that BerriAI’s private release pipeline built these wheels from an attacker-modified git source.

4 days ago

crossroadsguy

Every harness thinking right now… "if we want do something like this we won't even have to compromise anything."

4 days ago

zargon

Sarcasm?

4 days ago

pkulak

haha, yes, sorry.

4 days ago

rirze

you needed more exclamation marks :)

4 days ago

arcanemachiner

The gentleman's /s tag.

4 days ago

dofm

Or say "That's unpossible!" so we know for sure.

3 days ago

cromka

I mean, 55k commits in 3 years, that's about 55 a day, 7 days workweek. It's reasonably implied no one can be looking at all them thoroughly.

4 days ago

yujonglee

This incident is unrelated to what was committed to the repository. More info: https://docs.litellm.ai/blog/security-incident-oct-21-2026

4 days ago

threecheese

"Attackers compromised 3 unique developer github credentials" is a little scary; this seems systemic - if three developers had their keys stolen, it implies the real problem is elsewhere.

They revoked all but one developers, which is probably the right approach, but I'd pin LiteLLM to "something before this version" and keep it there until further notice!

4 days ago

detente18

Hi threecheese,

Pinning is a good idea.

So far, we believe it came from the github oauth cli tokens which are embedded in claude code tool responses - https://docs.litellm.ai/blog/security-incident-oct-21-2026#g...

It's something we're still investigating. I'll update my thread on this post, as we learn more!

3 days ago

madamelic

I am not following how someone snuck the files into the release.

All I read was that it creates a directory from the build file but seemingly no description how new files or added dependencies slipped by. Was `cook.pth` a dependency that got automatically downloaded from PyPI on install and then the build system put it into dependencies?

--

Leaving this for others:

> BerriAI’s private release pipeline built these wheels from an attacker-modified git source. The attacker could have used a compromised maintainer or bot credential, a private branch or mirror, or the reference the pipeline builds from. A stolen PyPI token used from the attacker’s own machines fits the evidence less well.

Makes sense. It was that a maintainer or the private pipeline was attacked and the malicious dependency was injected there by squatting on the maintainer's local copy rather than the dependency being hijacked for everyone. Very clever.

4 days ago

zahlman

> Was `cook.pth` a dependency that got automatically downloaded from PyPI on install and then the build system put it into dependencies?

This isn't the name of an entire package, but just a single file included in the package. .pth files are part of Python's import system. At startup, the standard library `site` runs automatically, and configures the (initial) places that Python will look for code to satisfy `import` statements. By design (because some people occasionally find it very useful), arbitrary Python code can be run from these[0]. The system searches the entire site-packages folder for these files.

The compromised build packages up a .pth file in such a way that installers will treat it as a "data file"; but because Python programmers are accustomed to looking for data files at paths relative to the current source file, installers put .pth files in places that will easily be picked up by the `site` scan.

So the point is simply that, once LiteLLM is installed, malware injected directly into the final package (not a dependency) runs at startup in the compromised environment even if you don't use the LiteLLM code explicitly.

Nothing "creates a directory from the build file". The attacker-modified source includes the entire "cook" payload directory and also modifies the build file so that the build system would include its contents. No dependencies are involved (unless you're thinking of LiteLLM as being a dependency for something else).

The Python ecosystem has been working on mitigations for this. Starting in 3.15 the .pth file system is being reworked[1], but a deprecation period is necessary. Also, the PyPI repository has an opt-in system called "Trusted Publishing"[2] which is supposed to allow verifying that a wheel was built from the appropriate source directly using GitHub's CI mechanism. (Opt-in because of the long tail of developers who don't use this or any other CI system and shouldn't be expected to learn them for a small hobby project.) However, since the attacker did use the CI system from a separate GitHub repository, and compromised multiple maintainer accounts, I imagine it was possible to defeat that protection as well.

If you don't need the `site` mechanism for your project (of course, this needs to be tested per project), it can be disabled by passing the -S flag to Python[3]. This has always been available, but awareness seems to be low.

Historically, Python packaging has overall had a pretty lax security posture, and new attempts to improve that are often delayed by the consensus process and by backward compatibility concerns.

[0]: It's possible to put arbitrary code directly in the file, but rather unwieldy. Interestingly, this payload is neatly organized: the .pth file simply runs an import statement and appears to keep hundreds of lines of code in two actual Python files: one that works directly, and one that sets up a sub-hook called `sitecustomize` (another explicit provision of the `site` configuration system).

[1]: https://docs.python.org/3/whatsnew/3.15.html#whatsnew315-sta...

[2]: https://docs.pypi.org/trusted-publishers/

[3]: https://docs.python.org/3/using/cmdline.html#cmdoption-S

3 days ago

aakashpawar

If that's right then attestations wouldn't have saved anyone here either. They prove which workflow published the wheel, not what source that workflow checked out. A private pipeline building from the same-looking repo still gives you a perfectly valid attestation.

What would catch it is someone rebuilding from the public tag and comparing the result, which almost nobody does for the python wheels. I have been treating attestations as the answer for my own project's releases, so this is a good reminder of where they stop.

4 days ago

mertcansaglamsh

[flagged]

2 hours ago

heyvaldemar

[flagged]

3 days ago

vibewright

[flagged]

4 days ago

detente18

Hi everyone, this has been contained. The attack was novel, and I'd love feedback on how we can improve from this.

Here's what we know so far: https://docs.litellm.ai/blog/security-incident-oct-21-2026#h...

- Krrish

4 days ago

fny

It's not a novel attack: it's novel negligence.

The best improvement would be to blame yourselves instead of Claude.

3 days ago

detente18

Hi fny,

That's fair feedback. What I'm trying to express is the complexity of it.

For this to work:

1. The attacker had to get access to our internal gateway

2. Claude code had to embed the github oauth cli token in tool call responses (not directly, but as part of an embedded url call).

3. We had to enable request/response logging.

4. The attacker had to be aware of this token logging circumstance, then harvest multiple user account tokens (one of which had to belong to an org admin)

5. The attacker had to wait for / hope that we would expand the release trust boundary (as we did on Monday)

among other things. This wasn't a "whoops we had shitty governance around our auth tokens" kind of situation, it involved a number of non-trivial circumstances many of which were beyond the attacker's direct control, including Claude Code behavior of which we were not previously aware of, and a sophisticated attacker.

- Krrish

3 days ago

eddythompson80

FYI - accidentally logging secrets is a fairly common thing that happens. Request/response bodies are fairly common place that would include secrets or PII. but in general you’d be surprised what other actually innocent places that end up accidentally including secrets. Having credential scanners on the logs is a way to catch these early.

As for sophisticated attackers, I think that phrase should probably be retired at this point. Every attacker is a fairly sophisticated now since you can automate that sophistication. Attackers before used to have a set of tools that would try to connect or execute a bunch of automated prescripted checks, while any attacker now can automate the generation of those tools and scripts on demand

3 days ago

joshheitzman

> specific situations in which Claude Code will embed a user's GitHub CLI OAuth token into a response body, and in situations where response bodies are logged, these tokens may appear in LiteLLM logs.

That's pretty hand wavy. What did y'all have CC doing that it had your GitHub CLI OAuth token in its context at all?

Stuff like this is why my coding agent is a coding agent not shell agent that basically own the machine. Mine has no tool for shell use narrow tools specific things like ripgrep, finding files, and deleting files that limit the paths to explicit workspace directory.

I take care of commits, push/pull, building wheels, and publishing wheels myself. It really doesn't take the much time and the coding agent frees up a lot of my time.

3 days ago

altbdoor

> I take care of commits, push/pull, building wheels, and publishing wheels myself.

I too was surprised when I see my colleagues telling me that these actions are "not maximizing your efficiency".

They'd rather hand all controls to the LLM, and then be surprised when shit hits the fan.

3 days ago

joshheitzman

Yeah, I don't get either. I look at source control as the first line of defense for when agents screw up to badly. I'm not going let the agent mess with the easiest way to undo its screw ups.

3 days ago

craftkiller

> narrow tools specific things like ripgrep

Ripgrep has the `--pre` flag and `--hostname-bin` flag that enables it to execute arbitrary commands.

> finding files, [...]

Does that tool invoke the `find` command? Because `find` has the `-exec` flag which could then be used to invoke arbitrary commands.

I really think the only safe option is putting your agent in a virtual machine with only a copy of your code (sans credentials), let the agent work, and then copy it back and manually audit it.

3 days ago

joshheitzman

The implementation of the tools don't execute processes (i.e. the implementations aren't wrappers for shell commands).

21 hours ago

neop1x

What about adding a human in the loop? So the actual releases to the package registry would be done manually after reviewing the package?

2 days ago

wisty

Original articles says it was likely a comprimised maintainer machine?

4 days ago

detente18

From our investigation, the token compromised was included by claude code in the tool call.

This was captured in the request/response logs.

The attacker, having gotten access to the logs, then looked for this and used it.

So - yes a local github cli oauth token, but not the machine.

4 days ago

franktankbank

Amateur hour again.

3 days ago

stbenjam

IMHO defaults for uv, npm, etc should all have a cooldown for 24 - 48 hours by default, you should have to go out of your way to install something newer.

4 days ago

robinhoodexe

We have 7d cooldown on packages on an internal pypi proxy but that blocks security updates also, which we now have to manually exclude from the cooldown.

I’m unsure how to automate this properly, maybe an LLM is the best starting point.

4 days ago

mbrumlow

Are security updates immune ?

4 days ago

tikkabhuna

They definitely wouldn't be if attackers knew that it would then bypass cooldowns!

Ultimately, you need someone to verify releases and that probably means paying for it. I know Sonatype paid solutions utilise their own internal research to allow/block new releases.

4 days ago

fmbb

It’s in the name. They are secure!

4 days ago

dlkasajiewo

No, but there's a bigger tradeoff for not upgrading. In one case it's that you're potentially exposed to a known vulnerability, vs the other case where you might just be missing out on some new or slightly reworked feature.

4 days ago

mattstir

But then an attacker marks their malicious release as a critical security update. It's really messy unfortunately.

4 days ago

happyPersonR

We need something external that actually checks to see if source code matches built artifacts, and also possibly something Llm based that checks to see if you’re making checked in code vulnerable or directly adding something malicious.

4 days ago

bramhaag

Ask your LLM about "reproducible builds".

3 days ago

happyPersonR

Yeah those exist… the trouble being we need something that doesn’t exist

There needs to be a service that does reproducible builds of opensource projects as they sit and publishes results for people to scrape freely.

To be robust itself against attacks, likely 3-4 services.

But then that needs funding to be sustainable … and companies are incredibly cheap. Especially right now. As such… shrug :(

Maybe something as a skill for LLMs to run that actually checks build reproduction from source for a project and all of its dependencies? But then that’s gonna burn some tokens ….

I do this for personal projects … but at work they’ve started rationing tokens… as such… idk?

2 days ago

trollbridge

Problem is, we leave ourselves open for LLM injection attacks.

4 days ago

freakynit

[flagged]

3 days ago

sunbum

I use safechain for this, because there is always gonna be another package manager configured wrong. https://github.com/AikidoSec/safe-chain

4 days ago

mstaoru

It's just next level:

"Installing the Aikido Safe Chain is easy with the installation script.

curl -fsSL https://github.com/AikidoSec/safe-chain/releases/download/1.... -o /tmp/install-safe-chain.sh \ && echo "9e4af7107f06d9c0d758cb08fb36804c2af88e24972f3a9d3f8c2bd0ca002adf /tmp/install-safe-chain.sh" | sha256sum -c - \ && sh /tmp/install-safe-chain.sh \ && rm /tmp/install-safe-chain.sh "

(facepalm)

3 days ago

rcleveng

I didn't realize they've all such added that, this was a change we made to our own projects last year when these first started - glad to see it's common.

Risk is ensuring enough security researchers are finding these and quarantining these before that default cooldown lapses.

4 days ago

stbenjam

Yea, I think all the tools have this now but its not a default. Install scripts should also be turned off by default and require some hassle to turn them on for the first time in a package.

4 days ago

0c3ca83

The last time I looked for information, I remember seeing that the median supply chain attack was in place for multiple months to years before being discovered. Cooldowns shorter than that seem like snake oil.

4 days ago

whatshisface

If we all implement a cooldown, nobody will find the attack. This implies the cooldown length must be the cooldown length plus one month.

4 days ago

stratos123

No, because there are malware-finding mechanisms other than "let some unlucky guy get affected first", like automated scanners. If everyone implements a cooldown, everyone will in fact be safer.

4 days ago

comboy

Plus some random value slow curve distribution. Basically incremental rollout from the client side. But if it's not caught at the commit time, attacker will just wait for a certain point on the curve.

4 days ago

0c3ca83

You tell people that the cooldown needs to be a day, while you implement a two month cooldown.

4 days ago

Muromec

Something something don't need to outrun the bear

4 days ago

woodruffw

My research did not show “months to years.” Even weeks of presence are an outlying event[1].

(This is a situation where everybody benefits from high quality data. If you have data that contradicts mine, please share it!)

[1]: https://blog.yossarian.net/2025/11/21/We-should-all-be-using...

3 days ago

lmeyerov

Median is < 1d for major ones

We did a small survey of major incidents after shai-hulud and came to 2d cooldowns through that

3 days ago

giancarlostoro

I wonder if in some cases it's not so much that people are updating too quickly, but rather, their deployment pipelines do not have pinned / frozen versions attached to their builds.

4 days ago

happyPersonR

I’d imagine there were probably fixes for critical CVEs that were included …. A lot of companies are on a 24 hour timeline to get them out.

4 days ago

galoisscobi

Then we’ll find out about malware 24 - 48 hours later.

4 days ago

yujonglee

4 days ago

1vuio0pswjnm7

1792590200 | LiteLLM Hijacked: Malicious 1.107.0 Wheels Steal Credentials | https://research.jfrog.com/post/litellm-hijacked-again/ | https://news.ycombinator.com/item?id=50180838 | 0 comments

1792590232 | Litellm 1.107.0 Compromised | https://github.com/BerriAI/litellm/issues/47826 | https://news.ycombinator.com/item?id=50180845 | 0 comments

4 days ago

rcleveng

Seriously again for litellm? I really with that things like Google ADK didn't add this to my deps.

4 days ago

cyanydeez

all popular, free, ungated software will be poisoned; like kesler syndrome but for the world of open source software.

4 days ago

taylorfinley

Horrifying framing, I will remember that analogy viscerally. Well done.

4 days ago

rcleveng

[dead]

4 days ago

tech_solver

its litellm again. I wonder about maintainer trustworthiness

4 days ago

baddash

litellm in the news again

4 days ago

TZubiri

What's interesting here is that this will only affect those who did not migrate off LiteLLM after it was attacked the first time.

I'm not saying everyong should migrate after there's a vuln, but I'm definitely saying that. Unless it's something huge and foundational like a programming language (PHP gets a second chance), just migrate within the next couple of weeks.

I didn't even have it installed the first time, so the ideal programmers, (read, your competition), actually suffered 0 vulns from this dep, if you suffered 1, that's already pretty bad, being burnt twice should be a wake up call. If you get hit 3 times by the same dependency, it's a lost case.

4 days ago

wongarsu

As someone who does not use litellm but instead a barely maintained underpowered alternative: what did everyone migrate to?

If it matters, I am mostly interested in the aspect of having a proxy that unifies multiple ollama and openai-compatible endpoints with model-based routing and failover. Optionally with api-keys, reporting, budgets, etc. I do not particularly care about the MCP gateway, Agent-To-Agent thingy, and whatever else LiteLLM bolted on over time

4 days ago

TZubiri

As with many dependencies, I wholeheartedly question the requirement and relentlessly cut them out without replacement.

Taking the definition from another user on this comment-chain:

"an OpenAI compatible endpoint routing across multiple providers based on latency"

If the endpoint is truly OpenAI compatible, you can just switch out the vendor by switching out the HTTP address, we are talking potentially about a single line change.

- LLM_HOST = "api.openai.com" + LLM_HOST = "api.competitor.com"

It sounds silly, but remember that software is a winner-takes all game, 99% of devs will lose, so if you are seeing 99% of devs do the same thing, it isn't surprising that it's a grossly wrong choice. It takes skill and guts to be in the 1%, reject the mainstream trends, and just write code without grasping at external imports to do even the most basic things.

I'd like to take this opportunity to sell out, I'm open for contracts if you are looking for a remote 1% dev. Early stage startups only, email in userpage.

4 days ago

tkz1312

Claude will write you something custom that fits your exact requirements with zero supply chain risk in an afternoon.

4 days ago

spike021

how is there zero supply risk? claude will still package dependencies into a homemade solution at some level...

4 days ago

tkz1312

sure zero was a bit strong, but I personally feel pretty good about depending on only the go stdlib, compared to pulling in litellm.

4 days ago

Cu3PO42

Even if it plays out that way (which I doubt, we have a ton of requirements around spend tracking), the problem I'm seeing is long term maintenance. I don't want to fudge around with my proxy every time one of the model providers comes up with a new parameter that works slightly differently on Bedrock than it does on Azure.

That said, I am looking into alternatives. Two times in one year is really a terrible look.

4 days ago

CamperBob2

"Claude, fudge around with my proxy."

4 days ago

staticman2

If Claude's custom solution is less secure than LiteLLM there will be no news report informing you of a vulnerability.

3 days ago

dannyboland

After the last LiteLLM compromise, I took my requirement (an OpenAI compatible endpoint routing across multiple providers based on latency) and built that as a small static Rust binary - https://github.com/sturnus-dev/sturnus

Now that AI makes it cheaper to build such tools in-house, the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker.

4 days ago

HumanOstrich

> and built that as a small static Rust binary

I don't think memory safety would have helped here.

> the case for adopting a maximalist framework and trusting that entire supply chain is a lot weaker

You mean the Rust ecosystem?

4 days ago

hoppp

I just installed it on my server in docker, then the UI was unusable and I deleted it and vibe coded a minimal alternative in golang. It works better.

4 days ago

connorbrinton

Pydantic AI has a great set of abstractions over LLM providers. The documentation tends to focus on its agent loop more, but I find the LLM abstractions quite nice even on their own

4 days ago

john_strinlai

>PHP gets a second chance

second? i haven't kept count, but we have to be somewhere in the hundreds of chances now

4 days ago

antonvs

> I'm not saying everyone should migrate after there's a vuln, but I'm definitely saying that.

Turns out Ancient Aliens was onto something all along.

4 days ago

valgaze

Google's ADK (agent development kit) uses LiteLLM extensively: https://adk.dev/agents/models/litellm/

Their docs list LiteLLM's vulnerability from March... https://i.imgur.com/TPVrVyW.png

3 days ago

saasisdead

Last time they had this, I just vibe coded a version of lite llm and it actually works better

4 days ago

starfallg

Yeah, thinking that after the last supply chain attack, people would be smarter than ship things with LiteLLM.

4 days ago

luke-stanley

Is that you Max?

4 days ago

nairboon

Isn't that at least the second time that LiteLLM got compromised?

4 days ago

mihaaly

Déjà vu for both of us, huh?

4 days ago

avereveard

Again this is the second time this year

4 days ago

hmokiguess

https://github.com/BerriAI/litellm/issues/47826#issuecomment...

> OSS Users > For the time being, please pin your version to 1.106.2 or any explicit version published prior to now.

> Enterprise Customers > There currently exists a private LiteLLM registry for enterprise customers, in which we are distributing an embargoed release that remediates a vulnerability whose disclosure date is scheduled for October 27. Our private registry is not affected. If you are an enterprise customer pulling from our private registry you are not affected.

4 days ago

wongarsu

So they sell advance access to information that will allow me to hack other people's LiteLLM instances?

Seems a little unethical

4 days ago

ctxc

I'm not following, what did you mean?

4 days ago

wongarsu

They are saying that

1. there is a second vulnerability that will be disclosed next Tuesday (can't be this one because talking about it as undisclosed in the GH issue disclosing it makes no sense)

2. the patch for that vulnerability is somewhere in the diff between $current_public_release and $current_enterprise_release

3. You have to pay to get $current_enterprise_release

They might not be selling the vulnerability directly, but they are selling access to the patch that fixes it. Which would be enough of a hint even in pre-LLM times. And unlike other vendors they don't just claim "you get faster updates and additional features" or something like that, they directly point out that if you buy now you get advance access to the fix to an unannounced vuln

I'm not saying they are advertising this to bad actors. But if they wanted to do that with a shred of plausible deniability, this would be the way to do it

4 days ago

chias

Note: I work at LiteLLM

There exists a critical vulnerability which affects only Enterprise LiteLLM deployments, and does not affect OSS deployments.

We have provided our customers with patched releases distributed via a private registry and are allowing them some time to update their deployments before we release the fix in our public registry and disclose the vulnerability details. Releasing the fix publicly in an open source product would be a de-facto vulnerability disclosure.

4 days ago

rvz

So in just 1 week:

1. Google Cloud Outage [0]

2. eMudhra CA private key Leak [1]

3. PingCap TiDB 10.6 TB data Leak [2]

4. PHP2Shell uncovers critical CVEs (with AI) [3]

...and now we have Round 2 of a LiteLLM compromise after the first one in [4].

[0] https://news.ycombinator.com/item?id=50163961

[1] https://news.ycombinator.com/item?id=50166579

[2] https://news.ycombinator.com/item?id=50173716

[3] https://news.ycombinator.com/item?id=50180836

[4] https://news.ycombinator.com/item?id=47501426

4 days ago

gcr

don't forget today's NodeJS security update addressing its own RCE bugs!

4 days ago

rolls-reus

is the google outage related to a security incident?

4 days ago

push0ret

No, it was a power event localized to one building.

4 days ago

petesergeant

> There currently exists a private LiteLLM registry for enterprise customers, in which we are distributing an embargoed release that remediates a vulnerability whose disclosure date is scheduled for October 27.

https://github.com/BerriAI/litellm/issues/47826

oh boy. Announcing that there's a private enterprise repo you should be paying for if you don't want a credential stealer in your build is quite the optic, and then announcing there's an undisclosed second vulnerability that's being distributed, just not to the poors...

4 days ago

sullivanmatt

That's not what's being said, at all. They are simply telling their enterprise customers that there is a different build pathway for the enterprise builds and that pathway wasn't impacted by this issue.

Also if you don't like organizations fixing vulns without releasing the open source first, you're going to be upset when you learn how every major vuln and associated patches are distributed long before the fixes hit the repos. A timely example of this is PHP, literally today.

4 days ago

petesergeant

> A timely example of this is PHP, literally today.

Sorry, there was a paid-for PHP release that fixed it I could have gotten a week ago, and that they publicly announced existed, and it was just people who don't pay for PHP who had to wait...?

4 days ago

sullivanmatt

Yeah, that's how many companies with open ecosystems work. If you're using e.g. RedHat, you are getting packages with embargoed fixes baked into them before those fixes hit public repos. Node.js, OpenSSL, Git itself all do this.

You can argue about whether the "publicly announced existed" part since they don't advertise for any single package, but that's the ecosystem you buy into.

4 days ago

petesergeant

> If you're using e.g. RedHat, you are getting packages with embargoed fixes baked into them before those fixes hit public git repos.

The way you've written this is substantially misleading. RedHat receives notice of the vulnerabilities so they can prepare packages to be available the moment the embargo lifts, which I guess technically could hit the public packages a tiny bit earlier, but they are not releasing fixes of embargoed vulnerabilities to their commercial clients before the embargo lifts.

This is substantially different from the policy I've quoted that I am objecting to.

4 days ago

chias

Note: I work at LiteLLM

There exists a critical vulnerability which affects only Enterprise LiteLLM deployments, and does not affect OSS deployments.

We have provided our customers with patched releases distributed via a private registry and are allowing them some time to update their deployments before we release the fix in our public registry and disclose the vulnerability details. Releasing the fix publicly in an open source product would be a de-facto vulnerability disclosure.

4 days ago

santiago-pl

I wish all the best to LiteLLM team with handling it again. I can imagine how challenging are situations like this one.

However, if anyone is looking for 20x faster and less "targeted" LiteLLM alternative - I'm work on one in Go. That's why GoModel it's called. https://github.com/ENTERPILOT/GoModel/

4 days ago

ghrl

Sounds like a reasonable choice of a language.

I myself have written a small llm proxy in Go, doing nothing but injecting the real API key to outgoing requests while itself validating virtual keys it generates, and it works great.

4 days ago

santiago-pl

Definitely it was a deliberated decision. Also enterprises adapt Go more and more.

4 days ago

TZubiri

If you don't use any LLM router, it's infinity times faster, and infitesimally as targetted.

4 days ago

verdverm

I have GoModel running at home, being deployed at work. There are many good reasons to run a gateway, some include

- being resilient to upstream provider downtime

- centrally managed config/controls

- visibility and allocation into usage and costs

We are deploying a gateway at work primarily for these three.

At home, GoModel's virtual models have been super helpful for testing out new open weights. I can change the model deployed in vLLM without having to change any of my downstream tools, which get a stable name.

https://gomodel.enterpilot.io/docs/features/virtual-models

4 days ago

[deleted]
4 days ago

toxik

Yeah, this DIY MITM attack is entirely avoidable.

4 days ago

Sherveen

In this thread: people who would blame the inventors of the computer for the impact of computer viruses.

4 days ago

abhikap

Crazy day - but seems like this was contained fairly quickly: https://docs.litellm.ai/blog/security-incident-oct-21-2026

4 days ago

[deleted]
3 days ago

callamdelaney

Why on earth do these people make it impossible to select the title of their article?

4 days ago

nkjoep

it's just the poor styling of the selection highlight colour.

4 days ago

hungryhobbit

To clarify a little further, you can select the title, you just can't tell that it's selected because the author made a "poor" (REALLY REALLY DUMB) decision to deliberately style away a basic browser feature (text highlighting).

4 days ago

worldsavior

What you don't know, it's actually a very clever trick by an LLM. By not highlighting the text, it makes you think harder and gives you actually some challenge. 99% of people want to be challenged when selecting text, so the AI decided to drop text highlighting.

/s

4 days ago

antonvs

"A way to provide users with a sense of pride and accomplishment."

4 days ago

tesnorindian

Earlier it was eMudhra and now BerriAI's internal failure? AI usage is both a boon and a bane. Shortsightedness due to continuous AI usage?

3 days ago

garou

https://github.com/gumieri/nenya

If someone just need an AI Proxy with no deps.

3 days ago

mc8098

That just shifts the risk

3 days ago

garou

[flagged]

3 days ago

1-6

This is the 2nd time. I'm glad I moved away from this platform.

4 days ago

wolvoleo

What worries me is that this seems to be the second time this happened and there's not even a notification on litellm.ai about it.

3 days ago

infecto

Any package hosts that scan and include cooldowns by default that operate on a consumption model? The ones I have seen all charge a lot.

4 days ago

abhisek

Not sure about a service, but PMG enforces dependency cooldown and sandbox on package install. https://github.com/safedep/pmg

4 days ago

edoceo

This one ? https://news.ycombinator.com/item?id=46909037

I use it, I don't cooldown, proper version pinning for me.

4 days ago

wallrat

Bytesafe (https://bytesafe.dev). Pure Dependency Firewall, not a package host, with usage based pricing.

4 days ago

TZubiri

Socket.dev

Tell em Tom Zubiri sent ya (I don't get a commission, just good karma)

4 days ago

cyanydeez

man, people are going to be sorely disappointed in what's free on the internet in a couple of years.

Agents and malware will eradicate all goodwill left.

Good by internet.

4 days ago

jacobalcock

[flagged]

4 days ago

syntaxing

I figured LLMs nowadays would be good enough to catch issues like this. Is it because it’s too costly to do it?

4 days ago

abhisek

Yes if you use LLM to analyse everything. Not really if you analyse diffs, with heuristics to select candidates for LLM/agent based analysis. But at the cost of missing out novel attacks.

4 days ago

hendersoon

I setup 8 day age gates on everything back in May. This is inobtrusive and pretty safe.

4 days ago

stuaxo

Oof.

LiteLLM has one of the nicest programmer UX to use but this is not good at all.

4 days ago

joshheitzman

Seems like pip doesn't have a way to exclude packages although it was requested at least once: https://github.com/pypa/pip/issues/10797

Just checked if litellm crept in with any packages I installed recently. Fortunately it didn't, but I'm working on an OSS AI agent harness so its certainly a concern and I'd like just flatout block litellm so I don't have to worry about picking it up as transitive dependency. Guess I'll have to vibe a script and remember to run it. Sigh.

3 days ago

wilkystyle

If you can use uv, try the --exclude flag, e.g.:

    uv pip install -r requirements.txt --exclude excludes.txt
excludes.txt is basically like an inverse requirements.txt. Your requirements.txt could have this...

    pandas
    scikit-learn
    matplotlib

...while excludes.txt has this...

    numpy
... and uv would resolve the dependencies in requirements.txt, excluding numpy from the dependency graph entirely and ignoring any dependencies that would have been introduced by numpy itself.

EDIT: You can also configure this for uv directly in a pyproject.toml file:

    [tool.uv]
    exclude-dependencies = ["numpy"]
3 days ago

joshheitzman

From the docs it seems like that would just silently not install the blocked package, so while it would protect my dev environments regular pip installs would still pick up the transitive dependency.

3 days ago

wilkystyle

Yes, and I did qualify my comment with "if you can use uv"

3 days ago

0xbadc0de5

Again! Yikes.

4 days ago

jiwidi

Twice? uiuiui

4 days ago

merelydev

A bad day for opensource, but I guess this is how we get stronger patch by patch.

4 days ago

[deleted]
4 days ago

jkercher

4 days ago

abletonlive

Well it seems like it’s over for litellm. You’d have to be negligent to be using them at this point. If you’re managing this at your company good luck explaining why you should continue to use litellm after this

4 days ago

detente18

Hi abletonlive,

I appreciate the feedback. We work hard to make every release safe and secure. The attack was novel, and I'd love your feedback on how we can improve our ci/cd further.

https://docs.litellm.ai/blog/security-incident-oct-21-2026#h...

4 days ago

abletonlive

"Novel" is doing a lot of work here. Save the PR for people that will fall for it. There's nothing novel about being internally compromised which you've demonstrated two times in a row

3 days ago

SomaticPirate

Second f**king time. Just torching their reputation at this point

4 days ago

bcvvvy

Fuck this week.

4 days ago

antonvs

There are over 8 billion people on Earth, and some subset of the computer systems that they all use are going to have issues, every day.

It's really not as bad as the accumulated headlines and exploitative hype make it sound.

It's similar to the mass hysteria and moral panics over things like child abduction. Media reports make certain dangers seem much more important than they really are, when considered in context.

4 days ago

zxilly

Welcome to the AI era, every week in the near future

4 days ago

arbor-group

[flagged]

3 days ago

tonyhines

[flagged]

3 days ago

smakosh

[flagged]

4 days ago

wunderwuzzi23

[dead]

3 days ago

azyc

[flagged]

4 days ago

latencyharbor

[flagged]

4 days ago

News2Use

[flagged]

4 days ago

grd_dev

[flagged]

3 days ago

deployvet

[flagged]

4 days ago

alexcodebytes

[flagged]

3 days ago

ibraryunus

[flagged]

4 days ago

luis_valerio

[flagged]

4 days ago

rao-v

[dead]

4 days ago

Jeeetendra

[flagged]

4 days ago

chelseahermes

[flagged]

4 days ago