Glassworm is back: A new wave of invisible Unicode attacks hits repositories
Comments
btown
andrewflnr
Regardless of the thorny question of whether it's Github's responsibility, it sure would be a good thing for them to do ASAP.
godelski
Here's the big reason GitHub should do it:
It makes the product better
So I'd argue it's the responsibility of the developers, hired by GitHub, to create this feature because it makes the product better. Because that's the thing you've been hired for: to make the product better. Your concern isn't about the money, your concern is about the product. That's what you're hired for.
btown
I'd say that this is also true from a money-and-costs-and-value perspective. Sure, all press is good press... but any number of stakeholders would agree that "we got some mindshare by proactively protecting against an emerging threat" is higher-ROI press than "Ars did a piece on how widespread this problem is, and we're mentioned in the context of our interface making the attack hard to detect."
And when the incremental cost to build a feature is low in an age of agentic AI, there should be no barrier to a member of the technical staff (and hopefully they're not divided into devs/test/PM like in decades past) putting a prototype together for this.
godelski
I agree and think it's extra important when you have specialized products. Experts are more sensitive to the little things.
Engineers and developers are especially sensitive. It's our job to find problems and fix them. I don't trust engineers that aren't a bit grumpy because it usually means they don't know what the problems are (just like when they don't dogfood). Though I'll also clarify that what distinguishes a grumpy engineer from your average redditer is that they have critiques rather than just complaints. Critique oriented is searching for solutions of problems, you can't just stop at problem identification.
> And when the incremental cost to build a feature is low in an age of agentic AI
Though we're also talking about an organization that couldn't merge a PR for a year that fixed a one liner. A mistake that should never have gotten through review. Seriously, who uses a while loop counter checking for equality?!? I'm still convinced they left the "bug" because it made them money
fingerlocks
>Though we're also talking about an organization that couldn't merge a PR for a year that fixed a one liner. A mistake that should never have gotten through review. Seriously, who uses a while loop counter checking for equality?!? I'm still convinced they left the "bug" because it made them money
What is this in reference to? I tried to search for it but only found this comment. “Github while loop fix that was in review for a year”?
rkagerer
At the end of the day it boils down to putting your users first.
Making the product better generally stems from acting in their interest, honing the tool you offer to provide the best possible experience, and making business decisions that respect their dignity.
Your comment talks a lot about product and I agree with it, I just mentioned this so we don't lose sight of the fact this is ultimately about people.
tapland
Tldr: Yeah it would make it better!
godelski
I hope I left the lead as the lead.
But I also think we've had a culture shift that's hurting our field. Where engineers are arguing about if we should implement certain features based on the monetary value (which are all fictional anyways). But that's not our job. At best, it's the job of the engineering manager to convince the business people that it has not only utility value, but monetary.
andrewflnr
> Your concern isn't about the money, your concern is about the product. That's what you're hired for.
According to whom? Certainly not the people did the hiring.
I somewhat agree that developers should optimize for something other than pure monetary value, but it has nothing to do with the hiring relationship, just the moral duty to use what power you have to make the world better. In general, this can easily conflict with "what you're hired for."
In this case I think showing suspicious (or even all) invisible Unicode in PRs is even a monetarily valuable feature, so the moral angle is mostly moot. And I would put the primary moral burden primarily on the product management either way, since they're the ones with the most power to affect the product, potentially either ordering the right thing to be done or stopping the devs when they try to do it on their own.
godelski
> According to whom? Certainly not the people did the hiring.
As a developer you have a de facto primary concern with the product. They hire you to... develop. They do not hire you to manage finances, they hire you to manage the product. Doing both is more the job of the engineering manager. But as a developer your expertize is in developing. I don't think this is a crazy viewpoint.
You were hired for your technical skills, not your MBA.
> In this case I think showing suspicious (or even all) invisible Unicode in PRs is even a monetarily valuable feature
Also note that I'm writing to my audience.
>> but HN is a space for developers, not the business people.
andrewflnr
They might say that your job is to make the product "better", and they might even think they mean it, but I think in practice you'll find that their definition of "better" as it relates to products is pretty closely related to money, and further that they are the authorities on what makes the product "better" so you should shut up and do what they say. If you want to make the product actually better, you're going to have to defy them occasionally. That's not what you were hired for, that's just being a human with principles.
jacquesm
It absolutely is. They are simply spreading malware. You can't claim to be a 'dumb pipe' when your whole reason for existence is to make something people deemed 'too complex' simple enough for others to use, then you have an immediate responsibility to not only reduce complexity but to also ensure safety. Dumbing stuff down comes with a duty of care.
OJFord
They advertise that they do do it, they just don't/it doesn't work.
See commenter on their 2025 bounty for reporting it, won't-fix resolution: https://news.ycombinator.com/item?id=47393393
zzo38computer
I think a "force visible ASCII for files whose names match a specific pattern" mode would be a simple thing to help. (You might be able to use the "encoding" command in the .gitattributes file for this, although I don't know if this would cause errors or warnings to be reported, and it might depend on the implementation.)
RVuRnvbM2e
Vigilant mode exists, and would have flagged the malicious commit as unverified in this case. Maybe it should be the default.
https://docs.github.com/en/authentication/managing-commit-si...
athrowaway3z
For some reason I was under the impression this was already the default.
I first heard about the possibility of this kind of attack >10 years ago, and I'll sometimes do a xxd if i'm feeling a bit paranoid.
iririririr
specially because it's literally a problem with their code viewer (and vscode, which is also theirs).
i see squares on a properly configured vim on xterm.
ocornut
It baffles me that any maintainer would merge code like the one highlighted in the issue, without knowing what it does. That’s regardless of being or not being able to see the “invisible” characters. There’s a transforming function here and an eval() call.
The mere fact that a software maintainer would merge code without knowing what it does says more about the terrible state of software.
dspillett
> It baffles me that any maintainer would merge code like the one highlighted in the issue, without knowing what it does.
I don't know if it is relevant in any specific case that is being discussed here, but if the exploit route is via gaining access to the accounts of previously trusted submitters (or otherwise being able to impersonate them) it could be a case of teams with a pile of PRs to review (many of which are the sloppy unverified LLM output that is causing a problem for some popular projects) lets through an update from a trusted source that has been compromised.
It could correctly be argued that this is a problem caused by laziness and corner cutting, but it is still understandable because projects that are essentially run by a volunteer workforce have limited time resources available.
mmlb
In this instance the PR that was merged was from 6 years ago and was clear https://github.com/pedronauck/reworm/pull/28. Looks to me like a force push overwrote the commit that now exists in history since it was done 6y later.
globular-toast
So who force pushed and why?
retropragma
likely a compromised github API token
pdonis
Wish I could upvote this more.
mmsc
GitHub advertises itself as warning about those Unicode characters: https://github.blog/changelog/2025-05-01-github-now-provides...
Of course, it doesn't work though. I reported this to their bug bounty, they paid me a bounty, and told me "we won't be fixing it": https://joshua.hu/2025-bug-bounty-stories-fail#githubs-utf-f...
The exact quote is "Thanks for the submission! We have reviewed your report and validated your findings. After internally assessing your report based on factors including the complexity of successfully exploiting the vulnerability, the potential data and information exposure, as well as the systems and users that would be impacted, we have determined that they do not present a significant security risk to be eligible under our rewards structure." The funny thing is, they actually gave me $500 and a lifetime GitHub Pro for the submission.
user_7832
Tangential, but that's quite interesting, I had no idea you could get GitHub Pro for life, and certainly not through something as "accessible" as bug bounties.
OJFord
That's bizarre. They won't be fixing it, and yet the changelog post is unretracted.
zzo38computer
I use non-Unicode mode in the terminal emulator (and text editors, etc), I use a non-Unicode locale, and will always use ASCII for most kind of source code files (mainly C) (in some cases, other character sets will be used such as PC character set, but usually it will be ASCII). Doing this will mitigate many of this when maintaining your own software. I am apparently not the only one; I have seen others suggest similar things. (If you need non-ASCII text (e.g. for documentation) you might store them in separate files instead. If you only need a small number of them in a few string literals, then you might use the \x escapes; add comments if necessary to explain it.)
The article is about in JavaScript, although it can apply to other programming languages as well. However, even in JavaScript, you can use \u escapes in place of the non-ASCII characters. (One of my ideas in a programming language design intended to be better instead of C, is that it forces visible ASCII (and a few control characters, with some restrictions on their use), unless you specify by a directive or switch that you want to allow non-ASCII bytes.)
amake
That’s great for you. Isn’t feasible for software development by teams that are native in a language with a non-Latin script.
1718627440
Do you write the code itself in a language other than English? Localizations typically are in different files.
amake
I do see a handful of people using non-ASCII identifiers in their code, but that's rare. Much more common is explanatory comments, docstrings, etc. in the local language. To require those to be ASCII would be a non-starter.
TacticalCoder
> ... and will always use ASCII for most kind of source code files
Same. And I enforce it. I've got scripts and hooks that enforces source files to only ever be a subset of ASCII (not even all ASCII codes have their place in source code).
Unicode chars strings are perfectly fine in resource files. You can build perfectly i18n/l10n apps and webapps without ever using a single Unicode character in a source file. And if you really do need one, there's indeed ASCII escaping available in many languages.
Some shall complan that their name as "Author: ..." in comments cannot be written properly in ASCII. If I wanted to be facetious I'd say that soon we'll see:
# Author: Claude Opus 27.2
userbinator
CP437 forever!
The biggest use of Unicode in source repos now might be LLM slop, so I certainly don't miss its absence at all.
nstart
I don't quite understand how this is working tbh. I looked at one of the affected repos, ironically named "reworm".
The malicious code was introduced in this commit - https://github.com/pedronauck/reworm/commit/d50cd8c8966893c6...
It says coauthored by dependabot and refers to a PR opened in 2020 (https://github.com/pedronauck/reworm/pull/28).
That PR itself was merged in 2020 here - https://github.com/pedronauck/reworm/commit/df8c1803c519f599...
But the commit with the worm (d50cd8c), re-introduces the same change from df8c180 to the file `yarn.lock`.
And when you look at the history of yarn.lock inside of github, all references to the original version bump (df8c180) are gone...? In fact if you look at the overall commit history, the clean df8c180 commit does not exist.
I'm struggling to understand what kind of shenanigans happened here exactly.
RVuRnvbM2e
Someone has maintainer/admin access to the repository and has force-pushed to master overwriting the git history.
Notice that the original commit is verified: https://github.com/pedronauck/reworm/commit/df8c1803c519f599...
While the malicious one is not: https://github.com/pedronauck/reworm/commit/d50cd8c8966893c6...
globular-toast
This reveals a deeper flaw in the whole git/npm pipeline (would apply to other systems like PyPI etc, not npm exclusively). These systems should operate on a "pull" model, not a push. The system should have rejected a build that wasn't derived from the latest in its repository. It would be quite easy in concept to set up one's own system to pull every source on npm and alert when the upstream has deviated.
jibal
The malicious code was added to package.json, not yarn.lock
nstart
Yup. That's correct. And I understand that. I was looking at the changes to yarn.lock that got reintroduced. I couldn't figure out what was happening. It turns out that not only was it force pushed, but GitHub also retains the old commit information even if it's been "deleted".
I still don't quite understand what GitHub is doing to allow someone to say that dependabot coauthored a spoofed commit. This isn't the commit message itself I'm talking about. It's the GitHub interface that officially recognizes this as a dependabot co authored commit. My hunch is that the malicious author squashed two commits, the original good commit to yarn.lock and a malicious change to package.json, and that somehow maintains the dependabot authorship instead of reassigning it fully to the squash-er.
vitus
Looks like the repo owner force-pushed a bad commit to replace an existing one. But then, why not forge it to maintain the existing timestamp + author, e.g. via `git commit --amend -C df8c18`?
Innocuous PR (but do note the line about "pedronauck pushed a commit that referenced this pull request last week"): https://github.com/pedronauck/reworm/pull/28
Original commit: https://github.com/pedronauck/reworm/commit/df8c18
Amended commit: https://github.com/pedronauck/reworm/commit/d50cd8
Either way, pretty clear sign that the owner's creds (and possibly an entire machine) are compromised.
chrismorgan
The value of the technique, I suppose, is that it hides a large payload a bit better. The part you can see stinks (a bunch of magic numbers and eval), but I suppose it’s still easier to overlook than a 9000-character line of hexadecimal (if still encoded or even decoded but still encrypted) or stuff mentioning Solana and Russian timezones (I just decoded and decrypted the payload out of curiosity).
But really, it still has to be injected after the fact. Even the most superficial code review should catch it.
vitus
Agreed on all those fronts. I'm just dismayed by all the comments suggesting that maintainers just merged PRs with this trojan, when the attack vector implies a more mundane form of credential compromise (and not, as the article implies, AI being used to sneak malicious changes past code review at scale).
jeltz
Yeah, the attack vector seems to be stolen credentials. I would be much more interested in an attack which actually uses Invisible characters as the main vector.
minus7
The `eval` alone should be enough of a red flag
whizzter
Sadly JS has ways around it that is far from obvious since you can chain effects over multiple files that leads to running code.
Like the following example (you can paste it into node to verify), could be spread out over multiple source files to make it even harder to follow:
// prelude 1, obfuscate the constructor property name to avoid raising simple analyser alarms
const prefix = "construction".substring(0,7);
const suffix = "tractor".substring(3);
const obfuscatedConstructorName = prefix + suffix; // innocent looking, but we have the indexing name.
// prelude 2, get the Function class by indexing a function object with our constructor property name (that does not show up in source-code)
const existingFunction = ()=>"nothing here";
const InnocentLookingClass = existingFunction[obfuscatedConstructorName];
// payload decoding elsewhere (this is where we decode our nasty source)
const nastyPayloadDisguisedAsData = "console.log('sourced string that could be malicious')";
// Unrelated location where payload gets executed
const hardToMissFun = new InnocentLookingClass(nastyPayloadDisguisedAsData);
hardToMissFun(); // when this function is run somewhere.. the nasty things happen.
jeltz
Yeah, I would have loved to see an example where it was not obvious that there is an exploit. Where it would be possible for a reviewer to actually miss it.
godelski
I'm not a JS person, but taking the line at face value shouldn't it to nothing? Which, if I understand correctly, should never be merged. Why would you merge no-ops?
kordlessagain
No it’s not.
simonreiff
OWASP disagrees: See https://cheatsheetseries.owasp.org/cheatsheets/Nodejs_Securi..., listing `eval()` first in its small list of examples of "JavaScript functions that are dangerous and should only be used where necessary or unavoidable". I'm unaware of any such uses, myself. I can't think of any scenario where I couldn't get what I wanted by using some combination of `vm`, the `Function` constructor, and a safe wrapper around `JSON.parse()` to do anything I might have considered doing unsafely with `eval()`. Yes, `eval()` is a blatant red flag and definitely should be avoided.
jacquesm
While there are valid use cases for eval they are so rare that it should be disabled by default and strongly discouraged as a pattern. Only in very rare cases is eval the right choice and even then it will be fraught with risk.
godelski
The parent didn't say "there's no legitimate uses of eval", they said "using eval should make people pay more attention." A red flag is a warning. An alert. Not a signal saying "this is 100% no doubt malicious code."
Yes, it's a red flag. Yes, there's legitimate uses. Yes, you should always interrogate evals more closely. All these are true
pavel_lishin
When is an eval not at least a security "code smell"?
SahAssar
It really is. There are very few proper use-cases for eval.
nswango
For a long time the standard way of loading JSON was using eval.
bawolff
Not that long, browsers implemented JSON.parse() back in 2009. JSON was only invented back in 2001 and took a while to become popular. It was a very short window more than a decade ago when eval made sense here.
Eval for json also lead to other security issues like XSSI.
creatonez
Problem is, it took until around 2016 for IE6 to be fully dead, so people continued to justify these hacks for a long time. Horrifying times.
_flux
And why do we not anymore make use of it, but instead implemented separate JSON loading functionality in JavaScript? Can you think of any reasons beyond performance?
bawolff
I'd be surprised if there is a performance benefit of processing json with eval(). Browsers optimize the heck out of JSON.
bulbar
Why did you opt in for such a comment while a straight forward response without belittling tone would have achieved the same?
gnabgib
Small discussion yesterday (9+9 points, 9+4 comments) https://news.ycombinator.com/item?id=47374479 https://news.ycombinator.com/item?id=47385244
bawolff
I feel like the threat of this type of thing is really overstated.
Sure the payload is invisible (although tbh im surprised it is. PUA characters usually show up as boxes with hexcodes for me), but the part where you put an "empty" string through eval isn't.
If you are not reviewing your code enough to notice something as non sensical as eval() an empty string, would you really notice the non obfuscated payload either?
loumf
The threat is that you depend on this library or use the VS Code Extension.
Arrowmaster
Honestly I was expecting more. There are many languages that support Unicode in variable or function names and I expected it to be used there.
It sounds like Python only allows approved Unicode characters to start a variable name but if it allowed any you could do something like `nonprintable = lambda x: insert exploit code here`. If that was hidden in what looked like a blank line between other additions would you catch it?
I'm sure there's some other language out there that has similar syntax and lax Unicode rules this could be used in.
The solution is that this and many other Unicode formatting characters should be ignored and converted to a visible indicator in all code views when you expect plain text.
tolciho
Attacks employing invisible characters are not a new thing. Prior efforts here include terminal escape sequences, possibly hidden with CSS that if blindly copied and pasted would execute who knows what if the particular terminal allowed escape sequences to do too much (a common feature of featuritis) or the terminal had errors in its invisible character parsing code.
For data or code hiding the Acme::Bleach Perl module is an old example though by no means the oldest example of such. This is largely irrelevant given how relevant not learning from history is for most.
Invisible characters may also cause hard to debug issues, such as lpr(1) not working for a user, who turned out to have a control character hiding in their .cshrc. Such things as hex viewers and OCD levels of attention to detail are suggested.
retropragma
If anyone's curious what the malware does: https://pastebin.com/raw/KiuwueMU
Looks like it's pilfering Solana wallets that don't belong to Russians.
DropDead
Why didn't some make av rule to find stuff like this, they are just plain text files
nine_k
The rule must be very simple: any occurrence of `eval()` should be a BIG RED FLAG. It should be handled like a live bomb, which it is.
Then, any appearance of unprintable characters should also be flagged. There are rather few legitimate uses of some zero-width characters, like ZWJ in emoji composition. Ideally all such characters should be inserted as \xNNNN escape sequences, and not literal characters.
Simple lint rules would suffice for that, with zero AI involvement.
hamburglar
I think there’s debate (which I don’t want to participate in) over whether or not invisible characters have their uses in Unicode. But I hope we can all agree that invisible characters have no business in code, and banishing them is reasonable.
WalterBright
> There are rather few legitimate uses of some zero-width characters, like ZWJ in emoji composition.
Emojis are another abomination that should be removed from Unicode. If you want pictures, use a gif.
_flux
Arguably them being in Unicode is an accessibility issue, unless we thought to standardize GIF names, and then that already sounds a lot like Unicode.
WalterBright
How is it an accessibility issue? HTML allows things like little gif files. I've done this myself when I wrote text that contained Egyptian hieroglyphs. It works just fine!
_flux
I mean if you don't have sight.
sghitbyabazooka
( ꏿ ﹏ ꏿ ; )
trollbridge
In our repos, we have some basic stuff like ruff that runs, and that includes a hard error on any Unicode characters. We mostly did this after some un-fun times when byte order marks somehow ended up in a file and it made something fail.
I have considered allowing a short list that does not include emojis, joining characters, and so on - basically just currency symbols, accent marks, and everything else you'd find in CP-1521 but never got around to it.
hrmtst93837
Automatic escaping sounds nice until you need to grep or diff across repos and get buried in opaque escapes that turn ordinary review into unreadable junk. Once that lands in a repo, even routine deps updates can turn into edge-case mismatch roulette.
Lint zero-width chars, sure. But if the actual sink is runtime string injection, banning eval is only half a fix because Function and friends still get you to the same bad place while the linter congratulates itself.
abound
Yeah it would have been nice to end with "and here's a five-line shell script to check if your project is likely affected". But to their credit, they do have an open-source tool [1], I'm just not willing to install a big blob of JavaScript to look for vulns in my other big blobs of JavaScript
nine_k
Something like this should work, assuming your encoding is Unicode (normally UTF-8), which grep would interpret:
grep -P '[\x{200B}\x{200C}\x{200D}\x{FEFF}]' code.ts
charcircuit
Isn't that what this article is about? Advertising an av rule in their product that catches this.
codechicago277
I wonder if this could be used for prompt injection, if you copy and paste the seemingly empty string into an LLM does it understand? Maybe the affect Unicode characters aren’t tokenized.
ancillary
There's at least one paper (though pretty recent) about it: https://arxiv.org/html/2603.00164v1
jibal
Yes, and that happens.
anesxvito
The scary part is how invisible this is in code review. Unicode direction overrides and zero-width characters don't show up in most editors by default. Anyone know a solid pre-commit hook config that catches this reliably?
Marsymars
New repo, but seems configurable to do what you're asking for: https://github.com/mit-d/check-unicode
Though also seems relatively straightforward to code up in your scripting language of choice to call from pre-commit.
invalidusernam3
eval is the major red flag here
herpdyderp
I keep seeing this and wondering if the ESLint default rules against weird characters would catch this? But I can’t figure out how to check.
CGamesPlay
Appears not to. https://claude.ai/share/ac070cf5-0034-4f3c-9a8c-1c43a58eea36
Claude’s analysis seems solid here based on reading the snippets it tested.
A purpose-built linter could be cross-language, it’s pretty reasonable to blanket ban these characters entirely, or at least allowlist them.
mhitza
Their button animations almost "crash" Firefox mobile. As soon as I reach them the entire page scrolls at single digit FPS.
P-MATRIX
This gets a lot worse when a coding agent is in the loop. A human at least has a review step—an autonomous agent that reads a Glassworm-infected file just acts on it. The fix probably needs to happen at the tool result layer, before the payload ever enters the agent's context, not just on what the agent writes out.
WalterBright
Unicode should be for visible characters. Invisible characters are an abomination. So are ways to hide text by using Unicode so-called "characters" to cause the cursor to go backwards.
Things that vanish on a printout should not be in Unicode.
Remove them from Unicode.
pvillano
Unicode is "designed to support the use of text in all of the world's writing systems that can be digitized"
Unicode needs tab, space, form feed, and carriage return.
Unicode needs U+200E LEFT-TO-RIGHT MARK and U+200F RIGHT-TO-LEFT MARK to switch between left-to-right and right-to-left languages.
Unicode needs U+115F HANGUL CHOSEONG FILLER and U+1160 HANGUL JUNGSEONG FILLER to typeset Korean.
Unicode needs U+200C ZERO WIDTH NON-JOINER to encode that two characters should not be connected by a ligature.
Unicode needs U+200B ZERO WIDTH SPACE to indicate a word break opportunity without actually inserting a visible space.
Unicode needs MONGOLIAN FREE VARIATION SELECTORs to encode the traditional Mongolian alphabet.
luke-stanley
So we need a new standard problem due to the complexity of the last standard? Isn't unicode supposed to be a superset of ASCII, which already has control characters like new space, CR, and new lines? xD
WalterBright
The only ones people use any more are newline and space. A tab key is fine in your editor, but it's been more or less abandoned as a character. I haven't used a form feed character since the 1970s.
tetha
That ship has sailed, but I consider Unicode a good thing, yet I consider it problematic to support Unicode in every domain.
I should be able to use Ü as a cursed smiley in text, and many more writing systems supported by Unicode support even more funny things. That's a good thing.
On the other hand, if technical and display file names (to GUI users) were separate, my need for crazy characters in file names, code bases and such are very limited. Lower ASCII for actual file names consumed by technical people is sufficient to me.
WalterBright
> That ship has sailed
Sure, but more crazy stuff gets added all the time.
WalterBright
Another dum dum Unicode idea is having multiple code points with identical glyphs.
Rule of thumb: two Unicode sequences that look identical when printed should consist of the same code points.
estebank
If anything, Unicode should have had more disambiguated characters. Han unification was a mistake, and lower case dotted Turkish i and upper case dotless Turkish I should exist so that toUpper and toLower didn't need to know/guess at a locale to work correctly.
WalterBright
Characters should not have invisible semantics.
nswango
So you think that the letters in the Greek and Cyrillic alphabets which are printed identically to the Latin A should not exist?
And, for example, Greek words containing this letter should be encoded with a mix of Latin and Greek characters?
WalterBright
> So you think that the letters in the Greek and Cyrillic alphabets which are printed identically to the Latin A should not exist?
Yes. Unicode should not be about semantic meaning, it should be about the visual. Like text in a book.
> And, for example, Greek words containing this letter should be encoded with a mix of Latin and Greek characters?
Yup. Consider a printed book. How can you tell if a letter is a Greek letter or a Latin letter?
Those Unicode homonyms are a solution looking for a problem.
bawolff
> Yes. Unicode should not be about semantic meaning, it should be about the visual. Like text in a book.
Do you think 1, l and I should be encoded as the same character, or does this logic only extend to characters pesky foreigners use.
WalterBright
They are visually distinct to the reader.
Yokohiii
Unicode is about semantics not appearance. If you don't need semantics then use something different.
WalterBright
> Unicode is about semantics not appearance.
And that's where it went off the rails into lala land. 'a' can have all kinds of distinct meanings. How are you going to make that work? It's hopeless.
Muromec
>Yup. Consider a printed book. How can you tell if a letter is a Greek letter or a Latin letter?
I can absolutely tell Cyrillic k from the lating к and latin u from the Cyrillic и.
>should not be about semantic meaning,
It's always better to be able to preserve more information in a text and not less.
WalterBright
> I can absolutely tell Cyrillic k from the lating к and latin u from the Cyrillic и.
They look visually distinct to me. I don't get your point.
> It's always better to be able to preserve more information in a text and not less.
Text should not lose information by printing it and then OCR'ing it.
ted_dunning
But these characters only look identical in some fonts. Are you saying that if you change font, some characters in a string should change appearance and others should not?
And what about the round-trip rule?
And ligatures? Aren't those a semantic distinction?
WalterBright
> But these characters only look identical in some fonts.
That's a problem with the fonts.
> And what about the round-trip rule?
Print Unicode on paper, then ocr it, and you'll get different Unicode. Oh, and normalization.
> ligatures
Generally an issue with rendering.
> semantic distinction
Unicode isn't about semantics (or shouldn't be). Consider 'a'. It's used for all kinds of meanings.
Yokohiii
What about numbers? Would they be assigned to arabic only? I guess someone will be offended by that.
While at it we could also unify I, | and l. It's too confusing sometimes.
WalterBright
> While at it we could also unify I, | and l. It's too confusing sometimes.
They render differently, so it's not a problem.
ted_dunning
They only render differently in some fonts, on some displays.
Yokohiii
totally not true :D
jeltz
I don't think that would help much. There are also characters which are similar but not the same and I don't think humans can spot the differences unless they are actively looking for them which most of the time people are not. If only one of two glyphs which are similar appear in the text nobody would likely notice, expectation bias will fuck you over.
WalterBright
I wonder how anybody got by with printed books.
wcoenen
As far as I know, glyphs are determined by the font and rendering engine. They're not in the Unicode standard.
WalterBright
Fraktur (font) and italic (rendering) are in the Unicode standard, although Hackernews will not render them. (I suspect that the Hackernews software filters out the nuttier Unicode stuff.)
ted_dunning
One of the ground rules of Unicode is the round trip rule. You have to be able to translate to and from Unicode without loss of information.
WalterBright
They threw that out the window with normalization.
ted_dunning
No need to remove them. Just make them visible for applications that don't need to render every language. Make that behavior optional as well in case you really want to name characters with Hangul or Tibetan.
Some middle ground so that you can use greek letters in Julia might be nice as well.
But I don't see any purpose in using the Personal Use Areas (PUA) in programming.
abujazar
Invisible characters are there for visible characters to be printed correctly...
WalterBright
I'll grant that a space and a newline are necessary. The rest, nope.
abujazar
You're talking about a subset of ASCII then. Unicode is supposed to support different languages and advanced typography, for which those characters are necessary. You can't write e.g. Arabic or Hebrew without those "unnecessary" invisible characters.
WalterBright
Please explain why an invisible zero width "character" is necessary.
ted_dunning
To prevent ligatures from forming when you need that.
krior
To mark linewrapping-breakpoints in strings.
slim
if you write كلب which is an arabic word written right to left in the middle of an english sentence, you want to preserve the order of the characters in the stream for computer processing purposes. meaning the chararacter ك must come before the ل and after the e and the space with respect to the memory layout. whereas when displayed, it must be inverted to be legible. the solution is to have an invisible character that indicates a switch in text direction. if you were wondering, the situation where you want to write text in a foreign language within your text is very common outside english speaking countries.
uhoh-itsmaciek
>Remove them from Unicode.
Do you honestly think this is a workable solution?
WalterBright
Yes, absolutely. See my other replies.
moritzruth
greatidea,whoneedsspacesanyway
WalterBright
Spaces appear on a printout.
ted_dunning
As do tabs, ems, ens and quads.
WalterBright
Unicode shouldn't be a typesetting language. The proper tool for that is Latex.
eviks
So you'd remove space and tab from Unicode?
bawolff
Good luck with that given there are invisible characters in ascii.
Also this attack doesnt seem to use invisible characters just characters that dont have an assigned meaning.
WalterBright
The only problematic one is CR which can be used to hide text on a glass terminal with a tty interface. I'd get rid of it if I could.
faangguyindia
Back in time I was on hacking forums where lot of script kiddies used to make malicious code.
I am wondering how that they've LLM, are people using them for making new kind of malicious codes more sophisticated than before?
Yokohiii
In this case LLMs were obviously used to dress the code up as more legitimate, adding more human or project relevant noise. It's social engineering, but you leave the tedious bits to an LLM. The sophisticated part is the obscurity in the whole process, not the code.
rvnx
This shows the failure of human reviews alone, an LLM-based reviewer would have caught it. Both approaches are complementary
rhysfonixone
Exactly this. I think a hybrid approach is going to be mandatory before long, if it's not already. A well-prompted frontier-lab LLM would catch things like this easily.
chairmansteve
eval() used to be evil....
Are people using eval() in production code?
like_any_other
Invisible characters, lookalike characters, reversing text order attacks [1].. the only way to use unicode safely seems to be by whitelisting a small subset of it.
And please, everyone arguing the code snippet should never have passed review - do you honestly believe this is the only kind of attack that can exploit invisible characters?
NoMoreNicksLeft
Why can't code editors have a default-on feature where they show any invisible character (other than newlines)? I seem to remember Sublime doing this at least in some cases... the characters were rendered as a lozenge shape with the hex value of the character.
Is there ever a circumstance where the invisible characters are both legitimate and you as a software developer wouldn't want to see them in the source code?
ted_dunning
Check out emacs for options like this.
And, yes, there is a circumstance if you want to include Arabic or Hebrew in comments or strings. You need the zero width left-right markers to make that work.
hananova
My hot take is that all programming languages should go back to only accepting source code saved in 7-bit ASCII. With perhaps an exception for comments.
krior
Yeah, fuck those non-english-speaking peasants /s.
hananova
I'm a non-english-speaking peasant. I code in English, because it's the lingua franca of coding, and because they form the only characters that you can reliably use everywhere.
Besides, that's why the ban only extends to syntax and string literals (use escapes instead), and not comments.
From my experience, the only two nationalities that insist on mixing their native languages with the mostly English syntax of programming languages are the French and the Japanese. And they can just suck it up for the other 8 billion of us.
max_
I don't have to worry about any of this.
My clawbot & other AI agents already have this figured out.
/s
IMO while the bar is high to say "it's the responsibility of the repository operator itself to guard against a certain class of attack" - I think this qualifies. The same way GitHub provides Secret Scanning [0], it should alert upon spans of zero-width characters that are not used in a linguistically standard way (don't need an LLM for this, just n-tuples).
Sure, third-party services like the OP can provide bots that can scan. But if you create an ecosystem in which PRs can be submitted by threat actors, part of your commitment to the community should be to provide visibility into attacks that cannot be seen by the naked eye, and make that protection the norm rather than the exception.
[0] https://docs.github.com/en/get-started/learning-about-github...