GitHub Merge Queue Silently Reverted Code

99 points
1/21/1970
3 days ago
by matthewbauer

Comments


cwkcw

The window to audit is 2026-04-23 16:05 to 20:43 UTC, about 4h 38m, and 'merged incorrectly' is unpacked in exactly no detail on the status page. Practical check is to compare the tree of each merge commit in that window against the tree of the rebased-and-tested commit that the queue ran CI on. GitHub Enterprise Cloud with Data Residency is still affected as of the latest status update, so DR customers shouldn't clear new queue merges yet. Would be kinda useful if GitHub published the specific failure mode, because content-drift and parent-linkage and dropped-changes all need different audit scripts.

3 days ago

matthewbauer

Might want to check your git logs if you are using the Merge Queue feature. This afternoon, we found that Merge Queue was silently reverting changesets in the merge queue. Acknowledged by GitHub, but could be a very hard problem to debug if you aren't looking for it.

3 days ago

EdwardDiego

Same here, ffs GH.

3 days ago

MarkMarine

4 people spent hours putting our repo back together at my company after this. GH has been unreliable and now they are breaking the core tenant of what I expect from this service.

3 days ago

ragall

> core tenant

tenet not tenant

3 days ago

mazeez

[flagged]

3 days ago

MarkMarine

You must be fun at parties

2 days ago

ragall

I don't care to associate with people who are offended by being corrected. They can sod off. And I never go to what you Americans call "parties".

a day ago

MarkMarine

This type of pedantry doesn’t add to the discussion. You’re correcting someone’s spelling on the internet, it’s pointless except to make you feel superior, adds nothing. Make a salient point, bring something to the discussion please. Sorry I hurt your feelings bud.

8 hours ago

acid__

Tip: identify PRs with this problem by looking for discrepancies between the metadata (files/lines affected) for PRs via GitHub API and associated commits in the time window.

This was a pain to clean up!

3 days ago

NelsonMinar

How did a bug like this get past their testing?

3 days ago

philpem

Given the push to AI/Copilot from MS I'd be impressed if they still did any testing.

3 days ago

estimator7292

They're at 88% uptime this year. I don't think they have testing.

2 days ago

tracker1

Microslop®, unmatched quality.

2 days ago

EdwardDiego

This has caused a morning of fun for our team, how do you break one of the most fundamental bits of your system? Time to look at alternatives...

3 days ago

ramon156

Self-hosting has been wonders. I thought I needed the social part of GH (i still kind of do, I like seeing what people are working on or liking) but overall its the same experience.

3 days ago

tracker1

I experienced a weird bug, where the github ci environment was checking out code that was throwing an error on build... the version tag definitely didn't have the code the error was coming from... I messed with the CI workflow yaml and finally cat'd the file in question... the error code was definitely there (2 lines repeated in the file) that was not in the repository.

I nuked the release tag and re-created it, and it started working... was just a really weird bug/experience.

2 days ago

sidewndr46

Github has been doing this for at least 5 years now. Merging a pull request merges whatever the latest commit on that branch is in the specific backend that handles the merge request. It does not merge what was just pushed into that branch.

3 days ago

jonnonz

Whats going on over there, is clippy in charge of the release button?

2 days ago

orpheansodality

for other folks currently in an incident trying to resolve the chaos this caused, the first commit we've found with issues in our repo is from ~10:30am pacific this morning

3 days ago

classified

How long will users put up with this before they finally leave? Microslop is inventing new failure modes on a weekly basis.

3 days ago

[deleted]
3 days ago

minraws

Wait what? How, what? This doesn't compute

3 days ago

nextlsj

[dead]

2 days ago