GCP automatically lowered our quota, caused an incident, and refused to upgrade
Comments
yashg
neilv
They lost a sale, but they got competitive data.
Maybe that's how that salesperson or their org thinks?
nordsieck
> Maybe that's how that salesperson or their org thinks?
Nah - it's just a numbers game. Sometimes they'll be wrong; that's a dead end.
Sometimes they'll be right. And some of those times, they'll make a sale.
MichaelZuo
I don't see how?
Assuming yashg has long term memory and will talk to coworkers and other acquaintences in the future about this topic, the one case of being duped will lead to hundreds of potential customers being more skeptical of GCP in the future.
The one missed sale here undermines the future prospects of hundreds of possible sales.
hansvm
That's true for GCP as a whole, but for that one salesperson the numbers probably do still make sense.
neilv
Given how hard it is to do enterprise sales as a startup vendor, it's disappointing to think of some salesperson, repping a big brand, just doing a "numbers game" drive-by spray, and seeing what they hit.
yashg
Cloud platforms are very much commoditized with all of them offering same core products at similar price points. Just different fancy names. Also shifting a cloud provider completely is not worth the small savings in most cases. So these days they don't pitch you to move entirely, they ask you to consider their platform for any upcoming projects.
One more trick used by channel partners it to move the billing under their name instead of billing directly with the platform. They promise to give a % of the commission they will receive from the platform back to customer. So they promise savings that way. They also promise to optimise your cloud infra, shut down unused instances, downsize some services that are not fully utilised and getting reserved instances for ones which are not yet on reserve. And for some companies it does end up with significant cost savings without actually switching the provider.
yjftsjthsd-h
> One guy once said if I can show him our AWS usage he can give me exact configuration in GCP and cost would be 20-25% less. I was intrigued so I shared. He came back with a price estimate higher than our AWS bill!
Now I have to know more! Did you call him on it? Did he have any sort of explanation/excuse?
yashg
I did call him out. He knew he was not making the sale on price factor so he became nonchalant. Yeah, this is what it will cost you on GCP. I think he had given up.
aledalgrande
I know that some of the same "partners" try to convert bigger companies/department with the promise of building something custom for them on top of GCP so that they can transition quickly out of AWS. What they don't say though is that after you are on GCP you are going to get zero engineers to support that custom product (because it doesn't get them a promo) and you are left to deal with spaghetti.
Demmme
Just an anecdotal counter point: very happy with gcp.
Best network from all and coherent modern ui.
Not the usability hell like azure... (You know when clicking on a often used resource on the start page which let's you jump directly to it but doesn't allow you to jump a level up of all the other resources of the same type which totally works fine when you navigate to it the normal way... Or the huge hassle and complexity of resource groups for f everything...)
But you know the tweet not even states what quota was reduced.
faizshah
I love the GCP UI and the DX in general of their cloud offerings. I also love the feature set of Cloud Run, BigQuery, and DataFlow.
My only problem with GCP is that their support is horrible, I much prefer AWS support that’s why I can’t use GCP beyond my hobby projects.
I’m trying out Cloudflare workers this weekend so we’ll see how that goes.
CommanderData
Azure's support has also been fantastic. Depending on priority of ticket, with an enterprise account things get picked up sometimes minutes after raising them.
Their AKS offering was a crap show during the first year of general release and I opened countless tickets and they were snappy at the response times which is interesting seeing at the scale they operate.
Judging by all the comments GCP seems like one to avoid? Which is a shame because I had a desire to train multi cloud. If they treat their support like the rest of their products then I'll advocate a different IaaS where I'm able to.
faizshah
I think most of the comments in this thread are probably about the lower support levels for GCP. I personally have never used enterprise support for GCP. For the lower support levels AWS is miles better than GCP. Not sure about the enterprise support levels. For the hobbyist/early stage, AWS support is the top pick for sure.
I read in a much older thread that GCP’s TAM and enterprise support is pretty good:
> Hey, thanks for all that you’ve done. My experience with GCP has been an incredibly positive one. GCP documentation has always seemed fantastic. Our TAMs were very responsive.
> GCP support has by far been the best support experience. I have to say that the initial days it seemed to suck. The UI was some 90s google group clone which wasn’t even accessible through the GCP console, it was its own separate site which I always found amusing. But over time, the UI and quality of support became more streamlined and predictable, and I consider it one of the best SaaS support experiences today.
> One particular incident I’ll never forget is a support person arguing with me why network tags based firewalls are better overall for security than service accounts based firewalls. I expected to have a very cut and dry exchange but the support engineer actually did convince me that tags are superior to using service accounts. I did not ever expect to have had such a discussion over enterprise support tickets.
brianwawok
I’ve been very happy with the support overall. Met some solid dudes. And I’m pretty cheap on the spend chart,
genmud
I would smash my head in a door before being reliant on google and their (lack of) support for any business critical services. It's fine as a secondary, but my god would it make me really nervous if that was the primary cloud platform.
TX81Z
I was happy with Azure but just got a bill with “other” charges for $1k on what is a basic VM used solely as a db replica.
Amasuriel
For what it’s worth there is a tabular view that has more item detail. Things like marketplace subscriptions can often appear as Other in the graph.
I’m pretty happy with azure. I’ve not used GCP but vs AWS I find permission management a lot nicer. I have yet to have a billing surprise that didn’t come down to me not reading closely enough.
What I don’t like is they sometimes lock private links (aka the ability to not have a service publically routable, only accessible on a vnet) behind premium SKUs, looking at you service bus.
richieartoul
Interzone bandwidth maybe? They used to not charge for that, but they started doing so recently
Shorn
Do you have any links to more info about this? I don't use Azure, so I'm not sure where I would look for something like this.
I've been aware of some enterprise projects that are running on Azure, where from the description of how their architecture worked, I couldn't understand how they weren't drowning in network cost. It makes a bit more sense now if I know that Azure weren't charging people for some types of network bandwidth.
elankart
This is certainly a troll post. I use all three cloud providers. GCP is the worst of all, just try their simple text to speech UI. It doesn’t work most of the times.
Don’t even get me started on a deployment story for GCP their deployment manager is deprecated and redirect you to use terraform.
I hate to swallow it but Azure was more usable and straightforward.
dimgl
> Azure was more usable and straightforward.
There is no world where Azure is more usable and straightforward. Just my two cents.
VirusNewbie
Azure has had multiple global outages for many services. I believe you can look at overall stability and see it's not in the same league as GCP and AWS.
predictabl3
Lol can you use ed25519 keys with Azure yet? I doubt it. It was and will continue to be a joke because MS can't pay or promote people that care before they leave.
bushbaba
Azure’s target customer group is non-tech enterprise. The type of group who generally buys solutions over builds them in house. Where needing ed25519 keys is not a common ask
_ktx2
Azure is like if a raging tire fire had a baby with Windows UI
Amasuriel
Do a lot of people really use the UI of any of the cloud services? Most people want managed infra, which means at least cli scripts if not terraform.
Not arguing the Azure UI is amazing, but it’s low on my list of concerns personally for cloud services.
wodenokoto
Yes. Most of our infra is setup using the Azure portal. MS customer satisfaction engineers constantly recommend we set up things using the portal.
AZ CLI is also terrible for interacting with Azure.
bushbaba
UI for r&d, terraform for long term ops
Demmme
And.i don't?
I have over 10 years of experience across all cloud providers and you just assume I'm a troll?
That's not a way to have a discussion in good faith...
bushbaba
Gcp used to have a big network offering lead. But not anymore. AWS has a more reliable network (hello no global outages) and there’s not the same performance gap since AWS and azure’s investments in private fiber connectivity panned out.
fnordpiglet
You lost me at UI. But I’m one who believes if you don’t write your software defined infrastructure as software, you’ll regret life pretty soon.
Waterluvian
A UI for cloud services is super helpful to have when exploring, troubleshooting, and noodling around. But yeah, massive red flag if anyone’s using it to deploy production services.
brianwawok
I deploy my sql with Ui cuz ugh… I do it once per year max, and it’s docker everywhere else.
paulgb
That’s a pretty myopic view of what a UI can be used for. I’m all in on IAC but still like having a UI to click around to observe state rather than memorize a bunch of CLI commands.
elguyosupremo
And even if I can get what I need via CLI it's nice that higher level people in the org can click around and get what they need without having to bother me to retrieve the data for them.
Demmme
We have 99% IaC.
I still don't mind upgrading a k8s cluster on the UI, monitoring it's healthy and than patching the tf code for the newest minor version.
davidgerard
flagged as obvious spam
Demmme
Spam from you?
textninja
[flagged]
seanhunter
I think they may be being downvoted not because their opinion is different but because usability of the web UI is possibly the least important attribute in choosing a cloud provider for a lot of people given the use of tools such as terraform.
LoganDark
They replied to an incident report with "works for me". (With an implied "sounds suspicious" in the last sentence, but I don't pretend to know whether that's what they meant.)
bluepizza
OP is being downvoted because his anecdotal defense of GCP due to nice UI is irrelevant to the link posted. And probably to cloud computing in general.
jiggawatts
A resource group is literally just a folder — a name — and is the best feature of Azure.
For comparison, any large AWS account is always a total mess. Just an endless list of randomly named things that are totally unrelated to each other.
If that is his criticism, he deserves the down votes.
I bet his desktop has a hundred icons on it strewn randomly where half of them are named “file.txt” and “document.doc”
deathanatos
Well, part of his criticism is that the UI is woefully inconsistent with regards to "upwards" navigation. If you navigate to a resource via the resource group, you can navigate back "up" to the parent RG. If you nagivate to the resource from the home screen, IIRC, you cannot. That's the criticism.
Yes, RG's are a pretty killer feature, and trying to understand the organization of resources in GCP is hard by comparison, and an utter nightmare in AWS. I'm not sure why he's knocking that. (And … GCP requires projects, which seem equivalent to the complaint against RGs.)
… that said … there are so many other things utterly and horrifically wrong with Azure that I wouldn't put them on a pedestal for resource groups. As much as I do like RGs.
(Also, "A resource group is literally just a folder" sigh, no, because they're not hierarchical. Azure goofed hard there.)
coredog64
AWS has Resource Groups: https://docs.aws.amazon.com/ARG/latest/userguide/resource-gr...
Resource Groups are supported pretty well within the CloudWatch ecosystem (you can create an AppInsights application from an RG and you can filter CW alarms by RG)
jiggawatts
In AWS, resource groups are just a special tag. In Azure they’re a part of the name.
In Azure you can have two resources with identical names, as long as they’re in different resource groups.
baxtr
They’re the top comment now. Sometimes it’s worth to wait and see how it settles.
HatchedLake721
UI in the world of terraform and pulumi where some companies even block UI access in prod?
kccqzy
Blocking UI access in prod is fine, but developers should be able to experiment rapidly and in an agile manner in dev environments. If you forcibly block UI access in those cases, developers simply do less exploration and stick to tried-and-true solutions, even when those solutions no longer suit the problem at hand.
throwaway2990
In AWS you can link 2 accounts for single billing and apply a budget to the account. Nice having a dev R&D account not attached to production.
Amasuriel
Azure has this too, you can have what they somewhat confusingly call subscriptions, which are logical units with their own billing, limits and permissions inside and organizations account.
throwaway2990
Ah yes I forgot about that. Don’t use azure much these days but we used to use that for dev vs production.
redman25
We had a similar thing happen with our company. Google had given a bigquery concurrent query limit extension to 300 queries. In January they removed the extension back down to 100 concurrent queries because they were introducing “query queues” that should cover the difference.
Unfortunately for us, query queues only affect spikes in concurrent queries and not overall throughput. We’ve been struggling with bigquery support ever since.
newhouseb
Quota management is indeed nonsensical. We serve large cash assistance programs that have stampedes of people applying all at once where we needed decent geocoding to determine eligibility. We were exceeding the default 50qps quota. I asked nicely (to double it, I think) and explained our use case and they said... no.
So... we left and switched to Smarty and haven't looked back since. We spend tens of thousands on geocoding annually. Really mind-boggling behavior from GCP.
betaby
What is 'Smarty'? Never heard about that company/product.
virtualpain
I too was confused, all Google search result was PHP templating library. So I re-read the comment and added "smarty geocoding" to display correct result.
newhouseb
Yeah, they used to be called SmartyStreets and then dropped the Streets. Not the greatest branding call.
parpfish
I’ve recently had an issue where GCP keeps shutting down my e2.micro because they think I’m crypto mining. I have no idea what I’m doing to trigger it, and I don’t blame them for not sharing their secret anti fraud heuristics, but it’s a freakin e2.micro. They think I’m mining on that?!
Nathanba
The cloud feels like a casino designed to fleece you. No way to set limits + no way to check your limit (via API). The price calculators are as complex as they can possibly be, just spin the wheel and hope for the best I guess. Price estimations for my instance in my month vary wildly from cents to over 1€. I just imagine what would happen if I actually spent money there and price estimations are 100€ per month when I check in the morning and then the next day for whatever reason x10 higher.
Currently my supposedly free micro instance on GCP is projected to costing 1.50€ per month. Obviously I'm very greatful for free computing, that's very nice. It's actually the main reason I'm even entertaining using the cloud for things and bothering to try things out. My complaint isn't about that, it's about the fact that their stated cost isn't the actual cost. It does seem to actually cost something. No, not network costs. It says "E2 instance core" and "E2 instance ram". It will be very interesting to see whether I get an explanation for this and whether they will take the money. In the first month they already took a few cents for the 2-3 days I ran it.
I don't know if irony is the right word but the GCP cost only has to go slightly higher and I can already switch off the cloud to a standard VM somewhere to have cheaper compute. Imagine that, I'm not a 100k/month spender like some here, I'm a free user and it's already almost cheaper for me to switch off the cloud.
bombcar
It might be worth trying that “perpetual free instance” that Oracle cloud offers. Apparently it’s actually free.
kyrra
I'd assume it's bandwidth costs.
missingdays
Why would they care if you mine crypto? You pay for resource usage, you use resources
aseipp
Fraud vector. It's very easy to turn compute into dollars if the compute is free or stolen. Cloud compute is very cost ineffective for something like coin mining, so any actual legitimate miners on any chain are going to just run everything in-house anyway. In the end, that means the people left in that pool predominantly fall into two groups: A) people using stolen credit cards B) accounts that had hacked credentials or leaked API keys running rampant.
Whatever heuristics cloud providers tend to use to discover and "remediate" such behavior is a totally different thing (e.g. obliterating an established account when an API key might have gotten leaked and a few GPUs let loose is a little overboard), but if I was offering a similar compute service, and its sole purpose wasn't just coin mining, I'd almost certainly ban it as well for similar reasons.
reaperman
It's a very convenient way to convert identity theft into cash. A huge amount of cloud crypto mining is paid for in fraudulent credit cards. Detecting and banning crypto mining greatly reduces % of fraudulent transactions as these attackers will move on to greener pastures.
They can also use an absolutely incredible amount of resources, because it's not their money. Two hundred A100 GPU's for a week? Sure, why not?
cavisne
Its impossible to profitably mine crypto on a public cloud so anyone doing it is either stupid, or has no intention of paying (much more common!).
Historically GCP/GAE has had a more generous free tier without credit card requirements, so they've always been a bit stricter on this than the other clouds.
professorsnep
GCE allows one e2.micro instance in their free tier
TX81Z
Maybe they’re waiting to finish the big roll out of NFT on YouTube!
herdrick
Providers often just ban all crypto mining. I think it might be because of pressure from authorities fighting money laundering.
Sebguer
No, it's because it's an extraordinarily common fraud vector, and causes noisy neighbor problem because despite GP's remark about 'paying for resource usage', cloud providers don't actually anticipate anyone using 100% of CPU 24/7/365. And the folks who really are crypto mining are never going to pay their monthly invoice.
Source: Worked at a cloud infra provider that struggled deeply with this problem.
jmaker
So you pay expecting full utilization but are expected to utilize on a fair-share basis only? And if I run my Monte Carlo simulation, which can take a week to complete, my instance is flagged for alleged crypto mining?
Sebguer
This generally depends on the tier of service you're paying for. Every infra provider offers tiers that provide dedicated resources. AWS handles this via CPU credits, other providers are less sophisticated.
Generally speaking you're not going to get banned unless you spin up max quotas and have other signals for fraud, such as zero payment history, connecting from regions with high fraud rates, etc. More often the provider will just limit your resources - either openly, like AWS does, or more subtly.
yjftsjthsd-h
> despite GP's remark about 'paying for resource usage', cloud providers don't actually anticipate anyone using 100% of CPU 24/7/365
Unless that's exposed to the customer (AWS burstable instances are actually okay IMO), that sounds like the cloud provider committing fraud and hoping nobody calls their bluff.
Sebguer
You don't know what the word fraud means, and I recommend reading these crazy things called 'terms of service'.
yjftsjthsd-h
> The crime of stealing or otherwise illegally obtaining money by use of deception tactics.
> Any act of deception carried out for the purpose of unfair, undeserved and/or unlawful gain.
(https://en.wiktionary.org/wiki/fraud#Noun)
Is the cloud provider getting money off of this arrangement? => Yes, obviously they're getting paid here.
Is the cloud provider getting that money as a result of deception? => Yes, by selling ex. the use of 1 CPU core when they actually have no intention of letting you use that core. Now, I'll grant that in an actual legal case it might well be possible to get away with this by burying it in the ToS, but this is HN, not a court room, so I'm comfortable setting the bar at "would most actual users expect that to happen based on marketing?", conclude that no, most users would consider that surprising even if there's something buried in the ToS claiming that it's allowed, and call a spade a spade.
parpfish
the problem is that the terms of service say "don't mine crypto", and I'm not. they've set up an imperfect detection mechanism and I'm one of the false positives but have no meaningful way to prove my innocence.
the worst part is that there's somebody in a GCP office somewhere that might secretly believe that I'm actually a cryptobro. shudder
sigstoat
> the worst part is that there's somebody in a GCP office somewhere that might secretly believe that I'm actually a cryptobro.
you're probably safe on that front, it isn't like the GCP folks are paying any attention to the customers.
cowsup
It's more akin to a gym membership. You can sign up to a gym promising you 24/7/365 access, but if you and a couple buddies decide to walk 1mph on the treadmill to work on your laptops from dusk til dawn, preventing anyone else from using them, you'll be asked to leave.
In either case (gym or GCP) you'll be entitled to a refund for the difference. They aren't allowed to just kick you out and pocket the difference.
sidcool
Why does GCP care if you are crypto mining as long as you are paying them for the CPU/GPU, memory, bandwidth and storage?
crescentfresh
Anecdotally, I know a paying GCP customer that recently got quota blocked on the number of global static IPs they are allowed to create. We could not figure out how to contact someone to increase it, and nobody knew why the quota was set so low (4). We just kept running into automated systems that gave circular information on how to request an increase. We kept getting links to sign up for a free tier account.
elguyosupremo
Search for "all quotas" in the GCP console, from there you can view all your quotas and request increases.
crescentfresh
Right. Unfortunately their right to request increases was removed. The exact wording: “Based on your service usage history you are not eligible for quota increase at this time” and then dead ends trying to reach someone.
They had billing accounts set up for all their projects. They were happy to hand over money but had no ability to.
ktta
Are you by chance referring to anycast IPs? Having a limit of 4 for static IPs sounds so incredulous
robin_reala
There was a follow-up Tweet that “explains” the problem:
https://twitter.com/JustJake/status/1667492928758095872
From the call with Google just now:
Them: "You exceeded the rate limit"
Us: "We did 5000/10min. The quota was approved at 18k/min"
Them: "That's not the rate limit"
Us: "What's the rate limit"
Them: "Not sure have to check with that team"
So, the quota is completely cosmetic...
holografix
A friend works at GCP and has told me this time and time again: if you don’t buy support from GCP _you have no support_.
azmodeus
Even if you buy support it is not comparable to AWS. I suffered GCP Dataflow support in the past it was: 1. slow 2. condescending 3. not helpful Really don't recommend GCP for long term use. If you get some free credits it's good to use it to train some ML models but AWS is much more customer friendly.
zeroclick
Based on the comments in the thread, it seems that even if you do buy support, you also have no support.
brianwawok
Buy support and generally disagree.
willtemperley
I had to close my GCP account because I was being spammed with "Action Required" emails every day for a non-existent tax admin issue. Bug reports were immediately closed and getting through to a human seems impossible.
jtchang
Why don't I hear about these incidents with AWS? Is it just less common or different business takes on quotas and rate limits?
oldtownroad
Amazon hate their employees. Google hate their customers.
neilv
Best of both worlds would be a place that takes great care of both customers and employees.
pclmulqdq
How about someone that hates both? That would explain Microsoft's rise in cloud.
yjftsjthsd-h
Well, yeah, obviously if you can have the best of all worlds then by all means do that, but in practice something's going to give; if you did somehow prioritize customer and employee well-being, it would almost certainly manifest in much higher prices, not actually sustaining quality, or both. (Now, charging more for a better outcome does seem like a good idea, but in practice >90% of customers will go for the cheaper option.)
robocat
Most people seem to equate employee well-being with more money.
Plenty of people care more about other job qualities than how much it pays, and a well run business can create a great environment without paying more than average. Look for companies where turnover is almost zero (assuming company is not growing) and you often find happy employees.
Places with worse employee conditions have to pay more, everything else held equal.
Anecdotally, I have certainly stayed in jobs with poorer pay because I liked my colleagues and the working conditions; but alternatively I have stayed in another job mostly because the pay was great.
koolba
No man can serve two masters: for either he. will hate the one, and love the other; or else. he will hold to the one, and despise the other.
iot_devs
I personally reply to customers inquiries on a very famous AWS product where the client was asking why the execution was interrupted after the timeout they set themselves expired and why it was a timeout error.
I am an engineer working on such products and it was a routine on-call shift where I got this kind of question.
Admittedly we should have this cover by our support engineers, that are very very good, but this one slip through and I took the time to answer such query.
I am not a fanboy or anything, I could not be further from a fanboy or a blind fan.
But after working on AWS, I do suggest it as very sensible choice. Again, not because it is my employer but because they really take care of operations and customers.
Screw ups can happen but they are very rare in my experience.
gurchik
My only problem with AWS support is that they start every ticket assuming it is a basic, RTFM-type of inquiry and it can take a bit of effort to guide them toward more helpful responses. Once you get there they eventually get to your answer.
Here is a ticket I opened recently (paraphrased):
> Me: I'm seeing an S3 charge in my billing and usage report called "AMZN-Out-Bytes" what does this mean? I already read the list of billing codes in this page (link) and it's not there.
> Support: Actually the majority of your report is "DataTransfer-Out-Bytes" charges which is caused by egress traffic to the internet. For more information see (link I already gave them).
> Me: Thanks but I don't need assistance with that billing code, please explain "AMZN-Out-Bytes." And can you explain how it's different from "AWS-Out-Bytes" as explained in (link)?
> Support: As explained in the page you linked, "AWS-Out-Bytes" refers to data transferred between different regions in the AWS network.
> Me: Please explain "AMZN-Out-Bytes."
> Support: Please close this ticket and reopen it with the S3 team. (who ended up answering my question)
In comparison, Google support could only be described as hostile, seemingly looking for any reason to close the ticket as my fault.
rhtgrg
> Is it just less common or different business takes on quotas and rate limits?
Neither, you're probably not seeking out such stories (there are plenty, even on HN). Google tends to get more negative attention on this particular forum, which might play a role, but the larger component is probably just chance.
CSMastermind
I disagree that it's simple selection bias.
AWS is _extremely_ customer friendly and if this happened would likely be offering dedicated support to make it right, credits for the business loss, etc.
Google's customer service is the worst I've experienced in the industry (like even speaking to a person is hard). While AWS is some of the best I've received.
kenhwang
I've seen my org send a silly amount of silly questions that were obviously our fault to AWS support and they always took it in stride. We very rarely needed to follow up with our client executive (but they always offer to jump in if necessary). When we did have serious issues, we had no problem scheduling time with the executive managing the product and the engineers that wrote the code.
Meanwhile we can't even get Google to answer an email about serious incidents that were very obviously their fault, much less assign a dedicated human point person (hell with AWS, we even had several backup contacts assigned to cover for vacation time). We were important enough to be featured on their client success frontpage, but that didn't make a difference in support quality we received.
I can't imagine why anyone would risk their business with GCE. Especially since it tends to cost more than AWS these days.
whoknowswhat11
Yes, AWS being patient with both customer fault (common) issues and even out of scope for support tier level is crazy - it’s got to be a purposeful cost center expenditure. I like some things about google w project model but we spend on AWS (not too much).
slowmovintarget
Amazon worked very hard to build a culture of customer service. Google worked very hard to create tech and create a developer-first culture.
You're seeing culture in operation.
rhtgrg
> AWS is _extremely_ customer friendly and if this happened would likely be offering dedicated support to make it right, credits for the business loss, etc.
It doesn't take much legwork to find counterexamples of that claim, even on HN (see below, I spent 2 minutes searching to find those).
> Google's customer service is the worst I've experienced in the industry (like even speaking to a person is hard). While AWS is some of the best I've received.
This is a bit too hyperbolic for me, but on balance I agree that Amazon has better customer experience than Google. That doesn't really answer what GP is asking, especially given all the posts made about AWS on this very forum that only sometimes get attention.
https://news.ycombinator.com/item?id=2478129
https://news.ycombinator.com/item?id=25224220
marcinzm
>That doesn't really answer what GP is asking, especially given all the posts made about AWS on this very forum that only sometimes get attention.
If these are your examples of angry AWS posts then I think it just proves that AWS has amazing in customer service.
>https://news.ycombinator.com/item?id=2478129
That's 12 years old but a legitimate complaint about AWS customer service.
>https://news.ycombinator.com/item?id=25224220
Sounds like AWS had an outage, not sure how this is about customer service?
>https://news.ycombinator.com/item?id=35375558
That's a list of AWS customer's having security incidents and, again, nothing about AWS customer service.
>https://news.ycombinator.com/item?id=16283547
Not sure what this is about since the link is dead except due to being 5 years old but, based on the upvotes, it's someone failing to get people to have angry opinions about AWS? Not sure how this is about AWS customer service given the context.
rhtgrg
Given that this isn't handpicked and just top results from a 2-minute search:
> That's 12 years old but a legitimate complaint about AWS customer service.
Should have at least prompted you to do your own search. The fact that it didn't is the end of this discussion.
marcinzm
So you made a claim that there's "there are plenty, even on HN" and then utterly failed to provide any proof even after trying. Then when called out you blamed the other person for not doing so on your behalf. I'm going to stick with my response.
neilv
I'm sure that GCP could turn around their HN customer service reputation, if they wanted to prioritize that.
Though, when it might not be baked into the culture as much as at AWS, I don't know offhand how to reconcile a customer happiness turnaround with the rush to avoid being an also-ran in the AI deployment frenzy.
(Are you going to assign conflicting KPIs in a company that has cultivated a career-driven culture, and hope that individuals strike optimal balances for the company? Or partition the goal assignments to separate teams, when the optimal outcome requires holistic thinking and behavior across teams?)
etse
Wasn’t this supposed to be Kurian’s prerogative—sell enterprise services? Google product seem fine to me, but I agree that they have no customer care DNA. I was expecting someone from the outside to bring it in.
pram
I worked in Kurian’s org at Oracle and I was shocked when Google hired him. OCI was a total disaster at the start. Like embarrassingly bad, they threw most of the original platform in the trash.
Literally failing upward.
bushbaba
AWS would also have a GM be involved for such an incident, and if this happened more than a few times the GM would risk being fired.
Amazon really is customer obsessed. If a department fails its customers that departments leadership will be replaced
JoshTriplett
I've seen a lot of incidents reported about AWS involving quotas, in the other direction: "I can't sign up new customers because I'm at a quota limit on resources", sometimes with "and AWS is taking a long time reviewing a quota increase", with the occasional "and AWS denied a quota increase".
marcinzm
Google wouldn't increase my quota unless I talked to a sales person. I wanted 4 GPUs. 4. That took at least a week to achieve.
coredog64
AWS quotas come in hard and soft varieties. A hard quota cannot be changed as it’s factored into service operations (e.g. APIGW’s 29 second integration timeout). A soft quota may or may not require review before being granted. However, if it is a soft quota, the best thing to do is escalate to your TAM after opening with the business impact of the request. The TAM is paid out of your enterprise support contract, so the least they can do is advocate for you.
cyclotron3k
My problem with the quote system is that it's largely opaque until you run into it. Yes, you can pore through the documentation and find the relevant quotas, but did you find them all? Spoiler alert: no, you didn't.
I know they are trying to improve visibility on quota limits, and they have a tool now, but in my experience it was half baked and only knew about a handful of the limits we were running into.
joecool1029
Some of the AWS services have clear quota limits and it works as expected, you give me 10k widget credits and if I use 10k widget credits it stops working and I need to open a ticket. Example of this would be SES.
Where it ends up being a bit of a nightmare is stuff like SMS on SNS where they would specify a quota but we could still hit it before their system reported it. We never did figure out if they were looking at a rolling monthly average and bursty campaigns could cause it to creep above their projection. This was always a manual review for this product and the only way we could avoid it was getting way higher quotas than we needed approved. Ultimately we ended up moving SMS over to Nexmo (now Vonage) for bulk SMS so we didn't have to have potential outages when the mystery quota of the now was reached.
bushbaba
Many quota systems are measured in qps cannot exceed the avg allotment. Hence yes, short lived large bursts deviating might not be allowed. Docs on the quota usually clarify this.
throwaway2990
They have a quota page in aws console which displays all quotas. Defaults are in all the documentation. And quota information can be queries from the cli and will tell you if it’s adjustable or not.
They have a support page for quota changes and are usually actioned in around 15-30m on average.
What more do you want?
adrr
Because Amazon will jump through hoops to make the customer happy. If I file ticket to get a quota raised it usually happens the same day and I have never had them push back on me.
theptip
Ran my startup on GCP. No regrets. Can’t recall a time I waited more than a few hours for a quota bump.
Only time I ever got pushback was trying to get them to make an unreasonable quota increase to work around a bug in early GKE that left stale network backend lying around, and when I explained they approved it.
I’ve found GCP support to be reasonable, but you have to pay for the enterprise tier. Their pricing model used to be insane ($250/seat??) but they fixed that a few years ago. I suspect a lot (not all!) of the complaints around here are for hobbyist / free tier, which sure, is garbage. If you spend $K/yr on support it’s fine though.
aledalgrande
Same. As a complete nobody account I got access to (more) large GPU instances, even before the GPT era, without even getting asked why I needed them.
balls187
AWS is great.
Gotta check out FBA/Amazin Marketplace to find all the nasty on Amazon.
elankart
In what world is AWS great? A world where they don’t have a clean way to group resources and clean them up? Their own twisted identity products in AWS that doesn’t integrate outside their cloud or their other products. A world where they leave lingering resources when you cleanup something.
AWS has to much hype riding behind it.
It’s not enterprise class and looks very incohesive.
ripper1138
Is this comment from 2015? AWS just had a $20B quarter.
whoknowswhat11
The point about lack of projects on AWS I agree with if that’s what you are talking a about. That said AWS is absolutely a cloud leader. We use azure and AWS now and there continue to be reasons AWS does well despite cost and quirks
balls187
> In what world is AWS great?
Mine. And Gartner’s.
I use GCP and AWS, and AWS outclasses GCP handily.
ineedasername
Hilariously reminiscent of Kafka:
Client: we were under the limit
GCP: No, you exceeded the limit
Client: Then what is the limit?
GCP: I don’t know
endisneigh
Tweet has basically no details
Havoc
That's presumably how they felt too...
invalidname
From my past experience Google is by far the worst when it comes to support. By far. Even if you pay for their gold support you're lucky if you reach an engineer and even then they blame you for the problem with no "proof" or direction. Most unhelpful ever... See: https://medium.com/hackernoon/why-and-how-we-left-app-engine...
AWS were surprisingly helpful by comparison and even pro-active. Not a fan of theirs overall but much better than Google. This is a deep cultural problem with Google that also expresses itself in mobile development and everywhere: https://dev.to/codenameone/google-play-kafkaesque-experience...
They aren't the cheapest and their service is terrible. I can't think of a good reason to use GCP.
Spooky23
Ymmv. I found a bug in their CA/PKI service, sent an example in and got a callback and feedback from a knowledgeable engineer who wrote me a workaround and put in a change request. I think the bug was fixed in a few weeks.
My team works with lots of companies, Google is one of the better ones in our experience. AWS is also excellent. I’m told Azure is good, but most of my experience with Microsoft is with enterprise product support, which is awful.
faizshah
Which Support level do you have with GCP?
I haven’t used GCP support at work yet only as a hobbyist, sounds like maybe you had their enterprise support tier?
fastest963
I've created several tickets against both AWS and GCP and had good and bad experiences with each. I actually used to have mostly bad experiences (a lot of useless back and forth, wrong answers, etc) with GCP but the last few years it's actually gotten significantly better. For example, we found a bug with autoscaling specific instance types (it's now posted as a known issue) and it was acknowledged and workarounds suggested without much hassle.
textninja
I find the self-serve UI and overall experience with GCP to be superior, so to me it’s a bit like the premium you pay for a Mac over a PC. Granted I never needed or wanted to speak to a human, but their infamy in that respect deserves a special nod of acknowledgment.
zmmmmm
That's sort of the flip side of the coin. Google firmly believes that technology can solve everything and humans shouldn't need to be involved out of principle. So they build a great UI, great tools for devs and engineers to use, and then provide absolutely zero humans to support that.
I have a hilarious situation at the moment where they decided to shut down my Workspace account because they retracted the grandfathered-in free tier. This has 15+years of history of a small business I ran in it. I tried to move it to a paid account, but because the account was created in a different country to my current billing country, the form doesn't work (can't enter address for credit card). And there's just literally NO way to do it. No avenue to any way to contact a human to quite literally give them money. So in the end I gave up : downloaded the emails and resigned myself to losing all the other history associated with the account.
But what's really hilarious is they can't seem to shut it down either, perhaps for the same reason. Every time they set a new billing deadline it warns me with another spate of emails and then the date goes by and its still there. I suspect their software doesn't know how to shut down a hybrid multi-country account either, and there are no humans so they are stuck.
justinclift
Sounds like you have the rarest of rare problems, being an immortal Google account that just won't die. ;)
whoknowswhat11
I had this exact type issue for a charity account. A bit higher impact given team size but NO WAY to talk to a human!! Or if we did they couldn’t do anything. I imagine this has since changed, but it does leave a sour taste
showdeddd
This medium article seems silly, just add custom metrics to your app for what was fetched from cache vs DB. And label the metric by route/query/pattern. To control costs, don't tick the metric for every single request, instead accumulate locally and post to metrics API every X minutes.
invalidname
That didn't exist back then. Adding logging all over the application would increase costs while losing a significant amount of money.
The problem is that Google had no problem billing for a metric that they didn't expose to the user and didn't provide tools to debug properly.
showdeddd
1. No it would not be expensive. Writing metrics is literally free: https://cloud.google.com/stackdriver/pricing . If metrics didn't exist yet then log 1/1000 requests to control the log volume and find the pattern from the averages.
2. It isn't up to google to tell you if you are querying against the cache or the DB. It's your code. You should know. Just tick something when you use the Redis/BQ/GCS/SQL client.
This task is so easy I would assign it to a junior engineer and expect code changes done in one day!
invalidname
1. Logs are not free. Stackdriver was not an option back then.
2. If the phone company charges me extra they tell me which numbers I called. Here a cache miss started happening. Only in production with no tooling available (at the time) to determine why this was happening. A single number of "data read" was all the information given. Not even the table name... That means you end up looking for a needle in a haystack.
I'm guessing you work for Google because your attitude seems similar. No they don't *have* to provide that service which is exactly why they suck. A service oriented company would make the *effort* to provide a user with this information. Especially a paying user at gold level.
tyingq
> accumulate locally
Is that an option in AppEngine? The memcache docs seem to indicate the free tier has undocumented eviction policies.
showdeddd
You have to implement that with your own code but it isn't much more than a dict/map and a timestamp for last update.
tyingq
Same-ish problem, though. You wouldn't know for sure the instance will run again...your dict/map data can be dropped. I don't see any sort of instance timeout callback where you could guard against that.
showdeddd
I think it's negligible. The only metrics you lose are on rare scaledowns and they are averaged out anyways. GCP likes to keep instances idle for a long time.
brigadier132
I'm only considering them because of GKE autopilot and cloud run which both seem like completely painless ways to deploy containers (I've setup prototypes with both).
klodolph
In my limited experience, quota is one of the pain points of GCP. It's where the cloud "leaks".
mvdtnz
Quota for what?
Spasnof
Warning this is rando internet detective work so YMMV JustJake at twitter would likely know the actual answer to this question.
It was mentioned that the service was artifact registery[1]. They were 5000/10min and approved at 18k/min [2] The ui appears to show a limit of 100 but not what of [3].
So reading the docs on artifact registery quota docs the only things that stands out is the "1 repository creation or deletion operation every 2 seconds."[4] so doing the math that is a rate limit of 30 per minute or 300 per 10 min so aroudn the ballpark of what they are. Looking at my own project this matches up and I can add request new quotas[4] but it needs to go through a human on googles end to approve.
Anyhoo if they were approved for 18k/min that is nearly 600x the normal quota. So they were likely (pure speculation) caught up in some kind of dragnet of quotas that were well beyond the norm. Skimming the site[6] it appears to be a rapid cloud based ide / deployment thing and I can't imagine how they are creating so many repos unless for each customer build they pass along to gcp's artifact registery 1-1. I use artifactory repo as well for managing docker images but mostly just make new tags off a handful of repos and am not typically creating them at such scale.
Sucks that they had the quota lowered without warning but "if" they were operating at the speculated quota that is flying pretty close to the sun.
[1] https://twitter.com/JustJake/status/1667660212902453248 [2] https://twitter.com/JustJake/status/1667492928758095872 [3] https://twitter.com/JustJake/status/1667478906591666176 [4] https://cloud.google.com/artifact-registry/quotas#project-qu... [5] https://console.cloud.google.com/apis/api/artifactregistry.g... [6] https://railway.app/
kaliszad
One thing I don't understand is how in 2023 people use GCP with the bad level of IPv6 support they have. Other Google services usually support IPv6 quite well. Only GCP, to some degree Chromecast, probably Nest and the DHCPv6 problem of Android are some well known exceptions.
Google also tend to put algorithms first and people and their business second so that is why I would be very particular about using GCP for anything serious. We do use Google Workspace and for what we do with it, it is mostly ok, however rather expensive.
aledalgrande
I've never used GCP for anything but demos, but on the other side AWS support is worth the (high) price: I always had my issues solved, quickly, and sometimes I was even able to put in feature requests that were shipped a few weeks later.
profwalkstr
It feels as if Google is intentionally trying to sabotage its own cloud business
Animats
Google giveth, and Google taketh away. Blessed be the name of Google.
civilitty
To our queries, it lights the way. In its Matrix, we are but a doodle.
Blessed be the name of Google, come what may.
pawelduda
Wonder what does GCP side have to say on this
mmanciop
[flagged]
slig
Computer says no.
uoaei
It is interesting to note the differences in apparent defensibility of companies such as Google and Reddit changing terms of use and behaviors of platforms.
According to the general sentiment on HN, Google is being mean to developers and shouldn't get away with things like this, but Reddit merely has a rug and developers are silly to build things on top of that rug when it can be pulled away at any time.
I wonder why there's a difference here.
anon84873628
Well, GCP is literally selling a "platform" to developers. Whereas Reddit is a consumer website?
uoaei
So?
exabrial
Once Again: Do not use Google for anything. Given they treat the consumer side of their business like cattle (Gmail, maps, etc) (,and despite the fact that an email address is an essential part of one's daily life: access to banking, investments, communication, cell phone, employment opportunities), they operate with absolutely zero support.
What makes you think they'll treat your small business any differently? You will be thrown under the bus the second you make them lift a finger to help you.
chlorion
Google services work perfectly for me.
I think the users of this website just really like to repeat anything that fits the "Google bad" narrative, and if you don't have interactions with people outside of the bubble, it seems like Google services are on fire and practically unusable, but in reality it's fine for 99% of people.
Arch485
From my experience, Google works perfectly most of the time, but when it doesn't work you're completely screwed and have no recourse. You just gotta hope the problem fixes itself.
x86x87
It works until it doesn't. Do you want to gamble with your business?
nathants
the purpose of not aws is two:
- put price/quality pressure on aws
- fill weird niches made by regulatory capture and other nonsense
not using aws is like not using linux. there are valid reasons, but you don’t want any of them.
aftbit
Well unless you just want to run your own hardware. That can be orders of magnitude cheaper (business trajectory changing stuff, think offering a free plan vs not) for the right kind of workloads.
nathants
optionality. having aws doesn’t mean using it for everything.
for most uses cases, netflix model seems like the right one. control plane on cloud, data plane on not cloud.
at a minimum you probably want to backup some high value data in s3 unless you have a more durable store somewhere.
cshokie
Or your business competes with Amazon and you don’t want to pay a competitor in order to compete with them.
showdeddd
This article seems silly, just add custom metrics to your app for what was fetched from cache vs DB. And label the metric by route/query/pattern. To control costs, don't tick the metric for every single request, instead accumulate locally and post to metrics API every X minutes.
I keep getting badgered by GCP channel partners to move over to GCP. One guy once said if I can show him our AWS usage he can give me exact configuration in GCP and cost would be 20-25% less. I was intrigued so I shared. He came back with a price estimate higher than our AWS bill! I haven't entertained them ever since.