Google outage – resolved

2316 pointsby abluecloud 2109 days ago827 comments

various services are broken - youtube returning error - gmail returning 502 - docs returning 500 - drive not working status page now reflecting outage: https://www.google.com/appsstatus -------------- services look to be restored.

Discussion

Loading discussion

pachico · 2109 days ago
Yup, suffering it right now.
manquer · 2109 days ago
Yeah hangouts/meet as well
niklasmtj · 2109 days ago
Same here in Germany. Gmail says that my account can not be found. Plus Google's support page is also down
asdojasdosadsa · 2109 days ago
Can verify - Finland
nashtheman · 2109 days ago
Yeah, drive and youtube seem to be down as well
_1tan · 2109 days ago
Drive too. Status board still green though.
Jumziey · 2109 days ago
Exciting!
0x54MUR41 · 2109 days ago
Yes, I can confirm this. I got an error when accessing Google Docs, Google Meet, and Gmail.
clusmore · 2109 days ago
Australia, same problem here.
domoritz · 2109 days ago
Yep, drive and gmail give me 500 errors.
noutella · 2109 days ago
Alors, Search queries taking up to 3 seconds here in France.
opdahl · 2109 days ago
Down in Norway too.
madjam002 · 2109 days ago
Google Home not working in UK
degosuke · 2109 days ago
Anyone else has this in the technical details? Numeric Code: 7444
- janpot · 2109 days ago
  7498
dkjaudyeqooe · 2109 days ago
Pretty much anything that requires login is broken.
maxekman · 2109 days ago
Meet is also down. From Sweden.
Avalaxy · 2109 days ago
The status page has proven itself useless again. According to that page, everything is working perfectly fine.
- kiechu · 2109 days ago
  What if it is a static website of corporate propaganda?
  - username90 · 2109 days ago
    It works 99.9% of the time, good enough eh?
    - RupertEisenhart · 2109 days ago
      better than a stopped clock
- daliusd · 2109 days ago
  It works fine. You can't login only.
tunnuz · 2109 days ago
Same (Austria).
proto-n · 2109 days ago
Hungary - gmail loads, can't send email (error #2014)
lwhi · 2109 days ago
Google home down too!
pvsukale3 · 2109 days ago
Google meet is also down
ivan1783 · 2109 days ago
Can confirm from Germany, youtube is broke.
tosh · 2109 days ago
Just checked https://www.google.com/appsstatus all green, which does not reflect reality for me (e.g. Gmail is down) edit: shows how incredibly difficult introspection is
- faeyanpiraat · 2109 days ago
  HN is my goto status page when these things happen, never failed to provide up-to-date reliable information.
  - jansan · 2109 days ago
    Google should write an AI application that checks on HN if their systems are working and display the result live on their status page. That would be more reliable than the current system they have in place (which is obviously not working at all).
    - jaimex2 · 2109 days ago
      It would probably just end up banning Googles own account and leave them with no human escalation option.
    - gostsamo · 2109 days ago
      won't help when the ai service itself is down.
  - polote · 2109 days ago
    Seems like everyone going on HN now. The site is so slow
    - davnicwil · 2109 days ago
      Interesting cascade effect of sites going down! I wonder where people will go next to check if HN is down? Also wondering if this is perhaps the fastest upvoted HN post ever? 8 mins -> ~350 votes, 15 mins -> ~750 votes. I wonder if @dang could chime in with some stats? Update: looks like it hit 1000 upvotes in ~25 mins! Update: 1500 in ~40 mins Update: 2000 in ~1 hour 20 mins (used the HN API for the timestamp)
      - TonyTrapp · 2109 days ago
        The recent youtube-dl takedown had a similar number of votes but still slightly slower, I think. I think it was >500 in 30min and >1000 in 60min if I remember correctly.
      - SSLy · 2109 days ago
        reddit, nanog, 4chan, twitter — pick your poison
        devenblake · 2109 days ago
        "poison" is a particularly apt description for all of the above
      - Macha · 2109 days ago
        Some politics topics have shot up pretty quickly in the past but user and/or mod flags send them back to page 2 about as quickly.
        throwawaylolx · 2109 days ago
        Funny this post is already #7. Are we seeing many reports again? Stats right now: 1985 points | 1 hour ago | 597 comments
        Macha · 2109 days ago
        Maybe, could have just seen a dropoff in upvotes once the issue was resolved also
      - mrtksn · 2109 days ago
        > I wonder where people will go next to check if HN is down? Probably: https://downforeveryoneorjustme.com/
    - tuxone · 2109 days ago
      Accessing the site anonymously fixes it (logged sessions are not chaced)
    - marcbarbosa · 2109 days ago
      HN uses Firebase, maybe that is the reason.
      - rmujica · 2109 days ago
        TIL. What for?
        porkbrain · 2109 days ago
        The very least public APIs are exposed via firebase. https://github.com/HackerNews/API
      - qwertox · 2109 days ago
        Does this site also use Firebase, or do they only feed Firebase's Database so that other developers can build upon it?
      - ratww · 2109 days ago
        Any source for that? Last information I have is that it's written in Arc Lisp and uses files rather than a database for storage. There is a public API on Firebase, but AFAIR it's just a mirror rather than the main storage.
    - ksec · 2109 days ago
      HN was having problem showing anything for the past 10 min or so. I was going to ask if HN was down, then I saw this.
  - throwawaylolx · 2109 days ago
    For many others it seems: https://i.ibb.co/ns8DgGq/Screenshot-2020-12-14-at-12-10-17.p...
  - sammorrowdrums · 2109 days ago
    I, and probably many others here are like a status page for friends and family too... I had my wife thinking that the internet was broken, and tried tethering to phone and things and still didn't work, then showed me and I saw the status code errors and was like "it's actually parts of Google that are down". I love (and am deeply scared by) the dependence of Google and the confusion of it with the entire internet.
    - throwawaylolx · 2109 days ago
      It gets even worse: https://twitter.com/joemfbrown/status/1338452107419148290 >I’m sitting here in the dark in my toddler’s room because the light is controlled by @Google Home. Rethinking... a lot right now. Some people are compiling more relevant events: https://twitter.com/internetofshit
      - mlindner · 2109 days ago
        If your home automation system requires a network connection to work then something is very wrong.
        m4rtink · 2109 days ago
        Right? Home automation is perfectly fine and mught even expose a secure authenticated API to the internet so you can say check and adjust the temperature remotely, but it shoukd never ever go down for lical use if the Internet connectivity or the remote service backend goes down.
      - tibu · 2109 days ago
        Home Assistant rulez with non-cloud sensors. I never by sensors, cameras, switches which require connection to an external host. But I'm sure that most people will do it even with such outages too.
      - fileeditview · 2109 days ago
        I still like my physical switch on the wall to turn the light on and off very much.. for me it's hard to beat. I can even turn it on/off semi-remotely with e.g. a tennis ball.
        i_am_new_here · 2109 days ago
        If my Alexa fails, then I can also just turn the light off with the switch and when flipped back "on" again , then it will be on. Fallback to 'classical mode' works for me.
  - ketamine__ · 2109 days ago
    There are other sites that can check for you
- espadrine · 2109 days ago
  It is all green if you do not need to be logged in . If you are logged in, the page crashes with an error. You can still browse all services from Incognito (which for some is not an option).
  - randompwd · 2109 days ago
    This is not true. i am logged in to gmail and getting 500 err edit:parent updated comment
    - StavrosK · 2109 days ago
      That's what the GP is saying.
      - randompwd · 2109 days ago
        the parent comment to mine at time of writing was just: > It is all green if you do not need to be log in. Giving the impression if you were logged in/didn't need to log in, that everything would be ok.
        StavrosK · 2109 days ago
        Oh, I see. Looks like their auth (state) servers are down.
        espadrine · 2109 days ago
        I apologize for my ambiguous phrasing, and I feel bad that you were downvoted (I reupvoted you, but that’s only one vote). I wrote too fast because I thought it could help people work around the problem.
  - skinnymuch · 2109 days ago
    Are you saying the Google status page is correct then? Being logged in is part of being green I’d think. Also, you can’t use many parts of Gmail, Drive, Photos, etc, without being logged in.
    - wffurr · 2109 days ago
      I think it just means the status page itself is not logged in. Apparently it should be...
  - fireattack · 2109 days ago
    Try YouTube, which is totally broken. But I guess it's technically not part of /appsstatus
    - shawabawa3 · 2109 days ago
      youtube is working in incognito mode
      - eddytrex_ · 2109 days ago
        Yeah for me too: youtube and gmail works in incognito mode
        eddytrex_ · 2109 days ago
        Is HN having issues too?
        oneeyedpigeon · 2109 days ago
        > gmail works in incognito mode I mean, that must be a generous definition of "works"! :)
        eddytrex_ · 2109 days ago
        Yeah xD
        deepujames · 2109 days ago
        drive too.
    - Intermernet · 2109 days ago
      YT seems to be working in Incognito / Private mode.
      - fireattack · 2109 days ago
        Ah, thanks! Kinda weird it would totally break if the auth failed, unlike other services like Search.
    - nkozyra · 2109 days ago
      Youtube Tv too
    - thunderrabbit · 2109 days ago
      Youtube is down for me if I am logged in. I can watch videos through duckduckgo search or an icognito window.
  - Kye · 2109 days ago
    Google made everyone get an account trying to force Google+ on to everyone. They don't get to pretend like that didn't happen by not factoring it into their status.
- fny · 2109 days ago
  https://downdetector.com/ is remarkably good at catching this too. But funnily enough, a lot of the votes come from traffic that searches for "is ____ down?" on Google. XD
  - alarmingfox · 2109 days ago
    https://downdetector.com/search/?q=google shows all the Google services and seems like everything other than search is reporting errors
    - londons_explore · 2109 days ago
      Even search seems to have issues. It can't seem to find rare pages at all, and ranking seems subpar for the past 15 minutes or so.
      - konschubert · 2109 days ago
        I think that's normal. ;)
  - Intermernet · 2109 days ago
    Amusingly, downdetector.com is now down. Gotta love that cascade effect :-)
  - wjdp · 2109 days ago
    Found this relatively recently, seems to do a good job. Do love how more consumer services (ISPs &c) always have some report of being down somewhere but its means nothing unless there's a big spike.
- skinnymuch · 2109 days ago
  Pretty funny. I would normally assume they are connected to some testing unit that real users interface with. Vs internal APIs or however Google is doing this with them still being green
- abalos · 2109 days ago
  Out of curiosity, what response time do you expect on a page like that? And what level of detail? I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes.
  - edf13 · 2109 days ago
    It's a big group of teams... one team is responsible for monitoring (And status reporting) alone
  - havetocharge · 2109 days ago
    There should be enough people on that to communicate out as well as resolve.
  - vineyardmike · 2109 days ago
    If they know its broken, one of the _many_ engineers/support across Youtube+Gmail+etc that are all known to be down related to this bug should be able to update it in first few minutes. Especially if this isn't a 5 min fix.
  - wbhart · 2109 days ago
    I'd expect within seconds that Google is alerted of a very large number of issues with their servers and that the status page would be updated (the green light going to red) within seconds. It's now quite some time after the start of the outage and everything is still green on that status page.
  - RichWalton · 2109 days ago
    I'd expect that status page to be automated based on a number of metrics / health checks. Our equivalent is.
  - JimDabell · 2109 days ago
    > what response time do you expect on a page like that? Faster than a free third-party website’s response time. Google should know they are down and tell people about it before Hacker News, Twitter, etc. Google should be the source of truth for Google service status, not social media. > And what level of detail? Enough to not tell people that there are “No issues” with services. > I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes. Google employs enough people to do both.
  - throwawaylolx · 2109 days ago
    How many employees do you think Google has? Do you think they're all working on the same task?
    - pjc50 · 2109 days ago
      Well, probably a lot of them are sitting around doing nothing because Google's down right now.
  - ric2b · 2109 days ago
    > I'd much rather have their team focus on fixing this as fast as possible than trying to update that dashboard in the first 5 minutes. It's not like they would be working on the status page right now, that work should have been done a long time ago...
  - kalleboo · 2109 days ago
    It should be faster than it takes TechCrunch to write an article about it
    - robertlagrant · 2109 days ago
      TechCrunch don't need to verify anything.
- pupdogg · 2109 days ago
  What's the point of having a status page when it can't reliably tell you the factual status?
  - woofie11 · 2109 days ago
    Helps Google meet a 99.999% SLA!
    - koheripbal · 2109 days ago
      Not really since now it's up, and the status page still says it's down.
      - keithnoizu · 2109 days ago
        right after I shamed them on twitter. I take responsibility despite it unlikely having any impact.
  - oneeyedpigeon · 2109 days ago
    Especially one that breaks the back button.
  - jaimex2 · 2109 days ago
    Customer confidence. Same as how everyone reports they have 99.99% uptime.
    - petre · 2109 days ago
      Then their status page is not unlike like target reporting in the USSR: completely fictional.
      - phaer · 2109 days ago
        From personal experience and other Ops peoples stories, I'd assume that could be the case for many status pages ;)
  - szatkus · 2109 days ago
    It's red already.
    - s_dev · 2109 days ago
      No -- it needs to be red when it's having the outage for people to have confidence in it as an indicator. The reality is Google have no real incentive to provide an actual external status page that is accurate -- to do so is an admission of not upholding SLAs. These status pages are updated retrospectively. Use a third party one like DownDetector.
- numpad0 · 2109 days ago
  Someone here in hn said at Amazon AWS they are required direct CTO approvals to reflect negative realities. Maybe that’s the case for anyone these days.
  - ThrawsAway · 2109 days ago
    Not true. Only for written things like postmortems. Automated systems need to be vetted but can update things like status pages on their own. The issue isn't "negative" realities, it is saying something while investigating that might break contracts only to find later that it wasn't true.
- prof-dr-ir · 2109 days ago
  Mistakes happen... that page is probably still correct 99,9999% of the time! /s
  - rightbyte · 2109 days ago
    These uptime numbers cloud providers gives out can't be nowhere near actual uptime.
- entropie · 2109 days ago
  Everything went red just now.
- akmittal · 2109 days ago
  It started showing red now
- oneeyedpigeon · 2109 days ago
  Everything red now: clean sweep
- papacho · 2109 days ago
  everything is red now.
- jim-greer · 2109 days ago
  It's correct for me now
- allie1 · 2109 days ago
  Now it's all red
- TimMeade · 2109 days ago
  Solid red now. Everything down
- biozdilek · 2109 days ago
  its not green anymore
- INTPenis · 2109 days ago
  I don't think it's difficult, I think they just lie. Monitoring is very simple, I even learned this from a document released by the Google devops team many years ago. Always alert from the end user perspective. So in other words have an external server test login to Gmail. Simple as that. They manually update that status page to not scare away stockholders.
- crosspanda · 2109 days ago
  Definitely i agree with you. Although the statuses are updated later, it may cause many problems if it is not instantaneous.
- senko · 2109 days ago
  It reflects the reality for them. The small print says: The problem with Gmail should be resolved for the vast majority of affected users. We will continue to work towards restoring service for the remaining affected users... At Google scale, "remaining affected users" probably number in tens of millions. Sucks to be one of them, tho. But hey, it happens. As a SaaS maintainer, I can sympathise with any SREs involved in handling this, and know that no service can be up 100% of the time.
rochacon · 2109 days ago
Google Meet and YouTube also 500 in Brazil
surajs · 2109 days ago
can confirmt this
akhiluk · 2109 days ago
YouTube shows a "something's broken" error. Similar errors when trying to access Gmail. Tried on multiple browsers and connections, got the same result. The search engine works fine, as of now.
- tonfa · 2109 days ago
  Youtube works when not logged in though.
  - akhiluk · 2109 days ago
    You are right, the YT homepage opens up in private tabs. It seems strange, I wonder might be causing this.
pharmhax · 2109 days ago
Same here, can't access gmail (currently on a trip in Poland).
d99kris · 2109 days ago
GMail IMAP is also down.
slmyldz · 2109 days ago
can verify - Turkey
decafbad · 2109 days ago
Yep. Gmail redirects to Gsuite Status Control and shows everything as fine there.
meheleventyone · 2109 days ago
Dead in Iceland.
henearkr · 2109 days ago
In Japan Youtube is down too.
- henearkr · 2109 days ago
  Everything is back and working fine now, here.
jerbearito · 2109 days ago
I can access youtube if I'm not logged in (Incognito)
- vineyardmike · 2109 days ago
  Thanks for the tip! Several people said its related to logins/auth so this helps getting access to videos/info. maybe this is their fingerprinting/cookies/tracking server side logic has some bug that is failing?
XOKP · 2109 days ago
Can't access my files on Google Drive, shit.
himujjal · 2109 days ago
Good. Now, at least a few people would search Google alternatives in DDG
- Veen · 2109 days ago
  Search is still working. It’s Drive, Meet, and other stuff that millions of people depend on for work, so maybe stow your glee for a moment.
  - himujjal · 2109 days ago
    Google still is the best when it comes to search. But its also important to know the alternatives. You will be surprised how many people dont know Google alternatives. lol
HelloNurse · 2109 days ago
I'm able to access GMail and Maps but not Youtube or Drive.
nicolapcweek94 · 2109 days ago
Gmail, Chat and Meet down from Italy as well, was in a Meet call and got disconnected, only getting 500s across all three now
behindai · 2109 days ago
Sweden. The same problem. Authorization doesn't work. Gmail doesn't work.
avibhu · 2109 days ago
Can verify - India. Gmail, Drive, Docs, Youtube not working.
- rk06 · 2109 days ago
  as another user mentioned, they work in incognito. looks like there is a problem with google auth which is causing the outage
milankragujevic · 2109 days ago
All down for me, Serbia. YouTube works for me when logged out (i.e. in Incognito).
dhbradshaw · 2109 days ago
Wow, I'm glad there are a few things that I do that don't require Google auth.
zubairov · 2109 days ago
Gmail, Calendar, Google Accounts down in Germany
- csunbird · 2109 days ago
  Same, my work mail does not work. I was supposed to be in a meeting right now.
- Kelteseth · 2109 days ago
  That's why my Thunderbird constantly popped up to enter my Google Mail credentials…
cabbagepatch · 2109 days ago
Google Cloud Platform is down in US.
Foivos · 2109 days ago
same here in Norway.
niico · 2109 days ago
Also down in Paraguay
tango12 · 2109 days ago
Google login also seems to be affected, meaning all services using google account login are inaccessible if you aren't logged in.
skinnymuch · 2109 days ago
Yeah weird it’s all green. I just noticed things down as well. Came to HN to see how quickly people posted. This is already #1 at 6:55 am EST
- nobleach · 2109 days ago
  Yeah, that status page is really frustrating. Didn't anyone consider building it to REALLY check if services were working? not just returning a 200 Status... but actually DOING things?
AbacusAvenger · 2109 days ago
The Nest endpoints are returning 500 as well.
farbrortumm · 2109 days ago
yeah, same in the UK
tibbydudeza · 2109 days ago
Was wondering why youtube can't show videos from the search page , it even reverted my language settings. South Africa
have_faith · 2109 days ago
Gmail said my account was "temporarily" unavailable... had a moment considering if it wasn't temporary. Good reminder to remove my reliance on gmail especially.
- tomglynch · 2109 days ago
  Same for me - and then I tried logging in to gmail in an incognito window and got "Couldn't find your Google account" which really scared me.
  - JorgeGT · 2109 days ago
    Time to stop postponing that large Google Takeout for me!
    - alpineidyll3 · 2109 days ago
      Clicking the stupid download links is so tedious...
- tux3 · 2109 days ago
  Thunderbird prompting me to login, and getting "Google does not recognize this email address" as a reply was a nice adrenaline spike, until I checked the status on HN!
  - lioeters · 2109 days ago
    This happened to me with a work email. For a moment I thought, "Well, I had a good run at this company.." My next thought was, "Ha, Google must be down - better check HackerNews."
  - BugWatch · 2109 days ago
    I am still experiencing this problem on all my synced-to-Thunderbird Gmail accounts, so it either hasn't been completely fixed yet, or there's another ongoing issue.
- imperfectcats · 2109 days ago
  https://console.cloud.google.com/ is down as well. Seems very much an identity issue.
- Ekaros · 2109 days ago
  I really should try to get my stuff moved to my new address. Not self-hosted, but a non-profit that has been around for a while... Finally paid the membership fee.
- cvrjk · 2109 days ago
  For a second I legit thought Google banned my account for some reason. And I don't like it that I feel relieved..
  - sn41 · 2109 days ago
    Underrated comment. If the government freezes your bank account, at least you can take them to court. If Google mistakenly disables your account, there does not seem to be any legal recourse. Considering the extent to which people depend on their services, perhaps there should be a more elaborate appeal process.
    - rightbyte · 2109 days ago
      Google doesn't disable ordinary rulefollowing accounts by mistake. They are aware of the false positive rate and don't care.
      - sn41 · 2109 days ago
        I agree, but the problem is that "ordinary rulefollowing" is not clearly defined by Google. Perhaps it was that innocent park frisbee party video with a Metallica sound played in the background by some other party there... and not only your video, but also your YouTube account, and worse, email account gets blocked. Maybe this scenario is dystopian - the point is that it is a black-box no-appeal/limited appeal system when such an event happens.
        sellyme · 2109 days ago
        Google account terminations typically only affect the one service. If your YouTube account gets terminated for ToS violations, your email account will still work perfectly fine.
    - noir_lord · 2109 days ago
      They can freeze the account but I own the domain so I can point it anywhere I want and I have copies locally via imap. At this point the only reason I use it is because I grandfathered in on an old plan it's still free, if that changes I'll go elsewhere.
- emilburzo · 2109 days ago
  I started this[1] the last time something similarly scary happened. It's very comforting to have a local copy of everything important in situations like this one. [1] https://wiki.emilburzo.com/backing-up-your-google-digital-li...
- timvisee · 2109 days ago
  Yes! It takes some work to switch, but it's worth it. Buy your own domain name, and link it to an existing service if you don't want to host yourself. You'll always be able to switch your mail alias when having issues with your email host.
- telesilla · 2109 days ago
  Our entire library of video tutorials disappeared for a while. I was not happy, and the thought of losing our email.. Currently working with the team to make backups of absolutely everything off-site and off-Google. A good wake up call for us.
n_time · 2109 days ago
The console for Google Cloud Platform is also down.
a012 · 2109 days ago
Youtube is up for me, but CANNOT login to Gmail
pantulis · 2109 days ago
Confirmed from Spain. Google Workspace seems to be down.
Betelgeuse90 · 2109 days ago
YouTube shows a monkey for me in Israel
kokotkohaha · 2109 days ago
without adblock it works
griffiths · 2109 days ago
What do you think, how much money are they losing right now every second?
- skinnymuch · 2109 days ago
  I was thinking the same thing. Also this funny bit that’s relevant: https://entertainment.theonion.com/google-shuts-down-gmail-f... “Google Shuts Down Gmail For Two Hours To Show Its Immense Power”
- vnxli · 2109 days ago
  Do tech companies post that kind of data? When I worked in the automotive industry, the Detroit Big 3 American auto makers had figured out their cost per minute of downtime. They would even charge that to their suppliers and anyone who shut down one of their plants. It was something like $10,000 per minute since they’re able to roll a complete vehicle off the assembly line about every minute
zupa-hu · 2109 days ago
Hungary - gmail down, youtube down if logged in, search works with 5s response time
kokotkohaha · 2109 days ago
In inkognito window (without Adblock) it works for me!
notyourplayer · 2109 days ago
Maybe it’s time to go outside?
- henearkr · 2109 days ago
  This outage is an idea from Calico then? (just kidding)
- jsemrau · 2109 days ago
  What is this bright thing in the sky?
- tiborsaas · 2109 days ago
  Let's not get ahead of ourselves :)
- paulmbw · 2109 days ago
  back inside we go guys:(
- muyuu · 2109 days ago
  let's not make panicked drastic decisions just yet
hemmert · 2109 days ago
Google Reader and Google Wave also seem to be affected.
- madjam002 · 2109 days ago
  This made me chuckle :)
- V-2 · 2109 days ago
  My FOMA on Google+ intensifies...
- fouronnes3 · 2109 days ago
  Google Buzz as well
  - scrollaway · 2109 days ago
    I can't even load Google+.
    - zapdrive · 2109 days ago
      Same with Orkut.com, can't even open it.
- nurettin · 2109 days ago
  my googles stopped working.
- vishnugupta · 2109 days ago
  I fell for it for a few seconds :-)