Ask HN: Is S3 down?

2589 pointsby iamdeedubs 3494 days ago1055 comments

I'm getting { "errorCode" : "InternalError" } When I attempt to use the AWS Console to view s3

Discussion

Loading discussion

mtdewulf · 3494 days ago
Yep, same here.
josephlord · 3494 days ago
Looks like it. Brief panic caused here.
baconomatic · 3494 days ago
Seeing it here as well.
jacobevelyn · 3494 days ago
We're getting errors indicative of an S3 outage too.
TheVip · 3494 days ago
Same problem bro...
mwambua · 3494 days ago
Not sure if it's related... but I'm having issues with Amazon Cloud-drive.
- iamdeedubs · 3494 days ago
  I would assume, I couldn't even share a screen shot of my evidence to my team on slack!
  - mwambua · 3494 days ago
    Lol... It's interesting how much depends on Amazon's infrastructure at the moment.
    - vinayan3 · 3494 days ago
      Also, interesting everyone is posting that various sites which are down in this thread. Feels like the internet is down!
kangman · 3494 days ago
any one get more info from AWS?
rhelsing · 3494 days ago
Having issues as well.. big issues..
alexleclair · 3494 days ago
Yup, same here. It has been a few minutes already. Wanna bet the green checkmark[1] will stay green until the incident is resolved? [1] https://status.aws.amazon.com/
- tuna-piano · 3494 days ago
  Still green now, 8 minutes in.
  - bpicolo · 3494 days ago
    I've had a few non-Amazon providers tell me AWS things are not working in the last 5 minutes, no note from Amazon though. Nice.
    - leesalminen · 3494 days ago
      Just sent out a notice to our customers via our status page. I really wanted to be able to add a link back to AWS detailing the issue but that's a pipe dream I suppose.
  - fudged71 · 3494 days ago
    ... still green
- cheeze · 3494 days ago
  Just went yellow Edit: nevermind
  - matthuggins · 3494 days ago
    Still green for me
    - leesalminen · 3494 days ago
      Just went yellow Increased Error Rates We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region. https://status.aws.amazon.com/
      - nkozyra · 3494 days ago
        Check individual services ... Amazon Simple Storage Service (US Standard) Service is operating normally
  - schneidmaster · 3494 days ago
    Did it? Still fields of green for me.
    - malloci · 3494 days ago
      While keeping the status green for s3, they have at least put up a notice at the top: Increased Error Rates We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.
      - schneidmaster · 3494 days ago
        Yeah I just now saw that. Probably regional cache clearing or something.
- jonstaab · 3494 days ago
  http://downdetector.com/status/aws-amazon-web-services looks like a reasonable alternative place to check/report downtime.
  - eicnix · 3494 days ago
    Which is coincidently down.
    - booleanbetrayal · 3494 days ago
      downdetectordown.com ?
      - pure_ambition · 3494 days ago
        yep, page won't load.
    - talawahdotnet · 3494 days ago
      Maybe they are hosted on S3 facepalm or maybe they just got a surge in traffic
  - zedpm · 3494 days ago
    I just check Twitter, since Amazon's status is always a lie. My personal dashboard is still showing no problems. It's bad enough that the main public status is always green even when there's clearly a problem, but you'd think they could at least make the private status accurate.
  - jonstaab · 3494 days ago
    Gah. It was up 3 minutes ago. Anyone have any suspicion this is another ddos episode? I saw that SO was down last night too: https://twitter.com/StackStatus/status/836450836322516992
    - cperciva · 3494 days ago
      Pretty confident that isn't it. S3 was returning InternalErrors for 22 seconds before it started timing out and/or returning 503s to all my requests. I'd bet that something broke (causing InternalError responses) and then nodes started marking themselves as failed (causing the timeouts and 503s soon after).
    - chx · 3494 days ago
      I want to see the botnet capable DDoSing S3. That would be something.
  - vjdhama · 3494 days ago
    Apparently, that's down too. Sigh.
- talawahdotnet · 3494 days ago
  There was an alert on the personal health dashboard[1] a second ago, it said S3 Operational issue in us-east-1 but when I tried to view the details it showed an error. Then I refreshed and the event disappeared altogether. [1] https://phd.aws.amazon.com/phd/home?region=us-east-1#/dashbo...
  - socialentp · 3494 days ago
    Same here. But it is in the general status dashboard: http://status.aws.amazon.com/
- hartleybrody · 3494 days ago
  I'm seeing green checkmarks across the board, but they just added a notice to the top of the page: > Increased Error Rates > We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.
  - syntheticcdo · 3494 days ago
    I guess sub-1% to 100% failure rate is technically an "increase".
    - Artemis2 · 3494 days ago
      I guess file uploads and downloads are technically "API calls".
  - samstave · 3494 days ago
    the worst thing is when your system cant handle these " increased error rates " as your control plane cascades failure due to something like this.... The worst " increased error rate " problem I had was when the API was failing and my autoscale system couldnt deal and launched thousands of instances because it couldnt tell when instances were launched (lack of API access) and the instances pummelled the fuck out of all other parts of the system and we basically had to reboot the entire platform.... Luckily, amazon is REALLY forgiving with respect to costs in these (and actually most) circumstance....
    - salvor · 3494 days ago
      recalls numerous times Yes. Yes they are. Thankfully.
- tlogan · 3494 days ago
  These service health boards are more like advertisement page then actual status of the service.
  - mwfj · 3494 days ago
    I guess their bizarre thinking is something along the lines of: "unless we have proof that noone can access the service, we won't change the indicator from green to yellow. Seriously: I don't understand why you guys stay with AWS.
    - rattray · 3494 days ago
      > I don't understand why you guys stay with AWS. Who do you recommend instead (assuming in-house or Hetzner-equiv is out of reach)? Google Cloud? Azure? Rackspace?
      - johansch · 3494 days ago
        Google Cloud if you're looking for something similar. It's just so much better and cheaper. I think a lot of the resistance here towards that kind of move is just because people are inherently lazy and they aren't paying the bill themselves. (I'm guessing a relatively large part is also selfish attachment to the market leader because of employment reasons. I hate wasting money, both for myself and for my employer, so I don't really understand this kind of thinking - but I do understand how it could flourish in a venture capital-rich time/locale.) I also recommend reading: https://thehftguy.com/2016/06/15/gce-vs-aws-in-2016-why-you-...
        joatmon-snoo · 3494 days ago
        Google also doesn't have the best record for developer tools.
        debaserab2 · 3494 days ago
        Google Cloud doesn't exactly have the greatest reliability/uptime either.
        eicnix · 3494 days ago
        https://status.cloud.google.com/summary tells a different story or do you have other information? I have used GCP for some time without being affected from any incident.
        briffle · 3494 days ago
        That is a pretty awesome page.. way better than a page full of green icons, during an obvious outage... I like that they have writeups a few days after the incidents....
        debaserab2 · 3494 days ago
        I'm not sure what you mean. If anything that link underscores my point. GCE has absolutely had it's own catastrophic errors. Remember last April when ALL instances in ALL regions went down? https://status.cloud.google.com/incident/compute/16007?post-...
        eicnix · 3494 days ago
        The GCP services are usually within their SLA target so I don't see the problem with the incidents. So you know what you buy and can take actions if you need a higher SLA for your application.
        debaserab2 · 3494 days ago
        > so I don't see the problem with the incidents All instances going down in all regions is an order of magnitude worse than a single service going down in a single region. You're deluding yourself if you think GCE is any more reliable than any other reputable cloud hosting platform.
        boulos · 3494 days ago
        While I agree that was a horrific outage for us, there's a big difference between no external connectivity for a few minutes (note: internal IPs still worked fine, as did access to APIs through that mechanism) and "ALL instances in ALL regions went down". Disclosure: I work on Google Cloud (and wouldn't want to be an incident responder at AWS today...)
        miken123 · 3494 days ago
        At least Google has a post-mortem, at AWS everything would still show up green with some random note about 'increased error rates'.
        chebum · 3494 days ago
        GC's CDN doesn't cache files bigger than 4Mb. No Windows VMs. Bound to AWS for these 2 reasons.
        dragonwriter · 3494 days ago
        GCE supports Windows VMs. https://cloud.google.com/compute/docs/instances/windows/
        vgt · 3494 days ago
        Someone else already mentioned Windows VMs. Looks like CDN has a 10MB limit: https://cloud.google.com/cdn/docs/caching (work at Google Cloud)
        mrmaximus · 3494 days ago
        As already mentioned, they do have Windows VM's but there are some caveats that indicate it's not fully baked yet. 1.) They require that each VM MUST have a public IP address so that Windows can talk to an activation server every 30 days. 2.) You cannot yet bring your own license.
      - pmalynin · 3494 days ago
        OVH
        LeoHaggins · 3494 days ago
        Bad idea there, support is horrible.
        jacquesm · 3494 days ago
        What is your last datapoint on that? The last year or two has seen a remarkable improvement according to those customers of mine that host there.
        rspeer · 3494 days ago
        OVH doesn't even want to take my money to keep my server running. Their auto-billing process is busted and when it goes wrong they just delete your server.
        fishnchips · 3494 days ago
        That's not what I've seen. I misconfigured my auto-billing and got paged in the middle of the night by nodes mysteriously disappearing, but they released those machines minutes after my CC went through. Not that I'm a big fan of OVH but if you design your system to allow for failure you can't match their value for money.
      - kohuma · 3494 days ago
        What about something like B2 from https://www.backblaze.com/ ?
        Twirrim · 3493 days ago
        S3 in a single region is based out of multiple data centres / availability zone, with data distributed so that the loss of a single availability zone won't impact either data availability or durability, even to the point of being comfortable with complete physical destruction of an AZ. The same applies for Azure, GCP etc. B2 is based out of a single DC (or at least, was at launch and I don't see anything that suggests that has changed?) You've got to decide what's most important to you. Data persistence or $$$.
    - tlogan · 3494 days ago
      > Seriously: I don't understand why you guys stay with AWS. I tried them all and Amazon is still the best.
    - gtsteve · 3494 days ago
      > Seriously: I don't understand why you guys stay with AWS. Personally I've been using it for ages and I know most services inside and out. They do suffer downtime in some regions occasionally, but it'd be too expensive at this point to move. And who doesn't suffer downtime? You can't avoid it; you just need a plan to deal with it. For example, having a backup replica bucket in another region and the ability to quickly switch your CDN over would probably be a good idea here; that's what I did. If you want to go further you can replicate your data to another cloud provider entirely and use low TTLs to switch to a backup CDN if your system is that mission-critical (in the event of a worldwide AWS failure doomsday scenario). All systems will fail you and it's our responsibility as IT professionals to have a plan to mitigate this.
      - johansch · 3494 days ago
        Sunk cost fallacy. I do agree that we should all plan for failures. However, I also think it's a sign of failure in planning and architecture foresight if it's too expensive to move away from a particular cloud provider.
        froogle · 3494 days ago
        The sunk cost fallacy is when you (irrationally) decide to stick with what you're doing purely because you've already spent a lot of resources on it. It doesn't apply when you've done an economic analysis and found out it doesn't make sense to swap. There are plenty of cases where it just wouldn't make sense to switch after looking at the costs, opportunity costs, etc. For example, if his site makes him $10 a month, outages cost him $1 a month that could be mitigated by moving, and it would cost $1000 of labor to swap providers. (Depends on interest rates.) Perhaps it was originally a failure to not have a plan to easily move from a provider, but it doesn't seem unreasonable to me that right now it may cost too many hours of work to justify the move.
        johansch · 3494 days ago
        (You're right, I used that term incorrectly.) Still stand behind the other two points I made in that post though.
        gtsteve · 3494 days ago
        It's not as though it would be impossible; our integration with AWS isn't that deep, it's not as though we use DynamoDB for our core data store or anything like that. But even migrating from one traditional datacenter to another isn't easy from an operational point of view. There needs to be a clear financial win. Even taking into account the failures we've seen so far, I don't see a compelling reason to leave AWS.
      - nicolaslem · 3494 days ago
        Low TTL on DNS entries might do more harm than good: if your DNS provider gets seriously DDoS, being able to rely on caches can save the day. Anyway, I agree with your conclusion.
    - cagataygurturk · 3494 days ago
      Because you perceive public clouds only as virtual machine providers, that you can replace with other provider in two days. A detailed cloud migration consists of replacing some parts of your software to use managed services provided by a specific cloud provider, and AWS is still has the best service offerings IMHO. When you use these services carefully also you will see that AWS is very cheap and reliable enough. Outages like today's are happening in every platform and it is possible to mitigate them. You can use Adwords as a self-service user. Without knowing so much of details you can run your ads but also you can bery easily ruin your budget. But many enterprise customers use it very differently than those users and they are extremely optimizing the cost. Cloud is the same. If you don't know how big customers use AWS, it is normal that you are surprised because AWS is still leading the market. You say GCP is better than AWS. Which part is better? GCP does not have many services of AWS we benefit from. How can you compare totally different providers? You can only say AWS EC2 is worse than GCP. But you cannot compare whole platforms in one sentence.
      - cagataygurturk · 3493 days ago
        Sorry for endless number of typos and mistakes. Obviously I was sleepy while I was writing this.
      - johansch · 3493 days ago
        (Sorry, I'm late to reply, but since you addended your comment you might still be listening...) After spending a year evaluating both AWS and GCP (with an emphasis on their managed database services; both SQL and no-SQL) my general feeling is this: "Microsoft Windows is to Unix as AWS is to GCP". (Or perhaps closer to the truth: "VMS is to Unix as AWS is to GCP".) Baically AWS services seem like they are badly designed by buerocratic mediocre engineers following some bureocratic template for "a service". GCP feels a lot saner (both API- and UI/console-wise). I often got the feeling it's designed by people who: a) are smart and well-rounded in terms of experiences. It does take cleverness and experience to design something elegant that is also useful. b) take pride in their work (it does show) (And then, as a bonus: It's cheaper!)
        cagataygurturk · 3493 days ago
        You talk about SQL and No-SQL as managed services and it shows that your experience is limited to a classical application consisting of virtual machines and some data storage. However these are not the only services offered by both platforms and currently AWS has a richer feature set. For example Lambda and its deep integration with whole AWS platform is the biggest game changer from my point of view. If we are talking about virtual machines and databases, I can accept this comparison. However we are talking about 30+ services, some of them are even not available somewhere else and solving serious business problems in production and at scale. It is very wrong to put everything into basket and compare. Maybe GCP has better pub/sub service and AWS has better object storage. These should be compared seperately. Answering to your question, why do we still stay at AWS, because it is solving our problems in the most cost effective way and with reduced complexity, we are happy with it.
        johansch · 3493 days ago
        You're probably assuming too much again :) I specifically spent a lot of time on Lambda and found it quite annoying compared to GCP AppEngine. So much bureaucracy. Just this thing that you have to specifically register every single Lambda API call and its parameters using an interface built by non-thinking people.. Sheesh. For on-demand processing I just want a single HTTP-ish entry point, like AppEngine provides. (That way I can I move my service between different providers, if I wanted to move away from e.g. AWS.)
        johansch · 3493 days ago
        Anyway, I just updated my HN profile with more details about my experience. Please visit it to judge if I might know what I'm talking about.
    - cgag · 3494 days ago
      Postgres on RDS
      - boulos · 3494 days ago
        Come to NEXT in a week! :).
        espeed · 3494 days ago
        Any chance UDF iterators for Cloud Bigtable are in the works? Being able to run distributed D4M/GraphBLAS queries in Cloud Bigtable would be killer. "From NoSQL Accumulo to NewSQL Graphulo: Design and Utility of Graph Algorithms inside a BigTable Database" https://arxiv.org/pdf/1606.07085.pdf
    - derefr · 3494 days ago
      I think it's more, "if the service can't do what people need it to do, that's a problem; if the service cluster gets wedged hard enough to stop responding to the requests of our monitoring system, that's a failure." Which would make sense (and is sorta-kinda a best-practice) if Amazon wrote services such that they "crashed early"—but instead they're seemingly written so the backend lock up and be rendered completely useless at "doing its job" but will continue to run just fine. Either of those two design decisions is potentially a good thing on its own, but they need to be considered in light of one-another if you want your status page to make any sense. If you want to report cluster failures, code your clusters to actually fail. If you want to keep your clusters up, write your monitoring checks as whole-stack acceptance tests.
    - notyourwork · 3494 days ago
      > Seriously: I don't understand why you guys stay with AWS. You don't seem to have enough experience to comment on the issue.
      - johansch · 3493 days ago
        Please visit this comment sub-tree: https://news.ycombinator.com/item?id=13765786
        notyourwork · 3491 days ago
        That is a regurgitation of your opinion without any facts. Comparing technology and saying "it seems" or "i feel" isn't really a good argument to convince me one way or the other.
- matwood · 3494 days ago
  I always joke that if one of those statuses ever went to red, it means the zombie apocalypse has begun.
  - cperciva · 3494 days ago
    The good news is, if Amazon's services are marked as offline, you're allowed to use Amazon Lumberyard to control nuclear power plants.
  - paulddraper · 3494 days ago
    The number of non-green marks is the number of ICBMs currently in flight towards an AWS data center.
  - chrisan · 3494 days ago
    In case anyone wants to see what mysterious the red icon looks like: https://status.aws.amazon.com/images/status3.gif At best when there are problems (not like now I guess) I will see the "note" green icon https://status.aws.amazon.com/images/status1.gif
  - krylon · 3494 days ago
    I've heard (on the Fnord new show on the most recent CCC congress, so take it with a grain of salt and a bucket of humor) that Amazon's TOS are more or less void when a Zombie Apocalypse breaks out. They had some convoluted but fairly specific wording in their TOS, whoever wrote must have had a lot of fun.
    - jfim · 3494 days ago
      From https://aws.amazon.com/service-terms/ > 57.10 Acceptable Use; Safety-Critical Systems. Your use of the Lumberyard Materials must comply with the AWS Acceptable Use Policy. The Lumberyard Materials are not intended for use with life-critical or safety-critical systems, such as use in operation of medical equipment, automated transportation systems, autonomous vehicles, aircraft or air traffic control, nuclear facilities, manned spacecraft, or military use in connection with live combat. However, this restriction will not apply in the event of the occurrence (certified by the United States Centers for Disease Control or successor body) of a widespread viral infection transmitted via bites or contact with bodily fluids that causes human corpses to reanimate and seek to consume living human flesh, blood, brain or nerve tissue and is likely to result in the fall of organized civilization.
    - grogenaut · 3494 days ago
      First the fall of human civilization has to be a real threat per the TOS so not sure they'll care. Second, I know the lawyer and yes he had fun.
  - obsurveyor · 3494 days ago
    Then I guess it has begun, the page is now showing red. I'd put a picture on imgur but it's not loading.
- clamprecht · 3494 days ago
  Calling @jeffbarr https://news.ycombinator.com/user?id=jeffbarr
  - adrenalinelol · 3494 days ago
    Looks like his personal site isn't loading... :)
    - johansch · 3494 days ago
      Yup, it is indeed hosted with Amazon.
- nostromo · 3494 days ago
  The red check mark is hosted on S3...
  - AtheistOfFail · 3494 days ago
    Comment of the year.
    - Kiro · 3494 days ago
      Fact of the year.
  - ak2196 · 3494 days ago
    The truth is stranger than fiction. https://twitter.com/awscloud/status/836656664635846656
    - crowbahr · 3494 days ago
      "Care to share the code as an anti pattern?" brilliant.
- ceejayoz · 3494 days ago
  Me right now: https://www.youtube.com/watch?v=_cHa063Mwos
- emrekzd · 3494 days ago
  In December 2015 I received an e-mail with the following subject line from AWS, around 4 am in the morning: "Amazon EC2 Instance scheduled for retirement" When I checked the logs it was clear the hardware failed 30 mins before they scheduled it for retirement. EC2 and root device data was gone. The e-mail also said "you may have already lost data". So I know that Amazon schedules servers for retirement after they already failed, green check doesn't surprise me.
  - smoodles · 3494 days ago
    So just as a FYI the reason that probably happened to you is that the underlying host was failing. I am assuming they wanted to give you a window to deal with it but the host croaked before then. I've been dealing w/ AWS for a long long time and I've never seen a maintenance event go early unless the physical hardware actually died...
  - problems · 3494 days ago
    That's completely ridiculous, get some fucking RAID Amazon. I order drives off newegg directly to my DC and I'm yet to lose data with the cheapest drives available in RAID10.
    - prdonahue · 3494 days ago
      Yes, solving problems at your scale and AWS' are quite comparable.
      - problems · 3494 days ago
        Not saying my scale is the the same at all - but the fact they can't do something so simple that I can do it as a single individual is embarrassing at best. Simple solutions to this do scale - Linode and DigitalOcean don't have such issues for example - and while they're not Amazon scale, they are quite large and I'd say they prove the concept.
        phonon · 3494 days ago
        EBS data is backed up in multiple redundant ways (using erasure encoding I think). Local storage is not intended for permanent storage, and is more use at your own risk. That's also why most of the new EC2 instances don't even support local storage. Availability =/= durability of course
        problems · 3494 days ago
        EBS is incredibly expensive and slow, not really a good solution. It'd be nice if they offered a better local storage option.
        foxylion · 3494 days ago
        I think most people rely on EBS and are happy with it. Sure it depends on the use case, but I think it works for most use cases.
        phonon · 3494 days ago
        Incredibly expensive and slow compared to what? A 500 GB SSD (gp2) costs $50/m, and has 1500 - 3000 IOPS. It's okay for most loads. For higher performance, you can use 1. EBS Provisioned IOPS (kind of expensive) 2. Aurora (for DB use) 3. The new I3 instances (super fast local storage at a reasonable price.)
        tomc1985 · 3494 days ago
        That's the cost of a new 500GB SSD per month! For the cost of three months' EBS storage and a couple of hours you could setup your own RAID array with a spare for backups and possibly get better uptimes than Amazon :P
        problems · 3494 days ago
        Not to mention you'll get 3-5x better IOPS off a dedicated SSD. I guess this just boils down again to Amazon not being cost effective enough for my use case in yet another way.
        phonon · 3494 days ago
        Actually, you can get way more than 3-5x better IOPS on your own SSD! Different types of storage have different types of trade-offs. EBS is great for some things, slow and overpriced for others...
        phonon · 3494 days ago
        Huh? What kind of 500 GB SSD costs $50? And again--500 GB on EBS is not stored on 500 GB of flash...they use erasure coding and distribute it over ~3x as much, roughly. Oh, and good luck creating snapshots of your home RAID!
        problems · 3494 days ago
        > Huh? What kind of 500 GB SSD costs $50? Definitely not $50 to my knowledge but for ~$170 you can get a Samsung 850 EVO which is rated for 98k IOPS. They're fairly reliable drives and much, much faster than anything you'll get on EBS. You could be running that full 3x replication in less than a year of paying for EBS. > Oh, and good luck creating snapshots of your home RAID! LVM, ZFS and Btrfs all do snapshotting quite nicely. FreeNAS - commonly used for consumer grade NASes will automatically manage ZFS snapshots for you too. Amazon will sell you extra space to store snapshots, sure, but increasing the size of your devices usually solves that problem. And quite cost effectively as you can probably tell by now...
        phonon · 3494 days ago
        ...and Dropbox can be replaced with SFTP and rsync. Can you roll your own? Sure. Will it work as reliably and effectively as EBS? Can I scale it easily? How many people are comfortable using snapshots on their own?
        problems · 3494 days ago
        It's tried and true tech that any competent ops person can use quite easily. Been around much longer than EBS quite frankly. Dropbox targets end users who don't have the knowledge required to use the alternative, if you're smart enough to use EBS you're probably smart enough to use ZFS snapshotting just as easily. Or could within a day or two. It's really not that hard. Like I said, there are systems that pretty much manage the whole thing for you and just warn you when something is about to blow up like FreeNAS.
        tomc1985 · 3494 days ago
        It could be replaced with Nextcloud Shadow volume replication is entirely possible with several filesystems or Hot Copy kernel mod. Also LVM does snapshotting fairly easily I may have got my prices a bit mixed up (I saw 120GB at Fry's for $60 last month) but my point stands. Also why is discomfort such a big problem for folks? Learn stuff.
        Sanddancer · 3494 days ago
        Not really that hard. zfs snap tank/data@$(date '+Y%m%d') zfs send tank/data@$(date '+Y%m%d') | zfs recv backup/data advanced magic for off-system backup zfs send tank/data@$(date '+Y%m%d') | ssh cheapdiskserver zfs recv tank/data
        Dylan16807 · 3494 days ago
        > 1500 - 3000 IOPS So about as many as this SD card, and nothing compared to a real SSD.
        phonon · 3494 days ago
        In practice, it's actually fine for most purposes. It's equivalent to multiple striped 15K RPM magnetic disks, which used to be high-end enterprise storage a few years ago. SD cards have much worse write IOPS.
        Dylan16807 · 3493 days ago
        > In practice, it's actually fine for most purposes. It is, yes, but I wouldn't refer to it as comparable to an SSD. > SD cards have much worse write IOPS. Surprisingly not! Testing in ATTO I got read and write speeds that were almost identical, and a peak of 2000 IOps.
        phonon · 3493 days ago
        >It is, yes, but I wouldn't refer to it as comparable to an SSD. EBS (gp2) is flash based, has far better performance than high end magnetic disks, with excellent latency and consistent performance. So, it's more comparable to SSD than anything else. >Surprisingly not! Testing in ATTO I got read and write speeds that were almost identical, and a peak of 2000 IOps. Really? Were you looking at 4K write? Typically that would be under 1 MB/s for an SD card.
        Dylan16807 · 3493 days ago
        https://www.amazon.com/gp/product/B013CP3MMM http://i.imgur.com/PRoxVPM.png It's a relatively high-quality SD card, unfortunately hampered by my reader's inability to use bus speed over 25MB/s.
        phonon · 3492 days ago
        Nice card! So about 1500 write IOPS at 4K. Performance might be worse (or better) at other queue depths though.
      - LoSboccacc · 3494 days ago
        but I never lost data off an usb stick how hard could it be!
        aurelianito · 3494 days ago
        Really?!?! Several times USB sticks (and USB HDs) failed on me and other people I work with.
    - vacri · 3494 days ago
      Yes, the only way a server can die is from non-raided disks.
      - problems · 3494 days ago
        Otherwise they should at least be providing customers their data back.
        foxylion · 3494 days ago
        I think you misunderstood the local storage. It is not intended to permanently store data. It's a volatile storage like RAM.
    - foxylion · 3494 days ago
      It's not just a RAID that can fail. And everyone who uses AWS should expect failures. You should build your infrastructure to handle such failures well.
      - problems · 3494 days ago
        They offer no RAID on local storage and only the expensive, IO restricted EBS as an alternative.
  - amaks · 3494 days ago
    That what happens when cloud provider doesn't support live migration for VMs.
- ak2196 · 3494 days ago
  We have a slack emoji for it called greenish. It's the classic AWS green checkmark with an info icon in the bottom. Apparently it's NOT an outage if you don't acknowledge it. It's called alt-uptime.
  - foxylion · 3494 days ago
    I really liked it. But when trying to add it to my HipChat group it failed to upload. Why? S3 outage, what an irony.
  - nhumrich · 3494 days ago
    AWS internal lingo calls this the "green-i"
- Fiahil · 3494 days ago
  So, global S3 outage for more than an hour now. Still green, still talking about "US East issue". I'm amazed.
  - gtsteve · 3494 days ago
    It doesn't appear to be global; my app in eu-west-1 appears unaffected. It's possible that the console won't work however as I believe that's served from us-east-1.
    - chebum · 3494 days ago
      My site hosted on S3 is also running.
- tuna-piano · 3494 days ago
  It's crazy how much better the communication (including updates and status pages) is of the companies that rely on AWS than AWS' communication itself. https://status.heroku.com/incidents/1059
  - busterarm · 3494 days ago
    Also, https://status.pantheon.io
  - tcsf · 3494 days ago
    Blake Gentry gave a full accounting of Heroku's response process here - http://www.heavybit.com/library/video/every-minute-counts-co... Amazon should take notes.
  - all_usernames · 3494 days ago
    I feel for them. Imagine, 40 or 50 different engineering teams all responsible for updating their statuses. At this moment on the AWS status page I see random usage of the red, yellow, and green icons, even though all the status updates are "Increased error rates." What that tells me is that there's no unified communication protocol across the teams, or they're not following it. And just imagine what it's like being on the S3 team right now. I notice even Cloudflare is starting to have problems serving up pages now.
  - SnowingXIV · 3494 days ago
    Font Awesome went out for me for a bit, but they did a great job getting back up and keeping their users in the loop. https://status.fortawesome.com/
- gordon_freeman · 3494 days ago
  Looks like they have fixed the issue with their health dashboard now. From https://status.aws.amazon.com/ : Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.
zitterbewegung · 3494 days ago
Can't access my website which is hosted on s3 ( http://joshuajherman.com ).
leesalminen · 3494 days ago
FreshDesk makes extensive use of S3 and it's been unbearably slow to load for the past hour or so. All on S3 requests.
willcodeforfoo · 3494 days ago
Uh-oh. Same here... and tried taking a screenshot of pinging s3.amazonaws.com and Slack upload hung.
cwe · 3494 days ago
Dropbox using this? Can't seem to sync
- rckclmbr · 3494 days ago
  They used to, but i think they are off it now.
GabeIsman · 3494 days ago
Yes.
meddlepal · 3494 days ago
Totally fucked.
jgacook · 3494 days ago
Yup - dead in the water
c4urself · 3494 days ago
It is, of course the checkmark will stay green throughout this as Amazon doesn't care about actually letting its customers know they have a problem.
danielmorozoff · 3494 days ago
Yea seeing the same thing
scrollaway · 3494 days ago
Down in US-East-1 as of 17:40 GMT. Amazon SES also down in US-East-1 as of a few minutes later. Hearing reports of EBS down as well.
- St-Clock · 3494 days ago
  I confirm for SES being down in US-East-1 :-(
- kureikain · 3494 days ago
  EBS is down for 30% of my servers as well
_callcc · 3494 days ago
SES also down.
pmalynin · 3494 days ago
Down from the outside; The internal access (from within EC2) APIs still work.
65827 · 3494 days ago
Dead as a doornail for me
djb_hackernews · 3494 days ago
Anyone else seeing ELB/ALB issues?
- asolove · 3494 days ago
  Yes
- redthrowaway · 3494 days ago
  Yup. Some of our machines in us-east-1e dropped out of the load balancers.
cryreduce · 3494 days ago
in the s3 web interface requests to S3 backend end with 503 Service Unavailable
booleanbetrayal · 3494 days ago
S3 and Elastic Beanstalk (S3 dependencies) ... no issues with RDS at the moment
4wmturner · 3494 days ago
Is cli working for anyone else? I can't use the console UI, but aws s3 ls and get commands seem to be working fine.
ARolek · 3494 days ago
Same here. US East (N. Virginia)
xvolter · 3494 days ago
Seeing the same here
twiss · 3494 days ago
S3 Ireland (eu-west-1) seems to be doing fine at first sight.
amcrouch · 3494 days ago
It appears to be down. My website runs on S3 and my monitors are going nuts!
talawahdotnet · 3494 days ago
Yup it looks so. My console says I have zero buckets, my Lambdas are timing out and https://aws.amazon.com/ returns a big: "500 The server encountered an error processing your request." message
- alanning · 3494 days ago
  Our lambda functions are also unavailable. Lucky for us we didn't move any of our critical functionality to lambda yet although we are planning to once we have an EC2 backup in place...
  - ec109685 · 3494 days ago
    Don't know if share911 is still your product (its about page is down), but you could run your critical services on top of something like PagerDuty to give you the reliability you need.
andrewfong · 3494 days ago
Yeah, we host on S3 (US-East-1 I think) with Cloudfront for caching / SSL. Some of our requests get through but it's been intermittent. Lots of 504 Gateway Time-Outs when retrieving CSS, JS.
rawrmaan · 3494 days ago
Incredible how much stuff this affected for me. Opbeat is not loading and I can't even deploy because CircleCI seems to depend on S3 for something and my build is "Queued". This seems so dangerous...
- piquadrat · 3494 days ago
  Hi, Beni from Opbeat here. Our community site is indeed down unfortunately, but the rest should be unaffected. Where do/did you experience issues?
  - rawrmaan · 3494 days ago
    Hi Beni! My dashboard wasn't loading (the JS assets from cloudfront.net seemed to be throwing an access-control-origin error) but it is now working again. Very slow, but it works. Fortunately nothing critical is happening right now so no worries. Love your service :)
    - piquadrat · 3494 days ago
      Ah right, static assets are an issue, didn't notice it right away due to local browser caching. Sorry about the trouble!
- vidarh · 3494 days ago
  circleci.com won't even load for me.
tbeutel · 3494 days ago
I'm getting this using s3cmd: $ s3cmd ls WARNING: Retrying failed request: / ([Errno 60] Operation timed out) WARNING: Waiting 3 sec... WARNING: Retrying failed request: / ([Errno 60] Operation timed out) WARNING: Waiting 6 sec...
greenhathacker · 3494 days ago
"I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror, and were suddenly silenced. I fear something terrible has happened."
ethanpil · 3494 days ago
What kills me is that their status page still shows nothing is wrong. https://status.aws.amazon.com/
- jeffijoe · 3494 days ago
  Their status page probably can't refresh because S3 is down.
  - amazon_throw · 3494 days ago
    correct.
jontro · 3494 days ago
I get this in my aws console. Increased API Error Rates 09:52 AM PST We are investigating increased error rates in the US-EAST-1 Region. Event data Event S3 operational issue Status Open Region/AZ us-east-1 Start time February 28, 2017 at 6:51:57 PM UTC+1 End time - Event category Issue
jrs235 · 3494 days ago
They don't show it on the status dashboard at https://status.aws.amazon.com/ (at least at the time I originally posted this comment). But if you go to your personal health dashboard ( https://phd.aws.amazon.com/phd/home#/dashboard/open-issues ) they report an S3 operational issue event there. Edit: Mine is reporting region us-east-1 Edit 2: And now the event disappeared from my personal health dashboard too. But we are still experiencing issues. WTH.
- chickenfries · 3494 days ago
  Also seeing the intermittent event on the personal dashboard. Wonderful.
  - jrs235 · 3494 days ago
    Apparently they need a special/separate status page for everyone's personal health dashboard too. SMH.
the_arun · 3494 days ago
Same here
kyleblarson · 3494 days ago
Any specific regions? us-west-2 seems fine to me. [edit] now I can't see any of my buckets in the web interface.
- jrs235 · 3494 days ago
  Appears to be us-east-1
magic_beans · 3494 days ago
Definitely experiencing non-loading for dependencies hosted on S3 at the moment...
jhaile · 3494 days ago
All of our S3 assets are unavailable. Cloudfront is accessible but returning a 504 status with the message: "CloudFront is currently experiencing problems with requesting objects from Amazon S3."
samgranieri · 3494 days ago
I'm running into timeouts trying to download elixir packages, and I'm willing to bet this is the cause
stevefram · 3494 days ago
Yes, affecting elb in us-east-1 right now. web services are down and unable to bring up the elb screen in the aws console.
chrisan · 3494 days ago
Down for us as well. We have cloudfront in front of some of our s3 buckets and it is responding with CloudFront is currently experiencing problems with requesting objects from Amazon S3. Can I also say I am constantly disappointed by AWS's status page: https://status.aws.amazon.com/ it seems whenever there is an issue this takes a while to update. Sometimes all you see is a green checkmark with a tiny icon saying a note about some issue. Why not make it orange or something. Surely they must have some kind of external monitor on these things that could be integrated here? edit: Since posting my comment they added a banner of "Increased Error Rates We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region." However S3 still shows green and "Service is operating normally"
- gtsteve · 3494 days ago
  To mitigate the effect of S3 going down I use cross-region replication to replicate objects to another S3 region. If S3 went down in my primary region I could update the app config to write back to the backup region and update the CDN configuration to use the backup region as an origin. I did that out of paranoia but it turns out this could happen to us. Does that sound like a sensible approach? Fortunately all my company's stuff is in eu-west-1 which still seems to be fine.
  - kaishiro · 3494 days ago
    I'm certainly not an expert here, but just to make you feel good (if nothing else), this is exactly what we did this morning (Melbourne time). Woke up to a bunch of flailing Lambda funcs on us-east-1. Luckily we're using Apex so cross deploying them to Singapore took all of 30 seconds. We were concerned about API Gateway since it was also sitting in us-east-1 but ended up not being an issue. Realized that redeploying S3 and Lambda across regions can be done in practically no time, but we would have been in trouble had we needed to replicated our APIs in another region. Going to start exploring spec'ing up our existing gateways in swagger to help with this.
kangman · 3494 days ago
what's the SLA for s3?