Z Plus Tornado Fun!
ZizzyDizzyMC
Hosting Provider
If you have service with us, your data is 99.999% probably-fine. ZFS + UPS meant mostly graceful shutdowns for all but the greediest VMs refusing to stop.
However, the entire city I'm in looks like confetti made of trees and silly string made of power lines. Announcing it here on LES since this is where 99% of our clients come from.
While our systems have power with a backup, our business line is down and network connectivity is a bust. With this knowledge, I've powered down the systems in as safe as state as possible and will power back up once I've restored connectivity.
I haven't decided what the compensation for the downtime will be exactly yet, but you will be compensated at least approximately double whatever the downtime is - possibly more. I'm small and can afford that since we own everything and aren't facing thousands of dollars an hour in losses.
I am in the process of getting a better power system in around july - we've settled on a 22kw system capable of running down to -40 degrees. We're also going to be replacing our front end router soon, just because I'm kind of over the BS that my CX3080 is giving us.
Apologies in advance,
Zizzy
Comments
Oh I should point out that in my accidental wisdom it appears that NS3 that's supposed to resolve our billing / admin panel got merced somewhere along the way so that won't load till I restore connectivity to NS1 and 2.
That's my bad, and everyone will get a free week of service (worth of account credit) as long as they ticket when this is resolved.
I don't care about uptime nearly as much as your safety, stay safe.
If you need to leave the hardware offline to stay safe, do it.
Hard to imagine to me how a tornado can wreck your city, as we don't have such things in my part of the world, but I have seen it do horrible damage in the US.
Good to see you are okay man. We did not get nearly what our downstate recieved. We have had some rain and thunder but not any winds or anything.
Let me know of there is anything I can to help out man.
Free Hosting at YetiNode | MicroNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?
I appreciate the kind words. We got hit with mostly straight-line winds, in excess of 70mph, other parts of the city / area had several tornados drop / scrape the wires etc.
I ended up seeing a wall of "WTF" in my bathroom after it got unusually calm outside and got everyone in the basement while it went through. Hail about the size of a quarter up to about a golfball. Saw what I think was a tornado on the ground about a mile away - considering it was to my west I wasn't fuckin around and got the hell in the basement, so no idea where it went - not that you could see it with the rain wrap once the wall of rain and hail hit.
I do feel bad for my clients though, at the very least for my own comfort and sanity - I didn't make any attempts to hide that I run a cornfield DC. Not like we're deadpooling but it'll suck for clients that want data right now.
Looking into possibly getting a 2nd backup service, but the cost is kinda high. Gonna wait to see how things pan out, and not make any rash decision.
I'd think that a tornado qualifies as force majeure.
Important thing is that you are safe.
A while back one of my lovliest providers got hit by a disaster which seemed pretty clearly not of their own making -- the exact same disaster could have hit almost any provider. They offered compensation, but I declined. Similarly, to me, it seems unfair to take compensation from Zizzy for the reason that his fully disclosed in advance cornfield datacenter and the surrounding community was ht by a tornado!
I hope everyone gets the servers they want!
Good news, nameserver 1/2 online so the billing panel can be accessed.
People can start ticketing for a week compensation in account credit, but if you hold off till after I've restored everything (when grid comes back) I'll be better at handling that.
Note for users of the Valentines Test special - your systems are ONLINE but the storage backer is OFFLINE - due to our dirty generator power and my UPS throwing error codes because of it, I do not want to turn on the 1.6KW worth of data storage and risk more equipment or data. Because of this all VMs on the node were gracefully shutdown. Our basic systems + the power efficient valentines node runs about 400W.
I got news from the power company that we may expect a minimum of 2 days without power, up to 4, maybe a week.
If it ends up being more than a couple days without grid (and therefor high speed network) I'll end up upping the compensation to 2 weeks.
In the future accidents like this will hopefully fail over automatically to far better generator power + standard backup network.
Alright we're back.
So, few things here.
I'm working towards getting automatic standby power - made the first step today by getting an Onan 100ENBA 100kw genset unit + transfer switch. Will take a while but will be building this onto a trailer so that I can leave it plugged in, or unplug it and move it to a different location if needed. It's way too big for what we have, but if I combine it with solar panels and a battery wall - it could charge the batteries in just an hour and then the batteries run everything for a day. Hooking it up to a transfer switch while it sits on a trailer will be a tad tricky though not impossible.
Starlink for emergency backup data will be happening, but I won't want to do that till they release their gen 4 home dish.
From what I can tell, data looks fine.
Thanks for riding with me through this, and the support. Good luck out there, and remember 3,2,1 backup rule!
This is starting to feel like Florin come-of-age story.
No hostname left!
I might not be aware of who this is referring to.