@cyforex said:
my whole service is down! I also can't open my service page in control panel, I don't know what node it is! the location is NYC.
my service ID: 691792
man your support system is a mess @VirMach can you check?
If you don't know the node name, but know it's NYC, and there is currently around the same time a node down in NYC which is NYCB028, you can at that point probably safely assume it's likely that one.
Update: unfortunately, the datacenter in Dallas has broken access to our out of bound management without notifying us previously, and then powered down the wrong server (DALZ005) causing a second outage during our maintenance request. We're still waiting for them to correct our access to DALZ004 so we can complete the reboot.
Update: unfortunately, the datacenter in Dallas has broken access to our out of bound management without notifying us previously, and then powered down the wrong server (DALZ005) causing a second outage during our maintenance request. We're still waiting for them to correct our access to DALZ004 so we can complete the reboot.
@cyforex said:
my whole service is down! I also can't open my service page in control panel, I don't know what node it is! the location is NYC.
my service ID: 691792
man your support system is a mess @VirMach can you check?
If you don't know the node name, but know it's NYC, and there is currently around the same time a node down in NYC which is NYCB028, you can at that point probably safely assume it's likely that one.
It looks like NYCB022 has also dropped by this time.
@JoeMerit said: Every NYC server is down. Dedipath offline?
The outage seems bigger than just VirMach. My VMs from another provider in the same DC when down at the same time as my VirMach VMs, and are still down.
The datacenter is experiencing a fire and is in emergency power off.
The other shoe definitely just dropped on this one. I hope everything gets sorted quickly with minimal damage to all parties involved, especially no one was, or is hurt.
Power remains off at our data center in Secaucus/EWR1 per the local fire marshal.
We have had an electrical failure with one of our redundant UPS' that started to smoke and then had a small fire in the UPS room. The fire department was dispatched and the fire was extinguished quickly. The fire department subsequently cut power to the entire data center and disabled our generators while they and the utility verify electrical system. We have been working with both the fire department and the utility to expedite this process.
We are currently waiting on the fire marshal and local utility to reenergize the site. We are completely dependent upon their inspection and approval. We are hoping to get an update that we can share in the next hour.
At the current time, the fire department is controlling access to the building and we will not be able to let customers in.
Got brief moments of seemingly working control panel (showing server as shut down).
But didn't manage to boot up yet.
Messages vary between timeout and
"There seems to have been a slight problem with our database, please try again later."
Update from the datacenter below. Their message did not specify a next update time, I've asked when we can expect to be updated.
Power remains off at our data center in Secaucus/EWR1 per the local fire marshal.
Current status update from DC Ops:
Our remediation vendor and our team has worked through the night to clean the UPS' at the request of the fire marshal. They have made significant progress and we hope to have the cleaning completed by mid-day, at which time we will engage the fire marshal to review the site. Following their review, we hope to get a sign off from them so that we can start the reenergizing process. The reenergizing process can take 4-5 hours, as we need to turn up the critical infrastructure prior to any servers.
Update: unfortunately, the datacenter in Dallas has broken access to our out of bound management without notifying us previously, and then powered down the wrong server (DALZ005) causing a second outage during our maintenance request. We're still waiting for them to correct our access to DALZ004 so we can complete the reboot.
( chuckle ) Interesting ... what would constitute "out of bound management"?
Comments
If you don't know the node name, but know it's NYC, and there is currently around the same time a node down in NYC which is NYCB028, you can at that point probably safely assume it's likely that one.
Seems fun?
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
When it rains, it pours it seems.
Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?
It looks like NYCB022 has also dropped by this time.
Every NYC server is down. Dedipath offline?
NYCB040 is down here
Operation Timed Out After 90000 Milliseconds With 0 Bytes Received
Ny Nodes are all offline, Operation Timed Out After 90000 Milliseconds With 0 Bytes Received
I guess it's this
https://status.dedicated.com/
aka
The datacenter is experiencing a fire and is in emergency power off.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
The outage seems bigger than just VirMach. My VMs from another provider in the same DC when down at the same time as my VirMach VMs, and are still down.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
The other shoe definitely just dropped on this one. I hope everything gets sorted quickly with minimal damage to all parties involved, especially no one was, or is hurt.
Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?
NYC server down.
why read when you can just post.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Moar fun that way?
Free Hosting at YetiNode | Cryptid Security | URL Shortener | LaunchVPS | ExtraVM | Host-C | In the Node, or Out of the Loop?
@virmach, none of the services can connect to Solus, are you doing any?
Yes... he is doing all servers now
My VPS from Dedipath directly (NY location) is also offline / unreachable.
This is from DediPath (because probably the same issue in the background):
It looks like the latest is:
I blame @Virmach. He got his ticket queue down too low so now the DC had a fire to pump the queue back up.
O'it is my little guilty pleasure to nag VirmAch during defecting,
Ooo... just my tummy hurts recently
smartass shitposting satirist
Weekend has ended, and the NYCB027 node is still displaying as locked.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
I feel guilty because I name my VPS in NYC fire.
Seems like NYCB028 pre-emptive shut itself down several hours before fire started
Life seems to be intermittedly returning to NYCB?
Got brief moments of seemingly working control panel (showing server as shut down).
But didn't manage to boot up yet.
Messages vary between timeout and
"There seems to have been a slight problem with our database, please try again later."
Hopefully SOON™
Lads, what's the newest gen Ryzen of VirmAche, 5900x 5950x or 7900x?
smartass shitposting satirist
5950, don't think 7000 series is deployed anywhere.
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
Posted 51 minutes ago by dedicated
fire season is for forests datacenters
( chuckle ) Interesting ... what would constitute "out of bound management"?