@Mason said:
I'm gonna reach out to Steve (Hivelocity COO) to see if there's anything he can do go address the Tampa issues. Not sure if you've spoken with him directly before, but he sometimes frequents the LE forums. The incompetence is beyond disturbing (and should be to HV leadership as well).
Already raised it with Steve, so you don't need to (but you can, if you want)
@Mason said:
I'm gonna reach out to Steve (Hivelocity COO) to see if there's anything he can do go address the Tampa issues. Not sure if you've spoken with him directly before, but he sometimes frequents the LE forums. The incompetence is beyond disturbing (and should be to HV leadership as well).
Already raised it with Steve, so you don't need to (but you can, if you want)
Ahh... good looks. Any response?
Not yet, I'll let @VirMach know if I get anything back.
Looks like it's back. Uptime matches up with when I clicked reboot in the panel to confirm the server wasn't hung, so it looks like panel controls were functional at the time. @rpollestad any luck with yours? I'm going to go ahead and close out my ticket, everything seems fine on mine now.
Back up for about ~4 hours now. Thanks @VirMach for checking on it.
Looks like it's back. Uptime matches up with when I clicked reboot in the panel to confirm the server wasn't hung, so it looks like panel controls were functional at the time. @rpollestad any luck with yours? I'm going to go ahead and close out my ticket, everything seems fine on mine now.
Back up for about ~4 hours now. Thanks @VirMach for checking on it.
So, the server is back up but I'm getting extremely poor network performance @VirMach. (Like, ~104KB/s downloads.) I rebooted the server to see if that would fix it but no dice. Any thoughts?
@Mason said:
I doubletapped him and got a response, he's looking into it
Oh man I feel bad for him, it's one of those things where to actually look into it he has to probably spend hours at this point and even then what is he going to do, restructure the entire company for us? I mean I appreciate the fact that he's looking into it but...
HAHAHAHAHA. Oh. My. God. Okay at this point I'm going to assume everyone's aware of the PDU sagas. Same guy again, how he constantly brought up PDUs while we were trying to get something else done. We opened a ticket to have them box up servers to send back to us and I'll get to that in a minute, but we also asked about taking back the PDU they billed us for and he's already planning his next derailment:
While I have you here, we do have these two packaged PDUs that you've shipped in the past. Please see ticket 881631 for reference.
And of course he doesn't mention the other PDU we asked about. Okay and then the servers they're sending back, dear god.
We packaged those servers in two separate boxes. Each box is 24x8x34 and one weighs 53lbs and 77 lbs.
Please just picture that for a moment. Six servers. And they managed to put 3 of the largest ones in one box, these are servers, as in like 1.75" or so height each. 77 pounds with an inch and a half of padding on each side.
WTF. Maybe I'm just paranoid from them breaking during shipping from the lightest breeze, for a moment I thought maybe it was a bad idea that I sent some off in 36 pound boxes double stacked for the shorter ones but it just seems like this guy's determined to do anything within his power to cause grief. What do you guys think, is it a good idea to ship 77 pounds of servers in a 24x8x34 box? Going to phone in my physics major friend on this one. I'm scared to even ask what they used for padding, their response (probably) "padding?"
At this point I should just insist they put it all into one box instead, I feel like that's more productive and possible versus getting into a weeklong debate about having them split off. Maybe it'll be so heavy that UPS can't even slam it around. Even when we bought just the empty chassis the sellers didn't have the balls to try to fit three in an 8 inch depth box.
Hey @VirMach an Uber X from Tampa to Miami would cost ~$300 which would save you worrying about breakage and the servers getting lost in a UPS warehouse. Not totally sure they would handle 77 lbs boxes, but the size looks fine. Just putting that out there.
@FrankZ said:
Hey @VirMach an Uber X from Tampa to Miami would cost ~$300 which would save you worrying about breakage and the servers getting lost in a UPS warehouse. Not totally sure they would handle 77 lbs boxes, but the size looks fine. Just putting that out there.
That's actually a smart idea, I was going to make a joke about a road trip and I just wanted to get them back here to see what they even did to the servers and ensure they'll last a week before crashing but it might be worth it having them going to Miami for minimal damage and then have QN ship out the scraps back to us.
At this point I can't tell if the average Uber driver is better at handling servers than Hivelocity's remote hands are, so I'd say it's not a horrible idea. Uber driver could always just steal the hardware if it's going unaccompanied though.
I see no one copy-pasted e-mail, so I will be first!
Dear VirMach Customers,
We have been attempting to get TPAZ005 back online after the server correctly powered itself off during a thermal event (to protect itself from hardware damage.) Unfortunately, a routine thermal paste re-application turned into the hardware being damaged (either by the datacenter technician or according to the technician due to the thermal event.) We'll investigate the cause, but in the meantime, unfortunately, the datacenter in Tampa has been unresponsive over the weekend and were not able to assist us with a replacement. The technician did replace the CPU, but that one was damaged as well.
At this point in time we are requesting the datacenter partner ship back the server with overnight delivery, and assuming they are able to perform this task in a timely manner, it means tomorrow morning we will receive TPAZ005 (the physical server) and immediately work on migrating your virtual server to a functional replacement, most likely nearby in Miami. Your service's IP address will change.
We sincerely apologize for any inconvenience caused and lack of updates.
Ok I was kind of joking about the Uber, but if you want to send them from HV Tampa to QN Miami I would suggest getting a quote from these guys https://www.floridacouriers.com/ They are bonded, have small vehicle types for small loads, and have a regular route between Miami and Tampa. Maybe you can get a good rate as a back haul from Tampa to Miami.
Looks like it's back. Uptime matches up with when I clicked reboot in the panel to confirm the server wasn't hung, so it looks like panel controls were functional at the time. @rpollestad any luck with yours? I'm going to go ahead and close out my ticket, everything seems fine on mine now.
Back up for about ~4 hours now. Thanks @VirMach for checking on it.
So, the server is back up but I'm getting extremely poor network performance @VirMach. (Like, ~104KB/s downloads.) I rebooted the server to see if that would fix it but no dice. Any thoughts?
Mine isn't THAT bad, but it does seem like some network issues have popped up. When I checked this morning after it came up everything was normal on the network, latency looked good across the board on ping.pe and was easily able to max out the connection up and down. Just checked it again and I'm struggling to do 300mbit, with locations that are normally 2ms-10ms looking more like 35ms-60ms and the rest noticeably worse. I'll wait and see if it gets better, this is the first it's been like this for me, when so many San Jose VPSs were having network issues my dedi was rock solid. At least it's not dropping packets, just slow.
@FrankZ said:
Ok I was kind of joking about the Uber, but if you want to send them from HV Tampa to QN Miami I would suggest getting a quote from these guys https://www.floridacouriers.com/ They are bonded, have small vehicle types for small loads, and have a regular route between Miami and Tampa. Maybe you can get a good rate as a back haul from Tampa to Miami.
I swear with your luck @VirMach, you might just be better off to hire an employee that spends their time flying between dc locations fixing issues that come up rather than the cost of remote hands (no brains included) and server damage from them and shipping damage / loss.
@Daevien said:
I swear with your luck @VirMach, you might just be better off to hire an employee that spends their time flying between dc locations fixing issues that come up rather than the cost of remote hands (no brains included) and server damage from them and shipping damage / loss.
What's the pay on a job like that? Asking for a friend. ;-)
You spend an alarming amount of time shit-talking your DCs. Maybe that time is better invested in finding in-house people to solve your problems effectively.
@Daevien said:
I swear with your luck @VirMach, you might just be better off to hire an employee that spends their time flying between dc locations fixing issues that come up rather than the cost of remote hands (no brains included) and server damage from them and shipping damage / loss.
@Daevien said:
I swear with your luck @VirMach, you might just be better off to hire an employee that spends their time flying between dc locations fixing issues that come up rather than the cost of remote hands (no brains included) and server damage from them and shipping damage / loss.
@Daevien said:
I swear with your luck @VirMach, you might just be better off to hire an employee that spends their time flying between dc locations fixing issues that come up rather than the cost of remote hands (no brains included) and server damage from them and shipping damage / loss.
Worker: $30/hr✖️12
Airfare: $461 (tomorrow, MEM to TPA)
Meal per-diem: $69 (Tampa)
SLA credits: $200
Total cost: $1120
I'm genuinely going to do this soon once the last few pieces finish falling in place. My only real fear/concern is the negative productivity impact when it comes to random hotel WiFi and lack of screen space. Once I figure that out I don't mind perpetually travelling every other few days for several years.
Although I will say I don't look forward to planning out the flights and hotels, obviously it has to be economical.
You spend an alarming amount of time shit-talking your DCs. Maybe that time is better invested in finding in-house people to solve your problems effectively.
I mean since we're ignoring a lot of variables such as cost for all these in-house people, we might as well purchase a few datacenters to save even more time. Think of all the one to two hours I'll save per month not "shit-talking" the DCs. All I've essentially done is state what happened in relation to a delay or outage, many times verbatim, and then just genuinely shocked with how far it goes and doing a reality check to ensure I'm reacting appropriately with some joking/venting.
I've also praised DCs as much as I've probably talked "negatively" but that doesn't end up being a long drawn out story that requires context or evidence, nor does it grab as much attention.
Perhaps the "alarming" amount of time is directly correlated with the time, money, and grief caused.
Comments
Ahh... good looks. Any response?
Head Janitor @ LES • About • Rules • Support
Not yet, I'll let @VirMach know if I get anything back.
I doubletapped him and got a response, he's looking into it
Head Janitor @ LES • About • Rules • Support
Back up for about ~4 hours now. Thanks @VirMach for checking on it.
Ack, stop shooting potentially helpful people Mason :P
So, the server is back up but I'm getting extremely poor network performance @VirMach. (Like, ~104KB/s downloads.) I rebooted the server to see if that would fix it but no dice. Any thoughts?
Oh man I feel bad for him, it's one of those things where to actually look into it he has to probably spend hours at this point and even then what is he going to do, restructure the entire company for us? I mean I appreciate the fact that he's looking into it but...
HAHAHAHAHA. Oh. My. God. Okay at this point I'm going to assume everyone's aware of the PDU sagas. Same guy again, how he constantly brought up PDUs while we were trying to get something else done. We opened a ticket to have them box up servers to send back to us and I'll get to that in a minute, but we also asked about taking back the PDU they billed us for and he's already planning his next derailment:
And of course he doesn't mention the other PDU we asked about. Okay and then the servers they're sending back, dear god.
Please just picture that for a moment. Six servers. And they managed to put 3 of the largest ones in one box, these are servers, as in like 1.75" or so height each. 77 pounds with an inch and a half of padding on each side.
WTF. Maybe I'm just paranoid from them breaking during shipping from the lightest breeze, for a moment I thought maybe it was a bad idea that I sent some off in 36 pound boxes double stacked for the shorter ones but it just seems like this guy's determined to do anything within his power to cause grief. What do you guys think, is it a good idea to ship 77 pounds of servers in a 24x8x34 box? Going to phone in my physics major friend on this one. I'm scared to even ask what they used for padding, their response (probably) "padding?"
At this point I should just insist they put it all into one box instead, I feel like that's more productive and possible versus getting into a weeklong debate about having them split off. Maybe it'll be so heavy that UPS can't even slam it around. Even when we bought just the empty chassis the sellers didn't have the balls to try to fit three in an 8 inch depth box.
One word: compensation.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Hey @VirMach an Uber X from Tampa to Miami would cost ~$300 which would save you worrying about breakage and the servers getting lost in a UPS warehouse. Not totally sure they would handle 77 lbs boxes, but the size looks fine. Just putting that out there.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
That's actually a smart idea, I was going to make a joke about a road trip and I just wanted to get them back here to see what they even did to the servers and ensure they'll last a week before crashing but it might be worth it having them going to Miami for minimal damage and then have QN ship out the scraps back to us.
At this point I can't tell if the average Uber driver is better at handling servers than Hivelocity's remote hands are, so I'd say it's not a horrible idea. Uber driver could always just steal the hardware if it's going unaccompanied though.
Just like UPS did with my out-of-state purchased VCR, back in '99. Paraphrasing their tagline, at the time: "UPS, consider it nicked".
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I see no one copy-pasted e-mail, so I will be first!
I forgot I have one there, rip idler.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Ok I was kind of joking about the Uber, but if you want to send them from HV Tampa to QN Miami I would suggest getting a quote from these guys https://www.floridacouriers.com/ They are bonded, have small vehicle types for small loads, and have a regular route between Miami and Tampa. Maybe you can get a good rate as a back haul from Tampa to Miami.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Mine isn't THAT bad, but it does seem like some network issues have popped up. When I checked this morning after it came up everything was normal on the network, latency looked good across the board on ping.pe and was easily able to max out the connection up and down. Just checked it again and I'm struggling to do 300mbit, with locations that are normally 2ms-10ms looking more like 35ms-60ms and the rest noticeably worse. I'll wait and see if it gets better, this is the first it's been like this for me, when so many San Jose VPSs were having network issues my dedi was rock solid. At least it's not dropping packets, just slow.
Contacted them.
This is like "free migration" from Tampa to Miami? :-)
Darn, I could have saved $3 if I was just a little more patient. Lesson learned.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
As long as it doesn't go MIA en route (geddit?).
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I would like my Tampa migrate to Amsterdam, thanks!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
I swear with your luck @VirMach, you might just be better off to hire an employee that spends their time flying between dc locations fixing issues that come up rather than the cost of remote hands (no brains included) and server damage from them and shipping damage / loss.
is someone playing hard with fm04 or my vps got weird?
)
What's the pay on a job like that? Asking for a friend. ;-)
@VirMach
You spend an alarming amount of time shit-talking your DCs. Maybe that time is better invested in finding in-house people to solve your problems effectively.
I think virmach should sell virmach and live his life.
Ryzen 5900: $390/ea✖️2
Asrock 1U2LW-X470: $674/ea✖️1
Brainless hands: $100/hr✖️4
SLA credits: $500
Total cost: $2354
Worker: $30/hr✖️12
Airfare: $461 (tomorrow, MEM to TPA)
Meal per-diem: $69 (Tampa)
SLA credits: $200
Total cost: $1120
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
I'm genuinely going to do this soon once the last few pieces finish falling in place. My only real fear/concern is the negative productivity impact when it comes to random hotel WiFi and lack of screen space. Once I figure that out I don't mind perpetually travelling every other few days for several years.
Although I will say I don't look forward to planning out the flights and hotels, obviously it has to be economical.
I'll start a GoFundMe for the private jet.
Some how I knew you were going to say this. I think maybe we have been hanging out together a lot more than I thought.
Why is there no gif with that ?
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
I mean since we're ignoring a lot of variables such as cost for all these in-house people, we might as well purchase a few datacenters to save even more time. Think of all the one to two hours I'll save per month not "shit-talking" the DCs. All I've essentially done is state what happened in relation to a delay or outage, many times verbatim, and then just genuinely shocked with how far it goes and doing a reality check to ensure I'm reacting appropriately with some joking/venting.
I've also praised DCs as much as I've probably talked "negatively" but that doesn't end up being a long drawn out story that requires context or evidence, nor does it grab as much attention.
Perhaps the "alarming" amount of time is directly correlated with the time, money, and grief caused.