Virmach did get my VM on Node: NYCB032 migrated to Node: NYCB028 and it is back online with new IP. So that is a big
For those of you who were on NYCB032 check the billing panel to see if you are also up. I did not receive any email notice.
Peace on earth will come to stay, when we all live as LESbians every day.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
@VirMach Is there a particular reason there is no information at all on the network status page with respect to LAX2Z017 being down. From this thread I understand LAX has issues, but IDK why the status page would not be telling customers of that. Ticket creation just seems to hang when it tries to read the service screen.
@tetech said:
Meanwhile, is there a plan on what to do next with the non-functional VPSes in LA? Or still up in the air?
Still waiting for them to even begin to respond to our request properly. They should all be functional to some degree, just not globally, and too many variables plus we're pretty blind on it because of the previous issue I mentioned above.
@tulipyun said:
Quadranet hasn't updated its status page in three days.
VirMach apparently did better than they did, they emptied their status page about 6 hours ago.
They say “There are no Network Issues Currently”.
There's no useful information we could provide on the status page and we would be unable to properly communicate to everyone who is affected on what routes. The system already displays the relevant state of the service in regards to controls being down. As in what's stated there is accurate so there does not need to be a status update re-stating the same thing. I don't believe we've added status updates (usually, recently) any time there are control issues unless it's not actually a control issue and therefore it needs to be corrected/clarified.
If you guys disagree with that approach for the controls portion, let me know.
We already have a large ticket backlog which we're going to use instead in this case rather than referring customers to the network status page, and actually go through them one by one for each specific problem reported once (a) we actually have some communication from QN on the networking portion, and (b) once we have some eyes, aka, they need to complete our request so we have our access back.
If you guys disagree with that approach and think we should instead post a network status update and then close all tickets and refer everyone to the network status page, let me know, but I think that'd be the wrong way to do it here. We're not going to both post a network status update and go through tickets one by one though, it's one or the other. I know it's not ideal we're not replying to every ticket to confirm we've received it and we're not just doing nothing about it, except, well, we are doing nothing about it right now because we literally cannot do anything until QN fulfills our request. So we could reply to every single ticket and inform customers we are currently doing nothing about it but that will just also be bad and unhelpful and then we'll have everyone responding to complain which will derail all the tickets from focusing on the actual issue once we can do something about it since customers we'll have to treat the tickets both as potential complaints that need to be responded to, as well as reported issues.
@grasple said: maybe. or maybe they didn't "purchase" a building, but bought an interest in a lease, or their veiled corp financed the purchase of property and can default without recourse etc etc, maybe they paid an upfront price and offered the previous owners earnouts etc.
That's a fair point. QN probably had an unusual structured lease-back agreement, which makes sense if they don't want to be managing property all day. Either way, the money they’d get for the IP space pales in comparison to the value of the business they just destroyed.
I mean, selling off the IPs you assigned to your customers isn't great. Doing it without warning is profoundly stupid.
I really don't think they sold off any IP space, yet. I think the people that bought QN really thought it was an amazing idea to consolidate the addresses so they can make their own jobs easier in the future when they want to use/assign IP addresses to their customers under their primary brand or whatever. They probably had this as a thing they had to get done in Q4 2024, missed it, and tried just rushing it Q1 2025 instead. Some higher up probably said "I don't care, just get it done now" so they did.
But I guess no one thought that... well, no one did any thinking. They forgot there's customers that use these IP addresses for IPMI (we're not currently using them for IPMI, technically still possible to get in, it's just not going to be feasible without spending the entire day only doing that, but I won't get into it, still trying. I'll be focusing on that more heavily now that I know QN just intends on pretending to respond but not actually able to do anything because not enough staff members exist that actually can do things.)
They're definitely regretting it right now after finally maybe thinking about the implications of what they tried doing without thinking because now it's a lot of credits, a lot of overtime pay, and a lot of general headache for them even if they do not care about the customers and only care about themselves.
As for the value of the IP space paling in comparison, I really don't think so. They seem like the type of business where the entire value is just the IP addresses and everything else is seen as a liability, at least to someone purchasing them because they do not charge a lot for their services. The rest is probably just a "bonus" but they do still need to obviously maintain whatever they're receiving to keep the lights on at the facilities and keep everyone on payroll. Once they actually use the IP addresses for whatever other thing they're trying to do and get in the customers they want to get in, they'll probably be trying to think of ways to get rid of these existing customers. I've spoken with them, they really did not seem interested in having us as a customer when we reached out to their primary brand, in fact they straight up just said they didn't even want to give us a quote or discuss anything basically in the nicest way they could. They're focusing on getting big companies in. Like really big. I disagree with that approach, but that does seem to be their approach. So they really would probably care even less if you're renting a small old Xeon server from them.
I have a lot of thoughts on the matter so what I said isn't perfect and it's oversimplified and of course it's all speculation and just how I feel based on a lot of other things I didn't explain. It could be completely wrong.
make official announcement about LAX and update its progress
answer all tickets about LAX and point them to the announcement and close the ticket so all affected can monitor it on the correct page without draining more support resources
send email to all LAX affected so those who didnt even know if their vps was down, now know
Here's basically what I've verified so far though, it might be useful to you guys.
Nothing is "down." At least based on the spot checking I've done, if your VPS was online before any of this occurred, as in you do not require controls and nothing was modified/broken specifically by you inside your virtual server since then (and you locked yourself out and require VNC) then it should have connectivity. There's a small chance I missed some very specific thing, but servers are powered on, and the way it's set up, all virtual servers have connectivity.
There ARE severe routing problems, as in dead routes. So depending on where you are accessing the service from and if that specific routing is broken or not, it might appear as if you are completely down. What you could do if you really need access to your service, instead of waiting for us to reply to your ticket right now, and if you want to try something on your own, is to try accessing it from another route/location so a VPN, a different server, and it should work. I'd say there's above a 50% chance each time you try as long as it's fully randomized (so you aren't just trying from the same network just on a different VPS.)
I'm going to disappear for a while to really try to focus on some workarounds to try to do this ourselves as it's been over 12 hours where we haven't even received a pretend response from them and it doesn't seem like anything is going to get done as I mentioned above. This is just so we can try to get controls working again and assess everything further, it does not mean I can do anything about the routing issues on your VPS specifically, nor can I do anything if there are also routing issues for the controls. I mean I guess if it comes down to that I could also do further workarounds but let's get through this first hoop.
@FrankZ said:
I have VMs that are not available on the interweb on nodes LAXA031, LAXA029 and LAXA026. Others on nodes LAXA022 and LAXA019 are available. I can ping the VMs that are down from the ones that are up so ......
So for example, I've 100% verified that all the nodes you listed here are working in the way I described above. However, the routing issues are NOT the same on each that's tested, otherwise everything I described vaguely above would be much simpler. This means for example, a random IP tested on LAXA031 has different routing issues than a random IP tested on LAXA029, and so on. For example one can be accessed through Cogent in Denver, and the other can be accessed through HE in Los Angeles but not vice versa.
make official announcement about LAX and update its progress
What should the announcement say? Just so we are clear, there is no current progress. And we do not have an extensive list of every IP address and what routes are or aren't affected. All I could think of is the status page would say "Your service in LAX may or may not have some undefined routing issue. The system message regarding controls being down is correct."
If that sounds helpful, and others agree, I suppose we could do that. I don't think it sounds helpful.
answer all tickets about LAX and point them to the announcement and close the ticket so all affected can monitor it on the correct page without draining more support resources
We cannot properly answer these tickets without establishing the mentioned communication with QN and without first being able to get controls back/access back to some degree to be able to look into it. Otherwise it would just be guessing and picking off those tickets where we can just refer them to the network status page, but the way things currently work, it would be difficult to keep track of which is what (as in, the customer that's replying back to the ticket, does this need to be put in the queue where it has to be checked further later, or just refer to status page again? Or is the ticket now about requesting SLA, or is it about a complaint?) We'd be stuck having to process each ticket multiple times over and complicate the process where they could be categorized once and responded to in batches. I do agree we need to do something about them soon if we're unable to make any progress or receive any help from QN.
I guess one important thing that needs to be clarified if it wasn't already is that there's a ticket backlog. It means it will take some time to process these even once due to the volume.
send email to all LAX affected so those who didnt even know if their vps was down, now know
If someone is using their VPS and it is functional for them and their use case, to where they do not know there are any routing issues, it means it effectively does not matter for them. Based on the number of tickets we've received compared to what we would usually receive per server that is completely offline, I believe the number of customers actually affected, while not low, is not anywhere close to being high. All this would do is strain the backlog even more as we now also have more tickets with questions when we need to focus on the ones that definitely have a major problem. Once we actually establish some communication with QN, and if we need to focus on specific issues for specific customers, that would create a huge mess and slow down progress for those customers that actually need critical assistance.
There are currently routing issues with our LAX servers which may affect internet access and also access via our control panel.
The upstream provider (Quadranet) has been made aware of the problem and we currently cannot provide an update as to when these problems will be resolved.
If you have already opened a ticket based on these LAX connectivity issues, could you kindly close the ticket so that we may concentrate on getting a resolution as quickly as possible.
You can view our network status page for further updates.
On the network page :-
Routing issues affecting LAX servers.
Certain Internet routes are causing server access issues including access via the Control panel. Further updates will follow when the upstream (Quadranet) provides more information.
I'll provide some more information to you guys on why it's complicated to try to do a workaround and fix what I need to fix, as I was vague about it in my previous messages.
QuadraNet also changed the IPv4 assignment that's used for the connection to our switch.
They probably also did this for all their network equipment so depending on how it's getting routed on their end, how much of it broke from when they were doing all these changes, it could potentially explain some of these problems. Now this creates many other problems just on our side, with how the switch was configured, my ability to access it and check what everything looks like, what needs to potentially be changed, what ports may need to be "reset" to try to get the motherboard to reconnect BMC.
What we usually use for access has routing issues to my network, so I have to basically figure out the connection to that. I do have the ability to make some important connections by now. However, due to how the "maintenance" or whatever you want to call it was performed by them, and due to how everything was specifically set up on our networking and server side, and the motherboards we use and how they work in these scenarios, it's astronomically more difficult to try to do something that would normally be possible and very simple to do and therefore relies on remote hands as mentioned, which is effectively non-existent. What bothers me the most about this is that I was in Los Angeles and ready to quickly resolve this portion of the problem as mentioned, but they just closed our access ticket as already discussed. I would have driven over anyway, but they also very specifically mentioned not to call them which is how more realtime communication would occur once I'm at the facility since there's at least one entry door I need to get past to even try to get to their suite to bang on the door.
I've already considered flying back again, but no good flights (as in expensive or very long layover and only slightly expensive) without knowing if I'll even be able to actually access the facility since they still have not even answered me on how I would get facility access in this scenario. Once is fine but if I have to do this daily then it'll cost thousands (just flying back and forth hoping we can maybe one day get in.)
One human being that exists with access to the facility needs to be able to spend at least 5 minutes to help us before much else can happen. And that's for my workaround at this point. Otherwise a lot more time has to be spent (which is probably why they do not have 5 minutes to spare) to try to do something that would have otherwise taken a few minutes to complete since they have to repeatedly plug in and unplug KVM into like 30 devices and that's just for us. That's the reason I still want to fly in but I need some type of confirmation I'll be able to actually visit the facility because they did finally add our datacenter location under their maintenance or whatever which now officially means I can't visit based on my understanding of their policy (great policy by the way, hopefully I'm just misinterpreting it. I kind of get it but at the same time if you do not have enough staff to get something done what's everyone supposed to do? It'd obviously be better if people could actually do their own work to reduce the workload.) And the implications are kind of outrageous, so pretty much the absolute most important times where the facility needs to be accessed it'd be inaccessible. Actually when I say officially I mean they still haven't actually declared maintenance on our facility but it was vaguely finally referenced in some way right after I left Los Angeles ("have affected multiple services at both locations.")
@Virmach There is so much focus on VPS that Shared Hosting seems to be unaddressed. That the service may be "UP" does NOT mean everything is working. I.E. My Shared Hosting on LAX2Z017 web site is "UP", but that's it. I cannot access control panel or webmail, and even email in general isn't working for me so all the troubleshooting tools being inaccessible makes this a significant outage. The statement that nothing is "DOWN" is really missing the point that actual services, important services, like e-mail, control panel, etc, are affected. I have made a ticket, but I still think its pretty bizarre not to have a status even if it is "a third-party provider is being an ass" message.
@Brayvin said: @Virmach There is so much focus on VPS that Shared Hosting seems to be unaddressed. That the service may be "UP" does NOT mean everything is working. I.E. My Shared Hosting on LAX2Z017 web site is "UP", but that's it. I cannot access control panel or webmail, and even email in general isn't working for me so all the troubleshooting tools being inaccessible makes this a significant outage. The statement that nothing is "DOWN" is really missing the point that actual services, important services, like e-mail, control panel, etc, are affected. I have made a ticket, but I still think its pretty bizarre not to have a status even if it is "a third-party provider is being an ass" message.
It's up and same thing I mentioned above applies as well. The only part that's different and definitely problematic is that I do not know what message is being displayed since we are using the default setup, so I do not know if it properly states that controls are down or not.
The statement that nothing is "down" has a very important other statement attached to it and of course just by itself, it would be lying by omission.
If not, then this would apply if you are interested:
@VirMach said: What you could do if you really need access to your service, instead of waiting for us to reply to your ticket right now, and if you want to try something on your own, is to try accessing it from another route/location so a VPN, a different server, and it should work. I'd say there's above a 50% chance each time you try as long as it's fully randomized (so you aren't just trying from the same network just on a different VPS.)
Not perfect but since your service as a shared hosting is essentially this server and not a VPS, you need to access this server. So actually shared hosting does technically still have controls assuming the login cookies are being set. I'll double check in a moment but I vaguely remember checking this earlier (haven't got much sleep, memory not great.)
I've gotten QN to reply (it's the single person I trust, networking wise so we got very lucky right now.) I'll hear what he has to say and see what he can do and then likely create a network status page one way or another since we'll at least finally have one human being we spoke with that has any information at all to have something to say.
@sh97 said:
Looks like its back up at most places now? Got alerts from my monitors on different networks.
May be related to this:
@VirMach said: I've gotten QN to reply (it's the single person I trust, networking wise so we got very lucky right now.) I'll hear what he has to say and see what he can do and then likely create a network status page one way or another since we'll at least finally have one human being we spoke with that has any information at all to have something to say.
I spoke about this person earlier to some degree but it might have been somewhere else. Basically, this is the one person I've ever seen do networking properly at QN. There are two other people I've noticed to some degree. One of them is the one that's broken Chicago a few times and also incorrectly said we had an incoming attack (they got attacked, he couldn't find the source so he just nullrouted all our blocks or something like that for a day maybe?) The other is someone who has "Network Engineer" or something in their signature but I've actually never seen them ever do anything and until a few days ago I didn't even notice they did network related tasks, I just assumed they were outsourced L1 or "general" support.
I'll be going through the ticket backlogs and won't be back here until that's cleaned up. If you do still have an issue and haven't opened a ticket, go ahead and put one in as I'd need to specifically check your service in any case and would not be able to be helpful here at this point.
If I notice any other more specific issues, I'll update the network status page with those specific issues now that we're able to identify what those issues may be.
Edit -- oh and I forgot to mention the most important part, I'm fixing controls now.
@Brayvin said: @Virmach There is so much focus on VPS that Shared Hosting seems to be unaddressed. That the service may be "UP" does NOT mean everything is working. I.E. My Shared Hosting on LAX2Z017 web site is "UP", but that's it. I cannot access control panel or webmail, and even email in general isn't working for me so all the troubleshooting tools being inaccessible makes this a significant outage.
If not, then this would apply if you are interested:
@VirMach said: What you could do if you really need access to your service, instead of waiting for us to reply to your ticket right now, and if you want to try something on your own, is to try accessing it from another route/location so a VPN, a different server, and it should work. I'd say there's above a 50% chance each time you try as long as it's fully randomized (so you aren't just trying from the same network just on a different VPS.)
Not perfect but since your service as a shared hosting is essentially this server and not a VPS, you need to access this server. So actually shared hosting does technically still have controls assuming the login cookies are being set. I'll double check in a moment but I vaguely remember checking this earlier (haven't got much sleep, memory not great.)
Thanks, I tried through a VPN and finally did manage to get control panel to work even with https still broken. I did eventually get email client issues sorted, and had to switch it to allow insecure connections, so there are still issues, but that's better than nothing!?
This server could not prove that it is lax2z017.virm.ac; its security certificate is not trusted by your device's operating system. This may be caused by a misconfiguration or an attacker intercepting your connection.
@VirMach said: I really don't think they sold off any IP space, yet.
Fair. I was effectively parroting the speculation on WHT.
@VirMach said: As for the value of the IP space paling in comparison, I really don't think so. [SNIP] I've spoken with them, they really did not seem interested in having us as a customer when we reached out to their primary brand, in fact they straight up just said they didn't even want to give us a quote or discuss anything basically in the nicest way they could. They're focusing on getting big companies in. Like really big. I disagree with that approach, but that does seem to be their approach. So they really would probably care even less if you're renting a small old Xeon server from them.
If they're trying to be like Crown Castle, or some equally big infrastructure player, this may have been the cheapest way to get rid of their small and medium size customers. But if their intention was to keep the business as it was, I think they really goofed up and will have a lasting impact on their revenues going forward.
But this does seem like the perfect time to run to Lowes, buy some 2x4s, and start a colocation company.
I just realized that my VM was migrated from PHXZ004 to LAX1Z017, so I installed the Debian 12 template. Is the template still broken?
I fixed the network issue by manually configuring /etc/network/interfaces, but apt update failed. Since I couldn’t resolve those issues, I installed Debian 11 instead…
Comments
mmhmm, more customers left, more free IP space
Virmach did get my VM on Node: NYCB032 migrated to Node: NYCB028 and it is back online with new IP. So that is a big
For those of you who were on NYCB032 check the billing panel to see if you are also up. I did not receive any email notice.
Peace on earth will come to stay, when we all live as LESbians every day.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
@VirMach Is there a particular reason there is no information at all on the network status page with respect to LAX2Z017 being down. From this thread I understand LAX has issues, but IDK why the status page would not be telling customers of that. Ticket creation just seems to hang when it tries to read the service screen.
The SLA credits will be insane.
No hostname left!
thats only if you paid
I bench YABS 24/7/365 unless it's a leap year.
Still waiting for them to even begin to respond to our request properly. They should all be functional to some degree, just not globally, and too many variables plus we're pretty blind on it because of the previous issue I mentioned above.
There's no useful information we could provide on the status page and we would be unable to properly communicate to everyone who is affected on what routes. The system already displays the relevant state of the service in regards to controls being down. As in what's stated there is accurate so there does not need to be a status update re-stating the same thing. I don't believe we've added status updates (usually, recently) any time there are control issues unless it's not actually a control issue and therefore it needs to be corrected/clarified.
If you guys disagree with that approach for the controls portion, let me know.
We already have a large ticket backlog which we're going to use instead in this case rather than referring customers to the network status page, and actually go through them one by one for each specific problem reported once (a) we actually have some communication from QN on the networking portion, and (b) once we have some eyes, aka, they need to complete our request so we have our access back.
If you guys disagree with that approach and think we should instead post a network status update and then close all tickets and refer everyone to the network status page, let me know, but I think that'd be the wrong way to do it here. We're not going to both post a network status update and go through tickets one by one though, it's one or the other. I know it's not ideal we're not replying to every ticket to confirm we've received it and we're not just doing nothing about it, except, well, we are doing nothing about it right now because we literally cannot do anything until QN fulfills our request. So we could reply to every single ticket and inform customers we are currently doing nothing about it but that will just also be bad and unhelpful and then we'll have everyone responding to complain which will derail all the tickets from focusing on the actual issue once we can do something about it since customers we'll have to treat the tickets both as potential complaints that need to be responded to, as well as reported issues.
I really don't think they sold off any IP space, yet. I think the people that bought QN really thought it was an amazing idea to consolidate the addresses so they can make their own jobs easier in the future when they want to use/assign IP addresses to their customers under their primary brand or whatever. They probably had this as a thing they had to get done in Q4 2024, missed it, and tried just rushing it Q1 2025 instead. Some higher up probably said "I don't care, just get it done now" so they did.
But I guess no one thought that... well, no one did any thinking. They forgot there's customers that use these IP addresses for IPMI (we're not currently using them for IPMI, technically still possible to get in, it's just not going to be feasible without spending the entire day only doing that, but I won't get into it, still trying. I'll be focusing on that more heavily now that I know QN just intends on pretending to respond but not actually able to do anything because not enough staff members exist that actually can do things.)
They're definitely regretting it right now after finally maybe thinking about the implications of what they tried doing without thinking because now it's a lot of credits, a lot of overtime pay, and a lot of general headache for them even if they do not care about the customers and only care about themselves.
As for the value of the IP space paling in comparison, I really don't think so. They seem like the type of business where the entire value is just the IP addresses and everything else is seen as a liability, at least to someone purchasing them because they do not charge a lot for their services. The rest is probably just a "bonus" but they do still need to obviously maintain whatever they're receiving to keep the lights on at the facilities and keep everyone on payroll. Once they actually use the IP addresses for whatever other thing they're trying to do and get in the customers they want to get in, they'll probably be trying to think of ways to get rid of these existing customers. I've spoken with them, they really did not seem interested in having us as a customer when we reached out to their primary brand, in fact they straight up just said they didn't even want to give us a quote or discuss anything basically in the nicest way they could. They're focusing on getting big companies in. Like really big. I disagree with that approach, but that does seem to be their approach. So they really would probably care even less if you're renting a small old Xeon server from them.
I have a lot of thoughts on the matter so what I said isn't perfect and it's oversimplified and of course it's all speculation and just how I feel based on a lot of other things I didn't explain. It could be completely wrong.
you should do both:
I bench YABS 24/7/365 unless it's a leap year.
Here's basically what I've verified so far though, it might be useful to you guys.
Nothing is "down." At least based on the spot checking I've done, if your VPS was online before any of this occurred, as in you do not require controls and nothing was modified/broken specifically by you inside your virtual server since then (and you locked yourself out and require VNC) then it should have connectivity. There's a small chance I missed some very specific thing, but servers are powered on, and the way it's set up, all virtual servers have connectivity.
There ARE severe routing problems, as in dead routes. So depending on where you are accessing the service from and if that specific routing is broken or not, it might appear as if you are completely down. What you could do if you really need access to your service, instead of waiting for us to reply to your ticket right now, and if you want to try something on your own, is to try accessing it from another route/location so a VPN, a different server, and it should work. I'd say there's above a 50% chance each time you try as long as it's fully randomized (so you aren't just trying from the same network just on a different VPS.)
I'm going to disappear for a while to really try to focus on some workarounds to try to do this ourselves as it's been over 12 hours where we haven't even received a pretend response from them and it doesn't seem like anything is going to get done as I mentioned above. This is just so we can try to get controls working again and assess everything further, it does not mean I can do anything about the routing issues on your VPS specifically, nor can I do anything if there are also routing issues for the controls. I mean I guess if it comes down to that I could also do further workarounds but let's get through this first hoop.
So for example, I've 100% verified that all the nodes you listed here are working in the way I described above. However, the routing issues are NOT the same on each that's tested, otherwise everything I described vaguely above would be much simpler. This means for example, a random IP tested on LAXA031 has different routing issues than a random IP tested on LAXA029, and so on. For example one can be accessed through Cogent in Denver, and the other can be accessed through HE in Los Angeles but not vice versa.
What should the announcement say? Just so we are clear, there is no current progress. And we do not have an extensive list of every IP address and what routes are or aren't affected. All I could think of is the status page would say "Your service in LAX may or may not have some undefined routing issue. The system message regarding controls being down is correct."
If that sounds helpful, and others agree, I suppose we could do that. I don't think it sounds helpful.
We cannot properly answer these tickets without establishing the mentioned communication with QN and without first being able to get controls back/access back to some degree to be able to look into it. Otherwise it would just be guessing and picking off those tickets where we can just refer them to the network status page, but the way things currently work, it would be difficult to keep track of which is what (as in, the customer that's replying back to the ticket, does this need to be put in the queue where it has to be checked further later, or just refer to status page again? Or is the ticket now about requesting SLA, or is it about a complaint?) We'd be stuck having to process each ticket multiple times over and complicate the process where they could be categorized once and responded to in batches. I do agree we need to do something about them soon if we're unable to make any progress or receive any help from QN.
I guess one important thing that needs to be clarified if it wasn't already is that there's a ticket backlog. It means it will take some time to process these even once due to the volume.
If someone is using their VPS and it is functional for them and their use case, to where they do not know there are any routing issues, it means it effectively does not matter for them. Based on the number of tickets we've received compared to what we would usually receive per server that is completely offline, I believe the number of customers actually affected, while not low, is not anywhere close to being high. All this would do is strain the backlog even more as we now also have more tickets with questions when we need to focus on the ones that definitely have a major problem. Once we actually establish some communication with QN, and if we need to focus on specific issues for specific customers, that would create a huge mess and slow down progress for those customers that actually need critical assistance.
The invoice indicates that it has been paid.
We received a sack of potatoes at our polar station last night.
Is that our SLA credits?
No hostname left!
How about email sent to LAX customers :-
On the network page :-
I'll provide some more information to you guys on why it's complicated to try to do a workaround and fix what I need to fix, as I was vague about it in my previous messages.
QuadraNet also changed the IPv4 assignment that's used for the connection to our switch.
They probably also did this for all their network equipment so depending on how it's getting routed on their end, how much of it broke from when they were doing all these changes, it could potentially explain some of these problems. Now this creates many other problems just on our side, with how the switch was configured, my ability to access it and check what everything looks like, what needs to potentially be changed, what ports may need to be "reset" to try to get the motherboard to reconnect BMC.
What we usually use for access has routing issues to my network, so I have to basically figure out the connection to that. I do have the ability to make some important connections by now. However, due to how the "maintenance" or whatever you want to call it was performed by them, and due to how everything was specifically set up on our networking and server side, and the motherboards we use and how they work in these scenarios, it's astronomically more difficult to try to do something that would normally be possible and very simple to do and therefore relies on remote hands as mentioned, which is effectively non-existent. What bothers me the most about this is that I was in Los Angeles and ready to quickly resolve this portion of the problem as mentioned, but they just closed our access ticket as already discussed. I would have driven over anyway, but they also very specifically mentioned not to call them which is how more realtime communication would occur once I'm at the facility since there's at least one entry door I need to get past to even try to get to their suite to bang on the door.
I've already considered flying back again, but no good flights (as in expensive or very long layover and only slightly expensive) without knowing if I'll even be able to actually access the facility since they still have not even answered me on how I would get facility access in this scenario. Once is fine but if I have to do this daily then it'll cost thousands (just flying back and forth hoping we can maybe one day get in.)
One human being that exists with access to the facility needs to be able to spend at least 5 minutes to help us before much else can happen. And that's for my workaround at this point. Otherwise a lot more time has to be spent (which is probably why they do not have 5 minutes to spare) to try to do something that would have otherwise taken a few minutes to complete since they have to repeatedly plug in and unplug KVM into like 30 devices and that's just for us. That's the reason I still want to fly in but I need some type of confirmation I'll be able to actually visit the facility because they did finally add our datacenter location under their maintenance or whatever which now officially means I can't visit based on my understanding of their policy (great policy by the way, hopefully I'm just misinterpreting it. I kind of get it but at the same time if you do not have enough staff to get something done what's everyone supposed to do? It'd obviously be better if people could actually do their own work to reduce the workload.) And the implications are kind of outrageous, so pretty much the absolute most important times where the facility needs to be accessed it'd be inaccessible. Actually when I say officially I mean they still haven't actually declared maintenance on our facility but it was vaguely finally referenced in some way right after I left Los Angeles ("have affected multiple services at both locations.")
In @jsg accent: the machine is totally up and the network is totally reachable.
No hostname left!
@Virmach There is so much focus on VPS that Shared Hosting seems to be unaddressed. That the service may be "UP" does NOT mean everything is working. I.E. My Shared Hosting on LAX2Z017 web site is "UP", but that's it. I cannot access control panel or webmail, and even email in general isn't working for me so all the troubleshooting tools being inaccessible makes this a significant outage. The statement that nothing is "DOWN" is really missing the point that actual services, important services, like e-mail, control panel, etc, are affected. I have made a ticket, but I still think its pretty bizarre not to have a status even if it is "a third-party provider is being an ass" message.
It's up and same thing I mentioned above applies as well. The only part that's different and definitely problematic is that I do not know what message is being displayed since we are using the default setup, so I do not know if it properly states that controls are down or not.
The statement that nothing is "down" has a very important other statement attached to it and of course just by itself, it would be lying by omission.
Are you able to access this:
https://lax2z017.virm.ac:2083/
If not, then this would apply if you are interested:
Not perfect but since your service as a shared hosting is essentially this server and not a VPS, you need to access this server. So actually shared hosting does technically still have controls assuming the login cookies are being set. I'll double check in a moment but I vaguely remember checking this earlier (haven't got much sleep, memory not great.)
I've gotten QN to reply (it's the single person I trust, networking wise so we got very lucky right now.) I'll hear what he has to say and see what he can do and then likely create a network status page one way or another since we'll at least finally have one human being we spoke with that has any information at all to have something to say.
Looks like its back up at most places now? Got alerts from my monitors on different networks.
The Ultimate Speedtest Script | Get Instant Alerts on new LES/LET deals | Cheap VPS Deals | VirMach Flash Sales Notifier
FREE KVM VPS - FreeVPS.org | FREE LXC VPS - MicroLXC
can confirm, all my LA VPS is now online
May be related to this:
I spoke about this person earlier to some degree but it might have been somewhere else. Basically, this is the one person I've ever seen do networking properly at QN. There are two other people I've noticed to some degree. One of them is the one that's broken Chicago a few times and also incorrectly said we had an incoming attack (they got attacked, he couldn't find the source so he just nullrouted all our blocks or something like that for a day maybe?) The other is someone who has "Network Engineer" or something in their signature but I've actually never seen them ever do anything and until a few days ago I didn't even notice they did network related tasks, I just assumed they were outsourced L1 or "general" support.
I'll be going through the ticket backlogs and won't be back here until that's cleaned up. If you do still have an issue and haven't opened a ticket, go ahead and put one in as I'd need to specifically check your service in any case and would not be able to be helpful here at this point.
If I notice any other more specific issues, I'll update the network status page with those specific issues now that we're able to identify what those issues may be.
Edit -- oh and I forgot to mention the most important part, I'm fixing controls now.
It's getting fragrant again.
No hostname left!
I have been having trouble with that URL.
Thanks, I tried through a VPN and finally did manage to get control panel to work even with https still broken. I did eventually get email client issues sorted, and had to switch it to allow insecure connections, so there are still issues, but that's better than nothing!?
Routing works, but certificate is bad.
This server could not prove that it is lax2z017.virm.ac; its security certificate is not trusted by your device's operating system. This may be caused by a misconfiguration or an attacker intercepting your connection.
No hostname left!
Fair. I was effectively parroting the speculation on WHT.
If they're trying to be like Crown Castle, or some equally big infrastructure player, this may have been the cheapest way to get rid of their small and medium size customers. But if their intention was to keep the business as it was, I think they really goofed up and will have a lasting impact on their revenues going forward.
But this does seem like the perfect time to run to Lowes, buy some 2x4s, and start a colocation company.
LA's network is finally back to normal.
VirMach should either offer the victims a million dollars in compensation or issue them coupons that are useless.
Right?
Have the honor of being the crybaby who pays $20 for a 128MB VPS at VirMach in 2023.
I just realized that my VM was migrated from PHXZ004 to LAX1Z017, so I installed the Debian 12 template. Is the template still broken?
I fixed the network issue by manually configuring /etc/network/interfaces, but apt update failed. Since I couldn’t resolve those issues, I installed Debian 11 instead…
Up since 13 hours ago. Should we save whatever we can from LA or will it go down again... 🤔
better to backup just in case. clearly out of Virmach control
I bench YABS 24/7/365 unless it's a leap year.