@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Shit... you're right. Won't be long until we get invaded worse than when this thread first started
@soulchief said:
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Someone had been updating the basic points last i checked over there but not the official Virmach account. I'm pretty sure he prob doesn't want to deal with getting bombarded and side tracked more by their insanity, it makes my head hurt going there for anything.
@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
I think the "I received the mail at xxxx hours while my vps has been down for over 10 hours... yadaa yadaa" messages can be read, so there is some chatter.
@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
A couple of guys have been posting a few bits of the information from here on OGF, so I guess they know a bit of what's going on.
I don't think it's just this issue, it seems VirMach has been a lot freer with explanations of what's been going on with the migrations here for weeks and people here are generally supportive, but on OGF it's just full of people moaning non-stop, so I can see why he's less keen to poke his head up there.
@Advin said:
I have a question, @VirMach, do you even sleep? You work damn hard from what I've gathered so far in this thread, jesus.
I can't even begin to fathom the volume of tickets that you've probably gotten so far.
Nah virmach just pretends to be working, I am trying to find new ip to which my vps was migrated to still the services detail page does not open, i get timeout after waiting too long for it to open?
I have been unable to use that vps for more than a month still I see no resolution,
Heck even boomer fixes his pannel in a week
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
I understand IPs change, and with rental being more common now and pricing becoming more competitive its likely to occur more frequency, but surely some notice should be provided for this.
@VirMach said: This is something we haven't addressed yet. I'm trying to think of every way we can bring it down to a meaningful level without just auto crediting and closing the tickets as it will of course upset a lot of people.
I hesitate to levy criticism as you're clearly working extremely hard and are going through a bad time, but has an alternative been considered to automation that often breaks things and auto-closing tickets which annoys people and probably makes them open even more tickets?
For example: automation that works and covers all/most cases, temporary increase in support to go through the tickets (even briefly), etc.
@adly said: I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
Sometimes no, sometimes yes. Can't say what % - if you clicked "Ryzen migrate" manually you were getting 2nd IP 'reserved' and it was likely used for migration. If your server ended in different location that Ryzen migrate button - you had no idea about IP [or sometimes you had - both were shown in panel, while server migrating, sometiems not]. If you got emergency migration - you had no idea about IP. Current batch is people from CC servers... so totally emergency migration, so no IPs known
@VirMach said: We finally set up Chicago Hivelocity. Since Chicago servers already got moved, a lot of people will be going there from former Buffalo location but it'll also be available on Ryzen button.
You likely have lots of other stuff on your mind right now, but will we get the Ryzen migration button enabled for services that have been auto-migrated to a (more or less) arbitrary location eventually?
This was already supposed to be enabled but there's so many segmentation of packages and locations that there's obviously some cases where it's not going through. I'm going to see if we can change the filter today to make it more of a free-for-all.
@vyas said:
Quality difference in response between here and LET to the situation . As expected, lot more PMS there
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Shit... you're right. Won't be long until we get invaded worse than when this thread first started
Location: NYC Metro. No other location currently immediately available. Future Locations: San Jose, Los Angeles. Potential Other Locations: Dallas, Las Vegas, Denver, Phoenix, Jacksonville.
Processors: E3. Mostly v2. Some may be substituted for Dual E5 as we run out.
RAM: 16GB, 32GB.
Disk Configurations: 2TB HDD, 240GB SSD, 1TB SSD, 2TB SSD, 4TB HDD, 500GB SSD Note: 2TB means 1x2TB or 2x1TB, and so on.
IPv4: /29 IPv4 Default.
Currently available are all: 16GB RAM, 2TB HDD, NYC Metro, /29 IPv4 with CentOS6/7 installed. Some have IPMI, some do not. At least one or two have no OS but IPMI. One doesn't have a main IP but has IPMI, so there's definitely some issues that need to be ironed out on a few.
In this case, I probably can wait as I prefer West coast DC like San Jose or Los Angeles for my replacement Dedi. Hope all these issues are sorted out, resolved so we can return back as normal again.
I had to end up delaying the provisions as we had to have them re-provisioned due to it missing a key piece of information. That's been done now and we have a lot of San Jose servers so I'll be able to honor those requests now.
@soulchief said:
I don't think VirMach has posted over there since this CC issue happened so there's probably a bunch of people who don't use LES going "omg zero communication!!"
Someone had been updating the basic points last i checked over there but not the official Virmach account. I'm pretty sure he prob doesn't want to deal with getting bombarded and side tracked more by their insanity, it makes my head hurt going there for anything.
It's a lot more difficult being ambiguous there given the hostile environment. It's easier communicating here with more like-minded individuals. @ OGF, I feel like everything would be misinterpreted right now.
@adly said:
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
For my vps7 migration from ColonCrossing ATL to Flexential ATL:
I pressed "Ryzen migrate to New York" button, it gave secondary IP but didn't migrate.
There was a notice that migration out of ColonCrossing may happen sometime, but not when.
One day, the machine went offline.
Billing panel shows new IP in Flexential ATL, which differs from the secondary IP assigned before.
Secondary IP disappears after several days.
For my vps1 migration from ColonCrossing LAX to DediPath SJC:
I pressed "Ryzen migrate to San Jose" button, it immediately migrated and showed IP, which worked.
There was a notice that some IPs in SJC may change, but not when.
One day, I rebooted the OS and it received different IP from DHCP. The previous IP worked up until this point.
Now I'm using the new IP from DHCP.
In vps1 step 3, I suppose the renumbering happened sometime ago but I'm unknowningly using somebody else's IP until I rebooted.
I was not told about when the renumbering would occur or what's the new IP, until I did the reboot myself.
@Advin said:
I have a question, @VirMach, do you even sleep? You work damn hard from what I've gathered so far in this thread, jesus.
I can't even begin to fathom the volume of tickets that you've probably gotten so far.
Nah virmach just pretends to be working, I am trying to find new ip to which my vps was migrated to still the services detail page does not open, i get timeout after waiting too long for it to open?
I have been unable to use that vps for more than a month still I see no resolution,
Heck even boomer fixes his pannel in a week
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
I understand IPs change, and with rental being more common now and pricing becoming more competitive its likely to occur more frequency, but surely some notice should be provided for this.
We couldn't do the usual rounds of communication given the situation. We couldn't do a lot of things we would normally ensure in terms of communication and notices. A lot of it was being directly misinterpreted/quoted to paint a false picture as well. I don't know if that makes sense but it was easy for one party to go "you mentioned XX, therefore YY, and we're going to apply this to everything else as well." I'll probably be able to reference this fully in a couple years and clear things up.
We were operating in a very hostile and unstable environment and had to be careful in multiple ways. If we were too specific, hostile actors could take advantage of it. I do not believe we would have been able to move everyone in that case without being sabotaged.
@VirMach said: We were operating in a very hostile and unstable environment and had to be careful in multiple ways. If we were too specific, hostile actors could take advantage of it. I do not believe we would have been able to move everyone in that case without being sabotaged.
@VirMach said: This is something we haven't addressed yet. I'm trying to think of every way we can bring it down to a meaningful level without just auto crediting and closing the tickets as it will of course upset a lot of people.
I hesitate to levy criticism as you're clearly working extremely hard and are going through a bad time, but has an alternative been considered to automation that often breaks things and auto-closing tickets which annoys people and probably makes them open even more tickets?
For example: automation that works and covers all/most cases, temporary increase in support to go through the tickets (even briefly), etc.
I understand what you're saying here and for this to have worked, for the most part, we would have had to invest a significant amount of time in doing a lot of that beforehand.
Temporary increase in support was difficult to coordinate. It would have ended up with people receiving frustratingly bad advice. I'm still trying to come up with a plan in sorting tickets and providing actual responses to avoid auto closing everything again. It didn't work out last time but this time we have the added advantage of everything actually finally settling down instead of other changes being stacked on that cause further grievances.
@adly said:
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
For my vps7 migration from ColonCrossing ATL to Flexential ATL:
I pressed "Ryzen migrate to New York" button, it gave secondary IP but didn't migrate.
There was a notice that migration out of ColonCrossing may happen sometime, but not when.
One day, the machine went offline.
Billing panel shows new IP in Flexential ATL, which differs from the secondary IP assigned before.
Secondary IP disappears after several days.
For my vps1 migration from ColonCrossing LAX to DediPath SJC:
I pressed "Ryzen migrate to San Jose" button, it immediately migrated and showed IP, which worked.
There was a notice that some IPs in SJC may change, but not when.
One day, I rebooted the OS and it received different IP from DHCP. The previous IP worked up until this point.
Now I'm using the new IP from DHCP.
In vps1 step 3, I suppose the renumbering happened sometime ago but I'm unknowningly using somebody else's IP until I rebooted.
I was not told about when the renumbering would occur or what's the new IP, until I did the reboot myself.
One major problem that caused a lot of these added issues was the fact that SVM or perhaps rather KVM virtualization in general operates in a way where several steps of reboots have to occur. When it's being done for a lot of services at once, things are bound to go wrong. We had to adjust timeouts based on each server's load and it wasn't always perfect.
We couldn't just set a 5 minute timeout per VM to ensure it goes 100% smoothly because if a hundred plus have to get done at once, that's a 10 hour queue.
If at any time this "broke" in any way, it left many servers in various partial states and there were multiple steps of adding and removing IPs. Sometimes one step would break but not another. We had to re-code everything over and over and over so depending on when it was done, and with the other added bugs, the situation was unpredictable. Now add in human error of having to keep track of these states over a hundred plus servers.
I didn't code out the IP change script for the migration without data, but when reviewing the code it looked fine. The multiple IP issue seems to be related to how SVM deals with timeouts and overlaps. Sometimes it'd assign an IP, but not really assign it in a way where it got locked away from it being re-assigned to another and the default behavior seems to be it'd just do it all over again, hence some people got multiple functional IPs.
@FrankZ said: Holy shit LET/LEB has gone completely vindictive on VirMach.
IMO, totally uncalled for, posting that. Raindog must be needing the extra content bilohbucks! Also found it interesting that LordSpock 'stole' my phrase of mass exodus (check post timings).
FrankZ should perhaps take the same stance as me, with (no) regards to OGF i.e. monitor but don't comment etc. just saying.
Now what a kind offer from @MikePT and one that genuinely may help, given the kudos that he's appropriated. Nice one.
@FrankZ said: Holy shit LET/LEB has gone completely vindictive on VirMach.
IMO, totally uncalled for, posting that. Raindog must be needing the extra content bilohbucks! Also found it interesting that LordSpock 'stole' my phrase of mass exodus (check post timings).
Now what a kind offer from @MikePT and one that genuinely may help, given the kudos that he's appropriated. Nice one.
I worked for Virmach for quite some time. I can't assist this and simply ignore it. They're good people especially the guy that posts here. They/he helped me a lot when I needed most.
@VirMach said: I can't wait until I can go back to being an alcoholic. I haven't been able to have a serious drink in over 1-2 years.
Dinnae do it, man. ( Us Scots know a thing or three about alchies! ) Goddamn Guinness got rid of my slim belly and with current meds, I'm limiting m'self to one can every two weeks, or so. Could do with a wee dram o' whisky though. [No single malt left in the house, for a while. Boo.]
@AlwaysSkint said: FrankZ should perhaps take the same stance as me, with (no) regards to OGF i.e. monitor but don't comment etc. just saying.
I've been mostly staying away from commenting at the OGF, other than trying to help folks out, for a while now. As @Fat32 has always had my respect, I was trying to be part of the solution, instead of just not being part of the problem. He also seems to be very quiet these days, outside of the Mega Threads at the OGF, so maybe your right and it's time I start thinking about what to do next.
@FrankZ said:
I've been mostly staying away from commenting at the OGF, other than trying to help folks out, for a while now. As @Fat32 has always had my respect, I was trying to be part of the solution, instead of just not being part of the problem. He also seems to be very quiet these days, outside of the Mega Threads at the OGF, so maybe your right and it's time I start thinking about what to do next.
Thanks FrankZ, I have seen you helping out in the thread a lot for people that are having issues with VirMach. You are amazing
For myself, I start to think about my life more... probably having my third/quarter-life crisis now
Comments
Shit... you're right. Won't be long until we get invaded worse than when this thread first started
Head Janitor @ LES • About • Rules • Support
Lol, just found out about LES yesterday.
I knew about LES for a couple years, but every time I went to LES it always had the same thread names as LET so I thought it was just a backup of LET.
Someone had been updating the basic points last i checked over there but not the official Virmach account. I'm pretty sure he prob doesn't want to deal with getting bombarded and side tracked more by their insanity, it makes my head hurt going there for anything.
I noticed that, caught me off guard a bit as well.
I think the "I received the mail at xxxx hours while my vps has been down for over 10 hours... yadaa yadaa" messages can be read, so there is some chatter.
blog | exploring visually |
A couple of guys have been posting a few bits of the information from here on OGF, so I guess they know a bit of what's going on.
I don't think it's just this issue, it seems VirMach has been a lot freer with explanations of what's been going on with the migrations here for weeks and people here are generally supportive, but on OGF it's just full of people moaning non-stop, so I can see why he's less keen to poke his head up there.
Virmach 2022 The Year Of The Perfect Storm.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
Virmach's new partnership will be with a whiskey company after all of this to recover and relax
I've not closely followed the migration process, but one thing I've noticed is that a few people seem unsure of their new IP following migration. Was the new IP not notified prior to the migration with some notice?
I understand IPs change, and with rental being more common now and pricing becoming more competitive its likely to occur more frequency, but surely some notice should be provided for this.
I hesitate to levy criticism as you're clearly working extremely hard and are going through a bad time, but has an alternative been considered to automation that often breaks things and auto-closing tickets which annoys people and probably makes them open even more tickets?
For example: automation that works and covers all/most cases, temporary increase in support to go through the tickets (even briefly), etc.
Sometimes no, sometimes yes. Can't say what % - if you clicked "Ryzen migrate" manually you were getting 2nd IP 'reserved' and it was likely used for migration. If your server ended in different location that Ryzen migrate button - you had no idea about IP [or sometimes you had - both were shown in panel, while server migrating, sometiems not]. If you got emergency migration - you had no idea about IP. Current batch is people from CC servers... so totally emergency migration, so no IPs known
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
This was already supposed to be enabled but there's so many segmentation of packages and locations that there's obviously some cases where it's not going through. I'm going to see if we can change the filter today to make it more of a free-for-all.
Yeah, we're not going to use OGF for a while.
I had to end up delaying the provisions as we had to have them re-provisioned due to it missing a key piece of information. That's been done now and we have a lot of San Jose servers so I'll be able to honor those requests now.
It's a lot more difficult being ambiguous there given the hostile environment. It's easier communicating here with more like-minded individuals. @ OGF, I feel like everything would be misinterpreted right now.
I can't wait until I can go back to being an alcoholic. I haven't been able to have a serious drink in over 1-2 years.
For my vps7 migration from ColonCrossing ATL to Flexential ATL:
For my vps1 migration from ColonCrossing LAX to DediPath SJC:
In vps1 step 3, I suppose the renumbering happened sometime ago but I'm unknowningly using somebody else's IP until I rebooted.
I was not told about when the renumbering would occur or what's the new IP, until I did the reboot myself.
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
We couldn't do the usual rounds of communication given the situation. We couldn't do a lot of things we would normally ensure in terms of communication and notices. A lot of it was being directly misinterpreted/quoted to paint a false picture as well. I don't know if that makes sense but it was easy for one party to go "you mentioned XX, therefore YY, and we're going to apply this to everything else as well." I'll probably be able to reference this fully in a couple years and clear things up.
We were operating in a very hostile and unstable environment and had to be careful in multiple ways. If we were too specific, hostile actors could take advantage of it. I do not believe we would have been able to move everyone in that case without being sabotaged.
I love your positive attitude.... "were"
I understand what you're saying here and for this to have worked, for the most part, we would have had to invest a significant amount of time in doing a lot of that beforehand.
Temporary increase in support was difficult to coordinate. It would have ended up with people receiving frustratingly bad advice. I'm still trying to come up with a plan in sorting tickets and providing actual responses to avoid auto closing everything again. It didn't work out last time but this time we have the added advantage of everything actually finally settling down instead of other changes being stacked on that cause further grievances.
One major problem that caused a lot of these added issues was the fact that SVM or perhaps rather KVM virtualization in general operates in a way where several steps of reboots have to occur. When it's being done for a lot of services at once, things are bound to go wrong. We had to adjust timeouts based on each server's load and it wasn't always perfect.
We couldn't just set a 5 minute timeout per VM to ensure it goes 100% smoothly because if a hundred plus have to get done at once, that's a 10 hour queue.
If at any time this "broke" in any way, it left many servers in various partial states and there were multiple steps of adding and removing IPs. Sometimes one step would break but not another. We had to re-code everything over and over and over so depending on when it was done, and with the other added bugs, the situation was unpredictable. Now add in human error of having to keep track of these states over a hundred plus servers.
I didn't code out the IP change script for the migration without data, but when reviewing the code it looked fine. The multiple IP issue seems to be related to how SVM deals with timeouts and overlaps. Sometimes it'd assign an IP, but not really assign it in a way where it got locked away from it being re-assigned to another and the default behavior seems to be it'd just do it all over again, hence some people got multiple functional IPs.
Seems LET/LEB has gone completely vindictive on VirMach now.
I can surely understand why he is not posting over there at the moment.
https://lowendbox.com/blog/virmach-teeters-at-the-edge/
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
@VirMach I have sent you a message in Skype. I will help you sort the millions of tickets free of charge after my shifts end every day.
I'm so sorry to hear these news. You really don't deserve all this.
fucking dog. nothing said about racknerdtalk.
I bench YABS 24/7/365 unless it's a leap year.
IMO, totally uncalled for, posting that. Raindog must be needing the extra content bilohbucks! Also found it interesting that LordSpock 'stole' my phrase of mass exodus (check post timings).
FrankZ should perhaps take the same stance as me, with (no) regards to OGF i.e. monitor but don't comment etc. just saying.
Now what a kind offer from @MikePT and one that genuinely may help, given the kudos that he's appropriated. Nice one.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I worked for Virmach for quite some time. I can't assist this and simply ignore it. They're good people especially the guy that posts here. They/he helped me a lot when I needed most.
Dinnae do it, man. ( Us Scots know a thing or three about alchies! ) Goddamn Guinness got rid of my slim belly and with current meds, I'm limiting m'self to one can every two weeks, or so. Could do with a wee dram o' whisky though.
[No single malt left in the house, for a while. Boo.]
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
I've been mostly staying away from commenting at the OGF, other than trying to help folks out, for a while now. As @Fat32 has always had my respect, I was trying to be part of the solution, instead of just not being part of the problem. He also seems to be very quiet these days, outside of the Mega Threads at the OGF, so maybe your right and it's time I start thinking about what to do next.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Thanks FrankZ, I have seen you helping out in the thread a lot for people that are having issues with VirMach. You are amazing
For myself, I start to think about my life more... probably having my third/quarter-life crisis now
食之无味 弃之可惜 - Too arduous to relish, too wasteful to discard.