I'm going to start providing updates here as we go, please note all these updates should be taken with a grain of salt, it hasn't fallen in place together to where we can make official announcements with confidence yet.
PHX is confirmed as close to 100% as possible safe, we've already spoken with Path. They stated they would not shut off services and are assisting us with moving to them directly.
Dedicated servers will receive an email very soon, we already winded down most of these but a few are left, we were going to continue those for another month as we split it off into multiple batches based on contract durations. We're going to wind up space for all dedicated servers so they can take backups as available, but probably only a dozen customers affected.
The majority of services were already backed up a month ago, pre-emptively in case something like this should occur. The backups are being updated and more thoroughly checked. I had a long wall of text written about all our plans that I decided not to post as it was very convoluted. As @Jab stated we're probably the best company you can be with right now as we have a lot of experience.
Need not panic, at the time my best guess is that everyone will be more cooperative than the first time we did something like this, and we have a ton of code prepared for all types of migrations everywhere. We also have some providers on standby that can provide us extra space, and we have extra servers, about 30 of them that we can deploy quickly. The most we're anticipating at this time would be up to maybe one day outage in emergency scenarios mostly related to switching over IPv4 blocks. Of course it could get worse in which case it'd go into loading from backups we have already taken or are updating/taking now.
Sorry for not updating you earlier, rest assured every hour spent since I woke up this morning has been dedicated to this situation and I definitely will be around for at least the next 2-3 days. We have a pretty good espresso machine at the office now.
@VirMach said: PHX is confirmed as close to 100% as possible safe, we've already spoken with Path. They stated they would not shut off services and are assisting us with moving to them directly.
@gritty said:
will the closure of dedipath affect Virmach's financial situation?
We'll have more money after the initial non-recurring extra costs incurred for the move. Most locations never got filled out because we didn't feel comfortable sending additional servers there for various reasons. And they increased IPv4 costs on us without much notice, so we'll be getting rid of the last remaining of about a dozen /24's with them. Dedipath was also charging us bogus overage fees which I don't know if I covered very well that they refused to refund, that didn't match monitors, so we'll no longer be paying those either.
Outside of obviously every negative thing surrounding such a move, the financials and everything else are technically quite positive. Forced consolidation to some degree, forced IP changes for those mentioned.
Speaking of which, my next update. Emails are going out for dedicated servers as mentioned soon and then for the approximately dozen servers that will also be getting IPv4 changes no matter what, we're going to attempt to emergency migrate those to nearest locations ASAP basically.
This may include any of the following: SJCZ001, SJCZ002, SJCZ003, SJCZ005, SJCZ006, SJCZ007, SJCZ008, SJCZ010, ATLZ002, SEAZ002, SEAZ005.
NYJ we have secured a new provider that's willing to assist us, it will look like possibly up to one day outage but depending on any grace period provided could be less and better done. We're waiting on Evocative's response still to give the go-ahead to move equipment. This location has the most servers and poses greatest difficulty in moving especially with a storage node being there so it's good we've been able to begin resolving this early on.
@VirMach said: NYJ we have secured a new provider that's willing to assist us, it will look like possibly up to one day outage but depending on any grace period provided could be less and better done. We're waiting on Evocative's response still to give the go-ahead to move equipment. This location has the most servers and poses greatest difficulty in moving especially with a storage node being there so it's good we've been able to begin resolving this early on.
Glad you were able to find someone so quickly for NJ
Also note, we haven't been selling new plans in any of their locations for several weeks now. This is another positive from multiple angles but I won't get too deep into it.
@Fritz said:
Should I move my BF-SPECIAL-2021 in NYC? @VirMach
I'd recommend you back it up of course as you would any crucial data at any time, but we won't be sending out any communication telling people to do that as we're already doing that and it's best any actions on the customer side is kept to a minimum to keep the nodes stable and running. With that being said, nothing so far to indicate a catastrophic scenario. Smooth sailing up until this point, if that changes everyone will hear about it immediately.
@VirMach said: NYJ we have secured a new provider that's willing to assist us, it will look like possibly up to one day outage but depending on any grace period provided could be less and better done. We're waiting on Evocative's response still to give the go-ahead to move equipment. This location has the most servers and poses greatest difficulty in moving especially with a storage node being there so it's good we've been able to begin resolving this early on.
Glad you were able to find someone so quickly for NJ
You should message me, I'll tell you all about it. An amazing LES member helped out.
Update for SJC and SEA - of course this can change but the current plan would be to consolidate these into one location. Sabey hasn't got back to us yet. Since San Jose does have mostly IPv4 blocks from Dedipath, the best course of action right now is what I discussed above. If we're able to move everyone to LAX, then it's possible that location may be at least temporarily discontinued.
Seattle we'd like to keep around and awaiting their response but also focusing on prioritizing everything for that in case of a disaster scenario. We have not yet secured any space here.
Atlanta will likely end up going to Miami, Chicago, and Tampa as a backup scenario. We should have enough space here to move everyone off and move them in from disaster backups if necessary. This location is Flexential, as is Denver which will likely be a retired location since there's not that many people there. Dallas will be kept as a location but we're still waiting for a response from Flexential on this one as well. We're still exploring colocation options for Dallas at this time.
Adding a note at the end here we've reached out to a company and waiting for further information. I don't want them to think we're no longer interested as they check the community, so for any location I just said above we're thinking about winding down, if that provider is able to help us out we'd much rather keep the location around.
(edit) We're getting into more vague territory here so I cannot stress enough, please do not take this as a finalized update.
@VirMach said: Update for SJC and SEA - of course this can change but the current plan would be to consolidate these into one location. Sabey hasn't got back to us yet. Since San Jose does have mostly IPv4 blocks from Dedipath, the best course of action right now is what I discussed above. If we're able to move everyone to LAX, then it's possible that location may be at least temporarily discontinued.
Seattle we'd like to keep around and awaiting their response but also focusing on prioritizing everything for that in case of a disaster scenario. We have not yet secured any space here.
Atlanta will likely end up going to Miami, Chicago, and Tampa as a backup scenario. We should have enough space here to move everyone off and move them in from disaster backups if necessary. This location is Flexential, as is Denver which will likely be a retired location since there's not that many people there. Dallas will be kept as a location but we're still waiting for a response from Flexential on this one as well. We're still exploring colocation options for Dallas at this time.
Adding a note at the end here we've reached out to a company and waiting for further information. I don't want them to think we're no longer interested as they check the community, so for any location I just said above we're thinking about winding down, if that provider is able to help us out we'd much rather keep the location around.
(edit) We're getting into more vague territory here so I cannot stress enough, please do not take this as a finalized update.
as is Denver which will likely be a retired location since there's not that many people there
it looks like the Denver vps will be opened to migration to other locations?
Dedicated server communication sent out, and refunds send out for those. Luckily only 6 people were affected as we already processed most of them earlier last month.
@yoursunny said:
We have a pretty good espresso machine now.
in the Starbucks across the street
I mostly go with this now. I think I've fried my brain, Starbucks doesn't do anything for me anymore.
Most fresh backups are completing right about now with a few asterisks.
@VirMach said: This may include any of the following: SJCZ001, SJCZ002, SJCZ003, SJCZ005, SJCZ006, SJCZ007, SJCZ008, SJCZ010, ATLZ002, SEAZ002, SEAZ005.
We've decided to shift focus away from this right now. As long as we have the backups we can spin them up from those anyway within several hours and have a similar end result.
@VirMach said:
Dedicated server communication sent out, and refunds send out for those. Luckily only 6 people were affected as we already processed most of them earlier last month.
My server has been suspended for half a year, and every time I apply to unsuspension of the server, you refused and provided no explanation, just closed my ticket. I am even willing to bear $25 fee to unsuspend my server, but still there is no response.
Ticket #441496, please check and give me a reasonable explanation, Thanks.
@VirMach said:
Dedicated server communication sent out, and refunds send out for those. Luckily only 6 people were affected as we already processed most of them earlier last month.
My server has been suspended for half a year, and every time I apply to unsuspension of the server, you refused and provided no explanation, just closed my ticket. I am even willing to bear $25 fee to unsuspend my server, but still there is no response.
Ticket #441496, please check and give me a reasonable explanation, Thanks.
Reasonable explanation comes along with actions which will result in more of your services being suspended, so I just recommend you close the ticket, cancel the service, and stop bringing more attention to yourself. Otherwise before you do at least make sure to purchase extra support level first on all your accounts, then we'll discuss it with you if you really want to pay the $25 suspension fee only for your service to still remain suspended since it was rejected for good reason.
So it's up to you if discussing it further is worth that to you but I don't think it would be.
Emails are going out. We were trying to wait to provide less useless information but it's reaching the point where that's not likely so we're going to funnel everyone into the network status page and slowly update that as we proceed. It's best people have a longer notice at this point even if it's vague, I don't want it to end up being a huge surprise for anyone so I'm basically calling it now.
Copy below:
Dear VirMach Customers,
Over the next week, one or more of your services will be affected by a physical migration. Unfortunately, we do not have an exact maintenance window yet but will provide an update on the network status page as soon as possible. Please visit your service details page, and make a note of the physical server it's located on (it will appear on the top left portion of the screen, on a tooltip.) If your service is on a server labeled as NYC, PHX, ATL, DAL, DEN, SEA, or SJC, then it will be affected. We have already been working in the background to take backups of all services in case of any emergency scenario, and also will be deploying additional physical servers, just in case.
Please understand that ticket volumes will be extremely high, and we may not be able to answer all your questions. Therefore we've answered some common questions below and will continue to update the network status page with additional answers.
Which locations are affected?
NYC Metro, Phoenix, Atlanta, Dallas, Denver, Seattle, and San Jose.
What does this mean?
There will be a maintenance window where your service will be temporarily unavailable. We are working hard to minimize the total downtime, and coordinating these windows for the fastest possible restoration. Therefore, we do not yet have specific times available.
Why is this happening?
One of our providers abruptly discontinued their business, and therefore it's required we move equipment, most likely within the same datacenter facility, but under a new contract with a different company.
Will my IP address change?
Most customers will still have the same IP address, but some may have to be changed. If we run ino any problems, the IP address may also change in an emergency scenario. We will provide more information on this on the network status page later today.
Will my data be migrated?
Yes, most servers are remaining exactly the same and we also have backups in case of any issues during the migration.
Will my location change?
For most services, the location will remain the same, but there's a small possibility that certain services will be moved to a nearby location. If this happens, you can contact us after the migration to move back. However, please check the network status page for any updates.
We apologize for any inconvenience. If there are any errors above, it will be corrected on the network status page.
NYCB004S is going to get a separate email. It will discuss how we can take backups if we're contacted, since most people are already using this as backups of another backup or do have "replaceable files" and not necessarily running anything major like a unique database.
This will allow us to prioritize those who need it most. We do have space to back everyone up, but we're going to be limited by port speeds since it's such a large amount of data. Keep in mind we have no reason to believe the server won't successfully be retrieved and moved, this is just as a precaution. We'll still proceed with backing everyone up on this node although it will take several days, after the prioritized backups for this node specifically.
Comments
I'm going to start providing updates here as we go, please note all these updates should be taken with a grain of salt, it hasn't fallen in place together to where we can make official announcements with confidence yet.
PHX is confirmed as close to 100% as possible safe, we've already spoken with Path. They stated they would not shut off services and are assisting us with moving to them directly.
Dedicated servers will receive an email very soon, we already winded down most of these but a few are left, we were going to continue those for another month as we split it off into multiple batches based on contract durations. We're going to wind up space for all dedicated servers so they can take backups as available, but probably only a dozen customers affected.
The majority of services were already backed up a month ago, pre-emptively in case something like this should occur. The backups are being updated and more thoroughly checked. I had a long wall of text written about all our plans that I decided not to post as it was very convoluted. As @Jab stated we're probably the best company you can be with right now as we have a lot of experience.
Need not panic, at the time my best guess is that everyone will be more cooperative than the first time we did something like this, and we have a ton of code prepared for all types of migrations everywhere. We also have some providers on standby that can provide us extra space, and we have extra servers, about 30 of them that we can deploy quickly. The most we're anticipating at this time would be up to maybe one day outage in emergency scenarios mostly related to switching over IPv4 blocks. Of course it could get worse in which case it'd go into loading from backups we have already taken or are updating/taking now.
Sorry for not updating you earlier, rest assured every hour spent since I woke up this morning has been dedicated to this situation and I definitely will be around for at least the next 2-3 days. We have a pretty good espresso machine at the office now.
Hope to get over this difficulty soon...
will the closure of dedipath affect Virmach's financial situation?
LMFTFY (seeing as I recently let @FrankZ off with an erroneous 'then')
We have faith in ya! (Well, some of us.)
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
My free-forever NVMe2G is safe!
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
We'll have more money after the initial non-recurring extra costs incurred for the move. Most locations never got filled out because we didn't feel comfortable sending additional servers there for various reasons. And they increased IPv4 costs on us without much notice, so we'll be getting rid of the last remaining of about a dozen /24's with them. Dedipath was also charging us bogus overage fees which I don't know if I covered very well that they refused to refund, that didn't match monitors, so we'll no longer be paying those either.
Outside of obviously every negative thing surrounding such a move, the financials and everything else are technically quite positive. Forced consolidation to some degree, forced IP changes for those mentioned.
Speaking of which, my next update. Emails are going out for dedicated servers as mentioned soon and then for the approximately dozen servers that will also be getting IPv4 changes no matter what, we're going to attempt to emergency migrate those to nearest locations ASAP basically.
This may include any of the following: SJCZ001, SJCZ002, SJCZ003, SJCZ005, SJCZ006, SJCZ007, SJCZ008, SJCZ010, ATLZ002, SEAZ002, SEAZ005.
NYJ we have secured a new provider that's willing to assist us, it will look like possibly up to one day outage but depending on any grace period provided could be less and better done. We're waiting on Evocative's response still to give the go-ahead to move equipment. This location has the most servers and poses greatest difficulty in moving especially with a storage node being there so it's good we've been able to begin resolving this early on.
SJCZ001 first!
I bench YABS 24/7/365 unless it's a leap year.
Should I move my BF-SPECIAL-2021 in NYC?
@VirMach
https://microlxc.net/
Hoping my locked NJ server will back then.
Glad you were able to find someone so quickly for NJ
Also note, we haven't been selling new plans in any of their locations for several weeks now. This is another positive from multiple angles but I won't get too deep into it.
I'd recommend you back it up of course as you would any crucial data at any time, but we won't be sending out any communication telling people to do that as we're already doing that and it's best any actions on the customer side is kept to a minimum to keep the nodes stable and running. With that being said, nothing so far to indicate a catastrophic scenario. Smooth sailing up until this point, if that changes everyone will hear about it immediately.
Replacement is already lined up for the location.
You should message me, I'll tell you all about it. An amazing LES member helped out.
Will do, sounds like an interesting story!
Update for SJC and SEA - of course this can change but the current plan would be to consolidate these into one location. Sabey hasn't got back to us yet. Since San Jose does have mostly IPv4 blocks from Dedipath, the best course of action right now is what I discussed above. If we're able to move everyone to LAX, then it's possible that location may be at least temporarily discontinued.
Seattle we'd like to keep around and awaiting their response but also focusing on prioritizing everything for that in case of a disaster scenario. We have not yet secured any space here.
Atlanta will likely end up going to Miami, Chicago, and Tampa as a backup scenario. We should have enough space here to move everyone off and move them in from disaster backups if necessary. This location is Flexential, as is Denver which will likely be a retired location since there's not that many people there. Dallas will be kept as a location but we're still waiting for a response from Flexential on this one as well. We're still exploring colocation options for Dallas at this time.
Adding a note at the end here we've reached out to a company and waiting for further information. I don't want them to think we're no longer interested as they check the community, so for any location I just said above we're thinking about winding down, if that provider is able to help us out we'd much rather keep the location around.
(edit) We're getting into more vague territory here so I cannot stress enough, please do not take this as a finalized update.
as is Denver which will likely be a retired location since there's not that many people thereit looks like the Denver vps will be opened to migration to other locations?
Dedicated server communication sent out, and refunds send out for those. Luckily only 6 people were affected as we already processed most of them earlier last month.
can we get new dedi offer once we get rid of Dedipath?
Yea man, that's absolute priority to deal with this moment.
Lovable drama, luckily I've abandoned vir and switched to OVH.
I'm wondering how long vir can hold on.
MicroLXC is lovable. Uptime of C1V
We have a pretty good espresso machine at the home now.
We have a pretty good espresso machine now.
in the Starbucks across the street
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
Have you considered VirMach owns a Starbucks?
食之无味 弃之可惜 - Too arduous to relish, too wasteful to discard.
We need Dedipath Refugee plan. lol
I mostly go with this now. I think I've fried my brain, Starbucks doesn't do anything for me anymore.
Most fresh backups are completing right about now with a few asterisks.
We've decided to shift focus away from this right now. As long as we have the backups we can spin them up from those anyway within several hours and have a similar end result.
My server has been suspended for half a year, and every time I apply to unsuspension of the server, you refused and provided no explanation, just closed my ticket. I am even willing to bear $25 fee to unsuspend my server, but still there is no response.
Ticket #441496, please check and give me a reasonable explanation, Thanks.
/direct https://lowendtalk.com/discussion/comment/3712633/#Comment_3712633
No idea how strict you on visiting LET, so screenshots. Worth the check?
Mod Edit: Removed link to LES banned member in above post.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Reasonable explanation comes along with actions which will result in more of your services being suspended, so I just recommend you close the ticket, cancel the service, and stop bringing more attention to yourself. Otherwise before you do at least make sure to purchase extra support level first on all your accounts, then we'll discuss it with you if you really want to pay the $25 suspension fee only for your service to still remain suspended since it was rejected for good reason.
So it's up to you if discussing it further is worth that to you but I don't think it would be.
Let it go dude, and move on. Time heals all wounds, if you just let it go.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Whose VPS in DediPath are still working? Mine's are!
Let's see till how long.
Fast as fuck Core i9 VPS (aff) | Powerful AMD Ryzen VPS (aff)
Emails are going out. We were trying to wait to provide less useless information but it's reaching the point where that's not likely so we're going to funnel everyone into the network status page and slowly update that as we proceed. It's best people have a longer notice at this point even if it's vague, I don't want it to end up being a huge surprise for anyone so I'm basically calling it now.
Copy below:
NYCB004S is going to get a separate email. It will discuss how we can take backups if we're contacted, since most people are already using this as backups of another backup or do have "replaceable files" and not necessarily running anything major like a unique database.
This will allow us to prioritize those who need it most. We do have space to back everyone up, but we're going to be limited by port speeds since it's such a large amount of data. Keep in mind we have no reason to believe the server won't successfully be retrieved and moved, this is just as a precaution. We'll still proceed with backing everyone up on this node although it will take several days, after the prioritized backups for this node specifically.