NYJ, SJC - We heard back from Evocative. We're placed in a queue to speak with someone, it looks like they may be overwhelmed with requests and they're sorting through all of them and trying to communicate with Dedipath to confirm server ownership.
Last night I went to a "juguería" (I don't know the term in English). Ironically I didn't but any juice but a papa rellena at $1. And I saw a glass of flan for $1! $1 WTF? It should be $0.4 max. The papa rellena has potato, yucca, meat, onion, a lot of oil and it takes like 10 minutes to get fried. And the fucking flan it's the flan powder (artificial BTW), water and milk and you get like 12 of those small plastic recipients they tried to sell me.
Last night I went to a "juguería" (I don't know the term in English). Ironically I didn't but any juice but a papa rellena at $1. And I saw a glass of flan for $1! $1 WTF? It should be $0.4 max. The papa rellena has potato, yucca, meat, onion, a lot of oil and it takes like 10 minutes to get fried. And the fucking flan it's the flan powder (artificial BTW), water and milk and you get like 12 of those small plastic recipients they tried to sell me.
@VirMach said:
Emails are going out. We were trying to wait to provide less useless information but it's reaching the point where that's not likely so we're going to funnel everyone into the network status page and slowly update that as we proceed. It's best people have a longer notice at this point even if it's vague, I don't want it to end up being a huge surprise for anyone so I'm basically calling it now.
Copy below:
Dear VirMach Customers,
Over the next week, one or more of your services will be affected by a physical migration. Unfortunately, we do not have an exact maintenance window yet but will provide an update on the network status page as soon as possible. Please visit your service details page, and make a note of the physical server it's located on (it will appear on the top left portion of the screen, on a tooltip.) If your service is on a server labeled as NYC, PHX, ATL, DAL, DEN, SEA, or SJC, then it will be affected. We have already been working in the background to take backups of all services in case of any emergency scenario, and also will be deploying additional physical servers, just in case.
Please understand that ticket volumes will be extremely high, and we may not be able to answer all your questions. Therefore we've answered some common questions below and will continue to update the network status page with additional answers.
Which locations are affected?
NYC Metro, Phoenix, Atlanta, Dallas, Denver, Seattle, and San Jose.
What does this mean?
There will be a maintenance window where your service will be temporarily unavailable. We are working hard to minimize the total downtime, and coordinating these windows for the fastest possible restoration. Therefore, we do not yet have specific times available.
Why is this happening?
One of our providers abruptly discontinued their business, and therefore it's required we move equipment, most likely within the same datacenter facility, but under a new contract with a different company.
Will my IP address change?
Most customers will still have the same IP address, but some may have to be changed. If we run ino any problems, the IP address may also change in an emergency scenario. We will provide more information on this on the network status page later today.
Will my data be migrated?
Yes, most servers are remaining exactly the same and we also have backups in case of any issues during the migration.
Will my location change?
For most services, the location will remain the same, but there's a small possibility that certain services will be moved to a nearby location. If this happens, you can contact us after the migration to move back. However, please check the network status page for any updates.
We apologize for any inconvenience. If there are any errors above, it will be corrected on the network status page.
I have temporarily transferred my sites to a different VPS so no problem even if there is a downtime or data loss. Wish you good luck. My body VPS is ready.
SJC will most likely be direct with Evocative. We're also waiting for a quote from xTom, if it's within budget we'll most likely go ahead with that for the existing servers to make the process smoother since we trust them/also have IPv4 already announced. In that scenario we'll then use Evocative for expanding in that location since we've been meaning to do that for a very long time now but were uncomfortable expanding with the previous provider.
For Dallas we wanted to go with DataPacket but they do not provide the right ports, we're trying to see if we can figure out the switch situation for that. So if anyone is in the area and has an EX4200 or similar let me know. I'm waiting for a response from them so we can just send them a switch just in case for whenever that arrives.
Sabey and Flex still haven't got back to us, I tried calling Flex but they weren't responding. We're trying to plan it out to where we could send Seattle and Denver to LAX if we can get equipment and IPv4 announced in either QN or HV.
We're still trying to figure out what to do about Atlanta, worst case scenario if they do not respond we may have to disperse that across all other locations near the end, from backups if necessary (worst case scenario.) Some of it could end up going to Tampa and Miami, still working out the specific details for that.
@VirMach said: Sabey and Flex still haven't got back to us, I tried calling Flex but they weren't responding. We're trying to plan it out to where we could send Seattle and Denver to LAX if we can get equipment and IPv4 announced in either QN or HV.
We're still trying to figure out what to do about Atlanta, worst case scenario if they do not respond we may have to disperse that across all other locations near the end, from backups if necessary (worst case scenario.) Some of it could end up going to Tampa and Miami, still working out the specific details for that.
You're talking DR strategy at the moment. Just wondering if long-term would those moves expect to be permanent. Realizing of course that 'long-term' is a variable length.
@VirMach said: Sabey and Flex still haven't got back to us, I tried calling Flex but they weren't responding. We're trying to plan it out to where we could send Seattle and Denver to LAX if we can get equipment and IPv4 announced in either QN or HV.
We're still trying to figure out what to do about Atlanta, worst case scenario if they do not respond we may have to disperse that across all other locations near the end, from backups if necessary (worst case scenario.) Some of it could end up going to Tampa and Miami, still working out the specific details for that.
Flex just replied. They can offer us space but timeline appears to be slower, I've asked them for confirmation on how the transition would work. We need to end up taking more space with them, which I'm not too excited about since their remote hands weren't great in the past.
Going to try to give Sabey a call again so we can knock out the remaining location, although that could just end up being the location we retire at least temporarily and move them to LAX if they don't respond, just so we don't risk anything.
You're talking DR strategy at the moment. Just wondering if long-term would those moves expect to be permanent. Realizing of course that 'long-term' is a variable length.
We've had extra servers in limbo for a while. We couldn't send them to QN because it was full and we didn't feel comfortable getting extra space with them especially after what happened in Chicago, but LAX is relatively good. Except we already got extra LAX space with HV but we were hesitant to fill that up because of what happened with them. Dedipath, we weren't comfortable sending any extra equipment to them pretty early on, like within the first couple months. xTom we've been wanting to send all our servers there (oh, not Tokyo), except a few got lost in customs and we needed more space in the US. So I'd say all of these moves are pretty permanent, until the next nightmare story.
Okay so it's pretty much end of the working day for probably most the people we were contacting, I'll do a quick update but it seems like two things are probably true: (1) the actual moves may end up pushing into next and, (2) I don't believe any of these providers will shut off services especially if we're awaiting a quote by them. The focus will most likely end up being the IP space that's not ours and the location that didn't yet reply (Seattle) but I doubt they'd work over the weekend to de-rack anything.
Backups - Pretty much complete, I have to go through some failed ones to fill in any gaps.
NYJ - We have new space with another provider in the same building, they're on standby to quickly move everything as soon as we get everything sorted on the other side.
Dallas - Awaiting response from DataPacket and Flexential. Flexential stated they'd do this next week.
Atlanta - Flexential stated they'd do this next week, for these I assume they understand it entails not powering down any existing services in their datacenters.
Denver - Same as above, but we may elect to retire this location as the minimum space Flexential requires we take would be excessive for the location. Likely will move to LAX as soon as possible. IPv4 may stay the same, what we'll likely do is assign one new IP and also keep the old one since there will be a gap in connectivity.
Seattle - No response from Sabey, we'll make sure we're ready to load in backups to LAX just in case.
San Jose - Going either direct with Evocative or xTom. Still waiting for equipment release and space to be finalized.
Phoenix - Going direct with Path.net and already finalized, the switch will happen sometime next week and they've confirmed they'll keep the services going until then.
@VirMach said: Let me know if there's any questions.
After everything re-balances, might there be a window to allow paid migrations even on specials ( via automation or via ticket )? Depending on how this plays out I may wind up several more machines in LAX than I care to is why I ask.
Phoenix - Going direct with Path.net and already finalized, the switch will happen sometime next week and they've confirmed they'll keep the services going until then.
Path.net top provider of the year month.
And it's the only location I have a service so I would be happy, especially if IP doesn't change and latency doesn't increase.
Flex just replied. They can offer us space but timeline appears to be slower, I've asked them for confirmation on how the transition would work. We need to end up taking more space with them, which I'm not too excited about since their remote hands weren't great in the past.
Remote hands without brains attached 👐.
Avoid if you don't want expensive hardware damaged.
Move everyone to TokyoFrankfurt Miami.
@skorous said:
After everything re-balances, might there be a window to allow paid migrations even on specials ( via automation or via ticket )? Depending on how this plays out I may wind up several more machines in LAX than I care to is why I ask.
Paid migrations could save some heart attack time :-)
Phoenix - Going direct with Path.net and already finalized, the switch will happen sometime next week and they've confirmed they'll keep the services going until then.
Path.net top provider of the year month.
And it's the only location I have a service so I would be happy, especially if IP doesn't change and latency doesn't increase.
Flex just replied. They can offer us space but timeline appears to be slower, I've asked them for confirmation on how the transition would work. We need to end up taking more space with them, which I'm not too excited about since their remote hands weren't great in the past.
Remote hands without brains attached 👐.
Avoid if you don't want expensive hardware damaged.
Move everyone to TokyoFrankfurt Miami.
@VirMach said: Denver - Same as above, but we may elect to retire this location as the minimum space Flexential requires we take would be excessive for the location. Likely will move to LAX as soon as possible. IPv4 may stay the same, what we'll likely do is assign one new IP and also keep the old one since there will be a gap in connectivity.
Oh LAX... I have one active server in DEN and do not want to have more servers in LAX anymore. Is it possible that allows customers to pick up another location based on their wishes if DEN would be retired at the end?
@VirMach said: Denver - Same as above, but we may elect to retire this location as the minimum space Flexential requires we take would be excessive for the location. Likely will move to LAX as soon as possible. IPv4 may stay the same, what we'll likely do is assign one new IP and also keep the old one since there will be a gap in connectivity.
Oh LAX... I have one active server in DEN and do not want to have more servers in LAX anymore. Is it possible that allows customers to pick up another location based on their wishes if DEN would be retired at the end?
They're emergency migrations because a vendor went out of business. I don't think slowing the process down while customers figure out their preferred destination is a good idea.
NYC may end up getting IPv4 changes as we cannot otherwise get LOA in time, more on that later.
(Edit) Oh this is not necessarily related but:
About 50 to 98 VMs are going to be moved from NYCB020, NYCB020, and NYCB018. Most of them will be NYCB020.
About 25 to 50 VMs are going to be moved from NYCB038, NYCB037, and NYCB036.
We'll focus on moving VMs that are offline first. I'll add it to network status if we start doing it with online people but I'll still try to be selective like find ones that are already abusing and would have gotten a powerdown anyway or something like that.
As for hardware moves into QN/HV, last I checked it wasn't ready yet IP-wise but that was some time ago, going to check again to see if taking hardware down there is a worthwhile endeavor. Oh, and I have gotten some sleep finally so I'm probably good for another 2 days although I may take a nap to make sure I'm there the entire day Sunday for the NYC move.
NYC IPv4 Changes
This is pretty much sitting at 100% in terms of likelihood, what we can do is change them, and then re-assign the old IP address, however, that may end up not being ideal to do (adding old one back) because of network configuration changes that are occurring simultaneously. We were able to obtain new blocks, and most of them are ready.
Potential upcoming IPv4 changes:
I didn't want to overwhelm you guys, but our /17 was supposed to auto-renew 2 months ago. They didn't properly update us on this at the time, basically making it sound like they didn't want to renew, and I tried asking for a confirmation, finally leading up to them letting us know on September 1st, that they're still unsure and would let us know on September 7th and allow us to renew for just one more month for now. If I were to make a cynical guess, it seems like they're just trying to line it up to where they can sell it with better margins and do not actually care about us, which was previously proven when we had that whole fiasco a few months back.
Luckily, with the essentially forced NYC changes above from them being unresponsive and not providing LOA, as well as another 1/4th chunk being unused from when we switched over xTom, I think any potential benefits of having an entire /17 are zero.
I am going to start planning everything as if they're not going to renew it, in any case, this is beneficial to us, and as far as I'm concerned, it's better if we just don't renew it a this point and stop giving companies the potential chance of screwing us over as it's been proven in the past that it's more likely to happen than not. Therefore, that means up to 60 other servers will end up getting changes. However, if we do this early and on our own terms, it means these will get multiple weeks notice.
@VirMach said: NYC IPv4 Changes
This is pretty much sitting at 100% in terms of likelihood, what we can do is change them, and then re-assign the old IP address, however, that may end up not being ideal to do (adding old one back) because of network configuration changes that are occurring simultaneously. We were able to obtain new blocks, and most of them are ready.
Potential upcoming IPv4 changes:
I didn't want to overwhelm you guys, but our /17 was supposed to auto-renew 2 months ago. They didn't properly update us on this at the time, basically making it sound like they didn't want to renew, and I tried asking for a confirmation, finally leading up to them letting us know on September 1st, that they're still unsure and would let us know on September 7th and allow us to renew for just one more month for now. If I were to make a cynical guess, it seems like they're just trying to line it up to where they can sell it with better margins and do not actually care about us, which was previously proven when we had that whole fiasco a few months back.
Luckily, with the essentially forced NYC changes above from them being unresponsive and not providing LOA, as well as another 1/4th chunk being unused from when we switched over xTom, I think any potential benefits of having an entire /17 are zero.
I am going to start planning everything as if they're not going to renew it, in any case, this is beneficial to us, and as far as I'm concerned, it's better if we just don't renew it a this point and stop giving companies the potential chance of screwing us over as it's been proven in the past that it's more likely to happen than not. Therefore, that means up to 60 other servers will end up getting changes. However, if we do this early and on our own terms, it means these will get multiple weeks notice.
Which company are you talking about here? Hivelocity?
Comments
NYJ, SJC - We heard back from Evocative. We're placed in a queue to speak with someone, it looks like they may be overwhelmed with requests and they're sorting through all of them and trying to communicate with Dedipath to confirm server ownership.
stop email spam
May I have write access to https://twitter.com/virmachcloud so I can post daily about... I don't know... servers? flan?
Last night I went to a "juguería" (I don't know the term in English). Ironically I didn't but any juice but a papa rellena at $1. And I saw a glass of flan for $1! $1 WTF? It should be $0.4 max. The papa rellena has potato, yucca, meat, onion, a lot of oil and it takes like 10 minutes to get fried. And the fucking flan it's the flan powder (artificial BTW), water and milk and you get like 12 of those small plastic recipients they tried to sell me.
Up to 5 emails lol.
Amazing SolusVM feature.
It lets you know when you have way too many services with us.
Recipes & important updates.
We've purchased additional space with QN and will also be taking servers down to HV today, so we have extra space available.
We're waiting on a quote for SJC.
Evocative has started responding for releasing equipment NYJ/SJC.
Still waiting on FLEX and Sabey.
Good luck! Hope everything goes well.
dnscry.pt - Public DNSCrypt resolvers hosted by LowEnd providers • Need a free NAT LXC? -> https://microlxc.net/
I have temporarily transferred my sites to a different VPS so no problem even if there is a downtime or data loss. Wish you good luck. My body VPS is ready.
Fast as fuck Core i9 VPS (aff) | Powerful AMD Ryzen VPS (aff)
SJC will most likely be direct with Evocative. We're also waiting for a quote from xTom, if it's within budget we'll most likely go ahead with that for the existing servers to make the process smoother since we trust them/also have IPv4 already announced. In that scenario we'll then use Evocative for expanding in that location since we've been meaning to do that for a very long time now but were uncomfortable expanding with the previous provider.
For Dallas we wanted to go with DataPacket but they do not provide the right ports, we're trying to see if we can figure out the switch situation for that. So if anyone is in the area and has an EX4200 or similar let me know. I'm waiting for a response from them so we can just send them a switch just in case for whenever that arrives.
Sabey and Flex still haven't got back to us, I tried calling Flex but they weren't responding. We're trying to plan it out to where we could send Seattle and Denver to LAX if we can get equipment and IPv4 announced in either QN or HV.
We're still trying to figure out what to do about Atlanta, worst case scenario if they do not respond we may have to disperse that across all other locations near the end, from backups if necessary (worst case scenario.) Some of it could end up going to Tampa and Miami, still working out the specific details for that.
You're talking DR strategy at the moment. Just wondering if long-term would those moves expect to be permanent. Realizing of course that 'long-term' is a variable length.
Flex just replied. They can offer us space but timeline appears to be slower, I've asked them for confirmation on how the transition would work. We need to end up taking more space with them, which I'm not too excited about since their remote hands weren't great in the past.
Going to try to give Sabey a call again so we can knock out the remaining location, although that could just end up being the location we retire at least temporarily and move them to LAX if they don't respond, just so we don't risk anything.
We've had extra servers in limbo for a while. We couldn't send them to QN because it was full and we didn't feel comfortable getting extra space with them especially after what happened in Chicago, but LAX is relatively good. Except we already got extra LAX space with HV but we were hesitant to fill that up because of what happened with them. Dedipath, we weren't comfortable sending any extra equipment to them pretty early on, like within the first couple months. xTom we've been wanting to send all our servers there (oh, not Tokyo), except a few got lost in customs and we needed more space in the US. So I'd say all of these moves are pretty permanent, until the next nightmare story.
Okay so it's pretty much end of the working day for probably most the people we were contacting, I'll do a quick update but it seems like two things are probably true: (1) the actual moves may end up pushing into next and, (2) I don't believe any of these providers will shut off services especially if we're awaiting a quote by them. The focus will most likely end up being the IP space that's not ours and the location that didn't yet reply (Seattle) but I doubt they'd work over the weekend to de-rack anything.
Backups - Pretty much complete, I have to go through some failed ones to fill in any gaps.
NYJ - We have new space with another provider in the same building, they're on standby to quickly move everything as soon as we get everything sorted on the other side.
Dallas - Awaiting response from DataPacket and Flexential. Flexential stated they'd do this next week.
Atlanta - Flexential stated they'd do this next week, for these I assume they understand it entails not powering down any existing services in their datacenters.
Denver - Same as above, but we may elect to retire this location as the minimum space Flexential requires we take would be excessive for the location. Likely will move to LAX as soon as possible. IPv4 may stay the same, what we'll likely do is assign one new IP and also keep the old one since there will be a gap in connectivity.
Seattle - No response from Sabey, we'll make sure we're ready to load in backups to LAX just in case.
San Jose - Going either direct with Evocative or xTom. Still waiting for equipment release and space to be finalized.
Phoenix - Going direct with Path.net and already finalized, the switch will happen sometime next week and they've confirmed they'll keep the services going until then.
Let me know if there's any questions.
Please don't move Seattle to any of the Asia Traffic Magnet locations - Chicago/Dallas would be better.
It wisnae me! A big boy done it and ran away.
NVMe2G for life! until death (the end is nigh)
After everything re-balances, might there be a window to allow paid migrations even on specials ( via automation or via ticket )? Depending on how this plays out I may wind up several more machines in LAX than I care to is why I ask.
7950X in Europe when?!
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Path.net top provider of the year month.
And it's the only location I have a service so I would be happy, especially if IP doesn't change and latency doesn't increase.
Remote hands without brains attached 👐.
Avoid if you don't want expensive hardware damaged.
Move everyone to Tokyo Frankfurt Miami.
Accepting submissions for IPv6 less than /64 Hall of Incompetence.
One of my DediPath VPS is now down. Other is still up.
Fast as fuck Core i9 VPS (aff) | Powerful AMD Ryzen VPS (aff)
Paid migrations could save some heart attack time :-)
forced enrollment in the miami beach club
Oh LAX... I have one active server in DEN and do not want to have more servers in LAX anymore. Is it possible that allows customers to pick up another location based on their wishes if DEN would be retired at the end?
They're emergency migrations because a vendor went out of business. I don't think slowing the process down while customers figure out their preferred destination is a good idea.
What is the future direction for NYCB027 node? It has been locked for several months.
me too.
You're locked? Or you don't know your future direction?
NYC may end up getting IPv4 changes as we cannot otherwise get LOA in time, more on that later.
(Edit) Oh this is not necessarily related but:
About 50 to 98 VMs are going to be moved from NYCB020, NYCB020, and NYCB018. Most of them will be NYCB020.
About 25 to 50 VMs are going to be moved from NYCB038, NYCB037, and NYCB036.
We'll focus on moving VMs that are offline first. I'll add it to network status if we start doing it with online people but I'll still try to be selective like find ones that are already abusing and would have gotten a powerdown anyway or something like that.
As for hardware moves into QN/HV, last I checked it wasn't ready yet IP-wise but that was some time ago, going to check again to see if taking hardware down there is a worthwhile endeavor. Oh, and I have gotten some sleep finally so I'm probably good for another 2 days although I may take a nap to make sure I'm there the entire day Sunday for the NYC move.
NYC IPv4 Changes
This is pretty much sitting at 100% in terms of likelihood, what we can do is change them, and then re-assign the old IP address, however, that may end up not being ideal to do (adding old one back) because of network configuration changes that are occurring simultaneously. We were able to obtain new blocks, and most of them are ready.
Potential upcoming IPv4 changes:
I didn't want to overwhelm you guys, but our /17 was supposed to auto-renew 2 months ago. They didn't properly update us on this at the time, basically making it sound like they didn't want to renew, and I tried asking for a confirmation, finally leading up to them letting us know on September 1st, that they're still unsure and would let us know on September 7th and allow us to renew for just one more month for now. If I were to make a cynical guess, it seems like they're just trying to line it up to where they can sell it with better margins and do not actually care about us, which was previously proven when we had that whole fiasco a few months back.
Luckily, with the essentially forced NYC changes above from them being unresponsive and not providing LOA, as well as another 1/4th chunk being unused from when we switched over xTom, I think any potential benefits of having an entire /17 are zero.
I am going to start planning everything as if they're not going to renew it, in any case, this is beneficial to us, and as far as I'm concerned, it's better if we just don't renew it a this point and stop giving companies the potential chance of screwing us over as it's been proven in the past that it's more likely to happen than not. Therefore, that means up to 60 other servers will end up getting changes. However, if we do this early and on our own terms, it means these will get multiple weeks notice.
Is the /17 you talking about the 47.87.192.0 - 47.87.199.255 range ?
Edit: changed to include complete /17 range instead of 47.87.xx.xx
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Which company are you talking about here? Hivelocity?
Fast as fuck Core i9 VPS (aff) | Powerful AMD Ryzen VPS (aff)