Networking is up in OKC, slight issue with one of the router modules but I bought a few backups, just had to drive them back and forth. I have an early flight tomorrow to Miami so I'll either make one more trip tonight to get a few servers up, or I'll try to squeeze it in before going to the airport. In either case though I don't think it will change when we'll have it set up for ordering and migrations: that'll probably be Tuesday (unless Miami is smooth and uneventful.)
Let me add the IPv6 block to RADb before someone makes fun of us again...
@VirMach said:
Networking is up in OKC, slight issue with one of the router modules but I bought a few backups, just had to drive them back and forth. I have an early flight tomorrow to Miami so I'll either make one more trip tonight to get a few servers up, or I'll try to squeeze it in before going to the airport. In either case though I don't think it will change when we'll have it set up for ordering and migrations: that'll probably be Tuesday (unless Miami is smooth and uneventful.)
Let me add the IPv6 block to RADb before someone makes fun of us again...
@VirMach
Hi, regarding this message:
Hello, providing you a warning that service may go offline indefinitely, until Hivelocity returns our equipment and it's redeployed to another facility. We're still working on setting up the Miami location at a different facility.
There is already a potential date for your solution? sorry if you said it before, I did not read the thread.
@kidoh said: @VirMach
Hi, regarding this message:
Hello, providing you a warning that service may go offline indefinitely, until Hivelocity returns our equipment and it's redeployed to another facility. We're still working on setting up the Miami location at a different facility.
There is already a potential date for your solution? sorry if you said it before, I did not read the thread.
What do mean "potential date for your solution"? The email says the "service may go offline indefinitely" because they can't guess what Hivelocity is going to do. It's entirely probable Hivelocity doesn't know what they're going to do since they're basically trying to put out the flaming bag of dog poo that is Quadranet and will likely need time to figure out what they have.
@VirMach it's strange but I found that when I add funds via PayPal,PayPal charged my twice while in your Invoice system there is only once,what should I do now
@JonyBee said: @VirMach it's strange but I found that when I add funds via PayPal,PayPal charged my twice while in your Invoice system there is only once,what should I do now
What I remember about this is it has to do with something you’re doing on PayPal’s side like clicking pay twice or back and we’ve never been able to reproduce it if we’re not reloading pages and clicking buttons like crazy throughout the entire process.
You have to open a ticket so we can credit you. Billing system doesn’t support refunding twice so it’ll get added as additional credit. If you don’t want that, the official policy is no refunds but I’d be willing to take a look and see if I can override it somehow.
@kidoh said: @VirMach
Hi, regarding this message:
Hello, providing you a warning that service may go offline indefinitely, until Hivelocity returns our equipment and it's redeployed to another facility. We're still working on setting up the Miami location at a different facility.
There is already a potential date for your solution? sorry if you said it before, I did not read the thread.
They’ll go back online in Oklahoma City in about a week. Then you can request a transfer to other available locations. We’re moving the IP blocks as well so it might stay the same but if it ends up being messy it might get changed.
Lumen set incorrect BGP configuration and did not follow our initial BGP request, we contacted them Monday and they have not yet corrected it. What this means is we don't have any functioning IP blocks. Cogent is still not set up. IX is ready as of yesterday.
Once Lumen responds/corrects the issue we'll have a few servers ready to go.
Going in again tonight and will provide some photos of where we're at with the cabinet.
@cybertech said: @VirMach your client login recaptcha is no good for MJJs.
am in Shanghai now and need VPN to be able to get past it.
It's on the list. Cloudflare's captcha was causing more problems and also pretty ineffective. It was also firing off on other pages and pretty annoying with no good workaround, plus sometimes you'd get stuck in an endless loop of like 8 captchas (if you did them perfectly, otherwise literally endless.)
@cybertech said: @VirMach your client login recaptcha is no good for MJJs.
am in Shanghai now and need VPN to be able to get past it.
It's on the list. Cloudflare's captcha was causing more problems and also pretty ineffective. It was also firing off on other pages and pretty annoying with no good workaround, plus sometimes you'd get stuck in an endless loop of like 8 captchas (if you did them perfectly, otherwise literally endless.)
try replace www.google.com/recaptcha to recaptcha.net/recaptcha
I was too tired to post an update last time, I basically had to remove and re-rack everything and reconfigure the router as a result since I forgot to finish setting up the sync configuration before powering it down. One good thing that came from that though is that I upgraded the rails for the router, while the last set was rated for its weight, I went with something else that's rated 2-3x higher just in case.
IX confirmed up, just waiting to be moved out of quarantine
Still waiting on (Major)
The subnets to show up and be usable, then we should have 4x /24 IPV4* and /40 IPv6 ready
*One /24 is reserved for IPv6+NAT and general critical requirements
Minor updates/confirmations
We can fit about 4x dual node Epycs per cabinet (originally may have had to get rid of them)
The dual node 2x Intel Golds do have some benefits but a) I'm not sure if they'll be as popular as Epyc, and b) with the split cabinets having more "normal" levels of power they make less sense. So we'll likely scale back any plans on those instead.
We'll just place old E5's as placeholders the second I figure out the rail situation for them, there's a lot of different brand chassis and I haven't been doing a great job at keeping inventory of specific rails over the last few years
List of what's currently racked in the first cabinet, top down:
Cable Management
MX240 Router
More cable management
EX4300 (48x 1G RJ45 + 4x 40Gbps + 4x 10Gbps)
EX4300 (48x 1G RJ45 + 4x 40Gbps + 4x 10Gbps)
Ryzen Node
Ryzen Node
Ryzen Node
Ryzen Node
[Empty space for 1-2 larger Epyc Nodes, need to locate rails]
[Empty space for about 10 more Ryzen Nodes, waiting on Miami to be delivered and splitting it across two cabinets]
QFX5100 goes somewhere here maybe to be closer to the nodes that may need network from it in the front
[Empty space for some Ryzen bare metal, previously was going to be lots of Intel Golds, potentially a second storage node]
One of the "problems" I've been dealing with is that initially the cabinet was switched to 42U, then we got 48U, but they're shorter depth and slim profile which I was not expecting since it's Oklahoma and they have a ton of available space around cabinets. This paired with the most power we've had in a single cabinet meant beefier PDUs and we want to use that power so it meant beefier servers. All of this added together meant it was difficult to organize everything as initially intended.
We decided to just get a second 48U cabinet while they still have them available and that it's better for now to keep it "normal" on the power so we don't have to worry about cramming huge servers. One PDU goes in second cabinet, which frees up much needed room in the first for actual cable management (I could barely screw things in as it is, now we can actually have rings and such for cables.)
Minor To-Do List:
We need to re-re-re-order cables since plan changed a few times, so for now it'll just be a little messy/not correct length.
What's Left:
See above but I'll be working on this today and Saturday.
Putting in more Ryzen nodes
Start getting Ryzen bare metal ready since that's been pushed back up on priority list with the cabinet split
Get larger Epyc nodes up, at least one
Finish configuring/organizing the switches and plugging everything in
Get hard drives in storage node
Setting up Virtfuuuuuuuuuuuuuuuuuuuuusion
Trying not to fall behind on everything else not OKC related
Prepare your wallets for a crazy sale, maybe the tail end of this weekend, barring any major issues. We'll likely have an additional early bird coupon that's an LES exclusive. But we'll want to migrate some people here first and test general stability.
@Jab said: @VirMach I DEMAND AN UPDATE
and photos, mostly photos.
Photos coming in, give me an hour or two to load whatever I need and make it to the facility. I did take a few other progress photos I think so those will be included as well.
@cybertech said:
will the NVMes be consumer or enterprise grade?
will it be RAID10?
Ryzen nodes are exactly the same, so higher end consumer. They also go under VirMach brand and SolusVM1.
Epyc and Intel Gold (if any) will have Enterprise NVMe SSD. So both for bare metal and VPS nodes.
E5's will likely be SAS SSD.
Bare metal Ryzen 5500's will be consumer and/or SATA.
Nothing in terms of NVMe SSDs is planned to be RAID10. Some nodes may be RAID1 but initially as we are short on enterprise NVMe SSDs, and most of people making initial purchases will be doing so at sale prices, I'd have to at least 1.5x the cost if we did RAID1 and I don't think many people want that. If enough people are adamant they want to pay more for that during a sale, I can always make another node and sell those specifically as RAID1 (or even RAID10, just 3-5x the cost, we'd need $1200 controllers.)
@cybertech said:
will the NVMes be consumer or enterprise grade?
will it be RAID10?
Ryzen nodes are exactly the same, so higher end consumer. They also go under VirMach brand and SolusVM1.
Epyc and Intel Gold (if any) will have Enterprise NVMe SSD. So both for bare metal and VPS nodes.
E5's will likely be SAS SSD.
Bare metal Ryzen 5500's will be consumer and/or SATA.
Nothing in terms of NVMe SSDs is planned to be RAID10. Some nodes may be RAID1 but initially as we are short on enterprise NVMe SSDs, and most of people making initial purchases will be doing so at sale prices, I'd have to at least 1.5x the cost if we did RAID1 and I don't think many people want that. If enough people are adamant they want to pay more for that during a sale, I can always make another node and sell those specifically as RAID1 (or even RAID10, just 3-5x the cost, we'd need $1200 controllers.)
i guess enterprise NVMe is a good start.
i believe enough people are adamant they do not want extended downtime(s) and/or data loss, which naturally translates into higher perceived value and hence higher allocated budget
Click "Contact us" on the submit ticket page and select the service you want to have moved. Select "Other Issue" and "Continue" -- Copy/paste the following subject and message and submit.
Subject: Migrate to Oklahoma
Message: I'd like to migrate the selected service to Oklahoma City. I understand it will be moved over with 1x IPv4 address only. I understand this is a beta and the service I have selected is non-critical and not associated with any multi-million dollar business. I understand there may be unannounced maintenance windows where my service is unavailable. I understand that for any OKC-specific related support, I will use the VirMach LowEndSpirit thread until beta ends.
My LES username: [Insert Your Username]
Replace [Insert Your Username] above with your LES username. It'll only be used for this beta test and cleared afterward just in case you don't want any association.
Whoever moves over will also be added to an "OKC beta testers" spreadsheet where we might do some other fun stuff or requests (if I have enough free time) such as trying to move people to Virtfusion/Epyc or just testing that with temporary provisioned services, beta testing the IPv6 system for SolusVM once that comes out, or even getting to take one of the bare metal servers for a spin to help test the provisioning system for that (and any other suggestions.)
Honorable mention @cybertech for saying Virtfusion a lot
(For any of the above tagged, if you don't have a service that can be moved per above, then just make a ticket anyway after selecting "No Service" instead and I might make you a temporary one just to help test.)
Okay I should probably actually head over to the datacenter now, I promised @Jab photos in 1-2 hours and that was nearly 2 hours ago... he's probably familiar with the VirMach® time warp though.
Comments
Virtfuuuuuuuuuuuuuuuuuuuuuuusion
I bench YABS 24/7/365 unless it's a leap year.
Tickets are no longer accepted online.
You must ring the 308 buzzer.
We are in Los Angeles for another 20 hours of so.
Send us $60 Lyft gift card and we can go there again to ring the buzzer on your behalf.
No hostname left!
Networking is up in OKC, slight issue with one of the router modules but I bought a few backups, just had to drive them back and forth. I have an early flight tomorrow to Miami so I'll either make one more trip tonight to get a few servers up, or I'll try to squeeze it in before going to the airport. In either case though I don't think it will change when we'll have it set up for ordering and migrations: that'll probably be Tuesday (unless Miami is smooth and uneventful.)
Let me add the IPv6 block to RADb before someone makes fun of us again...
@VirMach
Hi, regarding this message:
Hello, providing you a warning that service may go offline indefinitely, until Hivelocity returns our equipment and it's redeployed to another facility. We're still working on setting up the Miami location at a different facility.
There is already a potential date for your solution? sorry if you said it before, I did not read the thread.
What do mean "potential date for your solution"? The email says the "service may go offline indefinitely" because they can't guess what Hivelocity is going to do. It's entirely probable Hivelocity doesn't know what they're going to do since they're basically trying to put out the flaming bag of dog poo that is Quadranet and will likely need time to figure out what they have.
@VirMach it's strange but I found that when I add funds via PayPal,PayPal charged my twice while in your Invoice system there is only once,what should I do now
What I remember about this is it has to do with something you’re doing on PayPal’s side like clicking pay twice or back and we’ve never been able to reproduce it if we’re not reloading pages and clicking buttons like crazy throughout the entire process.
You have to open a ticket so we can credit you. Billing system doesn’t support refunding twice so it’ll get added as additional credit. If you don’t want that, the official policy is no refunds but I’d be willing to take a look and see if I can override it somehow.
They’ll go back online in Oklahoma City in about a week. Then you can request a transfer to other available locations. We’re moving the IP blocks as well so it might stay the same but if it ends up being messy it might get changed.
That’s the general plan but not finalized.
OKC Update
Lumen set incorrect BGP configuration and did not follow our initial BGP request, we contacted them Monday and they have not yet corrected it. What this means is we don't have any functioning IP blocks. Cogent is still not set up. IX is ready as of yesterday.
Once Lumen responds/corrects the issue we'll have a few servers ready to go.
Going in again tonight and will provide some photos of where we're at with the cabinet.
@VirMach your client login recaptcha is no good for MJJs.
am in Shanghai now and need VPN to be able to get past it.
I bench YABS 24/7/365 unless it's a leap year.
and a nice Debian 12 template that has some basic options on first login and is actually up to date
I bench YABS 24/7/365 unless it's a leap year.
very nice connection via Virmach TYO wireguard
I bench YABS 24/7/365 unless it's a leap year.
It's on the list. Cloudflare's captcha was causing more problems and also pretty ineffective. It was also firing off on other pages and pretty annoying with no good workaround, plus sometimes you'd get stuck in an endless loop of like 8 captchas (if you did them perfectly, otherwise literally endless.)
try replace www.google.com/recaptcha to recaptcha.net/recaptcha
recaptcha.net has china cdn
@VirMach Can you please process the work order? Thank you View ticket #455957
@VirMach I DEMAND AN UPDATE
and photos, mostly photos.
Haven't bought a single service in VirMach Great Ryzen 2022 - 2023 Flash Sale.
https://lowendspirit.com/uploads/editor/gi/ippw0lcmqowk.png
Photos and coffee... I'm basically out of milk
I was too tired to post an update last time, I basically had to remove and re-rack everything and reconfigure the router as a result since I forgot to finish setting up the sync configuration before powering it down. One good thing that came from that though is that I upgraded the rails for the router, while the last set was rated for its weight, I went with something else that's rated 2-3x higher just in case.
Major OKC Updates
Still waiting on (Major)
*One /24 is reserved for IPv6+NAT and general critical requirements
Minor updates/confirmations
List of what's currently racked in the first cabinet, top down:
More Info on Cab Split
One of the "problems" I've been dealing with is that initially the cabinet was switched to 42U, then we got 48U, but they're shorter depth and slim profile which I was not expecting since it's Oklahoma and they have a ton of available space around cabinets. This paired with the most power we've had in a single cabinet meant beefier PDUs and we want to use that power so it meant beefier servers. All of this added together meant it was difficult to organize everything as initially intended.
We decided to just get a second 48U cabinet while they still have them available and that it's better for now to keep it "normal" on the power so we don't have to worry about cramming huge servers. One PDU goes in second cabinet, which frees up much needed room in the first for actual cable management (I could barely screw things in as it is, now we can actually have rings and such for cables.)
Minor To-Do List:
What's Left:
See above but I'll be working on this today and Saturday.
Prepare your wallets for a crazy sale, maybe the tail end of this weekend, barring any major issues. We'll likely have an additional early bird coupon that's an LES exclusive. But we'll want to migrate some people here first and test general stability.
Photos coming in, give me an hour or two to load whatever I need and make it to the facility. I did take a few other progress photos I think so those will be included as well.
@VirMach Thanks for the update! Looking forward to the sale and/or migrating a service to OKC to try it out!
will the NVMes be consumer or enterprise grade?
will it be RAID10?
I bench YABS 24/7/365 unless it's a leap year.
Mentally strong people use consumer NVMe on disabled RAID.
We have lots of critical experiment data stored this way.
No hostname left!
i admire your mental resilience!
I bench YABS 24/7/365 unless it's a leap year.
Ryzen nodes are exactly the same, so higher end consumer. They also go under VirMach brand and SolusVM1.
Epyc and Intel Gold (if any) will have Enterprise NVMe SSD. So both for bare metal and VPS nodes.
E5's will likely be SAS SSD.
Bare metal Ryzen 5500's will be consumer and/or SATA.
Nothing in terms of NVMe SSDs is planned to be RAID10. Some nodes may be RAID1 but initially as we are short on enterprise NVMe SSDs, and most of people making initial purchases will be doing so at sale prices, I'd have to at least 1.5x the cost if we did RAID1 and I don't think many people want that. If enough people are adamant they want to pay more for that during a sale, I can always make another node and sell those specifically as RAID1 (or even RAID10, just 3-5x the cost, we'd need $1200 controllers.)
i guess enterprise NVMe is a good start.
i believe enough people are adamant they do not want extended downtime(s) and/or data loss, which naturally translates into higher perceived value and hence higher allocated budget
but i could be wrong
I bench YABS 24/7/365 unless it's a leap year.
Whoever wants to move to Oklahoma:
Click "Contact us" on the submit ticket page and select the service you want to have moved. Select "Other Issue" and "Continue" -- Copy/paste the following subject and message and submit.
Replace [Insert Your Username] above with your LES username. It'll only be used for this beta test and cleared afterward just in case you don't want any association.
Whoever moves over will also be added to an "OKC beta testers" spreadsheet where we might do some other fun stuff or requests (if I have enough free time) such as trying to move people to Virtfusion/Epyc or just testing that with temporary provisioned services, beta testing the IPv6 system for SolusVM once that comes out, or even getting to take one of the bare metal servers for a spin to help test the provisioning system for that (and any other suggestions.)
Tagging everyone who showed previous interest:
@AlwaysSkint @FrankZ @skorous @sh97 @JBB @localhost @Wonder_Woman @Kris @tenpera @animeiscool
Honorable mention @cybertech for saying Virtfusion a lot
(For any of the above tagged, if you don't have a service that can be moved per above, then just make a ticket anyway after selecting "No Service" instead and I might make you a temporary one just to help test.)
Okay I should probably actually head over to the datacenter now, I promised @Jab photos in 1-2 hours and that was nearly 2 hours ago... he's probably familiar with the VirMach® time warp though.
Virtfuuuuuuuuuuuuuuuuuuuuuuusion
will be making one work order in a bit.
I bench YABS 24/7/365 unless it's a leap year.
@FAT32 Do you want to join too?
Crash cart photo. I’m sad I lost our tiny keyboard that goes along with it. I’ll get another one and make a little cart.