I truly appreciate the time and advice you have offered so I want to thank you for that!
@host_c said: Focus more on customer stuff ( your products, what you bring to the table ), you can always upgrade devices on the go with announced downtime ( we did, oh boy did we change all in 2 years ), and sincerely, I do not think that anyone will mock you if you upgrade something on an announce downtime. ( there is always that 1% who will do )
I do worry about PPS not scaling well with software routing. I think I'll be able to achieve 10Gb, but that will definitely drop with small packets. It's difficult to estimate by how much.
Either way, both approaches have their tradeoffs in my opinion. However, I believe that swapping out the routers which I can configure everything so that it should* be as simple as plugging in the new routers with minimal or almost no noticeable disruption in the future. This should be easier than upgrading the hypervisor.
If I use VyOS, I don't intend for it to be a long term solution rather, it’s a temporary solution while the network is smaller, until I can justify the expense of purchasing high-performance routers that will exceed any future needs for this rack.
@host_c said: You could also not bother with routing, and ask you upstream to publish your sub-nets and you just do switching - this is useful at start and if you have only 1 upstream, so you do not spend the $$ on routing and only do switching; it would lower your costs considerably on start.
While that is certainly an option, I worry about the security implications of not being able to centrally inspect traffic, even if only lightly. I also wouldn’t want to allow customer and internal traffic to mix, as I prefer to keep them separate which would be impractical if I am just doing the switching.
@host_c said:
It is always better to over power routing and switching performance, that will assure you have leg-room to modify stuff on the go.
MX204 is a fine choice to start with if you like Junos.
Arista is King latency wise and in switching, they kinda beat all the rest. . Overkill if you will not do storage via NFS/ISCSI/CEPH or other storage services via Ethernet.
It's easy to build a redundant network on the cheap today.
If you do leaf and spine with 2 spines, and connect 2 MLAGed leaf switches to them, Connect all servers with MLAG/VRRP (one port in each MLAG leaf pair). Then you can upgrade network hardware without any customer impact.
Have gone through this myself a couple of months ago. It was a dream. Once again, Arista is dirt cheap (look at Arista 7050 series, you get 32x 40G switches around USD 200-300 each, 48x 10G SFP + 4x 40G for around the same price, 48x GE copper + 4x 40G for around USD 100 - 200 each. So it's a great way to get lots of power cheap.
Sure, there is no new software updates available, but that shouldn't be a problem for your use case.
When you need heavier stuff, then you can upgrade the part that needs to.
You can use breakout cables to get 4x 10G out of the 40G ports, so get DCS-7050QX-32 for 32x40G as spines.
If you want 10G SFP+ for you servers: DCS-7050SX-64 connect 40G to spine (recommend 2x 40G LAG to each spine (DACs are cheap).
For 1G copper: DCS-7050TX-64, connect the same upwards.
Went all in on Arista Leaf and Spine L3 setup, it's awesome. Only L3, each rack have their own AS (private), BGP all the way. It just works.
@host_c said: A dedicated device with specialized ASIC will always beat Software. Yes, that dedicated will have a higher price most of the time.
When it comes to routing performance, this is an area where I don't have extensive experience, but I’ve been researching it thoroughly to ensure I make the right choice.
Would we likely see better routing performance using Arista 7050 switches compared to a software-based router? From what I’ve read, these switches seem quite capable and much more capable than anything we would be able to do in software.
@host_c said: You could also not bother with routing, and ask you upstream to publish your sub-nets and you just do switching - this is useful at start and if you have only 1 upstream, so you do not spend the $$ on routing and only do switching; it would lower your costs considerably on start.
Customers demand routed /64 subnets and BGP sessions.
Having only on-link /64 would not land you in shame lists yet, but it's considered inferior than routed subnets.
@Pulsar67 said:
While that is certainly an option, I worry about the security implications of not being able to centrally inspect traffic, even if only lightly.
It is possible to inspect traffic via switch port mirroring.
This is how Intrusion Detection System works.
I also wouldn’t want to allow customer and internal traffic to mix, as I prefer to keep them separate which would be impractical if I am just doing the switching.
If customer A hosts a website and customer B wants to access this website, such traffic should be permitted.
@yoursunny said: If customer A hosts a website and customer B wants to access this website, such traffic should be permitted.
To clarify, by "internal traffic," I am referring to communication within our server infrastructure, not between customer VM traffic.
@yoursunny said: It is possible to inspect traffic via switch port mirroring.
This is how Intrusion Detection System works.
Yes this would allow for passive traffic inspection, however, it wouldn't allow us to enforce security policies or take action in real time with Layer 2 switching.
I will add one more thing before I wish you Good Winds and Sail.
You have 2 options when you start, with or without acceptance of fake accounts ( aka no fraud checks ). the later will bring you more customers at the start but will be a headache to clean out later.
Each one has it's benefits and weaknesses.
Choose wisely,
Now, we are waiting those promos, where are they???
@host_c said: You have 2 options when you start, with or without acceptance of fake accounts ( aka no fraud checks ). the later will bring you more customers at the start but will be a headache to clean out later.
This is for sure something we're looking into... For credit card transactions looks like 3DS might be the way to go + other fraud measures that we will incorporate behind the scenes. PayPal will be an alternative. As for fraud prevention on sign up right now probably just going to do email verification not much more.
@host_c said: Now, we are waiting those promos, where are they???
I can say we will have something by the end of this week... next week by the latest! Just finishing up the last few things
@host_c said: I just wish to see the formula of the device when inspecting traffic above 40GBPS
We're using sFlow for monitoring network flow and we aren't doing anything like Deep Packet Inspection my original comment was a little bit misleading.
@host_c said:
It is always better to over power routing and switching performance, that will assure you have leg-room to modify stuff on the go.
MX204 is a fine choice to start with if you like Junos.
Arista is King latency wise and in switching, they kinda beat all the rest. . Overkill if you will not do storage via NFS/ISCSI/CEPH or other storage services via Ethernet.
It's easy to build a redundant network on the cheap today.
If you do leaf and spine with 2 spines, and connect 2 MLAGed leaf switches to them, Connect all servers with MLAG/VRRP (one port in each MLAG leaf pair). Then you can upgrade network hardware without any customer impact.
Have gone through this myself a couple of months ago. It was a dream. Once again, Arista is dirt cheap (look at Arista 7050 series, you get 32x 40G switches around USD 200-300 each, 48x 10G SFP + 4x 40G for around the same price, 48x GE copper + 4x 40G for around USD 100 - 200 each. So it's a great way to get lots of power cheap.
Sure, there is no new software updates available, but that shouldn't be a problem for your use case.
When you need heavier stuff, then you can upgrade the part that needs to.
You can use breakout cables to get 4x 10G out of the 40G ports, so get DCS-7050QX-32 for 32x40G as spines.
If you want 10G SFP+ for you servers: DCS-7050SX-64 connect 40G to spine (recommend 2x 40G LAG to each spine (DACs are cheap).
For 1G copper: DCS-7050TX-64, connect the same upwards.
Went all in on Arista Leaf and Spine L3 setup, it's awesome. Only L3, each rack have their own AS (private), BGP all the way. It just works.
Have been slowly converting to full Arista (was full Juniper, but on value:port Arista is winning out at scale). It's been pretty seamless, though tedious. Final bit will require about 1-5m of downtime and I think one of the Juniper RE's complaining is going to force our hand earlier than we want, but this type of advice is $20k from a network consultant. Very well said. Getting into proper, clean, uniform MLAG and VRRP spine setup and a central leaf system with trunked cassettes didn't make sense when we first started, but now it's cutting down wasted switches by a significant amount. Now a rack with 10x GPU chassis doesn't need 3 switches deployed for HA anymore, it needs 1-2 mtp trunk cables and a single cassette for true A+B redundancy at central leafs.
My head is exploding reading all this that you guys have to go through this much to provides services to us [effort & $$$]. Good luck with your journey [I will keep reading this thread time by time]
Well, some we already did, some we read that others did so we not even tried, it is a mix.
I presume you had your share of mistakes/tests, we all did
Yeah, I think that is normal. Even simple things that I knew were wrong at the time like "Oh man, we shouldn't deploy this server like this. I'm going to regret fixing this later, but we have no choice and it can't keep waiting".
Well, some we already did, some we read that others did so we not even tried, it is a mix.
I presume you had your share of mistakes/tests, we all did
Yeah, I think that is normal. Even simple things that I knew were wrong at the time like "Oh man, we shouldn't deploy this server like this. I'm going to regret fixing this later, but we have no choice and it can't keep waiting".
we actually did that not so a few times.
But yes, ultimately you learn ( or some not ) . Either the option, it is fun when shit breaks loose and while you try to fix it, you remember "I'm going to regret fixing this later" and burst into .
Comments
I truly appreciate the time and advice you have offered so I want to thank you for that!
I do worry about PPS not scaling well with software routing. I think I'll be able to achieve 10Gb, but that will definitely drop with small packets. It's difficult to estimate by how much.
Either way, both approaches have their tradeoffs in my opinion. However, I believe that swapping out the routers which I can configure everything so that it should* be as simple as plugging in the new routers with minimal or almost no noticeable disruption in the future. This should be easier than upgrading the hypervisor.
If I use VyOS, I don't intend for it to be a long term solution rather, it’s a temporary solution while the network is smaller, until I can justify the expense of purchasing high-performance routers that will exceed any future needs for this rack.
While that is certainly an option, I worry about the security implications of not being able to centrally inspect traffic, even if only lightly. I also wouldn’t want to allow customer and internal traffic to mix, as I prefer to keep them separate which would be impractical if I am just doing the switching.
It's easy to build a redundant network on the cheap today.
If you do leaf and spine with 2 spines, and connect 2 MLAGed leaf switches to them, Connect all servers with MLAG/VRRP (one port in each MLAG leaf pair). Then you can upgrade network hardware without any customer impact.
Have gone through this myself a couple of months ago. It was a dream. Once again, Arista is dirt cheap (look at Arista 7050 series, you get 32x 40G switches around USD 200-300 each, 48x 10G SFP + 4x 40G for around the same price, 48x GE copper + 4x 40G for around USD 100 - 200 each. So it's a great way to get lots of power cheap.
Sure, there is no new software updates available, but that shouldn't be a problem for your use case.
When you need heavier stuff, then you can upgrade the part that needs to.
You can use breakout cables to get 4x 10G out of the 40G ports, so get DCS-7050QX-32 for 32x40G as spines.
If you want 10G SFP+ for you servers: DCS-7050SX-64 connect 40G to spine (recommend 2x 40G LAG to each spine (DACs are cheap).
For 1G copper: DCS-7050TX-64, connect the same upwards.
Went all in on Arista Leaf and Spine L3 setup, it's awesome. Only L3, each rack have their own AS (private), BGP all the way. It just works.
When it comes to routing performance, this is an area where I don't have extensive experience, but I’ve been researching it thoroughly to ensure I make the right choice.
Would we likely see better routing performance using Arista 7050 switches compared to a software-based router? From what I’ve read, these switches seem quite capable and much more capable than anything we would be able to do in software.
Customers demand routed /64 subnets and BGP sessions.
Having only on-link /64 would not land you in shame lists yet, but it's considered inferior than routed subnets.
It is possible to inspect traffic via switch port mirroring.
This is how Intrusion Detection System works.
If customer A hosts a website and customer B wants to access this website, such traffic should be permitted.
No hostname left!
To clarify, by "internal traffic," I am referring to communication within our server infrastructure, not between customer VM traffic.
This is how Intrusion Detection System works.
Yes this would allow for passive traffic inspection, however, it wouldn't allow us to enforce security policies or take action in real time with Layer 2 switching.
first time a provider admits to inspecting traffic?
The all seeing eye sees everything...
Your garden is being watched.
No hostname left!
I just wish to see the formula of the device when inspecting traffic above 40GBPS
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
"If there is no struggle there is no progress"
@Pulsar67
I will add one more thing before I wish you Good Winds and Sail.
You have 2 options when you start, with or without acceptance of fake accounts ( aka no fraud checks ). the later will bring you more customers at the start but will be a headache to clean out later.
Each one has it's benefits and weaknesses.
Choose wisely,
Now, we are waiting those promos, where are they???
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
"If there is no struggle there is no progress"
Netronome Agilio CX dual-port 40GbE SmartNIC
Run eBPF in hardware for traffic inspection at line speed.
No hostname left!
Not really inspecting traffic per se.
more so monitoring traffic for abuse using tools such as sFlow
This is for sure something we're looking into... For credit card transactions looks like 3DS might be the way to go + other fraud measures that we will incorporate behind the scenes. PayPal will be an alternative. As for fraud prevention on sign up right now probably just going to do email verification not much more.
I can say we will have something by the end of this week... next week by the latest! Just finishing up the last few things
We're using sFlow for monitoring network flow and we aren't doing anything like Deep Packet Inspection my original comment was a little bit misleading.
@host_c spitting so much truth, some of it hurts to read
Have been slowly converting to full Arista (was full Juniper, but on value:port Arista is winning out at scale). It's been pretty seamless, though tedious. Final bit will require about 1-5m of downtime and I think one of the Juniper RE's complaining is going to force our hand earlier than we want, but this type of advice is $20k from a network consultant. Very well said. Getting into proper, clean, uniform MLAG and VRRP spine setup and a central leaf system with trunked cassettes didn't make sense when we first started, but now it's cutting down wasted switches by a significant amount. Now a rack with 10x GPU chassis doesn't need 3 switches deployed for HA anymore, it needs 1-2 mtp trunk cables and a single cassette for true A+B redundancy at central leafs.
Welcome @Pulsar67 make sure you pay @yoursunny in a nice purple tshirt.
NVMe VPS | Ryzen 7950X VDS | Dedicated Servers -- Crunchbits.com
My head is exploding reading all this
that you guys have to go through this much to provides services to us [effort & $$$]. Good luck with your journey [I will keep reading this thread time by time] 
Well, some we already did, some we read that others did so we not even tried, it is a mix.
I presume you had your share of mistakes/tests, we all did
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
"If there is no struggle there is no progress"
Yeah, I think that is normal. Even simple things that I knew were wrong at the time like "Oh man, we shouldn't deploy this server like this. I'm going to regret fixing this later, but we have no choice and it can't keep waiting".
NVMe VPS | Ryzen 7950X VDS | Dedicated Servers -- Crunchbits.com
we actually did that not so a few times.
But yes, ultimately you learn ( or some not ) . Either the option, it is fun when shit breaks loose and while you try to fix it, you remember "I'm going to regret fixing this later" and burst into
.
@Pulsar67
Sorry for doing a hijack here, on your thread, but you reminded us ( and not just me as it seems ) of so meany fun memories.
FYI, Welcome to the club
Host-C - VPS & Storage VPS Services – Reliable, Scalable and Fast - AS211462
"If there is no struggle there is no progress"
No worries I’ve taken a lot of advice given so I truly do appreciate it! 🙂
Hahaha don't be so mean to the user, he's just starting out and you're already discouraging him!!
NohaVPS LLC - www.nohavps.com
VPS Servers | AS63025 | Location: USA, LONDON UK, FRANKFURT