I'm totally re-doing my whole home network.  I have a pfSense machine with two 10GbE ports connected to my main switch which has eight 1GbE Ethernet ports and two 10GbE Ethernet ports.  The 8x1GbE ports are for each of my VLANS and the router connects with the 10GbE.  Question is, do I set up the router/switch with

Option 1)  both 10GbE ports from the router to the two 10GbE ports on the switch with link aggregation?  or
Option 2)  One 10GbE port on the router to one 10GbE port on the switch and connect the other 10GbE port on the switch to my 12 port 10GbE switch which all my servers, workstations are connected to?

Paint file attached for simplification.

Attached File


Thoughts:

I don't think I'll need a trunk that consists of two 10GbE link aggregated lines.  On the other hand, do any of my 10GbE connected machines need that kind of speed through the router?  I have a dedicated Plex machine running FreeNAS, that it's possible I could have all friends/family watching at once needing 8 separate streams.

Thoughts?

I don't think a LAG between the pfSense box and the first switch is going to do much. Your devices will still be limted to 10GbE. And with the ARP tables in the switch those devices won't even use the router if they are connected to the same switch. Option one would limit your 10GbE switch to 1GbE to the rest of the network. I would go with option 2.

So my main office desktop has a 10GbE NIC and is connected to my 10GbE switch along with my servers.  What advantage would I have if it was 10GbE connected all the way to the router?

There wouldn't be any advantage to the router. I'm thinking you would be 10GbE to the whole network. So if there were multiple devices on the 1GbE switch using a gig they wouldn't be bottle necked by the 1GbE connection to the other switch. I would want the fastest connection between switches. But it all depends on what each device on each switch does.

Option 2.

What hangs off your VLAN ports?
How much traffic is there between the devices on the main switch and the VLANS on your 1g ports?
Maybe go router to main switch to VLAN switch?

If neither of your switches do L3, maybe run each one back to a router port? It's going to be controlling traffic between the VLANS anyway, so....

For home? Think about this for a second. What transfers will actually be >1gb? Maybe file copies from your primary desktop to a NAS/file server? Anything else? So the 2 ports should be to your desktop and the NAS, anything else is a waste of time. A business might want a bigger link to the router for faster concurrent cross vlan traffic? But at home? Come on....

I say this as a guy with a pfsense router myself, 3x24p switches for different areas of the house and the server rack, 6 vlans and who does a lot of this for a living....

I had originally planned to have the main 1/10GbE switch off the router do nothing but VLANs and have the “router on a stick” setup.  I currently only have four machines that are set up for 10GbE but I’d like to keep them on three separate VLANs.  (Main unRaid server used as primary home backup server and a small FreeNAS machine that I just mess around with on one VLAN only accessible from the network, a dedicated Plex machine running FreeNAS accessible by friends and family outside the network, and my main office desktop).

Both option 1 and option 2 leave a single point of failure at the gig switch (so hardware failure leaves the network fractured and segmented). If it were me, I'd go with option 2 and loop back up to the router with a L2 ring (if your switches allow for that kind of topology) or dedicate a 10G port on the pfSense box to each switch which moves your primary point of failure to the router/firewall itself.

Neither option is optimal,  but both will work to an extent.

Option 1 compresses all the traffic from the second switch through a single 1 gig port.
Option 2 is better from a bandwidth standpoint, but lacks a redundant path.

I would connect each switch to the router with a 10GbE connection,and then connect the unused 10GbE switch ports to each other. Assuming your switches support STP (Spanning Tree Protocol), they will automagically disable the interconnect ports to prevent a level 2 loop, but if either 10GbE link to the router fails, they will re-enable the interconnect link and keep the network up.

The switches should support STP, or a proprietary equivalent like PVST+, unless they are super basic consumer grade devices.

Single point of failure for home networking is having a single ISP, single router, etc. Are we really worried at home someone is going to break one of the switch to switch links? Pretty sure I'd just walk over and check out the cable in that scenario... Just sayin'

Agree with both of your posts.

I'd buy one of these to use as my core switch.

I currently use the ES-12F as my core switch but I can't do a LAG between my EdgeRouter Pro and that switch.

I have LAG's set up though between all of my other switches ansd the ES-12F via fiber or copper.

The bottom switch is going to a lab ER-X so it can be ignored.

Attached File


ETA: I do wonder though if you really need 10Gbe at home though OP?  I mean I want it but I can't justify the expense when I'd hardly ever use it since it takes either RAID or M.2 PCIe drives to actually transfer anywhere close to 10Gbe speeds.

I ended up stumbling on a new but open box Netgear 12 port 10GbE switch for $400.  Of course, spent more than than for NICs for servers, workstations, etc.  Do I need it?  Definitely not.  But it is cool seeing a multi-Gb movie transfer in only a few seconds.

Attached File

Option 2 makes the most sense in that crazy scenario.  I'm Jelly.

I went to all the trouble of running Cat-6A STP, away from power cables, and put the best connectors on it I could find.

I guess I should think abut 10 GbE at some point.

10gbe is such a waste (minus cool factor) for the home. Less some very unique ad improbable scenarios one could dream up.

Personally, I would use LACP to bundle multiple 1gig ports into 1 logical trunk port. So you could have a 2 gig or a 4 gig connection for backbones in your network.

While I don't disagree for the most part having 10Gbe links to my NAS from my PC where I convert DVD's/Blu-ray's would be extremely nice.  Also having the same between my main NAS and backup NAS would be nice.

Besides video or other large file transfers though it is pretty much just for bragging rights at home.

That being said I just put a Supermicro 5018D-FN8T server in with Untangle at home to server as my router because I wanted to and it has 10Gbe ports that I will use for inter-VLAN routing down the road.  I also wanted to be able to handle gigabit fiber with QoS, IPS, etc.

ETA: The worst part is that even with 10Gbe I'd still use 2-port LAG's for redundancy as I'm crazy like that

That's enough out of you!

Yeah, we don't need that kind of negativity around here.