pfSense setting up bridge networks for use in KVM

pfSense setting up bridge networks for use in KVM

My KVM host is Centos 6.6 with kernel upgrade to 3.19.8. I have an on-board and an add on PCIe 1GB NIC's for use.

I use this setup to install pfSense as a KVM guest using the two physical NIC's in my KVM host.

My first interface eth0 is a Qualcomm® Atheros 1GB on-board NIC with HWADDR=74:d4:35:ea:90:1f and using IP addr of 192.168.10.100 and network of 192.168.10.0/24
This interface is behind a 1GB switch then to my router attached to the Internet.

My second interface eth1 is a TP Link 1GB add on card with HWADDR=30:B5:C2:02:77:2B and using IP addr of 192.168.100.100 and network of 192.168.100.0/24
This interface is plugged into a standalone 1GB switch and has no other connectivity or networks attached.

My object is to be able to build guest images in my KVM host that can use either, or both, interfaces.
I want to install a pfSense guest image using both interfaces, eth0 as WAN and eth1 and LAN.
Then I can build two more Centos guest images, one with just eth0 and the other with eth1, and use the pfSense image as the router between the two.

First I make sure network manager is off, and stays off on reboot.

# chkconfig NetworkManager off
# service NetworkManager stop

Verify your resolv.conf file. Should look something like

# vi /etc/resolv.conf
search whittenberg.domain
nameserver 192.168.10.1

Verify your networks file

# vi /etc/networks
default 0.0.0.0
loopback 127.0.0.0
link-local 169.254.0.0

Verify your network file

# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=nas.whittenberg.domain
GATEWAY=192.168.10.1
NETWORKING_IPV6=no

Setup your ifcfg-eth0 file. Notice I rem out the IP numbers and add BRIDGE=br0 at the bottom .

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=74:d4:35:ea:90:1f
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
NETWORKING_IPV6=no
IPV6_AUTOCONF=no
#IPADDR=192.168.10.100
#NETMASK=255.255.255.0
#BROADCAST=192.168.10.255
#NETWORK=192.168.10.0
#GATEWAY=192.168.10.1
BRIDGE=br0

Setup your ifcfg-eth1 file. Notice I rem out the IP numbers and add BRIDGE=br1 at the bottom .

# vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=30:B5:C2:02:77:2B
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
NETWORKING_IPV6=no
IPV6_AUTOCONF=no
#IPADDR=192.168.100.100
#NETMASK=255.255.255.0
#BROADCAST=192.168.100.255
#NETWORK=192.168.100.0
#GATEWAY=192.168.10.1
BRIDGE=br1

Build a new ifcfg-br0 file. Notice this has the IP numbers in it and not rem'd out, DEVICE=br0, and TYPE=Bridge (Use upper-case B in Bridge or will not work)

# vi /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
NETWORKING_IPV6=no
IPV6_AUTOCONF=no
STP=yes
DELAY=0
IPADDR=192.168.10.100
NETMASK=255.255.255.0
BROADCAST=192.168.10.255
NETWORK=192.168.10.0
GATEWAY=192.168.10.1

Build a new ifcfg-br1 file. Notice this has the IP numbers in it and not rem'd out, DEVICE=br1, and TYPE=Bridge (Use upper-case B in Bridge or will not work)
I also have my GATEWAY at 192.168.10.1 so I can route out on the KVM host from either interface

# vi /etc/sysconfig/network-scripts/ifcfg-br1
DEVICE=br1
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
NETWORKING_IPV6=no
IPV6_AUTOCONF=no
STP=yes
DELAY=0
IPADDR=192.168.100.100
NETMASK=255.255.255.0
BROADCAST=192.168.100.255
NETWORK=192.168.100.0
GATEWAY=192.168.10.1

Now restart the network

# service network restart

Verify your setup

# ifconfig
br0       Link encap:Ethernet  HWaddr 74:D4:35:EA:90:1F
          inet addr:192.168.10.100  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::76d4:35ff:feea:901f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4364 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2023 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:742025 (724.6 KiB)  TX bytes:255958 (249.9 KiB)

br1       Link encap:Ethernet  HWaddr 30:B5:C2:02:77:2B
          inet addr:192.168.100.100  Bcast:192.168.100.255  Mask:255.255.255.0
          inet6 addr: fe80::32b5:c2ff:fe02:772b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2381 errors:0 dropped:0 overruns:0 frame:0
          TX packets:535 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:291677 (284.8 KiB)  TX bytes:38805 (37.8 KiB)

eth0      Link encap:Ethernet  HWaddr 74:D4:35:EA:90:1F
          inet6 addr: fe80::76d4:35ff:feea:901f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10464 errors:0 dropped:34 overruns:0 frame:0
          TX packets:3595 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2023784 (1.9 MiB)  TX bytes:481595 (470.3 KiB)
          Interrupt:18

eth1      Link encap:Ethernet  HWaddr 30:B5:C2:02:77:2B
          inet6 addr: fe80::32b5:c2ff:fe02:772b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5202 errors:0 dropped:34 overruns:0 frame:0
          TX packets:2479 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1166588 (1.1 MiB)  TX bytes:157098 (153.4 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2951 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2951 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:646709 (631.5 KiB)  TX bytes:646709 (631.5 KiB)
         

Verify the bridge

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.74d435ea901f       yes             eth0
br1             8000.30b5c202772b       yes             eth1
pan0            8000.000000000000       no


Check your routes

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         TEW-731BR       0.0.0.0         UG    0      0        0 br0
link-local      *               255.255.0.0     U     1011   0        0 br0
link-local      *               255.255.0.0     U     1012   0        0 br1
192.168.10.0    *               255.255.255.0   U     0      0        0 br0
192.168.100.0   *               255.255.255.0   U     0      0        0 br1

Notice my default gateway. TEW-731BR is my router name on 192.168.10.1 IP addr.

Test we have internet access bot FQDN and IP addr

# ping www.google.com
PING www.google.com (74.125.21.99) 56(84) bytes of data.
64 bytes from yv-in-f99.1e100.net (74.125.21.99): icmp_seq=1 ttl=42 time=90.1 ms

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=24.4 ms
         
Now you should be able to use br0 and br1 in your KVM guest images.

I will setup br0 as my WAN interface in pfSense guest image and give it IP addr of 192.168.10.10 with a route out of 192.168.10.1

I will also setup br1 as my LAN interface in pfSense guest image and give it IP addr of 192.168.100.10

Now any other guest image I build using the br1 interface I can set that image for a route out on 192.168.100.10 and it will go through the pfSense router for all request off the 192.168.100.0/24 network.

Hope this helps!

Added 6/1/2015 18:18 AZ
Well it worked! I used the e1000 NIC adapter on both interfaces when I built my pfSense image. The max transfer speed I was able to get out of it was 50MBs across the router. I think this may be do to the fact I am using the same zfs pool for source and destination file. My pfSense image is on the zfs pool. My test Centos image I put on 192.168.100.0 network is on the zfs pool. And the file I was retrieving is also on the zfs pool, but on a 192.168.10.0 network. I will try and get a share off the pool, and on my SSD, from the 192.168.10.0 network and trying the transfer again. BTW cpu was at 45% during the transfer. I used all 4 cores for all images. This is shared with the Host of course that was serving up the file off the zfs pool on 192.168.10.0 network. I may need another physical machine on the 192.168.10.0 network to get into the 100MB's (That's my goal).