Cloud, OpenStack

Openstack Multi-Node with Single NIC

This post is a summary of my experience getting a multi-node, single nic setup up and running.  I have two older laptops (relative term, they still are x64 with intel-vt so not that old) that I use in this scenario.  One is a levono (called stinkpad) and the other is a dell (called dell).  The purpose of all this is to enable neutron networking on a juno ubuntu openstack setup.

Let’s define what I’m doing here.

My switch/router is an Linksys N3000 which is running dd-wrt kong.  Basically it allows me to utilize VLANs. I actaully have it on the same LAN segment as my Comcast Business Router (which has awesome wifi as a side note).  This subnet is the 10.1.10.0/24 with the comcast router at .1 and the dd-wrt at .2.  It is important to note I am NOT NATing between these, .1 does the DHCP and routing. I am pluggin my laptops into the dd-wrt directly (I can’t do VLANs on the comcast router).  Why am I complicated? No clue, it’s just the setup that was there when I got this working.  (see here for someone who blogged about this a bit in depth, I would caution you should use ssh and command line for the VLANs, the GUI is not 100% at all!).

Just a side note, I wasted a ton of time trying to use wlan0. Not only does the centrino chip on the stinkpad have issues, but it will never work.  The wifi interface on the router doesn’t play well with masqueraded or changed packets, nor will it do VLAN trunking they way I want it.

Stepping back, we are going to have 3 networks but you only need two.  One is an external internet one that is my LAN 10.1.10/0/24 which all computers can reach behind my router.  This network will be the network with the floating ip range allocated and the “public” or “external” network in openstack. If you are using all yous IPs, you can have the dd-wrt NAT from the comcast router on a new subnet, however I didn’t feel like figuring out the translation from a computer on the Comcast subnet to the dd-wrt one.  I choose to allocated DHCP on 100-200 so I used the 10.1.10.10 to the 10.1.10.60 range for openstack (giving me 49 IPs (the router will use the first one)).

Utilizing the wiki for dd-wrt I configured two vlans (really just needed one but why stop at one?).  I used VLAN 5 and VLAN 6.  VLAN 5 is the 10.5.0.0/24 subnet and VLAN 6 is the 10.6.0.0/24 subnet.  They are trunked to the ports and my configuration for my dd-wrt is below.

#===========================================================#
# DD-WRT V24-K26 #
# Kong Mod #
#===========================================================#

BusyBox v1.21.0 (2014-06-07 21:53:22 CEST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

root@DD-WRT:~# nvram show | grep vlan.*ports
size: 27882 bytes (33558 left)
vlan6ports=4t 3t 2t 1t 8
vlan2ports=0 8
new_vlan2ports=0 8
vlan5ports=4t 3t 2t 1t 8
vlan1ports=4 3 2 1 8*
new_vlan1ports=1 2 3 4 8*
root@DD-WRT:~# nvram show | grep port.*vlans
port5vlans=1
port3vlans=1
port1vlans=1
port4vlans=1
port2vlans=1
port0vlans=2
size: 27882 bytes (33558 left)
root@DD-WRT:~# nvram show | grep vlan.*hwname
new_vlan1hwname=et0
vlan6hwname=et0
vlan2hwname=et0
vlan5hwname=et0
vlan1hwname=et0
new_vlan2hwname=et0
size: 27882 bytes (33558 left)

As for the servers, I used the debian wiki and Steve Weston’s blog for help here.

configure the eth0 for stinkpad (my controller/compute/network node) with a static IP

# interfaces(5) file used by ifup(8) and ifdown(8)

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 10.1.10.165
netmask 255.255.255.0
gateway 10.1.10.1
dns-nameservers 10.1.10.1 75.75.75.75

and for the dell

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 10.1.10.223
netmask 255.255.255.0
gateway 10.1.10.1
dns-nameservers 10.1.10.1 75.75.75.75

I reboot here (good time to do an apt-get update && apt-get dist-upgrade; and make sure everything came back up

After reboot add the following for packet forwarding

net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0

To load the values run the following

sudo sysctl -p

Install the following packages

sudo apt-get -y install ntp vlan openvswitch-switch

load the modules

sudo modprobe 8021q

sudo sh -c ‘echo 8021q >> /etc/modules’

Now create the vlans on both servers and assign an IP (these .2 are for the stinkpad, the .4 is for the dell)

sudo vconfig add eth0 5
sudo ip addr add 10.5.0.2/24 dev eth0.5
sudo ip link set dev eth0.5 up

sudo vconfig add eth 6
sudo ip addr add 10.6.0.2/24 dev eth0.6
sudo ip link set dev eth0.6

Do the same for the dell

sudo vconfig add eth0 5
sudo ip addr add 10.5.0.4/24 dev eth0.5
sudo ip link set dev eth0.5 up

sudo vconfig add eth 6
sudo ip addr add 10.6.0.4/24 dev eth0.6
sudo ip link set dev eth0.6

Make sure the interfaces are working and you can ping from one another

Let’s add them to the /etc/network/interfaces,  – this is for dell but do the same for the other node

auto eth0.5
iface eth0.5 inet static
vlan-raw-device eth0
address 10.5.0.4
netmask 255.255.255.0

auto eth0.6
iface eth0.6 inet static
vlan-raw-device eth0
address 10.6.0.4
netmask 255.255.255.0

Now at this point I am going to create an OVS bridge for my primary interface. If you aren’t on the console or using one of the VLAN ports to ssh then you’ll want to change over since this will disconnect your current session.

First let’s create the bridge, assign the port and assign the IP (this is for dell but you obviously need to do this on both nodes)

sudo ovs-vsctl add-br br0
sudo ovs-vsctl add-port br0 eth0 #or whatever your interface is called)

sudo ifconfig br0 10.1.10.223 netmask 255.255.255.0
sudo route add default gw 10.1.10.1
sudo ifconfig eth0 0

Double check here and make sure the IP is working. Then let’s commit this to the interfaces and reboot to ensure it sticks. This is the entire interfaces for the dell node, note that it replaces what is there and includes the vlan info (which is not in a ovs bridge since we will let neutron do this through the ml2 plugin.

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eth0
allow-br0 eth0
iface eth0 inet manual
ovs_bridge br0
ovs_type OVSPort
pre-up ifconfig $IFACE up
post-down ifconfig $IFACE down
address 0.0.0.0

auto br0
allow-ovs br0
iface br0 inet static
address 10.1.10.223
netmask 255.255.255.0
gateway 10.1.10.1
dns-nameservers 10.1.10.1 75.75.75.75
ovs_type OVSBridge
ovs_ports eth0
pre-up ifconfig $IFACE up
post-down ifconfig $IFACE down

auto eth0.5
iface eth0.5 inet static
vlan-raw-device eth0
address 10.5.0.4
netmask 255.255.255.0

auto eth0.6
iface eth0.6 inet static
vlan-raw-device eth0
address 10.6.0.4
netmask 255.255.255.0

Now let’s make sure everything is working by crossing our fingers and rebooting.  Especially with Ubuntu you’ll likely get a bit of a delay and then we should have the interfaces up. Log back in and let’s see what the final product looks like.

stack@dell:~$ ifconfig
br0 Link encap:Ethernet HWaddr 00:22:19:e2:e5:de
inet addr:10.1.10.223 Bcast:10.1.10.255 Mask:255.255.255.0
inet6 addr: 2601:0:9182:b000:f4ac:61c9:ef87:d6ae/64 Scope:Global
inet6 addr: fe80::4006:22ff:fee7:c09b/64 Scope:Link
inet6 addr: 2601:0:9182:b000:222:19ff:fee2:e5de/64 Scope:Global
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:362029 errors:0 dropped:60192 overruns:0 frame:0
TX packets:32782 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:98079943 (98.0 MB) TX bytes:19888165 (19.8 MB)

eth0 Link encap:Ethernet HWaddr 00:22:19:e2:e5:de
inet6 addr: fe80::222:19ff:fee2:e5de/64 Scope:Link
inet6 addr: 2601:0:9182:b000:4ac:2aa5:9f09:e7a6/64 Scope:Global
inet6 addr: 2601:0:9182:b000:222:19ff:fee2:e5de/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:744185 errors:0 dropped:0 overruns:0 frame:0
TX packets:351772 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:421597281 (421.5 MB) TX bytes:72956467 (72.9 MB)
Interrupt:17

eth0.5 Link encap:Ethernet HWaddr 00:22:19:e2:e5:de
inet addr:10.5.0.4 Bcast:10.5.0.255 Mask:255.255.255.0
inet6 addr: fe80::222:19ff:fee2:e5de/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:226093 errors:0 dropped:0 overruns:0 frame:0
TX packets:279632 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:305497247 (305.4 MB) TX bytes:48168955 (48.1 MB)


eth0.6 Link encap:Ethernet HWaddr 00:22:19:e2:e5:de
inet addr:10.6.0.4 Bcast:10.6.0.255 Mask:255.255.255.0
inet6 addr: fe80::222:19ff:fee2:e5de/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:257 errors:0 dropped:0 overruns:0 frame:0
TX packets:292 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:28742 (28.7 KB) TX bytes:37008 (37.0 KB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:69856 errors:0 dropped:0 overruns:0 frame:0
TX packets:69856 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:6204825 (6.2 MB) TX bytes:6204825 (6.2 MB)

stack@dell:~$ sudo ovs-vsctl show
722d1b65-7e9a-4cdc-a8b2-8a71448a507d
Bridge "br0"
Port "eth0"
Interface "eth0"
Port "br0"
Interface "br0"
type: internal
ovs_version: "2.0.2"

Make sure you modify /etc/hosts

for dell:

127.0.0.1 localhost
10.1.10.223 dell
10.5.0.4 compute1
10.1.10.165 stinkpad
10.5.0.2 controller network database compute0

for stinkpad:

127.0.0.1 localhost
10.1.10.165 stinkpad
10.5.0.2 controller database network compute0
10.1.10.223 dell
10.5.0.4 compute1

At this point you can install openstack or devstack

Again remembering my public network is 10.1.10.0/24 and I want to use .10-.60 for my floating ips
10.5.0.0/24 is my management
10.6.0.0/24 is my tunneling interface

here is my l3_agent.ini

[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
use_namespaces = True
external_network_bridge = br0
router_delete_namespaces = True

/etc/neutron/plugins/ml2/ml2_conf.ini

[ml2]
type_drivers = flat,gre
tenant_network_types = gre
mechanism_drivers = openvswitch

[ml2_type_flat]
Example:flat_networks = external

[ml2_type_vlan]

[ml2_type_gre]
tunnel_id_ranges = 1:1000

[ml2_type_vxlan]

[securitygroup]
enable_security_group = True
enable_ipset = True
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

[ovs]
local_ip = 10.6.0.2
enable_tunneling = True
bridge_mappings = external:br0

[agent]
tunnel_types = gre

We use gre to create tunnels on the 10.6.0.0 network between the nodes so the tenant (within the tenant) can talk to each other.

If you want to experience migration of the VMs on KVM, don’t forget to enable the ssh key for the nova account (you’ll need to set it to /bin/bash and actually create and import the key for passwordless auth between node)

Hope this helps someone, if you’re on a single machine, you can create vlans because they won’t go anywhere. If you can get switch VLANs down, this is actually pretty simple compared to trying to fake bridge and do some masquerading to get this to work.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s