C eph (Ceph Homepage - Ceph) is a great way to deploy persistent storage with OpenStack. Ceph can be used as the persistent storage backend with OpenStack Cinder (GitHub - openstack/cinder: OpenStack Block Storage (Cinder)) for: Volumes of nova VMs. Glance images. Without ceph, storage in OpenStack is ephemeral or temporary and will be deleted when we delete a nova VM.  Hence, ceph is great for: Persisting the data in the volume of a nova VM even after we delete the nova VM. Live migration of a nova VM to a different compute host. Data backup and replication of the data in a nova VM and glance images. Data reliability in OpenStack (data will still be available if a nova VM crashes). Here are some useful links about live migration in OpenStack using ceph: https://docs.openstack.org/nova/pike/admin/configuring-migrations.html https://docs.openstack.org/ha-guide/storage-ha-backend.html https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html This blog deploys ceph 0.94 (Hammer stable) with OpenStack Ocala. Ceph Releases — Ceph Documentation Below are the configurations needed for ceph, glance, cinder and nova in OpenStack Ocata: On the OpenStack controller host: /etc/ceph/ceph.conf: [global] osd pool default crush rule = 0 osd pool default size = 3 public_network = 172.18.0.0/16 err to syslog = true mon host = 172.18.7.160,172.18.7.161,172.18.7.162 auth cluster required = none osd pool default min size = 0 # 0 means no specific default; ceph will use (pool_default_size)-(pool_default_size/2) so 2 if pool_default_size=3 osd pool default pgp num = 128 auth service required = none mon client hunt interval = 40 log to syslog = true auth supported = none auth client required = none clog to syslog = true mon_initial_members = controller1,controller2,controller3 cluster_network = 172.18.0.0/16 log file = /dev/null max open files = 16229437 fsid = c29e5118-e0d3-c1c1-1d3d-fc32b8f011c5 osd pool default pg num = 128 [client] rbd default map options = rw rbd cache writethrough until flush = true log file = /var/log/rbd-clients/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor rbd default features = 3 # sum features digits admin socket = /var/run/ceph/rbd-clients/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor rbd concurrent management ops = 20 rbd default format = 2 rbd cache = true [mon.controller1] host = controller1 [mon.controller2] host = controller2 [mon.controller3] host = controller3 [mon] mon osd full ratio = 0.95 mon clock drift warn backoff = 30 mon cluster log file = /dev/null mon lease = 20 mon osd min down reporters = 7 # number of OSDs per host + 1 mon cluster log to syslog = true mon osd nearfull ratio = 0.9 mon pg warn max object skew = 10 # set to 20 or higher to disable complaints about number of PGs being too low if some pools have very few objects bringing down the average number of objects per pool. This happens when running RadosGW. Ceph default is 10 mon osd down out interval = 600 mon pg warn max per osd = 0 # disable complains about low pgs numbers per osd mon clock drift allowed = 0.15 mon lease ack timeout = 40 mon lease renew interval = 12 mon osd report timeout = 300 mon osd allow primary affinity = true mon accept timeout = 40 [osd] osd scrub sleep = 0.1 osd recovery threads = 1 osd scrub load threshold = 10.0 osd heartbeat grace = 30 filestore op threads = 2 osd scrub begin hour = 0 osd mon heartbeat interval = 30 osd disk thread ioprio priority = 7 osd mount options xfs = noatime,largeio,inode64,swalloc osd max backfills = 1 osd objectstore = filestore osd op threads = 2 osd scrub end hour = 24 osd max scrubs = 1 filestore merge threshold = 40 osd recovery max chunk = 1048576 osd mkfs options xfs = -f -i size=2048 osd recovery max active = 1 osd scrub chunk max = 5 osd deep scrub stride = 1048576 osd disk thread ioprio class = idle filestore max sync interval = 5 filestore split multiple = 8 osd crush update on start = true osd recovery op priority = 2 osd deep scrub interval = 2419200 osd mkfs type = xfs osd journal size = 4096 /etc/ceph/ceph.client.admin.keyring: [client.admin]   key = ABCDEMtZYzuLGBAAktFtABCDE/ABCDErOs2Nhg== /etc/glance/glance-api.conf: [glance_store] default_store=rbd stores=rbd, file, http rbd_store_ceph_conf=/etc/ceph/ceph.conf rbd_store_pool=nova-images1 rbd_store_chunk_size=8 /etc/glance/glance-scrubber.conf: [glance_store] default_store=rbd stores=rbd, file, http rbd_store_ceph_conf=/etc/ceph/ceph.conf rbd_store_pool=nova-images1 rbd_store_chunk_size=8 /etc/cinder/cinder.conf: [DEFAULT] default_volume_type=ceph /etc/cinder/cinder-volume.conf: [DEFAULT] enabled_backends=ceph-default [ceph-default] volume_driver=cinder.volume.drivers.rbd.RBDDriver rbd_pool=nova-images1 rbd_store_ceph_conf=/etc/ceph/ceph.conf rbd_flatten_volume_from_snapshot=false rbd_max_clone_depth=5 rbd_store_chunk_size=4 volume_backend_name=ceph-default /etc/rabbitmq/rabbitmq-monitor-queues.conf: autogen.controller.cinder-scheduler 20 5 autogen.cinder-volume.cinder-volume-1.ceph-default 20 5 On the OpenStack compute host: /etc/ceph/ceph.conf: [global] osd pool default crush rule = 0 osd pool default size = 3 public_network = 172.18.0.0/16 err to syslog = true mon host = 172.18.7.160,172.18.7.161,172.18.7.162 auth cluster required = none osd pool default min size = 0 # 0 means no specific default; ceph will use (pool_default_size)-(pool_default_size/2) so 2 if pool_default_size=3 osd pool default pgp num = 128 auth service required = none mon client hunt interval = 40 log to syslog = true auth supported = none auth client required = none clog to syslog = true mon_initial_members = controller1,controller2,controller3 cluster_network = 172.18.0.0/16 log file = /dev/null max open files = 16229437 fsid = c29e5118-e0d3-c1c1-1d3d-fc32b8f011c5 osd pool default pg num = 128 [client] rbd default map options = rw rbd cache writethrough until flush = true log file = /var/log/rbd-clients/qemu-guest-$pid.log # must be writable by QEMU and allowed by SELinux or AppArmor rbd default features = 3 # sum features digits admin socket = /var/run/ceph/rbd-clients/$cluster-$type.$id.$pid.$cctid.asok # must be writable by QEMU and allowed by SELinux or AppArmor rbd concurrent management ops = 20 rbd default format = 2 rbd cache = true [mon.controller1] host = controller1 [mon.controller2] host = controller2 [mon.controller3] host = controller3 [mon] mon osd full ratio = 0.95 mon clock drift warn backoff = 30 mon cluster log file = /dev/null mon lease = 20 mon osd min down reporters = 7 # number of OSDs per host + 1 mon cluster log to syslog = true mon osd nearfull ratio = 0.9 mon pg warn max object skew = 10 # set to 20 or higher to disable complaints about number of PGs being too low if some pools have very few objects bringing down the average number of objects per pool. This happens when running RadosGW. Ceph default is 10 mon osd down out interval = 600 mon pg warn max per osd = 0 # disable complains about low pgs numbers per osd mon clock drift allowed = 0.15 mon lease ack timeout = 40 mon lease renew interval = 12 mon osd report timeout = 300 mon osd allow primary affinity = true mon accept timeout = 40 [osd] osd scrub sleep = 0.1 osd recovery threads = 1 osd scrub load threshold = 10.0 osd heartbeat grace = 30 filestore op threads = 2 osd scrub begin hour = 0 osd mon heartbeat interval = 30 osd disk thread ioprio priority = 7 osd mount options xfs = noatime,largeio,inode64,swalloc osd max backfills = 1 osd objectstore = filestore osd op threads = 2 osd scrub end hour = 24 osd max scrubs = 1 filestore merge threshold = 40 osd recovery max chunk = 1048576 osd mkfs options xfs = -f -i size=2048 osd recovery max active = 1 osd scrub chunk max = 5 osd deep scrub stride = 1048576 osd disk thread ioprio class = idle filestore max sync interval = 5 filestore split multiple = 8 osd crush update on start = true osd recovery op priority = 2 osd deep scrub interval = 2419200 osd mkfs type = xfs osd journal size = 4096 /etc/nova/nova.conf: [libvirt] images_type=rbd images_rbd_pool=nova-images1 images_rbd_ceph_conf=/etc/ceph/ceph.conf The OpenStack controller hosts run ceph-mon, glance (glance-api, glance-registry) and cinder (cinder-api, cinder-scheduler) the following way: ceph-mon: /usr/bin/ceph-mon -i controller1 --pid-file /var/run/ceph/mon.controller1.pid -c /etc/ceph/ceph.conf --cluster ceph -f glance (glance-api, glance-registry): /usr/bin/python2 /usr/bin/glance-api /usr/bin/python2 /usr/bin/glance-registry cinder (cinder-api, cinder-scheduler): /usr/bin/python2 /usr/bin/cinder-api --config-file /etc/cinder/cinder.conf --config-file /etc/cinder/cinder-api.conf /usr/bin/python2 /usr/bin/cinder-scheduler --config-file /etc/cinder/cinder.conf --config-file /etc/cinder/cinder-scheduler.conf The OpenStack controller hosts also run the nova services like nova-api, nova-cert, nova-conductor, nova-scheduler, nova-console, nova-consoleauth and nova-novncproxy. The OpenStack compute hosts run ceph-osd and nova-compute the following way: ceph-osd: /usr/bin/ceph-osd -i 1 --pid-file /var/run/ceph/osd.1.pid -c /etc/ceph/ceph.conf --cluster ceph -f /usr/bin/ceph-osd -i 5 --pid-file /var/run/ceph/osd.5.pid -c /etc/ceph/ceph.conf --cluster ceph -f /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph -f /usr/bin/ceph-osd -i 13 --pid-file /var/run/ceph/osd.13.pid -c /etc/ceph/ceph.conf --cluster ceph -f nova-compute: /usr/bin/python2 /usr/bin/nova-compute --config-file /etc/nova/nova.conf --config-file /etc/nova/nova-compute.conf There are two ways to deploy ceph in OpenStack: Ceph-osd on the compute hosts (the way this blog describes). Ceph-osd on separate storage hosts and not on compute hosts. If we deploy ceph-osd on the compute hosts, we will lose a ceph host if a compute host crashes.  Also, this approach will increase the CPU and memory usage of the compute host as ceph-osd competes with nova-compute on the compute host. If we deploy ceph-osd on separate hosts and not on compute hosts, it is best to use fast 40 Gig links between the compute hosts and the storage hosts so that the ceph traffic moves fast between compute host and storage hosts.  Refer Jumbo Mumbo in OpenStack using Cisco's UCS servers and Nexus 9000 to configure jumbo frames in OpenStack as it increases the throughput of the ceph traffic between the compute host and storage hosts. This approach is expensive as we need separate storage hosts for ceph-osd. Below are the outputs of some useful ceph commands run on the controller host to check if the ceph cluster in OpenStack is in good condition: # ceph version ceph version 0.94.9-9.el7cp (b83334e01379f267fb2f9ce729d74a0a8fa1e92c) # ceph status     cluster c29e5118-e0d3-c1c1-1d3d-fc32b8f011c5      health HEALTH_WARN             clock skew detected on mon.controller2             1 mons down, quorum 0,1 controller1,controller2             Monitor clock skew detected      monmap e1: 3 mons at {controller1=172.18.7.160:6789/0,controller2=172.18.7.161:6789/0,controller3=172.18.7.162:6789/0}             election epoch 8, quorum 0,1 controller1,controller2      osdmap e172: 16 osds: 16 up, 16 in       pgmap v201041: 320 pgs, 3 pools, 583 MB data, 8218 objects             2746 MB used, 4730 GB / 4733 GB avail                  320 active+clean # ceph osd pool stats pool rbd id 0   nothing is going on pool nova-images1 id 1   nothing is going on pool backups id 2   nothing is going on # ceph pg dump | head -8 dumped all in format plain version 201067 stamp 2017-10-05 03:35:55.386523 last_osdmap_epoch 172 last_pg_scan 3 full_ratio 0.95 nearfull_ratio 0.9 pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 2.7d 0 0 0 0 0 0 0 0 active+clean 2017-10-04 21:54:02.754318 0'0 172:45 [9,8,14] 9 [9,8,14] 9 0'0 2017-10-04 21:54:02.754202 0'0 2017-09-27 02:42:12.189613 # ceph mon dump dumped monmap epoch 1 epoch 1 fsid c29e5118-e0d3-c1c1-1d3d-fc32b8f011c5 last_changed 0.000000 created 0.000000 0: 172.18.7.160:6789/0 mon.controller1 1: 172.18.7.161:6789/0 mon.controller2 2: 172.18.7.162:6789/0 mon.controller3 # ceph df GLOBAL:     SIZE      AVAIL     RAW USED     %RAW USED     4733G     4730G        2753M          0.06 POOLS:     NAME             ID     USED     %USED     MAX AVAIL     OBJECTS     rbd              0         0         0         1576G           0     nova-images1     1      583M      0.04         1576G        8218     backups          2         0         0         1576G           0 # ceph osd pool ls detail pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0 pool 1 'nova-images1' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 172 flags hashpspool stripe_width 0   removed_snaps [1~6,b~8d] pool 2 'backups' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 3 flags hashpspool stripe_width 0 # ceph -w     cluster c29e5118-e0d3-c1c1-1d3d-fc32b8f011c5      health HEALTH_WARN             clock skew detected on mon.controller2             1 mons down, quorum 0,1 controller1,controller2             Monitor clock skew detected      monmap e1: 3 mons at {controller1=172.18.7.160:6789/0,controller2=172.18.7.161:6789/0,controller3=172.18.7.162:6789/0}             election epoch 8, quorum 0,1 controller1,controller2      osdmap e172: 16 osds: 16 up, 16 in       pgmap v201177: 320 pgs, 3 pools, 664 MB data, 9515 objects             3097 MB used, 4730 GB / 4733 GB avail                  320 active+clean   client io 0 B/s wr, 90 op/s 2017-10-05 03:42:07.745506 mon.0 [INF] pgmap v201176: 320 pgs: 320 active+clean; 669 MB data, 3111 MB used, 4730 GB / 4733 GB avail; 377 kB/s wr, 96 op/s 2017-10-05 03:42:08.755441 mon.0 [INF] pgmap v201177: 320 pgs: 320 active+clean; 664 MB data, 3097 MB used, 4730 GB / 4733 GB avail; 0 B/s wr, 90 op/s # ceph quorum_status {"election_epoch":8,"quorum":[0,1],"quorum_names":["controller1","controller2"],"quorum_leader_name":"controller1","monmap":{"epoch":1,"fsid":"c29e5118-e0d3-c1c1-1d3d-fc32b8f011c5","modified":"0.000000","created":"0.000000","mons":[{"rank":0,"name":"controller1","addr":"172.18.7.160:6789\/0"},{"rank":1,"name":"controller2","addr":"172.18.7.161:6789\/0"},{"rank":2,"name":"controller3","addr":"172.18.7.162:6789\/0"}]}} # ceph mon_status {"name":"controller1","rank":0,"state":"leader","election_epoch":8,"quorum":[0,1],"outside_quorum":[],"extra_probe_peers":["172.18.7.161:6789\/0","172.18.7.162:6789\/0"],"sync_provider":[],"monmap":{"epoch":1,"fsid":"c29e5118-e0d3-c1c1-1d3d-fc32b8f011c5","modified":"0.000000","created":"0.000000","mons":[{"rank":0,"name":"controller1","addr":"172.18.7.160:6789\/0"},{"rank":1,"name":"controller2","addr":"172.18.7.161:6789\/0"},{"rank":2,"name":"controller3","addr":"172.18.7.162:6789\/0"}]}} # ceph auth list | head -5 installed auth entries: osd.0   key: AQCCF8tZf1O0HxAAXWSfmzMokX5QSPEHr00nvA==   caps: [mon] allow profile osd   caps: [osd] allow * osd.1 # ceph mds stat e1: 0/0/0 up # ceph mon stat e1: 3 mons at {controller1=172.18.7.160:6789/0,controller2=172.18.7.161:6789/0,controller3=172.18.7.162:6789/0}, election epoch 8, quorum 0,1 controller1,controller2 Refer How to attach cinder/ceph XFS volume to a nova instance in OpenStack horizon for the steps how to mount a ceph XFS volume inside a nova VM and read/write data from/to it. Hope this blog is useful!  Please let me know your comments below!  ... View more

T his is a blog about how to deploy multicast in an OpenStack cloud.  Unlike broadcast and unicast, multicast is used for one-to-many communications. What is multicast? One-to-many communication Driven by receivers (On the contrary, unicast is driven by senders) A multicast “group” is identified by a multicast IP address Without multicast, sender must duplicate each packet Analogy: TV and radio broadcasting Components of multicast Multicast addressing Group addresses range from 224.0.0.0 to 239.255.255.255 (class D addresses) Mac addresses range from 01-00-5E-00-00-00 to 01-00-5E-7F-FF-FF Multicast group management (IGMP) Multicast routing (Protocol Independent Multicast or PIM) and IGMP querying Unicast routing (OSPF, IS-IS, RIP) is needed for PIM Multicast clients that receive multicast traffic/streams Multicast use cases One-to-many corporate communications such as Employee training videos Quarterly company meetings Company-wide corporate communications Executive announcements Music/media streaming Video podcasts IPTV services One-to-many software updates/patches Social networking (Facebook, Twitter feeds, Instagram) Financial services, banks, stock exchange Government/Federal agencies RTP (Real Time Protocol) applications State of Multicast in OpenStack No multicast available out-of-the-box Open vSwitch 2.5 (virtual L2 switch) supports IGMP snooping (Open vSwitch) Linux Bridge 2.4 (virtual L2 switch) supports IGMP snooping (https://wiki.linuxfoundation.org/networking/bridge) For brN, the settings can be found under /sys/devices/virtual/net/brN/bridge. multicast_snooping Neutron virtual router (L3-Agent) does not support multicast routing, PIM and querying No CLI or API to configure multicast Multicast cannot be configured in the Horizon GUI Anti-spoofing rules and security groups drop multicast packets No plugin support/architecture for multicast How to deploy Multicast in OpenStack? Enable IGMP snooping in OVS/Linux bridge Add rules to allow multicast UDP port in security groups Disable neutron port-security for ports in multicast path Use neutron’s --allowed-address-pairs attribute and allow multicast group's destination IP address and MAC addresses SR-IOV ports may also be used Three recommended network architectures to deploy Multicast in OpenStack: Use provider networks without neutron router (L3-agent) and do multicast and unicast routing on upstream L3 devices connected to TOR outside OpenStack Use Cisco’s ASR1k plugin for neutron instead of L3-agent Use Cisco’s ACI and APIC driver for neutron Using provider networks without neutron router (L3-agent) Don’t use Neutron-L3-Agent  (OpenStack’s virtual router not used) Neutron does not route any multicast or unicast traffic Use Neutron provider network Use lab-routable public VLAN configured on top-of-rack switches Use lab's router outside OpenStack Use lab's gateway outside OpenStack Attach Nova instances directly to provider network with no neutron router Configure multicast routing / PIM and unicast routing (OSPF) on the lab upstream router (N9k) OVS does IGMP snooping Multicast source sends multicast UDP streams PIM on lab's upstream router (N9k) forwards multicast packets to OpenStack VMs Nova VMs receive/consume multicast streams Use Cisco’s ASR1k plugin for neutron instead of L3-agent Use Cisco’s ASR1k plugin for OpenStack Neutron instead of neutron-L3-agent GitHub - openstack/networking-cisco: Cisco Vendor Code for Neutron Neutron-L3-Agent (OpenStack’s virtual router) not used ASK1k plugin does both unicast and multicast routing Two ASR1000s used in VPC pair Neutron talks to ASR1k using Cisco’s config agent (using NETCONF) Neutron does not route any multicast or unicast routing traffic Configure multicast routing / PIM and unicast routing (OSPF) on the ASRs OVS does IGMP snooping Multicast source sends multicast UDP streams PIM on the ASRs forwards multicast packets to OpenStack VMs Nova VMs receive/consume multicast streams ACI and APIC driver for neutron UCSM multicast policy ACI/APIC multicast policy Multicast for media applications Cisco’s video conferencing apps: Telepresence Webex Jabber Cisco’s video and collaborations apps: Virtualized Video Processing (V2P) Videoscape AnyRes IPTV devices video encoders video surveillance VLC player http://www.videolan.org/projects/multicat.html Multicast limitations IGMP snooping degrades performance of layer-2 switch High bandwidth multicast traffic degrades unicast routing Needs plugin (ASR1k, APIC) integration with neutron Best-effort and out-of-sequence delivery (UDP is unreliable) Lack of TCP windowing results in network congestion Duplicate packets and occasional loops when unicast routing is broken Multicast testing tools Iperf - https://iperf.fr Nuttcp - http://www.nuttcp.net OpenStack VMTP - https://github.com/openstack/vmtp OpenStack Kloudbuster - https://github.com/openstack/kloudbuster Multicast talk I gave at Cisco Live in Las Vegas in 2016 Multicast in OpenStack Hope this blog is helpful to anyone deploying Multicast with OpenStack using Cisco's Nexus9000 and UCS! ... View more

T his blog has the steps to deploy OpenStack Newton with OpenDaylight Boron and Open vSwitch on CentOS-7 in VirtualBox on Mac laptop. Below are the versions used: OpenStack DevStack stable Newton (GitHub - openstack-dev/devstack at stable/newton) OpenDaylight Boron (https://nexus.opendaylight.org/content/repositories/opendaylight.release/org/opendaylight/integration/distribution-karaf…   or   https://www.opendaylight.org/downloads) CentOS-7 (http://mirror.cc.columbia.edu/pub/linux/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1611.iso) VirtualBox 5.0.30 (https://www.virtualbox.org/wiki/Downloads) Apple OS X El Capitan 10.11.6 Mac laptop Below is the architecture: VirtualBox is installed on Mac laptop and the CentOS-7 VM is created in VirtualBox. In VirtualBox, the CentOS-7-x86_64-Minimal-1611.iso image is used to boot a CentOS-7 VM with 4 GB RAM and the following two network adapters.  A host-only adapter is not needed. eth0 as a NAT adapter eth1 as an internal network adapter eth0 is a NAT adapter. eth1 is an internal network adapter. Run the following bash script to configure VirtualBox.  It will forward the required TCP ports from the host (Mac laptop) to the guest (CentOS-7 VM) and will also create eth1 as an internal network adapter. #!/bin/bash # Forward TCP port 3022 on host to TCP port 22 on guest VM so # that host can SSH into guest VM if ! VBoxManage showvminfo devstack-odl | grep 3022 > /dev/null then     VBoxManage modifyvm devstack-odl --natpf1 "SSH,TCP,,3022,,22" fi # Forward TCP port 8080 on host to TCP port 80 on guest VM so # that host can access OpenStack Horizon in browser if ! VBoxManage showvminfo devstack-odl | grep 8080 > /dev/null then     VBoxManage modifyvm devstack-odl --natpf1 "HTTP,TCP,,8080,,80" fi # Forward TCP port 6080 on host to TCP port 6080 on guest VM so # that host can access Nova VNC console in browser if ! VBoxManage showvminfo devstack-odl | grep 6080 > /dev/null then     VBoxManage modifyvm devstack-odl --natpf1 "CONSOLE,TCP,,6080,,6080" fi # Forward TCP port 8282 on host to TCP port 8181 on guest VM so # that host can access OpenDaylight web GUI at # http://localhost:8282/index.html (admin/admin) if ! VBoxManage showvminfo devstack-odl | grep 8282 > /dev/null then     VBoxManage modifyvm devstack-odl --natpf1 "ODL,TCP,,8282,,8181" fi # Forward TCP port 8187 on host to TCP port 8087 on guest VM so # that we can curl the OpenDaylight controller if ! VBoxManage showvminfo devstack-odl | grep 8187 > /dev/null then     VBoxManage modifyvm devstack-odl --natpf1 "ODL_neutron,TCP,,8187,,8087" fi # Add internal network adapter for guest VM if ! VBoxManage showvminfo devstack-odl | grep eth1 > /dev/null then     VBoxManage modifyvm devstack-odl --nic2 intnet     VBoxManage modifyvm devstack-odl --intnet2 "eth1" fi # Remove stale entry in ~/.ssh/known_hosts on host if [ -f ~/.ssh/known_hosts ]; then     sed -i '' '/\[127.0.0.1\]:3022/d' ~/.ssh/known_hosts fi Below are the forwarded ports through eth0 (NAT interface) in VirtualBox.  Host is the Mac laptop and the guest VM is CentOS-7 booted in VirtualBox. TCP port 3022 on host is forwarded to TCP port 22 on guest VM so that host can SSH into guest VM. TCP port 8080 on host is forwarded to TCP port 80 on guest VM so that host can access OpenStack Horizon in browser. TCP port 6080 on host is forwarded to TCP port 6080 on guest VM so that host can access Nova VNC console in browser. TCP port 8282 on host is forwarded to TCP port 8181 on guest VM so that host can access the OpenDaylight GUI in browser. TCP port 8187 on host is forwarded to TCP port 8087 on guest VM so that host can access neutron's ml2 ODL url in browser. Below is the screenshot of the forwarded ports through eth0 (NAT interface) in VirtualBox. Now, boot the CentOS-7 VM in VirtualBox.  Choose "VDI" as the disk format for the VM. When booting the CentOS-7 VM in VirtualBox, press the Tab key and type the following kernel boot options.  This keeps the interface names as eth0 and eth1 in the CentOS-7 VM instead of enp0s*. net.ifnames=0 biosdevname=0 Once the CentOS-7 VM boots, login into it and check the interfaces ("ip a" or "ifconfig").  eth0 will have an IP address like 10.0.2.15 and eth1 will not have any IP address.  Check the default gateway ("ip route").  10.0.2.2 will be the default gateway.  Make sure that you can ping a public DNS name like www.google.com and www.cisco.com. Below are the output snippets of "ip a" and "ip route" inside the CentOS-7 VM. $ ip a 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     link/ether 08:00:27:60:77:7e brd ff:ff:ff:ff:ff:ff $ ip route default via 10.0.2.2 dev eth0  proto static  metric 100 10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.15  metric 100 192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 From the Mac laptop, SSH into the CentOS-7 VM using the forwarded port 3022.  Use the root password to login. ssh -p 3022 root@127.0.0.1 Clone the DevStack Newton repository . git clone https://git.openstack.org/openstack-dev/devstack -b stable/newton Create stack user for DevStack.  Alternatively, you can use useradd and passwd to create a new stack user, and give sudo access to the stack user by typing visudo, adding " stack   ALL=(ALL)   ALL " under " root    ALL=(ALL)   ALL ", and saving the file. cd devstack ./tools/create-stack-user.sh su stack whoami echo $HOME cd pwd exit exit Copy the local.conf file below to the devstack directory.  It has the OpenStack core services (Horizon, Keystone, Nova, Neutron, Glance, RabbitMQ and MySQL) enabled.  It uses OpenvSwitch (OVS) as the virtual switch and VLAN for tenant networks.  It also enables the neutron ml2 ODL plugin to make neutron interact with OpenDaylight. [[local|localrc]] OFFLINE=True HORIZON_BRANCH=stable/newton KEYSTONE_BRANCH=stable/newton NOVA_BRANCH=stable/newton NEUTRON_BRANCH=stable/newton GLANCE_BRANCH=stable/newton ADMIN_PASSWORD=nomoresecret DATABASE_PASSWORD=stackdb RABBIT_PASSWORD=stackqueue SERVICE_PASSWORD=$ADMIN_PASSWORD LOGDIR=$DEST/logs LOGFILE=$DEST/logs/stack.sh.log LOGDAYS=2 ENABLED_SERVICES=rabbit,mysql,key ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch,n-novnc,n-cauth ENABLED_SERVICES+=,g-api,g-reg ENABLED_SERVICES+=,horizon # Neutron DISABLED_SERVICES=n-net ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-metering,neutron PUBLIC_INTERFACE=eth0 Q_PLUGIN=ml2 ENABLE_TENANT_VLANS=True # Enable neutron ODL plugin enable_plugin networking-odl http://git.openstack.org/openstack/networking-odl stable/newton ODL_MODE=allinone Q_ML2_PLUGIN_MECHANISM_DRIVERS=opendaylight,logger ODL_GATE_SERVICE_PROVIDER=vpnservice disable_service q-l3 ML2_L3_PLUGIN=odl-router ODL_PROVIDER_MAPPINGS=public:br-ex Now, exit and SSH back in as the stack user into the CentOS-7 VM. ssh -p 3022 stack@127.0.0.1 OpenDaylight requires Java 1.8.0 and Open vSwitch >= 2.5.0. Install Java 1.8.0. Java SE Development Kit 8 - Downloads wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u112-b15/jdk-8u112-linux-x64.rpm sudo yum localinstall jdk-8u112-linux-x64.rpm java -version rm -rf jdk-8u112-linux-x64.rpm Install OpenDaylight Boron. https://www.opendaylight.org/downloads http://docs.opendaylight.org/en/stable-boron/submodules/netvirt/docs/openstack-guide/openstack-with-netvirt.html#installing-openstack-and-opendaylight-using-devstack https://github.com/openstack/networking-odl wget https://nexus.opendaylight.org/content/repositories/opendaylight.release/org/opendaylight/integration/distribution-karaf/0.5.2-Boron-SR2/distribution-karaf-0.5.2-Boron-SR2.tar.gz tar xvfz distribution-karaf-0.5.2-Boron-SR2.tar.gz rm -rf distribution-karaf-0.5.2-Boron-SR2.tar.gz cd distribution-karaf-0.5.2-Boron-SR2/ export JAVA_HOME=/usr/java/jdk1.8.0_112 echo $JAVA_HOME Make sure that you are in the distribution-karaf-0.5.2-Boron-SR2 directory. Start the OpenDaylight server. sudo bash -c "export JAVA_HOME=/usr/java/jdk1.8.0_112 ; ./bin/start" Wait for 5 minutes so that the ODL boron server is up. Start OpenDaylight client and connect to the karaf shell. sudo bash -c "export JAVA_HOME=/usr/java/jdk1.8.0_112 ; ./bin/client" List the available ODL boron features in karaf shell. opendaylight-user@root>feature:list In the karaf shell, install the odl-netvirt-openstack bundle, dlux and their dependencies needed for OpenStack neutron. opendaylight-user@root>feature:install odl-netvirt-openstack odl-dlux-core odl-mdsal-apidocs List the installed ODL neutron northbound features. opendaylight-user@root>feature:list -i | grep -i neutron odl-neutron-service                            | 0.7.2-Boron-SR2  | x         | odl-neutron-0.7.2-Boron-SR2                | OpenDaylight :: Neutron :: API odl-neutron-northbound-api                     | 0.7.2-Boron-SR2  | x         | odl-neutron-0.7.2-Boron-SR2                | OpenDaylight :: Neutron :: Northbound odl-neutron-spi                                | 0.7.2-Boron-SR2  | x         | odl-neutron-0.7.2-Boron-SR2                | OpenDaylight :: Neutron :: API odl-neutron-transcriber                        | 0.7.2-Boron-SR2  | x         | odl-neutron-0.7.2-Boron-SR2                | OpenDaylight :: Neutron :: Implementation List the installed ODL OVS southbound features. opendaylight-user@root>feature:list -i | grep -i ovs odl-ovsdb-hwvtepsouthbound-api                 | 1.3.2-Boron-SR2  | x         | odl-ovsdb-hwvtepsouthbound-1.3.2-Boron-SR2 | OpenDaylight :: hwvtepsouthbound :: api odl-ovsdb-hwvtepsouthbound                     | 1.3.2-Boron-SR2  | x         | odl-ovsdb-hwvtepsouthbound-1.3.2-Boron-SR2 | OpenDaylight :: hwvtepsouthbound odl-ovsdb-southbound-api                       | 1.3.2-Boron-SR2  | x         | odl-ovsdb-southbound-1.3.2-Boron-SR2       | OpenDaylight :: southbound :: api odl-ovsdb-southbound-impl                      | 1.3.2-Boron-SR2  | x         | odl-ovsdb-southbound-1.3.2-Boron-SR2       | OpenDaylight :: southbound :: impl odl-ovsdb-library                              | 1.3.2-Boron-SR2  | x         | odl-ovsdb-library-1.3.2-Boron-SR2          | OpenDaylight :: library  List the installed ODL netvirt OpenStack features. opendaylight-user@root>feature:list -i | grep -i openstack odl-netvirt-openstack                          | 0.3.2-Boron-SR2  | x         | odl-netvirt-0.3.2-Boron-SR2                | OpenDaylight :: NetVirt :: OpenStack Hit CTRL+d to exit from karaf shell. Make sure that Open vSwitch's version is >= 2.5.0. $ ovs-vsctl --version ovs-vsctl (Open vSwitch) 2.5.0 Reboot the CentOS-7 VM and SSH in as the stack user. ssh -p 3022 stack@127.0.0.1 Now, you are ready to deploy! In the devstack directory, run stack.sh to deploy OpenStack Newton with OpenDaylight Boron and Open vSwitch. ./stack.sh Below is the output of stack.sh once it finishes. This is your host IP address: 10.0.2.15 This is your host IPv6 address: ::1 Horizon is now available at http://10.0.2.15/dashboard Keystone is serving at http://10.0.2.15/identity/ The default users are: admin and demo The password: nomoresecret Verify if OpenDaylight has been correctly deployed with OpenStack. Make sure that Open vSwitch is listening on TCP ports 6640 and 6653. $ sudo ovs-vsctl show 3ee26796-ce1a-44a8-83eb-ebb0269c94b8     Manager "tcp:10.0.2.15: 6640 "         is_connected: true     Bridge br-int         Controller "tcp:10.0.2.15: 6653 "             is_connected: true         fail_mode: secure         Port br-int             Interface br-int                 type: internal         Port "tap6caac5d1-9e"             Interface "tap6caac5d1-9e"                 type: internal     ovs_version: "2.5.0" $ sudo ovs-vsctl show | grep '6640\|6653'     Manager "tcp:10.0.2.15: 6640 "         Controller "tcp:10.0.2.15: 6653 " Make sure that OpenDaylight, Open vSwitch and the OVSDB server are listening on TCP ports 6640 and 6653. Note the PIDs of OpenDaylight (java), Open vSwitch and the OVSDB server. $ sudo netstat -pan | grep ': 6640 \|: 6653 ' tcp        0      0 10.0.2.15:38298         10.0.2.15: 6640           ESTABLISHED 18455/ovsdb-server  tcp        0      0 10.0.2.15:46744         10.0.2.15: 6653           ESTABLISHED 18465/ovs-vswitchd  tcp6       0      0 ::: 6640                  :::*                    LISTEN      2125/java           tcp6       0      0 ::: 6653                  :::*                    LISTEN      2125/java           tcp6       0      0 10.0.2.15: 6653           10.0.2.15:46744         ESTABLISHED 2125/java           tcp6       0      0 10.0.2.15: 6640           10.0.2.15:38298         ESTABLISHED 2125/java Make sure that these PIDs match what is seen in the output of "ps". $ ps aux | grep ' 18455 \| 18465 \| 2125 ' stack     2125 12.7 19.3 4342780 750692 pts/4  Sl   01:02   3:01 /usr/java/jdk1.8.0_112/jre/bin/ java -Djava.security.properties=/opt/stack/ opendaylight /distribution-karaf-0.6.0-SNAPSHOT/etc/odl.java.security -server -Xms128M -Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote -Djava.security.egd=file:/dev/./urandom -Djava.endorsed.dirs=/usr/java/jdk1.8.0_112/jre/lib/endorsed:/usr/java/jdk1.8.0_112/lib/endorsed:/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/lib/endorsed -Djava.ext.dirs=/usr/java/jdk1.8.0_112/jre/lib/ext:/usr/java/jdk1.8.0_112/lib/ext:/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/lib/ext -Dkaraf.instances=/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/instances -Dkaraf.home=/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT -Dkaraf.base=/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT -Dkaraf.data=/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/data -Dkaraf.etc=/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/etc -Djava.io.tmpdir=/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/data/tmp -Djava.util.logging.config.file=/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath /opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/lib/karaf-jaas-boot.jar:/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/lib/karaf-org.osgi.core.jar:/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/lib/karaf.branding-1.8.0-SNAPSHOT.jar:/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/lib/karaf.jar org.apache.karaf.main.Main root     18455   0.0  0.0  43724  1696 ?        S<   00:56   0:00 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor root     18465   0.0  0.9 268944 35496 ?        S<Ll 00:56   0:01 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor Curl the OpenStack Horizon dashboard and make sure that there are no errors in the output. $ curl localhost/dashboard $ Curl the OpenDaylight GUI.  Below is the expected output. $ curl localhost:8181/index.html <!DOCTYPE html> <html lang="en">   <head>     <meta charset="utf-8" />     <title>OpenDaylight Dlux</title>     <meta name="description" content="overview &amp; stats" />     <meta name="viewport" content="width=device-width, initial-scale=1.0" />     <script type="text/javascript"> var module = ['angular','ocLazyLoad','angular-ui-router','angular-translate', 'angular-sanitize', 'angular-translate-loader-static-files', 'angular-translate-loader-partial', 'angular-css-injector']; var deps = ['common/config/env.module','app/core/core.module','common/login/login.module','common/authentification/auth.module','common/navigation/navigation.module','common/topbar/topbar.module','common/general/common.general.module','app/topology/topology.module','common/layout/layout.module']; var e = ['oc.lazyLoad', 'ui.router', 'pascalprecht.translate', 'ngSanitize', 'angular.css.injector', 'app','app.core','app.common.login','app.common.auth','app.common.nav','app.common.topbar','app.common.general','app.topology','app.common.layout'];         // global variables     </script>     <!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->   <!--[if lt IE 9]>     <script src="assets/js/html5shiv.js"></script>     <script src="assets/js/respond.min.js"></script>     <![endif]-->     <!-- compiled CSS -->     <link rel="stylesheet" type="text/css" href="vendor/ng-grid/ng-grid.min.css" />     <link rel="stylesheet" type="text/css" href="vendor/select2-bootstrap-css/select2-bootstrap.css" />     <link rel="stylesheet" type="text/css" href="vendor/footable/css/footable.core.min.css" />     <link rel="stylesheet" type="text/css" href="vendor/footable/css/footable.standalone.min.css" />     <link rel="stylesheet" type="text/css" href="vendor/vis/dist/vis.min.css" />     <link rel="stylesheet" type="text/css" href="vendor/ng-slider/dist/css/ng-slider.min.css" />     <link rel="stylesheet" type="text/css" href="vendor/angular-material/angular-material.css" />     <link rel="stylesheet" type="text/css" href="vendor/material-design-icons/iconfont/material-icons.css" />     <link rel="stylesheet" type="text/css" href="assets/opendaylight-dlux-0.2.0.css" />     <link rel="stylesheet" href="assets/css/sb-admin.css" />     <script type="text/javascript" data-main="src/main.js" src="vendor/requirejs/require.js"></script>     <link rel="stylesheet" href="assets/css/font-awesome.min.css" />     <!-- the font-awesome is different from the 'official' one -->     <!-- application CSS -->   </head>   <body class="skin-3">     <div ui-view="mainContent" id="main-content-container"></div>   </body> </html> Check the OVS config. $ sudo ovs-vsctl get Open_vSwitch . other_config {local_ip="10.0.2.15", provider_mappings="public:br-ex"} Make sure that the neutron configuration file /etc/neutron/neutron.conf has the following ODL entries. [DEFAULT] service_plugins = odl-router,neutron.services.metering.metering_plugin.MeteringPlugin core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin Make sure that the neutron ml2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini has the following ODL entries. [ml2] mechanism_drivers = opendaylight,logger [ml2_odl] port_binding_controller = network-topology password = admin username = admin url = http://10.0.2.15:8087/controller/nb/v2/neutron Note the neutron ml2 ODL url: $ grep 8087 /etc/neutron/plugins/ml2/ml2_conf.ini url = http://10.0.2.15:8087/controller/nb/v2/neutron Make sure that neutron-server is using the right configuration files that have the ODL entries. $ ps aux | grep ml2 stack     7523  0.1  2.2 290596 88872 pts/8    S+   01:05   0:04 /usr/bin/python /usr/bin/ neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini Source userrc_early in the devstack directory and check neutron CLIs. Check if all the neutron agents are running fine. $ source userrc_early $ neutron agent-list +-----------------+----------------+--------------+-------------------+-------+----------------+--------------------+ | id              | agent_type     | host         | availability_zone | alive | admin_state_up | binary             | +-----------------+----------------+--------------+-------------------+-------+----------------+--------------------+ | 7a84f626-a656   | DHCP agent     | devstack-odl | nova              | :-)   | True           | neutron-dhcp-agent | | -426e-acae-     |                |              |                   |       |                |                    | | 5395cb56822a    |                |              |                   |       |                |                    | | 9a626977-97ed-  | Metering agent | devstack-odl |                   | :-)   | True           | neutron-metering-  | | 486b-b9c4-2fb60 |                |              |                   |       |                | agent              | | 9ebd69f         |                |              |                   |       |                |                    | | de275c04-18bc-  | Metadata agent | devstack-odl |                   | :-)   | True           | neutron-metadata-  | | 43f1-b01b-      |                |              |                   |       |                | agent              | | 4ee9affc3654    |                |              |                   |       |                |                    | +-----------------+----------------+--------------+-------------------+-------+----------------+--------------------+ Create a neutron network, subnet and a router. neutron net-create test-net neutron subnet-create --name test-subnet test-net 11.11.11.0/24 neutron router-create test-router $ neutron net-list | grep test-net | 66e9a2a1-de76-4c92-b84a-e9aafdf75ad7 | test-net | 65a0a59d-90b7-476f-b117-64d7c7ab4901 11.11.11.0/24      | $ neutron subnet-list | grep test-subnet | 65a0a59d-90b7-476f-b117-64d7c7ab4901 | test-subnet         | 11.11.11.0/24      | {"start": "11.11.11.2", "end": "11.11.11.254"}                              | $ neutron router-list | grep test-router | 618e0100-50a7-4251-94b3-029811789c1d | test-router | null Curl neutron's ml2 ODL url and check if the neutron networks, subnets, routers and ports can be successfully retrieved. $ curl -v -u admin:admin http://10.0.2.15:8087/controller/nb/v2/neutron/networks | grep '\"name\"'       "name" : "public",       "name" : "test-net",       "name" : "private", $ curl -v -u admin:admin http://10.0.2.15:8087/controller/nb/v2/neutron/subnets | grep '\"name\"'       "name" : "test-subnet",       "name" : "public-subnet",       "name" : "ipv6-public-subnet",       "name" : "ipv6-private-subnet",       "name" : "private-subnet", $ curl -v -u admin:admin http://10.0.2.15:8087/controller/nb/v2/neutron/routers | grep '\"name\"'       "name" : "router1",       "name" : "test-router", $ curl -v -u admin:admin http://10.0.2.15:8087/controller/nb/v2/neutron/ports Curl neutron's ml2 ODL url and check if the neutron network topology can be successfully retrieved. $ curl -v -u admin:admin http://10.0.2.15:8087/restconf/operational/network-topology:network-topology Check the OpenFlow 1.3 table in the OVS bridge br-int: $ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int OFPST_FLOW reply (OF1.3) (xid=0x2): cookie=0x0, duration=3340.050s, table=0, n_packets=0, n_bytes=0, dl_type=0x88cc actions=CONTROLLER:65535 cookie=0x0, duration=3265.069s, table=0, n_packets=7, n_bytes=558, in_port=1,dl_src=fa:16:3e:46:31:aa actions=set_field:0x17->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20 >NXM_OF_ETH_DST[],set_field:fa:16:3e:46:31:aa->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e4631aa->NXM_NX_ARP_SHA[],load:0xa000002->NXM_OF_ARP_SPA[],IN_PORT cookie=0x0, duration=899.385s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x51,arp_tpa=11.11.11.2,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:60:81:f2->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e6081f2->NXM_NX_ARP_SHA[],load:0xb0b0b02->NXM_OF_ARP_SPA[],IN_PORT cookie=0x0, duration=3340.083s, table=20, n_packets=16, n_bytes=1296, priority=0 actions=goto_table:30 cookie=0x0, duration=3340.050s, table=30, n_packets=16, n_bytes=1296, priority=0 actions=goto_table:31 Connect to the ODL karaf shell, and check if the neutron network, subnet and router that were created are captured in the ODL logs. cd ~/distribution-karaf-0.5.2-Boron-SR2/ sudo bash -c "export JAVA_HOME=/usr/java/jdk1.8.0_112 ; ./bin/client" opendaylight-user@root>log:display | grep test-net Network{getName= test-net , getStatus=ACTIVE, getTenantId=Uuid [_value=fdd867a9-e0e4-46c1-8985-70b1cc590d7d], getUuid=Uuid [_value=66e9a2a1-de76-4c92-b84a-e9aafdf75ad7], isAdminStateUp=true, isShared=false, augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.l3.ext.rev150712.NetworkL3Extension=NetworkL3Extension{isExternal=false}, interface org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.provider.ext.rev150712.NetworkProviderExtension=NetworkProviderExtension{getNetworkType=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.networks.rev150712.NetworkTypeVxlan, getSegmentationId=81}}} opendaylight-user@root>log:display | grep test-subnet Subnet{getAllocationPools=[AllocationPools{getEnd=IpAddress [_ipv4Address=Ipv4Address [_value=11.11.11.254]], getStart=IpAddress [_ipv4Address=Ipv4Address [_value=11.11.11.2]], augmentations={}}], getCidr=IpPrefix [_ipv4Prefix=Ipv4Prefix [_value=11.11.11.0/24]], getDnsNameservers=[], getGatewayIp=IpAddress [_ipv4Address=Ipv4Address [_value=11.11.11.1]], getHostRoutes=[], getIpVersion=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.IpVersionV4, getName= test-subnet , getNetworkId=Uuid [_value=66e9a2a1-de76-4c92-b84a-e9aafdf75ad7], getTenantId=Uuid [_value=fdd867a9-e0e4-46c1-8985-70b1cc590d7d], getUuid=Uuid [_value=65a0a59d-90b7-476f-b117-64d7c7ab4901], isEnableDhcp=true, augmentations={}} opendaylight-user@root>log:display | grep test-router Router{getName= test-router , getRoutes=[], getStatus=ACTIVE, getTenantId=Uuid [_value=fdd867a9-e0e4-46c1-8985-70b1cc590d7d], getUuid=Uuid [_value=618e0100-50a7-4251-94b3-029811789c1d], isAdminStateUp=true, isDistributed=false, augmentations={}} Check ODL OpenFlow statistics and session statistics in the karaf shell: opendaylight-user@root>ofp:showstats FROM_SWITCH: no activity detected FROM_SWITCH_TRANSLATE_IN_SUCCESS: no activity detected FROM_SWITCH_TRANSLATE_OUT_SUCCESS: MSG[MultipartType] -> +1208 | 1208 FROM_SWITCH_TRANSLATE_SRC_FAILURE: no activity detected FROM_SWITCH_PACKET_IN_LIMIT_REACHED_AND_DROPPED: no activity detected FROM_SWITCH_NOTIFICATION_REJECTED: no activity detected FROM_SWITCH_PUBLISHED_SUCCESS: MSG[PortStatusMessage] -> +6 | 6 FROM_SWITCH_PUBLISHED_FAILURE: MSG[MultipartReplyMessage] -> +6044 | 6044 TO_SWITCH_ENTERED: MSG[SetConfigInput] -> +1 | 1 TO_SWITCH_ENTERED: MSG[FlowModInputBuilder] -> +117 | 117 TO_SWITCH_ENTERED: MSG[RoleRequestInputBuilder] -> +4 | 4 TO_SWITCH_ENTERED: MSG[MultipartType] -> +7248 | 7248 TO_SWITCH_DISREGARDED: no activity detected TO_SWITCH_RESERVATION_REJECTED: no activity detected TO_SWITCH_READY_FOR_SUBMIT: MSG[] -> +4 | 4 TO_SWITCH_READY_FOR_SUBMIT: MSG[] -> +7248 | 7248 TO_SWITCH_READY_FOR_SUBMIT: MSG[] -> +118 | 118 TO_SWITCH_SUBMIT_SUCCESS: MSG[SetConfigInput] -> +1 | 1 TO_SWITCH_SUBMIT_SUCCESS: MSG[FlowModInputBuilder] -> +117 | 117 TO_SWITCH_SUBMIT_SUCCESS: MSG[RoleRequestInputBuilder] -> +4 | 4 TO_SWITCH_SUBMIT_SUCCESS_NO_RESPONSE: no activity detected TO_SWITCH_SUBMIT_FAILURE: no activity detected TO_SWITCH_SUBMIT_ERROR: no activity detected REQUEST_STACK_FREED: MSG[RpcContextImpl] -> +118 | 118 OFJ_BACKPRESSURE_ON: no activity detected OFJ_BACKPRESSURE_OFF: no activity detected opendaylight-user@root>ofp:show-session-stats SESSION : Uri [_value=openflow:185752284545496] CONNECTION_CREATED : 1 Check the ODL web end points in the karaf shell. opendaylight-user@root>web:list ID  | State       | Web-State   | Level | Web-ContextPath           | Name                                             ------------------------------------------------------------------------------------------------------------------------------ 269 | Active      | Deployed    | 80    | /moon                     | org.opendaylight.aaa.aaa-shiro (0.5.0.SNAPSHOT)  273 | Active      | Deployed    | 80    | /oauth2                   | aaa-authn-sts (0.5.0.SNAPSHOT)                   279 | Active      | Deployed    | 80    | /auth                     | aaa-idmlight (0.5.0.SNAPSHOT)                    289 | Active      | Deployed    | 80    | /controller/nb/v2/neutron | org.opendaylight.neutron.northbound-api (0.8.0.SNAPSHOT) 296 | Active      | Deployed    | 80    | /restconf                 | MD SAL Restconf Connector (1.5.0.SNAPSHOT)       299 | Active      | Deployed    | 80    | /apidoc                   | MD SAL Rest Api Doc Generator (1.5.0.SNAPSHOT) Make sure that the ODL configurations have the right entries for OpenStack neutron and Open vSwitch. opendaylight-user@root>config:list  | grep -i ovs    featuresBoot = config,standard,region,package,kar,ssh,management, odl-neutron-service ,odl-restconf-all,odl-aaa-authn,odl-dlux-core,odl-mdsal-apidocs, odl-ovsdb-openstack,odl-neutron-logger Pid:            org.opendaylight.ovsdb.library BundleLocation: mvn:org.opendaylight. ovsdb /library/1.4.0-SNAPSHOT    felix.fileinstall.filename = file:/opt/stack/opendaylight/distribution-karaf-0.6.0-SNAPSHOT/etc/ org.opendaylight.ovsdb.library.cfg    service.pid = org.opendaylight.ovsdb.library Hit CTRL+d to exit from karaf shell. Since we have setup port forwarding on VirtualBox, the following links can be accessed on the Mac laptop to retrieve the neutron networks, subnets, ports and routers from neutron's ml2 ODL url! http://localhost:8187/controller/nb/v2/neutron/networks http://localhost:8187/controller/nb/v2/neutron/subnets http://localhost:8187/controller/nb/v2/neutron/ports http://localhost:8187/controller/nb/v2/neutron/routers On the laptop, access the network topology at the ODL web endpoint using RESTCONF. http://localhost:8282/restconf/operational/network-topology:network-topology The OpenStack Horizon dashboard can be accessed on the Mac laptop at http://localhost:8080/.  Use the username admin and password nomoresecret to login into Horizon. Congratulations!  You've successfully deployed OpenStack Newton with OpenDaylight Boron and Open vSwitch! Please refer my blog How to stack DevStack Newton on CentOS-7 in VirtualBox on Mac for steps to boot a nova instance in the OpenStack Horizon dashboard. Boot a nova VM (test-vm) using cirros image, m1.tiny flavor, and attach it to the private network.  Also, create a floating IP in the public network and associate it to the nova VM.  Add security group rules to the "default" security group in order to SSH into and ping nova VMs. $ cd ~/devstack $ nova list +--------------------------------------+---------+--------+------------+-------------+--------------------------------------------------------------------+ | ID                                   | Name    | Status | Task State | Power State | Networks                                                           | +--------------------------------------+---------+--------+------------+-------------+--------------------------------------------------------------------+ | 3204114d-d3b2-4493-8115-abd0b463152a | test-vm | ACTIVE | -          | Running     | private=10.0.0.5, fd38:25d7:fb99:0:f816:3eff:fe35:90f0, 172.24.4.5 | +--------------------------------------+---------+--------+------------+-------------+--------------------------------------------------------------------+ $ openstack security group rule create --protocol tcp --dst-port 22 default  $ openstack security group rule create --protocol icmp --dst-port -1 default Find the DHCP namespace and SSH into the cirros VM from inside the DHCP namespace using the following credentials! username:  cirros password:  cubswin:) $ neutron net-list | grep private | f58ba1ee-9a21-4dea-ab96-10d06b2c46b5 | private | dda9ffee-b36c-400b-a93c-9ba3b36280ae fd38:25d7:fb99::/64 | $ ip netns | grep f58ba1ee-9a21-4dea-ab96-10d06b2c46b5 qdhcp-f58ba1ee-9a21-4dea-ab96-10d06b2c46b5 $ sudo ip netns exec qdhcp-f58ba1ee-9a21-4dea-ab96-10d06b2c46b5 ssh cirros@10.0.0.5 cirros@10.0.0.5's password: $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000     link/ether fa:16:3e:35:90:f0 brd ff:ff:ff:ff:ff:ff     inet 10.0.0.5/24 brd 10.0.0.255 scope global eth0     inet6 fe80::f816:3eff:fe35:90f0/64 scope link        valid_lft forever preferred_lft forever Here is the network topology you can see in http://localhost:8080/dashboard/project/network_topology/. Hope this blog is helpful! ... View more

I n an OpenStack deployment, when packets from a nova instance need to reach the external world, they first go through the virtual switch (OpenvSwitch or Linux Bridge) for layer-2 switching, then through the physical NIC of the compute host, and then leave the compute host.  Hence, t he path of the packets is: W hen, s ingle Root I/O Virtualisation (SR-IOV) and PCI pass-through are deployed in OpenStack, the packets from the nova instance do not use the virtual switch (OpenvSwitch or Linux Bridge).  They directly go through the physical NIC, and then leave the compute host, thereby bypassing the virtual switch.  In this case, the physical NIC does the layer-2 switching of the packets.  Hence, t he path of the packets is: A virtual switch (OpenvSwitch or Linux Bridge) may also be used with SR-IOV if an instance needs a virtual switch for layer-2 switching.  A virtual switch is just software and obviously does not scale well like a physical NIC when the packet rate is high. In order to deploy SR-IOV on Cisco's UCS servers, we need the UCSM ml2 neutron plugin developed by Cisco.  Here are few links about the UCSM ml2 neutron plugin: ML2 Mechanism Driver for Cisco UCS Manager — Neutron Specs ed6fc9a documentation OpenStack/UCS Mechanism Driver for ML2 Plugin Liberty - DocWiki Cisco UCS Manager ML2 Plugin for Openstack Networking : Blueprints : neutron SR-IOV needs specific NICs and currently works only on the following Cisco VICs: Cisco VIC 1380 on UCS B-series M4 blades Cisco VIC 1340 on UCS B-series M4 blades Cisco VIC 1240 on UCS B-series M3 blades Here is the Cisco VIC 1380 in the UCSM GUI: Below are the neutron and nova configurations needed to enable SR-IOV in OpenStack: In /etc/neutron/plugins/ml2/ml2_conf_sriov.ini: [sriov_nic] physical_device_mappings = physnet2:eth3 exclude_devices = In /etc/nova/nova.conf: [default] pci_passthrough_whitelist = { "devname": "eth3", "physical_network": "physnet2"} Pass /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as --config-file and start neutron-server. $ neutron-server \   --config-file /etc/neutron/neutron.conf \   --config-file /etc/neutron/plugin.ini \   --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini Pass /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as --config-file and start neutron-sriov-nic-agent. $ neutron-sriov-nic-agent \ --config-file /etc/neutron/neutron.conf -\ --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini OpenStack Docs: Using SR-IOV functionality OpenStack Docs: Attaching physical PCI devices to guests Not all OpenStack images can be booted with SR-IOV port.  The image must have the following ENIC driver that interacts with the VF (Virtual Function) for SR-IOV to work. $ modinfo enic filename:       /lib/modules/3.10.0-327.10.1.el7.x86_64/kernel/drivers/net/ethernet/cisco/enic/enic.ko version:        2.1.1.83 license:        GPL author:         Scott Feldman < scofeldm@cisco.com > description:    Cisco VIC Ethernet NIC Driver rhelversion:    7.2 srcversion:     E3ADA231AA76168CA78577A alias:          pci:v00001137d00000071sv*sd*bc*sc*i* alias:          pci:v00001137d00000044sv*sd*bc*sc*i* alias:          pci:v00001137d00000043sv*sd*bc*sc*i* depends:       intree:         Y vermagic:       3.10.0-327.10.1.el7.x86_64 SMP mod_unload modversions signer:         Red Hat Enterprise Linux kernel signing key sig_key:        E3:9A:6C:00:A1:DE:4D:FA:F5:90:62:8C:AB:EC:BC:EB:07:66:32:8A sig_hashalgo:   sha256 Create an SR-IOV neutron port by using the argument "--binding:vnic-type direct".  Note the port ID in the output below.  It will be needed to boot the nova instance and attach it to the SR-IOV port. neutron port-create provider_net --binding:vnic-type direct --name=sr-iov-port neutron port-list Boot a nova VM and attach the SR-IOV port to it. nova boot --flavor m1.large --image RHEL-guest-image --nic port-id=<port ID of SR-IOV port> --key-name mc2-installer-key sr-iov-vm nova list Once the instance is up, ping it and SSH into it (security groups do not work for SR-IOV ports.  Please refer the " Known limitations of SR-IOV ports" section at the end of this blog).  Make sure that you can ping the gateway of the instance.  The packets from the instance will not go through the virtual switch (OpenvSwitch or Linux Bridge).  They directly go through the physical NIC of the compute host, and then leave the compute host, thereby bypassing the virtual switch. Below are the steps to verify SR-IOV (VF) in the UCSM GUI. Below are the steps to verify that the SR-IOV port does not use OpenvSwitch and bypasses it when sending/receiving packets: 1.  Find the compute node on which sr-iov-vm is running. # nova show sr-iov-vm | grep host | OS-EXT-SRV-ATTR:host                 | mc2-compute-12             | | OS-EXT-SRV-ATTR:hypervisor_hostname  | mc2-compute-12             | 2.  Find the port ID of the SR-IOV neutron port.  d8379d3e-8877-4fec-8b50-ab939dbca797 is the port ID of the SR-IOV port in this case. # neutron port-list | grep sr-iov-port 3.  SSH into the compute host (mc2-compute-12 in this case) and make sure that:         1.  There is no interface with d8379d3e in its name on the compute host, and,         2.  There is no interface with d8379d3e in the output of "ovs-vsctl show". # ssh root@mc2-compute-12 [root@mc2-compute-12 ~]# ip a | grep d8379d3e [root@mc2-compute-12 ~]# [root@mc2-compute-12 ~]# ovs-vsctl show | grep d8379d3e [root@mc2-compute-12 ~]# The above outputs must be empty.  This shows that the SR-IOV port is not using OpenvSwitch for layer-2 switching and the physical NIC connected to the SR-IOV port does the layer-2 switching for packets from the instance sr-iov-vm . On the other hand, in case of an instance booted with a regular OVS (non-SR-IOV virtio) port, the above outputs will not be empty.  Below is an example of regular OVS (non-SR-IOV virtio) port (ID f9936cb7-885d-451d-9fc2-a86a670be732 ) attached to a instance.  This OVS (non-SR-IOV virtio) port is part of the br-int (OVS integration bridge) in the output of "ovs-vsctl show". # nova list +--------------------------------------+-----------+--------+------------+-------------+----------------------------+ | ID                                   | Name      | Status | Task State | Power State | Networks                   | +--------------------------------------+-----------+--------+------------+-------------+----------------------------+ | 206a42e9-fa07-4128-b276-4764beaf9a9b | test-vm   | ACTIVE | -          | Running     | mc2_provider=10.23.228.102 | +--------------------------------------+-----------+--------+------------+-------------+----------------------------+ # neutron port-list | grep 10.23.228.102 | f9936cb7-885d-451d-9fc2-a86a670be732 |             | fa:16:3e:df:30:32 | {"subnet_id": "9258aa43-59b2-4f57-8207-b7d30e7963e0", "ip_address": "10.23.228.102"} | SSH into the compute host running the instance. # ssh root@mc2-compute-4 [root@mc2-compute-4 ~]# ip a | grep f9936cb7 14: qbrf9936cb7-88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP 15: qvof9936cb7-88@qvbf9936cb7-88: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master ovs-system state UP qlen 1000 16: qvbf9936cb7-88@qvof9936cb7-88: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master qbrf9936cb7-88 state UP qlen 1000 17: tapf9936cb7-88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master qbrf9936cb7-88 state UNKNOWN qlen 500 [root@mc2-compute-4 ~]# ovs-vsctl show     Bridge br-int         fail_mode: secure         Port br-int             Interface br-int                 type: internal         Port "qvof9936cb7-88"             tag: 1             Interface "qvof9936cb7-88" Advantages of SR-IOV ports: Faster packet processing as the virtual switch (OpenvSwitch or Linux Bridge) is not used. Can be used for multicast, media applications, video streaming apps, and real time applications. Known limitations of SR-IOV ports: Security groups are not supported when using SR-IOV. When using Quality of Service (QoS), max_burst_kbps (burst over max_kbps) is not supported. SR-IOV is not integrated into the OpenStack Dashboard (horizon). Users must use the CLI or API to configure SR-IOV interfaces. Live migration is not supported for instances with SR-IOV ports. OpenStack Docs: SR-IOV Hope this blog is helpful! ... View more

O penStack cinder provides block storage for nova instances.  Cinder volumes are great to store files in a nova instance permanently.  Without cinder volume, the data/files in a nova instance are ephemeral (temporary) and will be lost once the instance in deleted.  Sometimes, we may need to save the data/files in an instance permanently even after the instance in deleted.  Cinder volumes can also be used to share files among multiple instances in OpenStack. This blog has the steps to: Create a cinder volume (using ceph backend) Attach this volume to a nova instance Check the attached volume in the instance Create an XFS filesystem/partition on this attached volume in the instance Mount the partitioned volume in the instance Write data into this mounted volume in the instance Unmount, detach the volume and make sure that data in the volume still exists even after the instance in deleted 1.  Create a cinder volume (using ceph backend): Here are few links about integrating OpenStack cinder with ceph backend. Block Devices and OpenStack — Ceph Documentation http://superuser.openstack.org/articles/ceph-as-storage-for-openstack/ In Horizon, Go to Project --> Compute --> Volumes --> Volumes --> Create Volume and enter all the info and click "Create Volume". Make sure that the created volume is in "Available" state. Create a nova instance in Horizon.  Hope you know the steps for this   If not, please contact me at vhosakot@cisco.com . Make sure that the instance is in "Active" state, can be pinged, and SSH'ed into with the right security groups (allow ICMP and SSH). 2.  Attach this volume to a nova instance: Go to Project --> Compute --> Volumes --> Volumes --> <Name of the volume> --> Edit Volume --> Manage Attachments. Select the name of the instance in the "Attach to Instance" drop-down and click "Attach Volume". Make sure that the volume is in "In-use" state and note the path to which the volume is attached in the "Attached To" column.  Also, check the "Size" column and make sure that it is right. 3. Check the attached volume in the instance: SSH into the instance and run the following commands. Check the volume at /dev/vdb and make sure that it is fine. $ lsblk NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda    253:0    0  40G  0 disk |---vda1 253:1    0  40G  0 part / vdb    253:16   0  20G  0 disk $ sudo parted /dev/vdb 'print' Error: /dev/vdb: unrecognised disk label Model: Virtio Block Device (virtblk)                                      Disk /dev/vdb: 21.5GB Sector size (logical/physical): 512B/512B Partition Table: unknown Disk Flags: $ df -h /dev/vdb Filesystem      Size  Used Avail Use% Mounted on devtmpfs        1.9G     0  1.9G   0% /dev $ ls -l /dev/vdb brw-rw----. 1 root disk 253, 16 Jun 29 05:25 /dev/vdb $ file /dev/vdb /dev/vdb: block special This attached volume is new and a raw block device without any filesystem.  Hence, this attached volume must be partitioned and mounted before the instance can write data into the volume.  The filesystem/partition should be created only the first time a new volume is attached to an instance, and NOT every time the volume is re-attached to an instance.  Creating the filesystem will erase all the data in the volume.  So, create the filesystem/partition only if you want to erase all the data in the attached volume. 4.  Create an XFS filesystem/partition on this attached volume in the instance: More information is at: 16.3. Accessing a Volume from a Running Instance Deployment Guide — swift 2.12.1.dev28 documentation $ sudo mkfs.xfs -f /dev/vdb meta-data=/dev/vdb               isize=256    agcount=4, agsize=1310720 blks          =                       sectsz=512   attr=2, projid32bit=1          =                       crc=0        finobt=0 data     =                       bsize=4096   blocks=5242880, imaxpct=25          =                       sunit=0      swidth=0 blks naming   =version 2              bsize=4096   ascii-ci=0 ftype=0 log      =internal log           bsize=4096   blocks=2560, version=2          =                       sectsz=512   sunit=0 blks, lazy-count=1 realtime =none                   extsz=4096   blocks=0, rtextents=0 Replace "xfs" above with "ext4" if an ext4 file system/partition is needed. Check the partitioned XFS filesystem: $ sudo parted /dev/vdb 'print' Model: Virtio Block Device (virtblk) Disk /dev/vdb: 21.5GB Sector size (logical/physical): 512B/512B Partition Table: loop Disk Flags: Number  Start  End     Size    File system  Flags 1      0.00B  21.5GB  21.5GB  xfs 5.  Mount the partitioned volume in the instance: Mount the volume at /dev/vdb to /mnt/volume. $ sudo mkdir -p /mnt/volume $ sudo mount /dev/vdb /mnt/volume Make sure that the volume is correctly mounted and the mounted partition's size (20 GB in this case) and file system (XFS) are correct. $ lsblk NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda    253:0    0  40G  0 disk |---vda1 253:1    0  40G  0 part / vdb    253:16   0  20G  0 disk /mnt/volume $ sudo parted /dev/vdb 'print' Model: Virtio Block Device (virtblk) Disk /dev/vdb: 21.5GB Sector size (logical/physical): 512B/512B Partition Table: loop Disk Flags: Number  Start  End     Size    File system  Flags 1      0.00B  21.5GB  21.5GB  xfs $ df -h /dev/vdb Filesystem      Size  Used Avail Use% Mounted on /dev/vdb         20G   33M   20G   1% /mnt/volume $ xfs_info /dev/vdb meta-data=/dev/vdb               isize=256    agcount=4, agsize=1310720 blks          =                       sectsz=512   attr=2, projid32bit=1          =                       crc=0        finobt=0 data     =                       bsize=4096   blocks=5242880, imaxpct=25          =                       sunit=0      swidth=0 blks naming   =version 2              bsize=4096   ascii-ci=0 ftype=0 log      =internal               bsize=4096   blocks=2560, version=2          =                       sectsz=512   sunit=0 blks, lazy-count=1 realtime =none                   extsz=4096   blocks=0, rtextents=0 Now that the attached volume is partitioned and mounted, the instance can write data into the volume! 6.  Write data into this mounted volume in the instance: Write data (two files test_file_on_volume and large_file.txt) into the mounted volume at /mnt/volume in the instance! $ cd /mnt/volume $ df -h . Filesystem      Size  Used Avail Use% Mounted on /dev/vdb         20G   33M   20G   1% /mnt/volume $ ls $ echo 'test test' | sudo tee test_file_on_volume test test $ ls -lrt total 4 -rw-r--r--. 1 root root 10 Jun 29 05:42 test_file_on_volume $ cat test_file_on_volume test test $ sudo dd if=/dev/urandom of=large_file.txt bs=102400 count=100 100+0 records in 100+0 records out 10240000 bytes (10 MB) copied, 0.773203 s, 13.2 MB/s $ ls -lrt total 10004 -rw-r--r--. 1 root root       10 Jun 29 05:42 test_file_on_volume -rw-r--r--. 1 root root 10240000 Jun 29 05:42 large_file.txt These two files test_file_on_volume and large_file.txt will be transferred from the compute node (on which the instance is scheduled) into the ceph nodes and also replicated on all the ceph nodes over the network.  These files should exist even after deleting the instance . NOTE: Data written/read in the attached volume travels on the network.  Hence, the network (NICs) must be fast in order to scale.  I found that configuring jumbo frames (with MTU 9000 bytes) in OpenStack improves performance when a lot of data is quickly written by the instance into ceph.  Please refer Jumbo Mumbo in OpenStack using Cisco's UCS servers and Nexus 9000 for steps to configure jumbo frames (with MTU 9000 bytes) in OpenStack. 7.  Unmount, detach the volume and make sure that data in the volume still exists even after the instance in deleted: Unmount the volume in the instance: $ sudo umount /mnt/volume $ lsblk NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda    253:0    0  40G  0 disk |---vda1 253:1    0  40G  0 part / vdb    253:16   0  20G  0 disk $ df /dev/vdb Filesystem     1K-blocks  Used Available Use% Mounted on devtmpfs         1922568     0   1922568   0% /dev Detach the volume from the instance: Go to Project --> Compute --> Volumes --> Volumes --> <Name of the volume> --> Edit Volume --> Manage Attachments. Select the instance and click "Detach volume" --> "Detach Volume". Make sure that the "Attached to" column is empty. How to make sure that the data/files in the volume still exists even after the instance in deleted: Attach this volume to a nova instance (step 2 above) Check the attached volume in the instance (step 3 above) DO NOT create the XFS filesystem/partition again (step 4 above).  The filesystem/partition should be created only the first time a new volume is attached to an instance, and NOT when the volume is re-attached to an instance.  Creating the filesystem will erase all the data in the volume.  So, create the filesystem/partition only if you want to erase all the data in the attached volume. Mount the partitioned volume in the instance (step 5 above) Read/write data in this mounted volume in the instance (step 6 above) Unmount and detach the volume from the instance (step 7 above) Re-attach and mount the volume (steps 2 and 3 above) to the same/different instance and the files test_file_on_volume and large_file.txt will still be seen at /mnt/volume in the instance: $ ls /mnt/volume test_file_on_volume  large_file.txt NOTE:   Always unmount and detach the attached volume before deleting/terminating the instance. To attach a cinder volume to an instance using cinder and nova CLIs, please refer OpenStack Docs: Launch an instance from a volume. Hope this blog is helpful! ... View more

T his is a blog about how to: Snapshot an OpenStack nova instance as a glance image Boot an instance from the snapshot (glance image) Download the snapshot (glance image) as a file onto disk Validate the downloaded snapshot file (image) using qemu-img Upload this file (image) on disk into OpenStack glance 1.  Snapshot an OpenStack nova instance as a glance image: Make sure that this snapshot is seen as an glance image. 2.  Boot an instance from the snapshot (glance image): Make sure that the flavor of the instance booted is at least equal to or bigger than the flavor of the instance that was snapshotted in step 1. Make sure to allow ICMP and SSH (TCP port 22) in nova security groups.  Once the above instance is up, make sure you can ping and SSH into the instance. 3.  Download the snapshot (glance image) as a file onto disk: # glance image-list +--------------------------------------+--------------------------------+ | ID                                   | Name                           | +--------------------------------------+--------------------------------+ | 56c4028c-57ab-4d12-8cf4-d96d874e90c3 | test-vm-snapshot               | +--------------------------------------+--------------------------------+ # glance image-download 56c4028c-57ab-4d12-8cf4-d96d874e90c3 --file test-vm-snapshot # # ls -l test-vm-snapshot -rw-r--r--. 1 root root 960692224 Jul 18 20:26 test-vm-snapshot This downloaded image can now be backed up or transferred over the network. 4.  Validate the downloaded snapshot file (image) using qemu-img The downloaded snapshot file (image) can be validated using: qemu-image check test-vm-snapshot qemu-image info test-vm-snapshot Sorry, unfortunately I do not have the output of the qemu-image commands above. qemu-img(1): QEMU disk image utility - Linux man page 5.  Upload this file (image) on disk into OpenStack glance # glance image-create --name 'image-from-downloaded-snapshot' --disk-format qcow2 --container-format bare --file test-vm-snapshot --visibility public --progress [=============================>] 100% +------------------+--------------------------------------+ | Property         | Value                                | +------------------+--------------------------------------+ | checksum         | 6c323b8eb0406d7f6097bfb821e9d68a     | | container_format | bare                                 | | created_at       | 2016-07-18T20:28:25Z                 | | disk_format      | qcow2                                | | id               | 44734abf-60c1-42ee-ac2a-c3018f43f003 | | min_disk         | 0                                    | | min_ram          | 0                                    | | name             | image-from-downloaded-snapshot       | | owner            | 3f61ded7cb7b46d594cbff5899e54ead     | | protected        | False                                | | size             | 960692224                            | | status           | active                               | | tags             | []                                   | | updated_at       | 2016-07-18T20:28:37Z                 | | virtual_size     | None                                 | | visibility       | public                               | +------------------+--------------------------------------+ # glance image-list +--------------------------------------+--------------------------------+ | ID                                   | Name                           | +--------------------------------------+--------------------------------+ | 44734abf-60c1-42ee-ac2a-c3018f43f003 | image-from-downloaded-snapshot | +--------------------------------------+--------------------------------+ This uploaded image will be seen in Horizon as well under "Images".  This image can now be used to boot a nova instance! ... View more

T his is a blog about configuring jumbo frames in OpenStack in order to scale neutron and nova in a production datacenter using Cisco's UCS servers, UCS Fabric Interconnects and Nexus 9000. In computer networking, MTU (Maximum Transmission Unit) is the size (in bytes) of the largest packet that can be transferred by an interface without IP fragmentation. MTU is: Fixed before transmitting data (Ethernet) Negotiated during handshake/connect time (point-to-point serial links) Dynamically determined on-the-fly while transmitting data The default MTU is 1500 bytes.  Jumbo MTU is 9000 bytes. Here is an IP packet. IP fragmentation is an process that breaks datagrams into smaller pieces (fragments), so that packets may be formed that can pass through a link with a smaller maximum transmission unit (MTU) than the original datagram size. The fragments are reassembled by the receiving host.  IP fragmentation adds additional processing overhead on the interface. IP fragmentation is controlled by the DF (Don't Fragment) bit. IP Fragmentation (DF bit in IP header): Here are the neutron configurations needed to enable jumbo MTU (9000 bytes) in OpenStack. In neutron.conf:     [DEFAULT]     global_physnet_mtu = 9000     advertise_mtu = true In openvswitch_agent.ini:     [ovs]     bridge_mappings = provider1:eth1,provider2:eth2,provider3:eth3 In ml2_conf.ini:     [ml2]     physical_network_mtus = provider2:4000,provider3:1500     path_mtu = 9000 Enable the DHCP MTU option (26) in /etc/neutron/dnsmasq-neutron.conf:     dhcp-option-force=26,1454 Restart dnsmasq (DHCP server) on all network nodes. The MTU values only apply to new network resources.  network_device_mtu in nova.conf is deprecated in OpenStack Juno. Use case to test jumbo frames in OpenStack: Boot a nova instance Create a cinder volume using ceph backend Attach volume to instance and mount it SSH into instance and write lots of data in this volume Measure traffic drop on the OpenStack nodes and ceph nodes Refer How to attach cinder/ceph XFS volume to a nova instance in OpenStack horizon for more information about the above steps in the use case. Below are the steps to configure jumbo frames on the UCS Fabric Interconnects in Cisco's UCSM GUI: Test jumbo MTU on the OpenStack nodes (controller and compute nodes): # ip a | grep mtu 2: mx: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 3: t: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 4: p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 7: br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN 8: br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN 9: br-int: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN 10: phy-br-inst@int-br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 11: int-br-inst@phy-br-inst: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 12: phy-br-prov@int-br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 13: int-br-prov@phy-br-prov: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 Send jumbo frames with 8972 bytes as payload using ping from OpenStack nodes.  These frames will not be fragmented due to -M do . # ping -M do -s 8972 10.7.8.2 PING 10.7.8.2 (10.7.8.2) 8972(9000) bytes of data. 8980 bytes from 10.7.8.2: icmp_seq=1 ttl=64 time=0.118 ms 8980 bytes from 10.7.8.2: icmp_seq=2 ttl=64 time=0.066 ms --- 10.7.8.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.062/0.082/0.118/0.022 ms -M above is used to select Path MTU Discovery strategy. -M above can be: do (prohibit fragmentation, even local one) want (do PMTU discovery, fragment locally when packet size is large) dont (do not set DF flag) Test jumbo frames on Cisco's UCS Fabric Interconnects: UCS-FI-A# connect nxos UCS-FI-A(nxos)# show queuing interface | grep MTU     q-size: 360640, HW MTU: 9216 (9216 configured) Send jumbo frames with 9000 bytes as payload using ping from the UCS Fabric Interconnect. UCS-FI-A(local-mgmt)# ping 10.23.223.20 count 5 packet-size 9000 PING 10.23.223.20 (10.23.223.20) from 10.23.223.45 : 9000(9028) bytes of data. 9008 bytes from 10.23.223.20: icmp_seq=1 ttl=255 time=0.741 ms 9008 bytes from 10.23.223.20: icmp_seq=2 ttl=255 time=0.796 ms 9008 bytes from 10.23.223.20: icmp_seq=3 ttl=255 time=0.740 ms 9008 bytes from 10.23.223.20: icmp_seq=4 ttl=255 time=0.775 ms 9008 bytes from 10.23.223.20: icmp_seq=5 ttl=255 time=0.814 ms --- 10.23.223.20 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4033ms rtt min/avg/max/mdev = 0.740/0.773/0.814/0.034 ms Configure and test jumbo frames on Cisco's Nexus 9000: N9k# configure terminal N9k(config)# interface Ethernet1/3 N9k(config-if)# mtu 9216 N9k(config-if)# end N9k# show running-config interface ethernet 1/3 interface Ethernet1/3   mtu 9216 N9k# show interface ethernet 1/3 Ethernet1/3 is up   MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec Jumbo MTU can also be configured for port-channels. Send jumbo frames with 9000 bytes as payload using ping from Nexus 9000. N9k# ping 10.23.223.21 vrf management packet-size 9000 count 5 PING 10.23.223.21 (10.23.223.21): 9000 data bytes 9008 bytes from 10.23.223.21: icmp_seq=0 ttl=254 time=1.384 ms 9008 bytes from 10.23.223.21: icmp_seq=1 ttl=254 time=0.993 ms 9008 bytes from 10.23.223.21: icmp_seq=2 ttl=254 time=0.919 ms 9008 bytes from 10.23.223.21: icmp_seq=3 ttl=254 time=0.927 ms 9008 bytes from 10.23.223.21: icmp_seq=4 ttl=254 time=1.002 ms --- 10.23.223.21 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.919/1.044/1.384 ms Jumbo packet (with MTU 9000 bytes) captured in Wireshark packet sniffer: Jumbo frames in OpenStack with neutron SR-IOV ports: I observed that nova VMs attached to a neutron SR-IOV port will also pass jumbo MTU packets.  Below is how jumbo MTU is seen on the interface of a nova VM attached to a neutron SR-IOV port. [cloud-user@sr-iov-vm ~]$ ip a | grep 9000 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000 Path MTU Discovery: Path MTU Discovery (PMTUD) is a technique to determining the MTU of the network path between source and destination.  PMTUD works by setting the Don't Fragment (DF) flag bit in the IP headers of outgoing packets. Then, any device along the path whose MTU is smaller than the packet will drop it, and send back an Internet Control Message Protocol (ICMP) Fragmentation Needed (Type 3, Code 4) message containing its MTU, allowing the source host to reduce its Path MTU appropriately.  This process is repeated until the MTU is small enough to traverse the entire path without fragmentation. Advantages of jumbo frames: Greater efficiency Lesser packet drop Interfaces process fewer packets per second Disadvantages of jumbo frames: Slower per-packet processing since each packet is big (9000 byes) Bigger packets on the wire may increase lag and latency Packet corruption causes huge retransmits and congestion ... View more

I n the movie Ghostbusters, Zuul is the demigod and gatekeeper of Gozer, the destructor. In OpenStack, ZUUL is a program used to gate the source code repository of a project so that changes are only merged if they pass tests.  Zuul is a pipeline-oriented project gating system that f acilitates running tests and automated tasks in response to gerrit events. Zuul - A Project Gating System — Zuul 2.5.2.dev44 documentation Zuul — OpenStack Project Infrastructure 0.0.1.dev12032 documentation GitHub - openstack-infra/zuul Components of Zuul: Connections (Gerrit, SMTP) Triggers (Gerrit, Timer, Zuul) Reporters (Gerrit, SMTP) Zuul Cloner Launchers (Gearman Jenkins Plugin) Statsd reporting (Metrics) Zuul Client Zuul architecture: Zulu-Jenkins Integration: The Gearman Jenkins Plugin makes it easy to use Jenkins with Zuul by providing an interface between Jenkins and Gearman ( https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin ). Gearman Plugin passes Zuul Parameters as Jenkins build parameters ( http://docs.openstack.org/infra/zuul/launchers.html#zuul-parameters ). Example Jenkins Git SCM plugin configuration: Source Code Management:     Git Repositories:          Repository URL: <your Gerrit or Zuul repository URL>         Advanced:         Refspec: ${Zuul_REF}         Branches to build:         Branch Specifier: ${Zuul_COMMIT}     Advanced:         Clean after checkout: True Zuul v3 supports multi-node.  Example multi-node playbook: ### openstack-infra/zuul-playbooks/zuul-multinode.yaml ---    hosts: controller    roles:      - zuul-controller ---    hosts: compute    roles:      - zuul-compute To deploy Zuul on bare-metal, devstack-vm-gate-wrap.sh must be replaced with bare-metal orchestration. devstack-vm-gate-wrap.sh - openstack-infra/devstack-gate - Run DevStack in the gate OpenStack Kolla deploys OpenStack cloud in Docker containers in gate using Zuul. Kolla - OpenStack project-config/kolla.yaml at master · openstack-infra/project-config · GitHub Other useful links needed to deploy Zuul: https://github.com/openstack-infra/zuul/tree/master http://status.openstack.org/zuul/ http://docs.openstack.org/infra/zuul/ https://github.com/openstack-infra/project-config/tree/master https://jenkins.io OpenStack Ironic - https://wiki.openstack.org/wiki/Ironic ... View more

T his blog has the steps to stack DevStack Newton on CentOS-7 in VirtualBox on Mac laptop. Below are the versions used. OpenStack DevStack stable Newton (GitHub - openstack-dev/devstack at stable/newton) CentOS-7 (http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1511.iso) VirtualBox 5.0.30 (Downloads – Oracle VM VirtualBox) Apple OS X El Capitan 10.11.6 Mac laptop VirtualBox is installed on Mac laptop and the CentOS-7 VM is created in VirtualBox. In VirtualBox, the CentOS-7-x86_64-Minimal-1511.iso image is used to boot a CentOS-7 VM with 4 GB RAM and the following two network adapters.  A host-only adapter is not needed. eth0 as a NAT adapter eth1 as an internal network adapter eth0 is a NAT adapter. eth1 is an internal network adapter. Run the following bash script to configure VirtualBox.  It will forward the required TCP ports from the host (Mac laptop) to the guest (CentOS-7 VM) and will also create eth1 as an internal network adapter. #!/bin/bash # Forward TCP port 3022 on host to TCP port 22 on guest VM so # that host can SSH into guest VM if ! VBoxManage showvminfo devstack-odl | grep 3022 > /dev/null then     VBoxManage modifyvm devstack-odl --natpf1 "SSH,TCP,,3022,,22" fi # Forward TCP port 8080 on host to TCP port 80 on guest VM so # that host can access OpenStack Horizon in browser if ! VBoxManage showvminfo devstack-odl | grep 8080 > /dev/null then     VBoxManage modifyvm devstack-odl --natpf1 "HTTP,TCP,,8080,,80" fi # Forward TCP port 6080 on host to TCP port 6080 on guest VM so # that host can access Nova VNC console in browser if ! VBoxManage showvminfo devstack-odl | grep 6080 > /dev/null then     VBoxManage modifyvm devstack-odl --natpf1 "CONSOLE,TCP,,6080,,6080" fi # Add internal network adapter for guest VM if ! VBoxManage showvminfo devstack-odl | grep eth1 > /dev/null then     VBoxManage modifyvm devstack-odl --nic2 intnet     VBoxManage modifyvm devstack-odl --intnet2 "eth1" fi # Remove stale entry in ~/.ssh/known_hosts on host if [ -f ~/.ssh/known_hosts ]; then     sed -i '' '/\[127.0.0.1\]:3022/d' ~/.ssh/known_hosts fi Below are the forwarded ports through eth0 (NAT interface) in VirtualBox.  Host is the Mac laptop and the guest VM is CentOS-7 booted in VirtualBox. TCP port 3022 on host is forwarded to TCP port 22 on guest VM so that host can SSH into guest VM. TCP port 8080 on host is forwarded to TCP port 80 on guest VM so that host can access OpenStack Horizon in browser. TCP port 6080 on host is forwarded to TCP port 6080 on guest VM so that host can access Nova VNC console in browser. Below is the screenshot of the forwarded ports through eth0 (NAT interface) in VirtualBox. Now, boot the CentOS-7 VM in VirtualBox.  Choose "VDI" as the disk format for the VM. When booting the CentOS-7 VM in VirtualBox, press the Tab key and type the following kernel boot options.  This keeps the interface names as eth0 and eth1 in the CentOS-7 VM instead of enp0s*. net.ifnames=0 biosdevname=0 Once the CentOS-7 VM boots, login into it and check the interfaces ("ip a" or "ifconfig").  eth0 will have an IP address like 10.0.2.15 and eth1 will not have any IP address.  Check the default gateway ("ip route").  10.0.2.2 will be the default gateway.  Make sure that you can ping a public DNS name like www.google.com and www.cisco.com. Below are the output snippets of "ip a" and "ip route" inside the CentOS-7 VM. $ ip a 2: eth0 : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     inet 10.0.2.15 /24 brd 10.0.2.255 scope global dynamic eth0 3: eth1 : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000     link/ether 08:00:27:60:77:7e brd ff:ff:ff:ff:ff:ff $ ip route default via 10.0.2.2 dev eth0  proto static  metric 100 10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.15  metric 100 192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 From the Mac laptop, SSH into the CentOS-7 VM using the forwarded port 3022.  Use the root password to login. ssh -p 3022 root@127.0.0.1 Clone the DevStack Newton repository. git clone https://git.openstack.org/openstack-dev/devstack -b stable/newton Create stack user for DevStack.  Alternatively, you can use useradd and passwd to create a new stack user, and give sudo access to the stack user by typing visudo , adding " stack   ALL=(ALL)   ALL " under " root    ALL=(ALL)   ALL ", and saving the file. cd devstack ./tools/create-stack-user.sh su stack whoami echo $HOME cd pwd exit exit Copy the local.conf file below to the devstack directory.  It has the OpenStack core services (Horizon, Keystone, Nova, Neutron, Glance, RabbitMQ and MySQL) enabled.  It uses OpenvSwitch (OVS) as the virtual switch and VLAN for tenant networks.  Remove "OFFLINE=True" below in local.conf if you are deploying DevStack for the very first time as it prevents DevStack from pulling the required OpenStack repos and configuration files. [[local|localrc]] OFFLINE=True HORIZON_BRANCH=stable/newton KEYSTONE_BRANCH=stable/newton NOVA_BRANCH=stable/newton NEUTRON_BRANCH=stable/newton GLANCE_BRANCH=stable/newton ADMIN_PASSWORD=nomoresecret DATABASE_PASSWORD=stackdb RABBIT_PASSWORD=stackqueue SERVICE_PASSWORD=$ADMIN_PASSWORD LOGDIR=$DEST/logs LOGFILE=$DEST/logs/stack.sh.log LOGDAYS=2 ENABLED_SERVICES=rabbit,mysql,key ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch,n-novnc,n-cauth ENABLED_SERVICES+=,g-api,g-reg ENABLED_SERVICES+=,horizon # Neutron DISABLED_SERVICES=n-net ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-metering,neutron PUBLIC_INTERFACE=eth0 Q_PLUGIN=ml2 ENABLE_TENANT_VLANS=True Now, exit and SSH back in as the stack user into the CentOS-7 VM. ssh -p 3022 stack@127.0.0.1 Run stack.sh and stack DevStack Newton. ./stack.sh Below is the output of stack.sh once it finishes. This is your host IP address: 10.0.2.15 This is your host IPv6 address: ::1 Horizon is now available at http://10.0.2.15/dashboard Keystone is serving at http://10.0.2.15/identity/ The default users are: admin and demo The password: nomoresecret Add security group rules to the "default" security group in order to SSH into and ping nova VMs.  This can be done in Horizon as well. cd devstack source openrc openstack security group rule create --protocol tcp --dst-port 22 default openstack security group rule create --protocol icmp --dst-port -1 default Check the two neutron networks and the neutron router created by DevStack. On the Mac laptop, enter http://localhost:8080 in a browser to access the Horizon GUI.  Login as admin and use nomoresecret as the password.  After logging into Horizon, in the upper left, select "demo" as the project. Boot a nova VM (test-vm) using cirros image, m1.tiny flavor, and attach it to the private network.  Also, create a floating IP in the public network and associate it to the nova VM.  We can now SSH into the nova VM using the floating IP! ssh cirros@172.24.4.7 password: cubswin:) $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000     link/ether fa:16:3e:28:05:85 brd ff:ff:ff:ff:ff:ff     inet 10.0.0.9/24 brd 10.0.0.255 scope global eth0     inet6 fdc7:3411:13c6:0:f816:3eff:fe28:585/64 scope global dynamic        valid_lft 86375sec preferred_lft 14375sec     inet6 fe80::f816:3eff:fe28:585/64 scope link        valid_lft forever preferred_lft forever 172.24.4.7 above is the floating IP in the public network associated to test-vm and 10.0.0.9 is the private IP address of test-vm. Below is the network topology of the nova VM test-vm attached to the private network.  DevStack creates the public network, private network and the neutron router router1 that connects them. In Horizon, go to Project --> Compute --> Instances --> test-vm --> Console and right click on the link "Click here to show only console" and click on "Open Link in New Tab". Now, change the IP address in the new tab's web address from 10.0.2.15 to localhost as VirtualBox redirects TCP port 6080 from the host (Mac laptop) to the guest (CentOS-7 VM).  Now, the nova VM's console can be accessed in Horizon! If a Cirros image is used to boot the nova VM, use the default credentials below to login into the console. username: cirros password: cubswin:) In the CentOS-7 VM, the devstack logs will be /opt/stack/logs .  Run ./unstack.sh to unstack Devstack. The VirtualBox VDI file will be at /Users/<userID>/VirtualBox\ VMs/ on the Mac laptop.  This VDI file is portable and can be re-used to boot the CentOS-7 VM in VirtualBox on any machine. Hope this blog is helpful! ... View more

The blog about Cisco's scalable CPNR DHCP and DNS neutron plugin for OpenStack is at Cisco's scalable, enterprise-class DHCP and DNS solution for OpenStack Neutron ... View more

T his is a blog about the best practices to develop, test, debug and scale REST APIs in an OpenStack cloud.  OpenStack is a great way to deploy a cloud in data centers and is one of the most popular open-source projects to deploy highly-scalable clouds.  OpenStack is all about RESTful services.  Every service in OpenStack talks to every other service using REST APIs.  As a cloud operator deploying OpenStack, it is very important to understand how REST APIs work in order to debug and scale an OpenStack cloud effectively. What is a RESTful service? Representational State Transfer (REST) is a stateless, client-server based, cacheable web service Uses HTTP protocol for communication Data is considered as a “resource” and accessed using Uniform Resource Identifiers (URIs) that typically look like a web link URI is RFC3986 (RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax) Uses HTTP clients like curl, browsers (Google Chrome Postman) and wget REST in OpenStack I gave a talk "Best REST in OpenStack" at Cisco Live in Las Vegas in 2016.  My talk covered the following topics: Run OpenStack Neutron CLIs and analyze REST packets in WireShark REST for big data use cases – REST pagination in Neutron Implementing a RESTful server using Python Flask The slides from my talk are at http://www.slideshare.net/Vikram_Hosakote/best-rest-in-openstack. The scripts I used in my talk are at GitHub - vhosakot/Cisco-Live-Workshop: Scripts for Cisco Live Workshops. Troubleshooting / debugging REST APIs curl -v (verbose mode) wget -v (verbose mode) wget -d  or  wget --debug (debug mode) neutron -v  or  neutron -d  or  neutron --debug (neutron debug mode) http://www.httpdebugger.com https://www.wireshark.org Sample script to troubleshoot / debug REST APIs using Python import httplib import logging import requests httplib.HTTPConnection.debuglevel = 1 logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True result = requests.get('http://www.cisco.com/') Output of above Python script with REST API debug messages >>> result = requests.get('http://www.cisco.com/') INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.cisco.com send: 'GET / HTTP/1.1\r\nHost: www.cisco.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.10.0\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Server: Apache header: ETag: "10839-53789488e5b33" header: Accept-Ranges: bytes header: Content-Encoding: gzip header: CDCHOST: wemxweb-publish-prod1-01 header: Content-Length: 14215 header: Content-Type: text/html header: Expires: Wed, 13 Jul 2016 20:25:27 GMT header: Cache-Control: max-age=0, no-cache, no-store header: Pragma: no-cache header: Date: Wed, 13 Jul 2016 20:25:27 GMT header: Connection: keep-alive header: access-control-allow-origin: * DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 200 14215 >>> print result <Response [200]> Sample script to mock a REST server to unit-test Python REST APIs $ python >>> import requests >>> import requests_mock >>> session = requests.Session() >>> adapter = requests_mock.Adapter() >>> session.mount('mock', adapter) >>> adapter.register_uri('GET', 'mock://test.com', text='data') >>> resp = session.get('mock://test.com') >>> resp.status_code, resp.text (200, 'data') https://github.com/openstack/requests-mock https://pypi.python.org/pypi/mock-server/0.3.7 https://github.com/gabrielfalcao/HTTPretty http://python-mock-tutorial.readthedocs.io/en/latest/mock.html REST API caching and cache-aware REST clients RFC 7234 - https://tools.ietf.org/html/rfc7234#section-5 REST supports the following HTTP cache-control headers: max-age max-stale min-fresh no-cache no-store no-transform only-if-cached must-revalidate public private Bulk REST operations and REST API batching Bulk REST operations allow us to perform REST operations (GET, POST, PUT, DELETE) on bulk (more than one) objects (networks, ports, subnets, etc) at once.  This reduces the number of REST messages between client and the server. http://developer.openstack.org/api-ref-networking-v2.html#bulkCreateNetwork https://wiki.openstack.org/wiki/Neutron/APIv2-specification#Bulk_version Bulk REST API JSON request format to create five neutron networks in OpenStack using POST operation: POST v2.0/networks.json Content-Type: application/json Accept: application/json { "networks": [    {      "name": "sample_network_1",      "admin_state_up": false    },    {      "name": "sample_network_2",      "admin_state_up": false    }    {      "name": "sample_network_3”,      "admin_state_up": true    }    {      "name": "sample_network_4”,      "admin_state_up": true    }    {      "name": "sample_network_5”,      "admin_state_up": false    }] } ... View more

This blog is in the OpenStack area at Un-virtualizing the soft cloud - software vs hardware ... View more