OpenStack Interview Questions and Answers


Last updated on 03rd Jul 2020, Blog, Interview Questions

About author

Damodharan (Sr Solution Engineer - Cloud Platform )

Highly Expertise in Respective Industry Domain with 10+ Years of Experience Also, He is a Technical Blog Writer for Past 4 Years to Renders A Kind Of Informative Knowledge for JOB Seeker

(5.0) | 16547 Ratings 2029

Nowadays most of the firms are trying to migrate their IT infrastructure and Telecom Infra into private cloud i.e OpenStack. If you are planning to give interviews on Openstack admin profile, then the below list of interview questions might help you to crack the interview.

1)   Define OpenStack and its key components?


It is a bundle of open source software, which all in combine forms a cloud software known as Open Stack. Open Stack is known as Stack of Open source Software or Projects.

Following are the key components of OpenStack

  • Nova – It handles the Virtual machines at compute level and performs other computing tasks at compute or hypervisor level.
  • Neutron – It provides the networking functionality to VMs, Compute and Controller Nodes.
  • Keystone – It provides the identity service for all cloud users and openstack services. In other words, we can say Keystone is a method to provide access to cloud users and services.
  • Horizon – It provides a GUI (Graphical User Interface), using the GUI Admin can all day to day operations task at ease.
  • Cinder – It provides the block storage functionality, generally in OpenStack Cinder is integrated with Chef and ScaleIO to service block storage to Compute & Controller nodes.
  • Swift – It provides the object storage functionality. Generally, Glance images are on object storage. External storage like ScaleIO can work as Object storage too and can easily be integrated with Glance Service.
  • Glance – It provides Cloud image services, using glance admin used to upload and download cloud images.
  • Heat – It provides an orchestration service or functionality. Using Heat admin can easily VMs as stack and based on requirements VMs in the stack can be scale-in and Scale-out
  • Ceilometer – It provides the telemetry and billing services.

2)  What services generally run on a controller node?


Following services run on a controller node:

  • Identity Service ( KeyStone)
  • Image Service ( Glance)
  • Nova Services like Nova API, Nova Scheduler & Nova DB
  • Block & Object Service
  • Ceilometer Service
  • MariaDB / MySQL and RabbitMQ Service
  • Management services of Networking (Neutron) and Networking agents
  • Orchestration Service (Heat)
3)  What are the services generally run on a Compute Node?


Following services run on a compute node,

  • Nova-Compute
  • Networking Services like OVS
4) What is the default location of VMs on the Compute Nodes?


VMs in the Compute node are stored at “/var/lib/nova/instances”

5) What is default location of glance images?


As the Glance service runs on a controller node, all the glance images are store under the folder “/var/lib/glance/images” on a controller node.

6) Tell me the command how to spin a VM from Command Line?


We can easily spin a new VM using the following openstack command,

  • # openstack server create –flavor {flavor-name} –image {Image-Name-Or-Image-ID}  –nic net-id={Network-ID} –security-group {Security_Group_ID} –key-name {Keypair-Name} <VM_Name>

7) How to list the network namespace of a tenant in OpenStack?


Network namespace of a tenant can be listed using “ip net ns” command

  • ~# ip netns list 
  • qdhcp-a51635b1-d023-419a-93b5-39de47755d2d
  • haproxy
  • vrouter

8) How to execute command inside network namespace in open stack?


  • Let’s assume we want to execute “ifconfig” command inside the network namespace “qdhcp-a51635b1-d023-419a-93b5-39de47755d2d”, then run the beneath command,
  • Syntax : ip netns exec {network-space} <command>
  • ~# ip netns exec qdhcp-a51635b1-d023-419a-93b5-39de47755d2d “ifconfig”
9) How to upload and download a cloud image in Glance from command line?


A Cloud image can be uploaded in glance from command using beneath openstack command,

  • ~# openstack image create –disk-format qcow2 –container-format bare   –public –file {Name-Cloud-Image}.qcow2 <Cloud-Image-Name>

Use below openstack command to download a cloud image from command line,

  • ~# glance image-download –file <Cloud-Image-Name> –progress  <Image-ID>
10)  How to reset error state of a VM into active in OpenStack env?


There are some scenarios where some VMs went to error state and this error state can be changed into active state using below commands,

  • ~# nova reset-state –active {Instance_id}
11) What are the commands used to generate key pairs in OpenStack?


Commands used to generate key pairs in OpenStack:

  • ssh-keygen
  • cd.ssh
  • nova keypair-add -pub_key mykey
12) Which hardware is required for networking in OpenStack?


In Open Stack,networking can be done with following hardware:

  • Networks
  • Routers
  • Subnets
  • Ports
  • Vendor Plugins
13) Which command is used to manage floating IP addresses in OpenStack?


nova floating-ip-*

14) Explain the usage of Cinder in OpenStack?


Open Stack Cinder is used to handle block storage in the context of OpenStack.

15) What is the use of $ nova floating-ip-pool-list command in OpenStack?


The $ nova floating-ip-pool-list command is used to list IP address information in Open Stack.

16) Explain the term “flavor” in OpenStack?


The term “flavor” is an available hardware configuration for a server, which defines the size of a virtual server that can be launched.

17) By default, which cloud infrastructure layer is supported by openstack, when install? 


By default, openstack lies into infrastructure layer called – IaaS (Infrastructure as a Service)

18) What is IaaS ?


Iaas (Infrastructure as a Service) – in this service, cloud vendor only provides the underlying hardware support to it’s clients. Other than the hardware resources, client has to take care of everything like – OS, application, security.

19)  How soon openstack releases happens ?


Openstack releases happens twice a year and after each six months. Openstack releases happens in month of April and October.

20)  Which configuration management tool is used by pack-stack utility ?


Pack-stack utility uses the puppet as the configuration management tool for automating the openstack installation. In background, puppet pushes the configuration to the nodes based on the answer file configuration.

    Subscribe For Free Demo

    21)  What is floating IP ?


    Floating IP is an IP address which is acts as a public ip address. In localhost, floating ip address are any ip address which you have defined for your infrastructure. But in the real scenario, floating IP address are the real publi ip address which are provided by the ISP (Internet Service Provider)

    22)  What is Cinder ?


    Cinder is a block based storage in which we create the LUNs and use it for the file system storage. First, we will create a LUN, then scan it on the OS. Then, we create the file system on it and mount it. This is how we can use cinder volume for storing the data. Cinder is equivalent to Amazon EBS.

    23)  What is Swift ?


    Swift is the object based storage which by passes the file system structure. We can directly upload and download the different objects on swift storage. It is uses where read are more and write are less to objects. Swift is equivalent to Amazon S3.

    24)  What is Nova Scheduler ?


    Nova scheduler dispatches the request for new virtual machines to the correct compute nodes based on configured weights and filters. Basically, it checks the different compute nodes based on some algorithms which helps it to create virtual machine on which compute node.

    25)  What is Neutron ?


    Neutron is the service which provides the functionality of the network layer. Here, you can create your own isolated network for your own infrastructure. You can create your own network, subnets, route tables and many more.

    26) Define ‘users,’ ‘role’ and ‘tenant’ in OpenStack.


    Users can be members of multiple projects:

    • Tenant is a group of users and an alternative term for Project/accounts where projects are organizational units in cloud processing
    • Role is the position to which a user is mapped (the authorization level). Roles are usually assigned to project-user duos.

    27) Define the Networking Managers in OpenStack Cloud.


    • Flat Network Manager: This places all VMs on a single network utilizing the same subnet and bridge as created by the administrator. Thus, all VMs share the same network that can be interconnected and are known to have Flat Network Manager.
    • Flat DHCP Network Manager: Much similar to the above except that the IP addresses to VM are assigned via DHCP (Dynamic Host Configuration Protocol).
    • VLAN: Unlike the single network concept, VLAN facilitates more secure and separate network to VMs. It has a physical switch to offer separate virtual network and separate IP range and bridge for each tenant. This is indeed most preferable choice for multi-tenant/project environment.

    28) Define Identity Service in OpenStack.


    Keystone is the most important and preferred Identity Service in OpenStack and executes the complete OpenStack Identity API. The Keystone Identity Service is responsible for user management and service catalog. In user management, it tracks users and their permissions while Service Catalog offers a list of services available with their API. The former provides authentication credential details of users, tenants and roles.Internal services like Token and Policy are also part of Keystone Identity

    29) Name the commands used to pause and un-pause(resume) an instance


    • # novaunpause INSTANCE_NAME
    • # nova pause INSTANCE_NAME

    30) List the storage locations for VM images in OpenStack


    •  OpenStack Object Storage
    •  Filesystem
    •  S3
    •  HTTP
    •  RBD or Rados Block Device
    • GridFSMaster Openstack from industry experts. Find out more in this openstack Blog now.
    Course Curriculum

    Learn Hands-On Practical OpenStack Certification Course By Certified Professionals

    Weekday / Weekend BatchesSee Batch Details

    31) What is OpenStack Python SDK?


    Python SDK (Software Development Kit) helps users to write applications for performing automation tasks in Python by calling Python objects. It provides a platform to work with multiple OpenStack services at one place. It consists of language bindings to access OpenStack clouds, complete API reference, easy interaction with REST API and sample code for initial applications.

    32) Describe the function of Filter Scheduler.


    The Filter Scheduler facilitates filtering and weighting to notify where a new instance can be created. It supports working with Compute Nodes. Filter Scheduler firstly creates an unfiltered dictionary of hosts and then filter them using related properties and makes the final selection of hosts for the number of instances as needed.

    33) Define the Networking option in OpenStack.


    •  CapacityFilter: filtering based on volume host’s capacity consumption
    •  DifferentBackendFilter: Scheduling volumes to a different back-end
    •  DriverFilter: filters based on ‘filter function’ and ‘metrics’
    • InstanceLocalityFilter
    • JSONFIlter
    • RetryFilter: Filter the previously attempted hosts
    • SameBackendFilterMost in-depth, industry-led curriculum in Openstack.

    34) List down the type of Hypervisors supported by OpenStack.


    • KVM: (Kernel-based Virtual machine)
    • LXC: Linux Containers having Linux-based VMs
    • QEMU: Quick EMUlator used for development purposes
    • UML: User Mode Linux used for development purposes
    • VMware vSphere: VMware-based Linux and Windows via vCenter server connection.
    • Hyper-V: Server virtualization with Microsoft’s Hyper-V

    35) Explain in brief the modular architecture of OpenStack.


    The three important components of OpenStack modular architecture are:

    • OpenStack Compute: For managing large networks on the virtual machine
    • Image Service: The delivery service provides discovery and registration for virtual disk images
    • OpenStack Object Storage: A storage system that provides support for both block storage and object storage

    36) What are the biggest opportunities for folks who want to create something awesome and outstanding in the OpenStack Network Project?


    • There will be a big push in Havana for VPN-as-a-Service in several different deployment modes. Also, we’ll extend load balancing. In Grizzly we took the baby steps of getting it out and there’s several vendors who are now trying to leverage that API.  IPv6 support is also going to be big as well. More internet service providers are offering v6 services for business deployments. Ensuring that OpenStack Network Project works for the various deployment modes of IPv6 is going to be important as well.
    • We also have excitement around folks who are working on bare metal with OpenStack Networking and on device management in larger scale: If I’m a hardware vendor – how do I integrate my piece of hardware into OpenStack Networking? Also, we are focused on deployer topics such as: How can I provide different level service level offerings?

    37) List down the components of OpenStack Compute


    Nova (Compute) Cloud comprises following components:

    • API server
    • Message Queue (Rabbit-MQ Server)
    • Compute Workers (Nova-Compute)
    • Network controller (Nova-Network)
    • Volume Worker
    • Schedule

    38)  What Will You Do In Case Of Server Failure?


    • If a server is having hardware issues, it is a smart thought to ensure the Object Storage services are not running. This will permit Object Storage to work around the disappointment while you investigate.
    • If the server simply needs a reboot or a little measure of work that should just last two or three hours, at that point, it is most likely best to let Object Storage work around the disappointment and recover the machine settled and on the web. At the point, when the machine returns online, replication will ensure that anything absent amid the downtime will get refreshed.
    • If you can’t supplant the drive instantly, at that point, it is best to abandon it unmounted and expel the drive from the ring. This will permit every one of the limitations that were on that drive to be reproduced somewhere else until the point when the drive is supplanted. Once the drive is supplanted, it can be re-added to the ring.

    39)  How can you overcome any type of sudden server failure?


    During the failure of the server or when the server is not seen to be perfectly functioning then the Object Storage services should not be running. Hence, it is best to close them to this problem arises. This is because to know the solution, troubleshooting must be done. Closing the object storage service will allow it to work or function while troubleshooting is still going on. However, often such failure just needs rebooting and hence for this work which doesn’t require much time, the object storage service doesn’t need to be closed. Update to this service will be added once the machine gets online after rebooting. Removing the drive from the ring is the best idea when the driver can’t be replaced. But if it can be replaced, then it can be added back to the ring.

    40)  What Is Sanitization Process?


    The sanitization process removes information from the media such that the information cannot be retrieved or reconstructed. Sanitization techniques, including clearing, purging, cryptographic erase, and destruction, prevent the disclosure of information to unauthorized individuals when such media is reused or released for disposal.

    41)  What Is Alarm In Openstack?


    Alarms provide user-oriented Monitoring-as-a-Service for resources running on OpenStack. This type of monitoring ensures you can automatically scale in or out a group of instances through the Orchestration module, but you can also use alarms for general-purpose awareness of your cloud resources’ health.

    42)  What Is Orchestration?


    Orchestration is an orchestration engine that provides the possibility to launch multiple composite cloud applications based on templates in the form of text files that can be treated like code.

    43)  What is CPU Pinning in OpenStack Cloud Computing?


    The CPU Pinning is defined as the process of reserving the physical core parts of virtual machines specified as per the requirement. It is also called the process of isolating the CPU.

    44)  What are the secluded design segments of OpenStack?


    OpenStack is an open source and free arrangement of programming instruments or distributed computing stage which is used for overseeing and building distributed computing stage for the private and open cloud.

    45)  Give An Example Where Logs Help In Openstack Security?


    For instance, analyzing the access logs of Identity service or its replacement authentication system would alert us to failed logins, frequency, origin IP, whether the events are restricted to select accounts and other pertinent information. Log analysis supports detection.

    46) What Are Data Privacy Concerns In OpenStack? How Can Those Be Remediated?


    Data residency: Concerns over who possesses data in the cloud and whether the cloud administrator can be at last trusted as a caretaker of this data have been critical issues previously.Data disposal: Best practices recommend that the administrator cleans cloud framework media (non-digital and digital) before to disposal, discharge out of organization or release for reuse.Data not safely deleted: This might be remediated with the database as well as framework setup for auto vacuuming and intermittent free-space wiping.Instance memory scouring, Cinder volume data, Image service delay delete feature.

    47)  Name the commands used to pause & un-pause (resume) an instance?


    • $ Novaunpause INSTANCE_NAME
    • $ Nova pause INSTANCE_NAME
    48)  Describe the function of Filter Scheduler?


    The Filter Scheduler facilitates filtering & weighting to alert where a new instance can be created. It supports working with gauge Nodes. Filter Scheduler firstly creates an unfiltered dictionary of hosts and then filter those using linked properties& makes the final selection of hosts for the number of instances as needed.

    49)  Analyze Hypervisor?


    For all cloud computing platforms, Hypervisor is a term to characterize virtual machine monitor (VMM) including hardware, software & firmware factor running on a virtual machine. Host machine is the one having hypervisor with one or more virtual machines. OpenStack Compute grant multiple hypervisors. There are functionalities to choose one among them for a specific purpose.

    50)  List down the type of Hypervisors financed by OpenStack?


    KVM (Kernel-based Virtual machine)

    • LXC: Linux Containers get Linux-based VMs
    • QEMU: Quick Emulator used for development purposes
    • UML: User Mode Linux used for development purposes
    • VMware vSphere: VMware-based Linux & Windows via venter server connection.
    • Hyper-V: Server virtualization with Microsoft’s Hyper-V
    Course Curriculum

    Get Best OpenStack Training to Become An Certified OpenStack Developer

    • Instructor-led Sessions
    • Real-life Case Studies
    • Assignments
    Explore Curriculum

    51) What is the role of Integration Bridge(br-int) on the Compute Node ?


    • The integration bridge (br-int) performs VLAN tagging and untagging for the traffic coming from and to the instance running on the compute node.
    • Packets leaving the n/w interface of an instance goes through the linux bridge (qbr) using the virtual interface qvo. The interface qvb is connected to the Linux Bridge & interface qvo is connected to integration bridge (br-int). The qvo port on integration bridge has an internal VLAN tag that gets appended to packet header when a packet reaches to the integration bridge.

    52) What is the role of Tunnel Bridge (br-tun) on the compute node?


    • The tunnel bridge (br-tun) translates the VLAN tagged traffic from integration bridge to the tunnel ids using OpenFlow rules.
    • br-tun (tunnel bridge) allows the communication between the instances on different networks. Tunneling helps to encapsulate the traffic travelling over insecure networks, br-tun supports two overlay networks i.e GRE and VXLAN

    53) What is the role of external OVS bridge (br-ex)?


    As the name suggests, this bridge forwards the traffic coming to and from the network to allow external access to instances. br-ex connects to the physical interface like eth2, so that floating IP traffic for tenants networks is received from the physical network and routed to the tenant network ports.

    54)  Describe Troubleshooting checklist 


    • Identify & reproduce the problem 
    • What was the user / admin interaction what triggered it 
    • Collect information 
    • Client tools being used, versions, debug output
    • Services being involved, configuration, logs, debug output
    • Check environment: networking, OS, dependent services, storage disk space, etc.
    • Fix trivial issues ○ Fix it on the spot, experiment with dev/test environment, home lab
    • Ask for help
    • Use web search, reach out to docs, support, developers 
    • Mitigate carefully
    • Plan and test the steps of the mitigation procedure (aka “do not break prod”)
    • Document everything for future reference

    55)  How can you transfer volume from one owner to another in OpenStack?


    We can transfer a volume from one owner to another by using the command

    56) What Is The Command Used For Unpause And Pause An Instance?


    To unpause an instance:

    • $ nova unpause INSTANCE_NAME

    To pause an instance:

    • $ nova pause INSTANCE_NAME

    57) Explain How You Can Transfer Volume From One Owner To Another In Openstack?


    You can transfer a volume from one owner to another by using the command cinder transfer

    58) What Are The Main Components Of Identity User Management?  


    • Users: It is a digital representation of a person, service or system who uses OpenStack cloud services
    • Roles: A role includes a set of rights and privileges. A role determines what operations a user is permitted to perform in a given tenant
    • Tenants: A container used to group or isolate resource or identity objects. Depending on service operator a tenant may map to a customer, account, organization or project.

    59) What Are Data Privacy Concerns In Openstack, How Those Can Be Remediated?


    • Data residency: Concerns over who owns data in the cloud and whether the cloud operator can be ultimately trusted as a custodian of this data have been significant issues in the past.
    • Data disposal: Best practices suggest that the operator sanitize cloud system media (digital and non-digital) prior to disposal, release out of organization control or release for reuse.
    • Data not securely erased: This may be remediated with database and/or system configuration for auto vacuuming and periodic free-space wiping.

    Instance memory scrubbing, Cinder volume data, Image service delay delete feature.

    60) How To Create A Normal User In Openstack ?


    sudo nova-manage user create user-name 

    61) How You Assign A Project To A User ?


    sudo nova-manage project add –project=project_name


    62) How You Can Remove A Rule From Security Group ?


    nova secgroup-delete-rule webserver tcp 443 443

    63) How To Display Images Using Nova Client ?


    nova image-list

     64) How To See A List Of Roles And The Associated Ids I n Our Environment ?


    keystone role-list

    65) What Is Job Of User Crud ?


    The user CRUD filter enables users to use a HTTP PATCH to change their own password.

    66) Where Caching Configuration Is Stored ?


    The majority of the caching configuration options are set in the [cache] section of the keystone.conf file.

    67)  What Is Alarm In Openstack ?


    Alarms provide user-oriented Monitoring-as-a-Service for resources running on OpenStack. This type of monitoring ensures you can automatically scale in or out a group of instances through the Orchestration module, but you can also use alarms for general-purpose awareness of your cloud resources’ health.

    68) How To Migrate Running Instances From  One Openstack Compute Server To Another Openstack Compute Server ?


    Check the ID of the instance to be migrated

    • Check the information associated with the instance
    • Select the compute node the instance will be migrated to.
    • Check that Host has enough resources for migration
    • Migrate the instance using the $ nova live-migration SERVER HOST_NAME command.

    69) How You Can Change Behavior Of Dhcp  Server ?


    The behavior of dnsmasq can be customized by creating a dnsmasq configuration file

    70) What Is Use Of Account Reaper ?


    In the background, the account reaper removes data from the deleted accounts.

    Open Stack Sample Resumes! Download & Edit, Get Noticed by Top Employers! Download

    71) What You Will Do In Case Of Drive Failure ?


    In the event that a drive has failed, the first step is to make sure the drive is unmounted. This will make it easier for Object Storage to work around the failure until it has been resolved. If the drive is going to be replaced immediately, then it is just best to replace the drive, format it, remount it, and let replication fill it up.

    72) What You Will Do In Case Of Server Failure ?


    • If a server is having hardware issues, it is a good idea to make sure the Object Storage services are not running. This will allow Object Storage to work around the failure while you troubleshoot.
    • If the server just needs a reboot, or a small amount of work that should only last a couple of hours, then it is probably best to let Object Storage work around the failure and get the machine fixed and back online. When the machine comes back online, replication will make sure that anything that is missing during the downtime will get updated.
    • If you cannot replace the drive immediately, then it is best to leave it unmounted, and remove the drive from the ring. This will allow all the replicas that were on that drive to be replicated elsewhere until the drive is replaced. Once the drive is replaced, it can be re-added to the ring.

    73)  How You Will Install Dhcp Agent ?


    • $ apt-get install neutron-dhcp-agent

    74) What Does It Mean For The Cloud Ecosystem?


    • Wide adoption of an open-source, open-standards cloud should be huge for everyone. It means customers won’t have to fear lock-in and technology companies can participate in a growing market that spans cloud providers. Companies are already using OpenStack to provide public clouds, support, training and system integration services and hardware and software products.
    • A great analogy comes from the early days of the Internet: the transition away from fractured, proprietary flavors of UNIX toward open-source Linux. An open cloud stands to provide the same benefits for large-scale cloud computing that the Linux standard provided inside the server.
    75) What is iptables?


    Through nova-network or neutron, OpenStack Compute automatically manages iptables, including forwarding packets to and from instances on a compute node, forwarding floating IP traffic, and managing security group rules. In addition to managing the rules, comments (if supported) will be inserted in the rules to help indicate the purpose of the rule.

    The following comments are added to the rule set as appropriate:

    • Perform source NAT on outgoing traffic.
    • Default drop rule for unmatched traffic.
    • Direct traffic from the VM interface to the security group chain.
    • Jump to the VM specific chain.
    • Direct incoming traffic from VM to the security group chain.
    • Allow traffic from defined IP/MAC pairs.
    • Drop traffic without an IP/MAC allow rule.
    • Allow DHCP client traffic.
    • Prevent DHCP Spoofing by VM.
    • Send unmatched traffic to the fallback chain.
    • Drop packets that are not associated with a state.
    • Direct packets associated with a known session to the RETURN chain.
    • Allow IPv6 ICMP traffic to allow RA packets.

    Run the following command to view the current iptables configuration:

    • # iptables-save

    76) How would you advise people who want to get started contributing to OpenStack Networking? What steps should they specifically take?


    The first step is to obviously join the OpenStack development mailing list. It gives you a sense of what the topics are that the OpenStack Networking developers are discussing.

    1. The OpenStack Networking team also maintains a Wiki page for starter bugs. On that page we keep track of simple links for: Here’s how to find the code reviews for the OpenStack Networking server side or the OpenStack Networking client etc. As bugs are reported we will tag the easier ones as low hanging fruit which are an excellent opportunity for new developers and contributors to jump in on the project.
    2. That means whoever is triaging the bugs can say: “Hey, this is something that is not overly complicated and is a good way to become familiar with the OpenStack Networking code base.” We also maintain a list of community projects that would be good to start working on. By being a member of the OpenStack Networking mailing list you can recruit other members of the community and work together. It also builds up trust within the community so that those folks who were reviewing your code are working with you. You have a sense of rapport with them.
    77) What are the Tools for automated neutron diagnosis?


    • easyOVS is a useful tool when it comes to operating your OpenvSwitch bridges and iptables on your OpenStack platform. It automatically associates the virtual ports with the VM MAC/IP, VLAN tag and namespace information, as well as the iptables rules for VMs.
    • Don is another convenient network analysis and diagnostic system that provides a completely automated service for verifying and diagnosing the networking functionality provided by OVS.

    78) How to Dealing with Network Namespaces?


    Linux network namespaces are a kernel feature the networking service uses to support multiple isolated layer-2 networks with overlapping IP address ranges. The support may be disabled, but it is on by default. If it is enabled in your environment, your network nodes will run their dhcp-agents and l3-agents in isolated namespaces. Network interfaces and traffic on those interfaces will not be visible in the default namespace.

    To see whether you are using namespaces, run ip netns:

    • # ip netns
    • qdhcp-e521f9d0-a1bd-4ff4-bc81-78a60dd88fe5
    • qdhcp-a4d00c60-f005-400e-a24c-1bf8b8308f98
    • qdhcp-fe178706-9942-4600-9224-b2ae7c61db71
    • qdhcp-0a1d0a27-cffa-4de3-92c5-9d3fd3f2e74d
    • qrouter-8a4ce760-ab55-4f2f-8ec5-a2e858ce0d39
    • L3-agent router namespaces are named qrouter-<router_uuid>, and dhcp-agent name spaces are named qdhcp-<net_uuid>. This output shows a network node with four networks running dhcp-agents, one of which is also running an l3-agent router. It’s important to know which network you need to be working in. A list of existing networks and their UUIDs can be obtained by running openstack network list with administrative credentials.
    • Once you’ve determined which namespace you need to work in, you can use any of the debugging tools mention earlier by prefixing the command with ip netns exec <namespace>. For example, to see what network interfaces exist in the first qdhcp namespace returned above, do this:
    • # ip netns exec qdhcp-e521f9d0-a1bd-4ff4-bc81-78a60dd88fe5 ip a
    • 10: tape6256f7d-31: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    •     link/ether fa:16:3e:aa:f7:a1 brd ff:ff:ff:ff:ff:ff
    •     inet brd scope global tape6256f7d-31
    •     inet brd scope global tape6256f7d-31
    •     inet6 fe80::f816:3eff:feaa:f7a1/64 scope link
    •     valid_lft forever preferred_lft forever
    • 28: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    •     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    •     inet scope host lo
    •     inet6 ::1/128 scope host
    •     valid_lft forever preferred_lft forever

    From this you see that the DHCP server on that network is using the tape6256f7d-31 device and has an IP address of Seeing the address, you can also see that the dhcp-agent is running a metadata-proxy service. Any of the commands mentioned previously in this chapter can be run in the same way. It is also possible to run a shell, such as bash, and have an interactive session within the namespace. In the latter case, exiting the shell returns you to the top-level default namespace.

    79) What is Troubleshooting Open vSwitch?


    • Open vSwitch, as used in the previous OpenStack Networking examples is a full-featured multilayer virtual switch licensed under the open source Apache 2.0 license. Full documentation can be found at the project’s website. In practice, given the preceding configuration, the most common issues are being sure that the required bridges (br-int, br-tun, and br-ex) exist and have the proper ports connected to them.
    • The Open vSwitch driver should and usually does manage this automatically, but it is useful to know how to do this by hand with the ovs-vsctl command. This command has many more subcommands than we will use here; see the man page or use ovs-vsctl –help for the full listing.
    • To list the bridges on a system, use ovs-vsctl list-br. This example shows a compute node that has an internal bridge and a tunnel bridge. VLAN networks are trunked through the eth1 network interface:
    • # ovs-vsctl list-br
    • br-int
    • br-tun
    • eth1-br
    • Working from the physical interface inwards, we can see the chain of ports and bridges. First, the bridge eth1-br, which contains the physical network interface eth1 and the virtual interface phy-eth1-br:
    • # ovs-vsctl list-ports eth1-br
    • eth1
    • phy-eth1-br
    • Next, the internal bridge, br-int, contains int-eth1-br, which pairs with phy-eth1-br to connect to the physical network shown in the previous bridge, patch-tun, which is used to connect to the GRE tunnel bridge and the TAP devices that connect to the instances currently running on the system:
    • # ovs-vsctl list-ports br-int
    • int-eth1-br
    • patch-tun
    • tap2d782834-d1
    • tap690466bc-92
    • tap8a864970-2d
    • The tunnel bridge, br-tun, contains the patch-int interface and gre-<N> interfaces for each peer it connects to via GRE, one for each compute and network node in your cluster:
    • # ovs-vsctl list-ports br-tun
    • patch-int
    • gre-1
    • .
    • .
    • .
    • gre-<N>
    • If any of these links are missing or incorrect, it suggests a configuration error. Bridges can be added with ovs-vsctl add-br, and ports can be added to bridges with ovs-vsctl add-port. While running these by hand can be useful debugging, it is imperative that manual changes that you intend to keep be reflected back into your configuration files.
    80) How do you Debug DHCP Issues with nova-network?


    • One common networking problem is that an instance boots successfully but is not reachable because it failed to obtain an IP address from dnsmasq, which is the DHCP server that is launched by the nova-network service.
    • The simplest way to identify that this is the problem with your instance is to look at the console output of your instance. If DHCP failed, you can retrieve the console log by doing:
    • $ openstack console log show <instance name or uuid>

    If your instance failed to obtain an IP through DHCP, some messages should appear in the console. For example, for the Cirros image, you see output that looks like the following:

    • udhcpc (v1.17.2) started
    • Sending discover…
    • Sending discover…
    • Sending discover…

    No lease, forking to background

    • starting DHCP forEthernet interface eth0 [ [1;32mOK[0;39m ]
    • cloud-setup: checking
    • wget: can’t connect to remote host ( Network is unreachable

    After you establish that the instance booted properly, the task is to figure out where the failure is.

    • A DHCP problem might be caused by a misbehaving dnsmasq process. First, debug by checking logs and then restart the dnsmasq processes only for that project (tenant). In VLAN mode, there is a dnsmasq process for each tenant. Once you have restarted targeted dnsmasq processes, the simplest way to rule out dnsmasq causes is to kill all of the dnsmasq processes on the machine and restart nova-network. As a last resort, do this as root:
    • # killall dnsmasq
    • # restart nova-network

    81) Manually Disassociating a Floating IP


    Sometimes an instance is terminated but the floating IP was not correctly disassociated from that instance. Because the database is in an inconsistent state, the usual tools to disassociate the IP no longer work. To fix this, you must manually update the database.

    First, find the UUID of the instance in question:

    • mysql> select uuid from instances where hostname = ‘hostname’;

    Next, find the fixed IP entry for that UUID:

    • mysql> select * from fixed_ips where instance_uuid = ‘<uuid>’;

    You can now get the related floating IP entry:

    • mysql> select * from floating_ips where fixed_ip_id = ‘<fixed_ip_id>’;

    And finally, you can disassociate the floating IP:

    • mysql> update floating_ips set fixed_ip_id = NULL, host = NULL where
    •        fixed_ip_id = ‘<fixed_ip_id>’;

    You can optionally also deallocate the IP from the user’s pool:

    • mysql> update floating_ips set project_id = NULL where
    •        fixed_ip_id = ‘<fixed_ip_id>’;

    82) Network Configuration in the Database for nova-network


    With nova-network, the nova database table contains a few tables with networking information:

    • fixed_ips
      • Contains each possible IP address for the subnet(s) added to Compute. This table is related to the instances table by way of the fixed_ips.instance_uuid column.
    • floating_ips
      • Contains each floating IP address that was added to Compute. This table is related to the fixed_ips table by way of the floating_ips.fixed_ip_id column.
    • instances
      • Not entirely network specific, but it contains information about the instance that is utilizing the fixed_ip and optional floating_ip.

    From these tables, you can see that a floating IP is technically never directly related to an instance; it must always go through a fixed IP.

    83) Elaborate tcpdump


    One great, although very in-depth, way of troubleshooting network issues is to use tcpdump. We recommended using tcpdump at several points along the network path to correlate where a problem might be. If you prefer working with a GUI, either live or by using a tcpdump capture, check out Wireshark.

    For example, run the following command:

    • # tcpdump -i any -n -v ‘icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo’

    Run this on the command line of the following areas:

    1. An external server outside of the cloud
    2. A compute node
    3. An instance running on that compute node

    In this example, these locations have the following IP addresses:



    Compute Node


    External Server


    Next, open a new shell to the instance and then ping the external host where tcpdump is running. If the network path to the external server and back is fully functional, you see something like the following:

    On the external server:

    • 12:51:42.020227 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF],
    • proto ICMP (1), length 84)
    • > ICMP echo request, id 24895, seq 1, length 64
    • 12:51:42.020255 IP (tos 0x0, ttl 64, id 8137, offset 0, flags [none],
    • proto ICMP (1), length 84)
    • > ICMP echo reply, id 24895, seq 1,
    •     length 64

    On the compute node:

    • 12:51:42.019519 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
    • proto ICMP (1), length 84)
    • > ICMP echo request, id 24895, seq 1, length 64
    • 12:51:42.019519 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
    • proto ICMP (1), length 84)
    • > ICMP echo request, id 24895, seq 1, length 64
    • 12:51:42.019545 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF],
    • proto ICMP (1), length 84)
    • > ICMP echo request, id 24895, seq 1, length 64
    • 12:51:42.019780 IP (tos 0x0, ttl 62, id 8137, offset 0, flags [none],
    • proto ICMP (1), length 84)
    • > ICMP echo reply, id 24895, seq 1, length 64
    • 12:51:42.019801 IP (tos 0x0, ttl 61, id 8137, offset 0, flags [none],
    • proto ICMP (1), length 84)
    • > ICMP echo reply, id 24895, seq 1, length 64
    • 12:51:42.019807 IP (tos 0x0, ttl 61, id 8137, offset 0, flags [none],
    • proto ICMP (1), length 84)
    • > ICMP echo reply, id 24895, seq 1, length 64

    On the instance:

    • 12:51:42.020974 IP (tos 0x0, ttl 61, id 8137, offset 0, flags [none],
    • proto ICMP (1), length 84)
    • > ICMP echo reply, id 24895, seq 1, length 64

    Here, the external server received the ping request and sent a ping reply. On the compute node, you can see that both the ping and ping reply successfully passed through. You might also see duplicate packets on the compute node, as seen above, because tcpdump captured the packet on both the bridge and outgoing interface.

    84) How will you Find a Failure in the Path ?


    • Use ping to quickly find where a failure exists in the network path. In an instance, first see whether you can ping an external host, such as If you can, then there shouldn’t be a network problem at all.
    • If you can’t, try pinging the IP address of the compute node where the instance is hosted. If you can ping this IP, then the problem is somewhere between the compute node and that compute node’s gateway.
    • If you can’t ping the IP address of the compute node, the problem is between the instance and the compute node. This includes the bridge connecting the compute node’s main NIC with the vnet NIC of the instance.
    • One last test is to launch a second instance and see whether the two instances can ping each other. If they can, the issue might be related to the firewall on the compute node.

    85) Visualizing nova-network Traffic in the Cloud


    If you are logged in to an instance and ping an external host, for example, Google, the ping packet takes the route shown in Figure. Traffic route for ping packet.

    OpenStack Interview Questions and Answers

    Figure. Traffic route for ping packet

    1. The instance generates a packet and places it on the virtual Network Interface Card (NIC) inside the instance, such as eth0.
    2. The packet transfers to the virtual NIC of the compute host, such as, vnet1. You can find out what vnet NIC is being used by looking at the /etc/libvirt/qemu/instance-xxxxxxxx.xml file.

    From the vnet NIC, the packet transfers to a bridge on the compute node, such as br100.
    If you run FlatDHCPManager, one bridge is on the compute node. If you run VlanManager, one bridge exists for each VLAN.
    To see which bridge the packet will use, run the command:

    • $ brctl show

    Look for the vnet NIC. You can also reference nova.conf and look for the flat_interface_bridge option.

    1. The packet transfers to the main NIC of the compute node. You can also see this NIC in the brctl output, or you can find it by referencing the flat_interface option in nova.conf.
    2. After the packet is on this NIC, it transfers to the compute node’s default gateway. The packet is now most likely out of your control at this point. The diagram depicts an external gateway. However, in the default configuration with multi-host, the compute host is the gateway.

    Reverse the direction to see the path of a ping reply. From this path, you can see that a single packet travels across four different NICs. If a problem occurs with any of these NICs, a network issue occurs.

    86) What is it that makes OpenStack Networking so special? Why does it matter?


    With Nova you can spin up virtual machines and it provides basic network capabilities. But when you want to use newer technologies, say, tunneling to provide network isolation between tenants or VXLAN – you can’t really leverage those with Nova networking. It limits you to VLAN’s or flat networking. The new technologies enable the biggest benefits of OpenStack Networking: scalable tenant isolation.

    87)  What has the OpenStack Networking community accomplished so far, and what are your plans for the Havana release?


    • OpenStack Networking was originally created at the Diablo Summit. It was an incubator project during Essex and it was integrated in Folsom. During the Essex and Folsom time frame the community really spent a lot of time trying to reach feature parity and build many L2 and L3 features into OpenStack Networking.
    • In Grizzly we were able to shift focus to adding new services, and also closing the parity gap with Nova Networking. In the Grizzly cycle we added overlapping metadata services, migrations, and security groups. Another big feature of Grizzly was load balancing.
    • As a matter of fact, several folks from Mirantis actually helped contribute to load balancing. That was a big community project that involved multiple people from various vendors who all worked together to produce an API and the foundation for load balancing.
    • In Havana we’re going to extend the load balancing service and add more features. Looking forward there are vendors in the community working to improve OpenStack Networking’s by adding VPN support, enterprise level ACL support, and IPv6 support. Right now the IPv6 functionality is pretty basic and folks want to add some high level services on top of that. Also there are companies and community members working on bare metal support, full multi-host support, providing HA in a little smaller context similar to what Nova multi-host is … also there are several community members who come together to work on other user facing features.

    88)  What is genuinely unique about OpenStack Networking or is it just an open source version of Networking as a Service as it already exists?


    I think the most unique thing about OpenStack Networking compared to almost any of the other Networking as a Service solutions is a very vibrant vendor community. During the Grizzly cycle we added five new plug-ins from different vendors. That’s one of the unique things about OpenStack Networking. It also shows a lot of vendor momentum because most of the vendors have chosen to put their energy and their efforts behind OpenStack.

    89)  What are some use cases where OpenStack Networking really shines?


    • In multi-tenant environments where isolation and security are a must you can get those systems up and running rapidly and provide those services to tenants, whether it’s a public or private cloud and you can get them running at scale fairly quickly.
    • In the case of smaller deployments OpenStack Networking can be configured to support even smaller private clouds fairly easily using open source tools. Smaller shops that have limited resources still can take advantage of many of the same features that the folks who are deploying at scale can as well.
    • If you were to compare that to other networking solutions in some other cloud stacks, they have maybe one or two options if you’re lucky. You’re pretty much stuck with the networking option of the stack.

    90)  How about the know-how base that you need to have to get OpenStack Networking up and running?


    • It’s the same set of skills that you would find if you had a network engineer or even a DevOps type of position. You really only need a basic familiarity for deploying IP networks.
    • So there’s really no special skills that are needed because a lot of what the plug-in authors have done – both open source and proprietary – is abstracted out a lot of the details of knowing the extreme specifics of certain protocols so that the deployer can focus on the API’s.

    91)  How about OpenStack Networking scalability?


    There are several large deployments running OpenStack Networking. Some are running versions of Folsom, and some people are actually running trunk which is really interesting because it speaks to the maturity of codebase.

    92) Who would you like to see contributing to OpenStack Network Project ?


    • We’re very fortunate. We have contributions from some very well respected companies including: Arista, BigSwitch, Brocade, Cisco, HP, IBM, NEC, Nicira, Juniper, Midokura, Plumgrid, and VMware. During the Grizzly cycle we added even more companies and some new start-ups who are offering their solutions so that drives innovation in the community.
    • As far as the ideal contributor … it’s somebody who is excited about networking and wants to participate in the OpenStack community, and is willing to trade ideas back and forth amongst the different contributors so that at the end of the day the community benefits as a whole.

    93) Does OpenStack Network Project still have any “childhood ailments”?


    We spent a lot of time in Grizzly working on isolated metadata services. A lot of the support questions we got and bug reports after the Folsom release were: How does metadata service work? – and so we spent a lot of time making sure that metadata service was a lot easier to configure and just worked out of the box for a wide variety of deployments. In Grizzly that’s probably one of the biggest diseases that we’ve gotten rid of.

    94) Are there any misconceptions about OpenStack Networking?


    • We don’t battle too many misconceptions with OpenStack Networking. I think most people understand what it does. Some folks will choose to still deploy Nova networking for new installations because they’re concerned about OpenStack Networking’s complexity, maturity or stability.
    • Now the support materials for installation have caught up and the distributions have done a really good job of packaging OpenStack, so that is no longer the case. For new deployments, you should use OpenStack Networking from the beginning and leverage those features now, versus starting with Nova-network and eventually having to migrate to OpenStack Networking, which is a non-trivial migration.
    • During Havana, the OpenStack Networking and Nova teams are going to be discussing: How do we bridge that gap and how does OpenStack Networking become the default network provider for Nova?

    95) When you say “fairly quickly,” can you quantify that?


    • For a smaller shop, just following the OpenStack guides – if you’re familiar with OpenStack – that would be half a day. If you’re unfamiliar with OpenStack, maybe a day or two. The guides will walk you through and get you set up with a pretty realistic set-up that works well for the majority of cases.
    • The nice thing is you can do that with the minimal level staff and it all runs on commodity hardware. You don’t need special switches or servers. For those who are trying to experiment and figure out if OpenStack or OpenStack Networking is the solution for them, they can use gear that most businesses have in their labs anyway for testing.

    96) What are your responsibilities? What do you find yourself doing on a day to day basis?


    • During the Grizzly release, I was a core contributor focusing on improving the metadata service functionality when using overlapping IP networks. I also worked on database migration so that folks who were deploying OpenStack can seamlessly upgrade from Folsom to Grizzly. During Grizzly, I also led several sub-teams including the L3, database, and bug triage teams.
    • Now as PTL, I take a much larger view of the project. I’m responsible for running our weekly team meeting and organizing the Network track at the design summit.  On a daily basis, I’ll correspond with the community, coordinate with sub-team leads, review code submissions, triage bugs, and review blueprints. I’ll also coordinate with other members of the Foundation and Technical Committee on cross-project issues.
    97) Explain about Troubleshoot OpenStack Networking Issues.


    This section discusses the different commands you can use and procedures you can follow to troubleshoot the OpenStack Networking service issues.

    Debugging Networking Device

    • Use the ip a command to display all the physical and virtual devices.
    • Use the ovs-vsctl show command to display the interfaces and bridges in a virtual switch.
    • Use the ovs-dpctl show command to show datapaths on the switch.

    Tracking Networking Packets

    • Use the tcpdump command to see where packets are not getting through.
      # tcpdump -n -i INTERFACE -e -w FILENAME
      Replace INTERFACE with the name of the network interface to see where the packets are not getting through. The interface name can be the name of the bridge or host Ethernet device.
      The -e flag ensures that the link-level header is dumped (in which the vlan tag will appear).
      The -w flag is optional. You can use it only if you want to write the output to a file. If not, the output is written to the standard output (stdout).
      For more information about tcpdump, refer to its manual page by running man tcpdump.

    Debugging Network Namespaces

    • Use the ip netns list command to list all known network namespaces.

    Use the ip netns exec command to show routing tables inside specific namespaces.

    • # ip netns exec NAMESPACE_ID bashs
    • # route -n

    Start the ip netns exec command in a bash shell so that subsequent commands can be invoked without the ip netns exec command.

    98) Why to troubleshoot ?


    • Complexity increases room for errors 
    • OpenStack – the software ○ Easy concept: “Just a bunch of python scripts with a nice WebGUI” ○ Yet complex: >20M LOC (including docs), ~65K commits in a year across ~60 projects 
    • OpenStack – the platform 
    • Deployed on hundreds / thousands of servers in a DC (horizontal complexity) 
    • Components layered on top of each other (vertical complexity) ○ Services communicate across clusters (mesh complexity) ○ Redundancy for high availability (temporal complexity)

    99) Best approach to troubleshooting


    • Avoid troubles! 
    • Monitoring, logging 
    • Alerting 
    • Blue-Green deployments 
    • Dev / staging environments
    • Infrastructure-as-code 
    • Log analytics, etc.

    Are you looking training with Right Jobs?

    Contact Us
    Get Training Quote for Free