Permission Denied While Starting VM Instance in Openstack

When working with OpenStack, you may encounter a “Permission denied” error when trying to start a VM instance on a compute node. This error typically appears in the Nova compute log and is caused by incorrect file ownership or permissions in the libvirt/QEMU configuration. In this guide, we’ll walk through the root cause and how to fix it.

Understanding the Error

The error surfaces when you attempt to start an existing instance using the OpenStack CLI or the legacy Nova command:

# Using OpenStack CLI
openstack server start <instance-name-or-id>

# Using legacy Nova CLI
nova start <instance-name-or-id>

Checking the Nova compute log on the affected compute node reveals the underlying issue:

$ tail -f /var/log/nova/nova-compute.log

ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
ERROR oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
ERROR oslo_messaging.rpc.dispatcher libvirtError: internal error: process exited while connecting to monitor:
  qemu-kvm: -chardev file,id=charserial0,path=/var/lib/nova/instances/<instance-uuid>/console.log:
  Could not open '/var/lib/nova/instances/<instance-uuid>/console.log': Permission denied

The key part of this error is: Could not open '...console.log': Permission denied. This means the QEMU process cannot access the instance’s console log file because it’s running as the wrong user.

Root Cause

This issue occurs when the QEMU process user and group in /etc/libvirt/qemu.conf are set incorrectly. In an OpenStack environment, Nova manages instance files under /var/lib/nova/instances/, which are owned by the nova user. If QEMU is configured to run as a different user (e.g., qemu or root), it won’t have permission to access these files.

A common scenario is when you’ve manually modified /etc/libvirt/qemu.conf while troubleshooting KVM issues and set:

# Incorrect configuration for OpenStack
user = "qemu"
group = "qemu"
dynamic_ownership = 0

With dynamic_ownership = 0, libvirt will not automatically change file ownership to match the configured user, which compounds the permission problem.

How to Fix the Permission Denied Error

Step 1: Stop the libvirt daemon

sudo systemctl stop libvirtd

Step 2: Edit the QEMU configuration

Open the QEMU configuration file:

sudo vim /etc/libvirt/qemu.conf

Find and update the user, group, and dynamic_ownership settings to:

user = "nova"
group = "nova"
dynamic_ownership = 1

This ensures that:

• QEMU processes run as the nova user, which owns the instance files under /var/lib/nova/instances/
• dynamic_ownership = 1 allows libvirt to automatically adjust file ownership as needed when starting VMs

Step 3: Fix ownership of existing instance files

If instances were previously started with the wrong user, their files may have incorrect ownership. Fix it with:

sudo chown -R nova:nova /var/lib/nova/instances/

Step 4: Restart libvirt and Nova compute

sudo systemctl restart libvirtd
sudo systemctl restart openstack-nova-compute

Step 5: Verify the fix

Try starting the instance again:

openstack server start <instance-name-or-id>

Monitor the log to confirm the instance starts without errors:

tail -f /var/log/nova/nova-compute.log

The instance should now boot successfully on the compute node.

Additional Troubleshooting Tips

If the issue persists after the above fix, check the following:

• SELinux context: On RHEL/CentOS systems, SELinux may block access. Check with sudo ausearch -m avc -ts recent and fix with sudo restorecon -Rv /var/lib/nova/instances/
• AppArmor profiles: On Ubuntu, ensure the libvirt AppArmor profile allows access to Nova instance directories
• NFS-backed instance storage: If /var/lib/nova/instances is an NFS mount, ensure it’s mounted with no_root_squash or proper UID mapping for the nova user
• Verify the nova user exists: Run id nova to confirm the user is present on the compute node

Conclusion

The “Permission denied” error when starting OpenStack instances is almost always caused by a mismatch between the QEMU process user and the file ownership under /var/lib/nova/instances/. Setting user = "nova", group = "nova", and dynamic_ownership = 1 in /etc/libvirt/qemu.conf resolves the issue in most cases.

More guides on OpenStack:

• Generate Rocky Linux 8 Qcow2 Image for OpenStack / KVM / Qemu
• Scale up Worker Nodes in OpenStack Magnum Kubernetes Cluster
• Deploy VM instance on OpenStack using Terraform
• Remove admin tenant compute quota limits in OpenStack