I have been working on a proof of concept project to check a cloud solution and tested CloudStack as one of the solutions.
Although setting up CloudStack seems simple and the installation documentation is good, there were issues that I encountered and took some time to figure out as the error logging does not always tell the cause of the issue.
I have tested CloudStack with KVM hypervisor – so some of the issues may be specific to this configuration.
What to check during installation
When using NFS storage as the primary storage, CloudStack will try to mount the storage on the physical hosts automatically by calling Libvirt and tell it to create a new storage pool with the NFS server host and path.
Libvirt will mount the primary storage NFS at /mnt/<CloudStack’s Primary Storage GUID> on the physical host.
So, when using an NFS storage, make sure that the primary storage is NOT left mounted or configured in /etc/fstab on:
- any of the physical hosts.
- the management server (not needed).
If you use NFS, make sure to choose NFS in the protocol drop down menu (avoid the SharedMountPoint option).
Primary storage protocol
When creating a primary storage in the GUI, make sure to choose NFS from the protocol drop down menu.
The SharedMountPoint option is available for KVM hosts that use OCFS2 or other shared file systems and NOT for NFS. You will need to specify the path where the mount point is on the physical host (for example: /mnt/primary). When this option is specified, CloudStack will not try to mount the primary storage on the physical hosts.
Note: If ShareMountPoint is specified and it points to an existing NFS mount point, CloudStack will get confused:
It will ask Libvirt to define a storage pool for /mnt/primary (or the mount point path that was specified in the GUI) and also ask libvirt to define another storage pool again under /mnt/<CloudStack’s Primary Storage GUID>. Libvirt will fail and will put the physical host into the avoid set.
The secondary storage is an NFS share that is used to store CloudStack’s VM templates.
When configuring secondary storage in the GUI:
- In “NFS Server” field – put the IP of the NFS server as the host name (DNS will not work well in CloudStack 4).
- Put the path to the files on the NFS server.
Make sure not to mount the secondary storage on the physical hosts (or have an entry for it in /etc/fstab) – CloudStack will mount the storage automatically.
Using physical host’s local storage as primary storage
If you have plenty of storage on the physical host, you can store the VM images on the host itself instead of on a shared storage that might be slower (like NFS).
The downside of this is that it will not be possible to move VMs from one physical host to another.
To enable storing VM images on the local disks, go to “Global Settings” in CloudStack’s GUI and enable:
- system.vm.use.local.storage = true
- use.local.storage = true
When enabled, CloudStack will ask Libvirt to create a storage pool that points to “/var/lib/libvirt/images”. Make sure to create a symlink from /var/lib/libvirt/images to a disk or partition that has enough room for the VM images.
VM expunge values
When a VM is being deleted, CloudStack will keep its details in the database as well as it’s disk images on the primary storage for 24 hours.
If you want to shorten that period, go to “Global Settings” in the GUI and set the expunging delay to 60 seconds or other value:
- expunge.delay = 60
- expunge.interval = 60
Issue #1: When trying to remove a POD receive an error: “There are private IP addresses allocated for this POD”
Check that the IP range is not already taken by other computers on the network.
Sometimes, CloudStack will keep old VMs in the database and will not release their IPs until they are expunged (see above about shortening the expunge period).
There are two options here:
- If it is not a production environment, the fastest option is to drop the database and recreate it.
- For production environments:
Check which hosts are left over in the host table for a given POD:
select * from host where pod_id = ’1′;
delete from host where pod_id = ’1′;
See what is left for a given host in the op_ha_work table:
select * from op_ha_work;
delete from op_ha_work where host_id = ’1′;
Remove the host from the vm_instance table:
delete from vm_instance where pod_id = ’1′ and host = ’1′;
Issue #2: In management-server.log: GetRootDir for nfs://nfs-server/secondary failred due to java.net.UnknownHostException
Put the full name as the secondary storage host, or use IP instead.
Issue #3: Templates stuck during download
Restart the secondary storage VMs.
Issue #4: Unable to create system VMs or other VMs. In the management-server.log: Host name: phys04, hostId: 1 is in avoid set, skipping this and trying other available hosts
- Restart the cloud-management service.
- Check that other computers on the network do not occupy IPs in CloudStack’s IP range for VMs or system VMs.
If you liked this article, please share it on Digg, Stumble Upon or del.icio.us. Thank you.