This post should give beginners a rough overview over the storage concepts in a usual Proxmox installation. It will not go in depth on all possibilities and caveats.
Storage
By default, Proxmox uses LVM for its storage system. LVM is the Logical Volume Manager, which adds an additional logical abstraction layer between the hardware (the disks) and the file system.
There is the “local” storage, which stores the actual root filesystem of the PVE OS, as well as ISO images, backups, container templates and any other files.
The “local-lvm” storage on the other hand, stores the filesystems of your LXC containers and VMs. This one uses lvm-thin, which means only those blocks that are actually used will be written to disk. This allows for overprovisioning (assigning more space to the VMs / containers in total than the physical disk actually provides). Excerpt from the Proxmox Wiki: “LVM normally allocates blocks when you create a volume. LVM thin pools instead allocates blocks when they are written. This behavior is called thin-provisioning, because volumes can be much larger than physically available space. […] LVM thin pools cannot be shared across multiple nodes, so you can only use them as local storage.”
Thin overprovisioning can lead to problems, when the physical disk is full. You won’t get any errors, but your containers and systems will just produce disk I/O errors and data loss will occur. So it should generally be avoided.
Proxmox storage configuration is stored in the file /etc/pve/storage.cfg.
When working with LVM, you should know these acronyms and their meanings:
| VG | Volume Group |
| LV | Logical Volume |
| PV | Physical Volume |
In LVM, the hierarchy works like this: First, there are your usual partitions on a hard disk or SSD. LVM handles these partitions as physical volumes (PV). LVM also manages metadata areas for each of these PVs. Multiple PVs can be grouped into a volume group (VG). And within such a group, multiple logical volumes (LV) can be created. Only at the level of logical volumes the actual file system is created.
Useful storage commands
- lsblk: lists the available block devices in a tree view. Example: lsblk –output NAME,FSTYPE,LABEL,UUID,MODE
- fdisk -l: lists the partitions on the physical drive.
- pvcreate /dev/sdX: creates a physical volume based on the given physical drive.
- pvs or pvdisplay: print information about the available physical volumes.
- vgcreate [name] /dev/sdbX /dev/sdcY: creates a new volume group consisting of the given physical volumes.
- vgs or vgdisplay: print information about the available volume groups.
- lvcreate -n [name] -L[size] [vg-name]: creates a new logical volume with the name and size in the volume group. Example: lvcreate -n data -L1G vg00
- lvs or lvdisplay: print information aboute the available logical volumes.
- mkfs.ext4 /dev/[vg-name]/[lv-name]: creates an ext4 file system on the given logical volume. Note how the path to the drive resembles the hierarchy of VG and LV. Example: mkfs.ext4 /dev/vg00/data
- lvremove /dev/[vg-name]/[lv-name]: removes a logical volume.
- vgreduce [vg-name] /dev/sdXn: Removes a physical volume from a volume group. Example: vgreduce vg00 /dev/sda1
- pvremove /dev/sdXn: Removes the physical volume from LVM.
LVM attributes
When you use the commands pvs, vgs or lvs to show information about your volumes, you will likely see some letters which are attributes of that entity. Here is a full list what each attribute means.
Attributes of PVs
| a | physical volumes can be allocated to the volume |
| x | denotes that the volume group that the PV is attached to is exported (e.g. for migration to another system) |
Attributes of VGs
| r,w | read and/or write permissions |
| z | resizeable |
| x | the group is exported |
| p | partial |
| c,n,a,i | policy of the allocation (contiguous, cling, normal, anywhere, inherited) |
| c | cluster |
Attributes of LVs
| t | type of volume is “thin” |
| m | mirrored |
| M | mirrored, without intial sync |
| o | origin |
| p | pvmove |
| s | snapshot |
| S | invalid snapshot |
| v | virtual |
| i | mirror image |
| l | mirror image without sync |
| c | under construction |
| – | simple volume |
Increasing the size of an LVM volume
To extend the size if an LVM volume, proceed like this:
- (optional) Add a partition to the volume group if you need more physical space.
- Extend the size of the logical volume.
lvextend -L +2G /dev/mapper/your-volume - Extend the file system to match the size of the LV.
resize2fs -p /dev/mapper/your-volume
If this does not work, try the following command:
lvextend –resizefs -l +100%FREE /dev/mapper/your-volume
Increasing the size of an LVM thin pool
- Find the name of the thin pool. This can be achieved by using the “lvs” command.
- Extend the size of the pool like this:
lvextend -L +2G [vg-name]/[pool-name]