Backup related upgrade failure from 4.20.1/4.20.2 to 4.21.0/4.22.0 #12769
Description
problem
When upgrading from either 4.20.1 or 4.20.2 to 4.21.0 or 4.22.0, upgrade could fail when backup framework is in use.
An error like the following is seen,
2026-03-09 01:56:55,403 DEBUG [c.c.u.d.ScriptRunner] (main:[]) (logid:) -- Add diskOfferingId, deviceId, minIops and maxIops to backed_volumes in backups table
2026-03-09 01:56:55,403 DEBUG [c.c.u.d.ScriptRunner] (main:[]) (logid:) UPDATE `cloud`.`backups` b INNER JOIN `cloud`.`vm_instance` vm ON b.vm_id = vm.id SET b.backed_volumes = ( SELECT CONCAT("[", GROUP_CONCAT( CONCAT( "{\"uuid\":\"", v.uuid, "\",", "\"type\":\"", v.volume_type, "\",", "\"size\":", v.`size`, ",", "\"path\":\"", IFNULL(v.path, 'null'), "\",", "\"deviceId\":", IFNULL(v.device_id, 'null'), ",", "\"diskOfferingId\":\"", doff.uuid, "\",", "\"minIops\":", IFNULL(v.min_iops, 'null'), ",", "\"maxIops\":", IFNULL(v.max_iops, 'null'), "}" ) SEPARATOR "," ), "]") FROM `cloud`.`volumes` v LEFT JOIN `cloud`.`disk_offering` doff ON v.disk_offering_id = doff.id WHERE v.instance_id = vm.id )
2026-03-09 01:56:55,458 ERROR [c.c.u.d.ScriptRunner] (main:[]) (logid:) Error executing: UPDATE `cloud`.`backups` b INNER JOIN `cloud`.`vm_instance` vm ON b.vm_id = vm.id SET b.backed_volumes = ( SELECT CONCAT("[", GROUP_CONCAT( CONCAT( "{\"uuid\":\"", v.uuid, "\",", "\"type\":\"", v.volume_type, "\",", "\"size\":", v.`size`, ",", "\"path\":\"", IFNULL(v.path, 'null'), "\",", "\"deviceId\":", IFNULL(v.device_id, 'null'), ",", "\"diskOfferingId\":\"", doff.uuid, "\",", "\"minIops\":", IFNULL(v.min_iops, 'null'), ",", "\"maxIops\":", IFNULL(v.max_iops, 'null'), "}" ) SEPARATOR "," ), "]") FROM `cloud`.`volumes` v LEFT JOIN `cloud`.`disk_offering` doff ON v.disk_offering_id = doff.id WHERE v.instance_id = vm.id )
2026-03-09 01:56:55,475 ERROR [c.c.u.d.ScriptRunner] (main:[]) (logid:) java.sql.SQLException: Row 19 was cut by GROUP_CONCAT()
2026-03-09 01:56:55,495 ERROR [c.c.u.DatabaseUpgradeChecker] (main:[]) (logid:) Unable to execute upgrade script java.sql.SQLException: Row 19 was cut by GROUP_CONCAT()
at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:186)
at com.cloud.utils.db.ScriptRunner.runScript(ScriptRunner.java:88)
This is due to the following DB changes in the 4.20.1to4.21.0 upgrade path,
https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/resources/META-INF/db/schema-42010to42100.sql#L686-L709
The error is due to the length limit for group_concat_max_len in MySQL and it could be seen if there are backups for VMs having multiple volumes (possible 3 or more).
Workaround is setting value for group_concat_max_len:
SET GLOBAL group_concat_max_len = 1000000;
versions
Upgrade 4.20.x to 4.21/4.22