ASM disk group re-balancing manually

One of my favorite things about ASM is the ability to add and remove disks/luns from a diskgroup with no down time and an automatic rebalancing of the data on the luns.    This creates all kinds of possibilities.  I have even used this process to move an entire very large database from one SAN to another with no down time.

To remove a lun, use the following command:

SQL> alter diskgroup dgroup1 drop disk VOL004;
Diskgroup altered.

This will kick off a rebalance, that is moving data off of VOL004 and distributing it across the remaining disks. To see this operation in progress, query the v$asm_operation tables:

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -----------
           1 REBAL RUN           1          1        163      29082        576          50


Among the data from this query we can see the diskgroup, the operation and how much work is left to do. Also we can see the asm_power_limit of 1. This “power” is a function of the amount of resources that ASM will use to complete the operation. The higher the number, faster it will be done, but the more resources taken away from other processes. This range is 0-11. 11 you say? Why not 10? The truth is it was an inside joke referring to a scene in “Spinal Tap” and an amplifier that went not to 10, but to 11. The limit of 0 was first made available in 10gR2.


To alter the limit there are two possibilities, altering at the system level:

SQL> alter system set asm_power_limit=11;
System altered.

This will alter the power limit for any new operations, but not existing ones. To change an existing operation, you must directly alter the power limit on the diskgroup involved in the operation:

SQL> select * from v$asm_operation

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -----------
           1 REBAL RUN           1          1       4354      32842        665          42


SQL> alter diskgroup dgroup1 rebalance power 11;

Diskgroup altered.

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- -----------
           1 REBAL RUN          11         11        331      26085       2682           9

As you can see above, changing the power from 1 to 11 changed the estimated completion (EST_MINUTES) of the rebalance operation from 42 minutes down to 9. This is an estimate only and can vary quite a bit during an operation.

Note: The power can be set from the start in the add/drop statement with the following syntax:

SQL>  alter diskgroup dgroup1 drop disk vol005 rebalance power 11;
Diskgroup altered.


The big advantage to being able to set the power limit to 0 comes when it is necessary to add/drop several disks at the same time. If the power limit is already set, then each add/drop will be serialized and each operation will have to complete before the next can begin. If, however, the power limit is set to 0 first, then submit all the add/drop operations and finally set the power limit to a non-zero number for the disk group, the operations are paralellized and run at the same time. For large moves, such as SAN to SAN, this can take hours off a move, because all the operations are done together. For example, if you had to move 1TB from one SAN to another, if the power limit is non-zero when the luns from the new SAN are added and the luns from the old SAN are removed, then the add operation will run first rebalancing so there are 500G on each SAN when complete. Then the remove operation will start, rebalancing the remaining disk from the old SAN the new one. If parallelized, the entire operation is done at the same time moving the entire 1TB of data from the old SAN to the new SAN.

This ability is one of my favorite features of ASM. Power(limits) to the people!