QUESTION

How can I set up and configure a network aggregation under Solaris 10?

ANSWER
An aggregation is similar to an IPMP link created for failover, but has other advantages

• An aggregation is treated as one device for configuration and management, and uses only one IP address.
• In-bound as well as outbound load spreading is provided, so extra bandwidth is available.
• Automatic failover is provided, to the remaining working links in the aggregation
• Policies can be applied to the traffic flows.
• Parallel aggregations between servers can be created, without going through a switch

Early versions of Solaris 10 do not have this facility - you would need to download the Sun Trunking software to get it.

To show the available network interfaces

# dladmin  show-link
eri0            type: legacy    mtu: 1500       device: eri0
qfe0            type: legacy    mtu: 1500       device: qfe0
qfe1            type: legacy    mtu: 1500       device: qfe1
qfe2            type: legacy    mtu: 1500       device: qfe2
qfe3            type: legacy    mtu: 1500       device: qfe3

If the device shows up as type:legacy you cannot use it in an aggregation, therefore devices such as eri, hme and qfe are not supported in current Solaris production releases. In Opensolaris (and presumably later production releases) a new driver is included which does support the older devices.

Output from dladm will then show the following

# dladm    show-link
LINK  CLASS  MTU  STATE  OVER
eri0   phys  1500   up    --
hme0   phys  1500   up    --
qfe0   phys  1500  down   --

There are conditions to consider

• If you need to run an aggregation through a network switch, that switch must support aggregation technology.

• An interface that has been plumbed cannot become a member of an aggregation.

• Currently, interfaces must be of the GLDv3 type: xge, e1000g, and bge.

• NOTE: Later releases of Solaris 10 will support aggregations on what are now considered legacy devices such as eri. qfe and hme.

• All interfaces in the aggregation must run at the same speed and in full duplex mode.

To set up an aggregation using dladm, the basics steps are

# dladm  create -aggr  -d  bge0  -d  bge1  1

Create an aggregation number 1 from interfaces bge0 and bge1.

# ifconfig  aggr1   plumb  192.168.200.98  up

Brings the aggregation interface up.

# dladm   show-aggr
key: 1 (0x0001) policy: L4      address: 0:3:ba:7:84:5e (auto)

device   address           speed         duplex  link    state
bge0     0:3:ba:7:84:5e    1000  Mbps    full    up      attached
bge1     0:3:ba:7:84:5e   1000  Mbps    full    up      attached 

To add another interface to the aggregation

# dladm  add-aggr  -d  bge2  -d  bge3  1
# dladm  show-aggr  1
key: 1 (0x0001) policy: L4      address:0:3:ba:7:84:5e (auto)
           device       address                 speed           duplex  link    state
           bge0      0:14:4f:1:c8:b0   1000  Mbps    full    up      attached
           bge1      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached
           bge2      0:14:4f:1:c8:b1   1000  Mbps    full    up      attached
           bge3      0:14:4f:1:c8:b2   1000  Mbps    full    up      attached

To remove a device from the aggregation

# dladm  remove-aggr  -d  bge0  1
# dladm  show-aggr  1
key: 1 (0x0001) policy: L4      address: 0:3:ba:7:84:5e (auto)
           device       address                 speed           duplex  link    state
           bge1      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached
           bge2      0:14:4f:1:c8:b1   1000  Mbps    full    up      attached
           bge3      0:14:4f:1:c8:b2   1000  Mbps    full    up      attached

It is possible to set policies for aggregations, such as whether to use LACP protocol or not.
Here is an example which sets LACP mode to active, and the timer interval to short.

# dladm   modify-aggr  -l   active  -t  short  1

The default outbound load spreading is based on ports, but this can also be changed - see the dladm man page for full details.
Aggregations can be assigned to zones using IP instances (Update 4 onwards) - see zonecfg for details.
dladm can assign an aggregation temporarily to a zone, until the next reboot:-

#  dladm  set-linkprop   -t  -p  zone=apache  aggr1

The -t option must be used.
Use zonecfg to assign aggr1 permanently to the apache zone.

Show statistics

dladm also provides functions to display statistics, and not just for aggregations:-

# dladm  show-dev  -s  eri0
                ipackets  rbytes         ierrors opackets        obytes      oerrors
eri0            806693    122733634        0       808629     336565292         0

# dladm  show-aggr  -s
key: 1                   ipackets    rbytes      opackets   obytes          %ipkts %opkts
           Total        158030     31173663            12     504
           qfe1         132878     26014941            12     504            84.1    100.0
           qfe2          8818       1818890             0       0             5.6     0.0
           qfe3          8174       1671697             0       0             5.2     0.0
           qfe0          8160       1668135             0       0             5.2     0.0