Presentation #: 380

Building Basic Configurations with the HP FC-AL Storage Enclosure

William Hooper and Thomas Moore

Hewlett Packard

8000 Foothills Blvd.

Roseville, CA 95747

Phone: 916/785-5135

Fax: 916/785-5943

bill_hooper@hp.com

Abstract:

Fibre Channel provides new features and functionality that are valuable in creating powerful and flexible storage configurations. While these new features and functionality can cause Fibre Channel to appear to be complicated and confusing, it actually isn't. Basic Fibre Channel configurations can be constructed that are simple and very effective. Specific Fibre Channel configurations can also allow for high availability, provided these systems are constructed correctly.

By presenting some basic configurations, we can show the simplicity of Fibre Channel and the ease in which we can create high availability solutions. Including illustrations and providing HPUX commands will help the user in producing these configurations and provide a foundation for creating solutions that are more complex.

Fibre Channel Features

Fibre Channel brings a new level of flexibility to mass storage configurations, but unfortunately, a new level of complexity also appears. Fibre Channel presents a variety of technological advancements over Parallel SCSI and it allows for new configurations that were not previously possible.

With Parallel SCSI, configurations are rather simple. Because of cable distances of at most 12 meters, host systems and disks are never far from each other. A Parallel SCSI configuration can have only 15 devices per host adapter and many configurations contain an average of only 8 disks. Since nearly all available multiple disk enclosures can contain at least eight disks, the entire configuration is usually comprised of a single enclosure.

Using FC-AL (Fibre Channel Arbitrated Loop) a very simple configuration can be created that would be very similar to a basic Parallel SCSI configuration. However, with FC-AL, the number of possible devices on a single host adapter rises to 126 and cable distances have increased to 500 meters and greater. With a disk enclosure that can contain up to 10 disks, a single FC-AL configuration could involve up to12 enclosures along a 500 meter cable.

FC-AL disks also provide two independent connections, or ports, unlike Parallel SCSI disk devices that provide a single SCSI connection. These two ports allow for greater performance and improved availability. This feature can also create additional complexity.  A host system with two FC-AL adapters can now connect to a dual-port disk enclosure and be able to have  increased availability and performance. A dual loop configuration can provide an enhanced level redundancy, such that if a host adapter experiences a fault or a cable becomes disconnected the host can still access the disks via the other FC-AL adapter.

HP FC-AL Storage Enclosure

The HP FC-AL storage enclosure provides power, cooling, and protection for Fibre Channel disks. In addition, the enclosure provides external LED indicators about the state of the enclosure and devices, a built-in hub and redundant SCSI Enclosure Services devices. The front of the enclosure provides slots for 10 disks along with LED status indicators.

Figure 1 HP FC-AL Enclosure Front

At the rear of the HP FC-AL Storage Enclosure, there are the two power supplies, two fans and two Link Control Cards (LCC). There is one LCC for each loop within the enclosure. Each LCC provides a built-in hub with a Primary and Expansion port, along with a SCSI Enclosure Services  (SES) device. The SES device provides enclosure information and status to the host system.

Figure 2 HP FC-AL Enclosure Rear

With the built-in hub, additional enclosures can be easily attached without external hubs or additional cost. Configurations can be created by attaching the Primary port of the first enclosure to the host adapter and attaching the Expansion port of the first enclosure to the Primary port of the second. Additional enclosures can be attached in the same fashion. Also included in the embedded hub design is the ability to mix optical and copper interconnects. This makes it possible to improve costs and ease configurations by using optical to connect the host to the first enclosure and using copper to connect the remaining expansion enclosures.

Figure 3 Enclosure Primary and Expansion Connections

The first and most basic Fibre Channel configuration uses a host system with a single FC-AL host adapter and a single FC-AL disk enclosure. This configuration is very similar to a Parallel SCSI configuration, with the added features of up to 100 disks, a 100 MB/s interconnect and long-distance capabilities up to 10 kilometers.

Basic Fibre Channel Configuration

The following figure shows the most basic Fibre Channel configuration. Because this is a very basic configuration there is no availability provided. The enclosure does provide some protection with redundant power supplies and fans, but any access to the disks, though, could be terminated as a result of a disk fault, a host adapter or LCC fault, or a disconnected cable. This configuration would typically be used for boot, swap or temporary disk, or file space.

Figure 4 Basic Enclosure Configuration

A Fibre Channel cable is used to connect the host FC-AL adapter to the Primary port of the loop A LCC. None of the other Fibre Channel connections are used.

Using this configuration with ten disks installed within the enclosure, executing an ioscan command produces the output shown in Figure 5.

Figure 5 Basic Enclosure Configuration ioscan Output

Figure 6 Basic Enclosure Configuration Using pvcreate

The commands vgcreate and vgextend are then used to create a volume group that contains a group of physical volumes.

Figure 7 Basic Enclosure Configuration Using vgcreate and vgextend

In the example shown in Figure 7, the vgcreate command is used to create a volume group using a single physical volume. The vgextend command is then used to extend the volume group with the remaining four physical volumes.

The command lvcreate is used to create individual logical volumes within a volume group, as shown in Figure 8. If a logical volume needs to be modified or additional space added the lvextend command is used to perform these tasks.

Figure 8 Basic Enclosure Configuration Using lvcreate

Additional HP-UX commands can be used to create a file system on each logical volume and mount the volume in the appropriate location.

Figure 9 Basic Multiple Enclosure Configuration

With this first configuration, additional enclosures may be added to the Fibre Channel loop. This is done by connecting the Expansion port of the first enclosure with the Primary port of the second enclosure and so on until all of the enclosures are connected.  Up to ten HP FC-AL enclosures may be chained together in this manner.

Basic Mirrored Fibre Channel Configuration

Using the basic Fibre Channel configuration, a mirrored environment can be easily built. Although this is still a basic configuration, high availability is provided through the mirrored enclosures. Each enclosure provides additional protection with redundant power supplies and fans. Access to one side of the mirror could still be terminated with a disk device fault, a host adapter or LCC fault, or even a disconnected cable. The mirrored enclosure pair provides fail-over access and data protection when these events occur.

Figure 10 Basic Mirrored Enclosure Configuration

Two Fibre Channel cables are used to connect each host FC-AL adapter to the Primary port of the loop A LCC on each enclosure. None of the other Fibre Channel connections are used.

Using this configuration with ten disks installed within each enclosure, executing an ioscan command produces output as shown in Figure 13. Note that this output is similar to Figure 4 with the exception that twenty disks are listed in the output. The disk devices 2 through 11 are contained within a single enclosure and attached to one host adapter. The disk devices  12 through 21 are contained within the other enclosure and attached to the remaining host adapter.

Here again the command pvcreate is used to create physical volumes for each device. The commands vgcreate and vgextend are then used to create a volume group that contains a group of physical volumes.  At the time that this volume group is created two physical volume groups must be produced. The first physical volume groups must include disk devices from one enclosure and the second physical volume groups must contain disk devices from the second enclosure. These two physical volume groups must be created in this fashion to allow for PVG-strict mirroring. In the example shown in Figure 11 these two physical volume groups are created and identified as PRIMARY-1 and MIRROR-1 respectively.

Figure 11 Basic Mirrored Enclosure Configuration Using vgcreate

The command lvcreate is used to create individual logical volumes with their mirrored copy as shown in Figure 12. If a logical volume needs to be modified or additional space added the lvextend command is used to perform these tasks.

Figure 12 Basic Mirrored Enclosure Configuration Using lvcreate

Class     I  H/W Path            Driver      S/W State   H/W Type     Description
==================================================================================
disk     12  8/4.8.0.255.0.0.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t0d0   /dev/rdsk/c8t0d0
disk     13  8/4.8.0.255.0.1.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t1d0   /dev/rdsk/c8t1d0
disk     14  8/4.8.0.255.0.2.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t2d0   /dev/rdsk/c8t2d0
disk     15  8/4.8.0.255.0.3.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t3d0   /dev/rdsk/c8t3d0
disk     16  8/4.8.0.255.0.4.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t4d0   /dev/rdsk/c8t4d0
disk     17  8/4.8.0.255.0.5.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t5d0   /dev/rdsk/c8t5d0
disk     18  8/4.8.0.255.0.6.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t6d0   /dev/rdsk/c8t6d0
disk     19  8/4.8.0.255.0.7.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t7d0   /dev/rdsk/c8t7d0
disk     20  8/4.8.0.255.0.8.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t8d0   /dev/rdsk/c8t8d0
disk     21  8/4.8.0.255.0.9.0   sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c8t9d0   /dev/rdsk/c8t9d0
disk      2  8/12.8.0.255.0.0.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t0d0   /dev/rdsk/c7t0d0
disk      3  8/12.8.0.255.0.1.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t1d0   /dev/rdsk/c7t1d0
disk      4  8/12.8.0.255.0.2.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t2d0   /dev/rdsk/c7t2d0
disk      5  8/12.8.0.255.0.3.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t3d0   /dev/rdsk/c7t3d0
disk      6  8/12.8.0.255.0.4.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t4d0   /dev/rdsk/c7t4d0
disk      7  8/12.8.0.255.0.5.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t5d0   /dev/rdsk/c7t5d0
disk      8  8/12.8.0.255.0.6.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t6d0   /dev/rdsk/c7t6d0
disk      9  8/12.8.0.255.0.7.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t7d0   /dev/rdsk/c7t7d0
disk     10  8/12.8.0.255.0.8.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t8d0   /dev/rdsk/c7t8d0
disk     11  8/12.8.0.255.0.9.0  sdisk       CLAIMED     DEVICE       SEAGATE ST118202FC
                                /dev/dsk/c7t9d0   /dev/rdsk/c7t9d0

Figure 13 Basic Mirrored Enclosure Configuration ioscan Output

Additional HP-UX commands can be used to create a file system on each logical volume and mount the volume in the appropriate location.

Figure 14 Basic Mirrored Multiple Enclosure Configuration

Expansion can be achieved by chaining additional enclosures to each host adapter, as shown in Figure 14. Note that in a mirrored configuration additional disks must be added in pairs, one for each host adapter.

Dual Port Fibre Channel Configuration

The next configuration requires a host system with two FC-AL host adapters and a single FC-AL disk enclosure. This configuration takes advantage of the dual port feature of Fibre Channel.

Additional availability can be achieved in Fibre Channel based systems by cabling the two loop connections through the enclosure with the system's two FC-AL host adapters. This configuration provides two independent paths for data access. When a host is connected to a dual port device, if a fault on one loop occurs the host can still access the attached devices over the other loop. When both loops are used simultaneously, a combined performance of 200 MB/s can be achieved.

Because this is a very basic configuration limited availability is provided. The enclosure provides some protection with redundant power supplies and fans, and access to the disks could persist with a host adapter or LCC fault or a disconnected cable. No protection, though, is provided  in the event of a disk device fault. This configuration would typically be used for boot, swap or temporary disk, or file space.

Figure 15 Basic Dual Loop Enclosure Configuration

Using this configuration with ten disks installed within the enclosure, executing an ioscan command produces output identical to Figure 13 with twenty disks listed in the output. It must be realized though that there are not twenty different disks attached to the system, but rather ten disks attached through two host adapters.  Of the disk devices identified in Figure 13, disks 2 through 11 would be connected to one loop within the enclosure and attached to one host adapter. The disk devices 12 through 21 would then be the same disks within the enclosure connected via the second loop and attached to the remaining host adapter.

As in previous configurations, the command pvcreate is used to create physical volumes for each device. The commands vgcreate and vgextend are then used to create a volume group that contains a group of physical volumes. It is recommended that the volume group be first created with disks attached to the same host adapter. The vgextend command can then be used to include the disk devices attached to the second host adapter (actually the same disk devices attached to the first host adapter). The volume manager will automatically recognize each of these disks as having an alternate connection, or link, through the second host adapter.  If during the course of operation, the primary link to a disk experiences a fault, the volume manager will continue accessing the disk via the alternate link.

The command lvcreate is then used to create individual logical volumes within a volume group. If a logical volume needs to be modified or additional space added, the lvextend command is used to perform these tasks.

Additional HP-UX commands can be used to create a file system on each logical volume and mount the volume in the appropriate location.

As with other the configurations additional enclosures may be added to both Fibre Channel loops. Connecting the Expansion port of the first enclosure with the Primary port of the second enclosure and so on does this.  Up to ten enclosures may be chained together in this manner.

Figure 16 Basic Dual Loop Multiple Enclosure Configuration

Mirrored Dual Port Fibre Channel Configuration

Disk drives within the enclosure have dual Fibre Channel ports that make them perfect for high availability configurations. Utilizing the dual ports in a Fibre Channel configuration a mirrored environment that provides greater availability can be readily constructed.

As with the basic mirrored configuration, high availability is provided through the mirrored enclosures. Each enclosure provides additional protection with redundant power supplies and fans. Access to one side of the mirror could still be terminated with a disk device fault, a host adapter or LCC fault, or even a disconnected cable. A dual port mirrored enclosure pair would provided fail-over access and data protection when these events occur.

In this configuration, both loops are connected to all disks. This is a fundamental difference from the basic mirrored configuration shown in Figure 10. When properly configured, the volume manager will automatically recognize each of these loops as having an alternate link through the appropriate host adapter. During the course of operation, when the primary link to a disk experiences a fault, the volume manager will continue accessing the disk via the alternate link. While this may mean a slight reduction in performance when a fault occurs, the integrity of the mirrored pair is maintained. This prevents a costly rebuild that would occur in basic mirrored configuration when a faulted component has been replaced.

Figure 17 Mirrored Dual Loop Enclosure Configuration

Using this configuration with ten disks installed within the enclosure, executing an ioscan command produces output similar to Figure 13 with forty disks listed in the output. It must be realized that there are not forty different disks attached to the system, but rather twenty disks attached through two host adapters

As in previous configurations, the command pvcreate is used to create physical volumes for each device. The commands vgcreate and vgextend are then used to create a volume group that contains a group of physical volumes. It is recommended that the volume group be first created with disks attached to the same host adapter. The vgextend command can then be used to include the disk devices attached to the second host adapter (actually the same disk devices attached to the first host adapter). The volume manager will automatically recognize each of these disks as having an alternate link through the second host adapter. If during the course of operation, the primary link to a disk experiences a fault, the volume manager will continue accessing the disk via the alternate link.

The command lvcreate is then used to create individual logical volumes within a volume group. If a logical volume needs to be modified or additional space added, the lvextend command is used to perform these tasks.

Additional HP-UX commands can be used to create a file system on each logical volume and mount the volume in the appropriate location.

Highly Redundant Mirrored Dual Loop Configuration

This last configuration is a combination of the basic mirrored configuration and the dual loop configuration. High availability is provided in a variety of ways. The enclosure itself, with redundant components, provides a basic level of hardware availability. Fault protection is provided with redundant Link Control Cards, power supplies and fans within each enclosure. The mirrored enclosure disk pairs provide fail-over access and data protection when a single disk fault occurs.

Providing two host adapters for each portion of the mirror permits multiple data paths that provides an increased level of redundancy. The dual loops insure that access to the disks would persist with a host adapter or LCC fault or a disconnected cable. When both loops are used simultaneously, a combined performance of 200 MB/s can be achieved.

Figure 18 Highly Redundant Mirrored Dual Loop Configuration

As with all of the previous configurations the command pvcreate is used to create physical volumes for each device. The commands vgcreate and vgextend are then used to create a volume group that contains a group of physical volumes.

As with any mirrored configuration, it is recommended that the volume group be first created with disks attached to the same host adapter. The vgextend command can then be used to include the disk devices attached to the second host adapter (actually the same disk devices attached to the first host adapter). The volume manager will automatically recognize each of these disks as having an alternate link

The command lvcreate is then used to create individual logical volumes within a volume group. If a logical volume needs to be modified or additional space added, the lvextend command is used to perform these tasks.

Additional HP-UX commands can be used to create a file system on each logical volume and mount the volume in the appropriate location.

Summary

Fibre Channel provides new features and functionality that are valuable in creating powerful and flexible storage configurations. It is not necessary, though, for these new features and functionality to be complicated or confusing. With the illustrations, it has been shown that simple and very effective basic Fibre Channel configurations can be produced. By constructing these systems correctly, basic configurations can be created that provide availability, flexibility and functionality.

With these basic configurations, we have shown the simplicity of Fibre Channel and the ease in which we can create high availability solutions. The illustrations and descriptions of pertinent HPUX commands will help the user in producing these configurations and provide a foundation for creating solutions that are more complex.

Acknowledgements

Many thanks to Sami Waheed for reviewing this paper and actually building and testing each illustrated configuration.
Biographical Sketch:

William Hooper is a Storage Systems Architect working for HP's Enterprise Storage Solutions Division. William has been working at HP for eleven years and has a BS in Computer Science from California State University, Sacramento.

Tom Moore Jr. is a Product Manager working at HP's Enterprise Storage Solutions Division. Tom has been working at HP for ten years, focusing primarily on disks and high availability solutions. Tom has a BS in Computer Science from Eastern Washington University.