Highly Available Storage:
Multipathing and the Microsoft MPIO Driver Architecture
White Paper
Published: October 2003
Abstract
High reliance on the Internet and e-commerce means that organizations are increasingly requiring that their data be available 24x7, year round. Multipathing, the ability of a system to use more than one read/write path to a storage device, is a high availability solution that provides fault tolerance against single-point-of failure in hardware components. Multipathing can also provide load balancing of I/O traffic, thereby improving system and application performance.
This white paper is of interest to the storage administrator who is ultimately responsible for ensuring that an organization’s data remains highly available, and to storage vendors who target multipathing solutions using the Microsoft MPIO (multipath input/output) solution in Windows 2000 Server, Windows Server 2003, and Windows Storage Server 2003. The paper begins with a discussion of the situations and environments in which multipathing is a smart business solution, then describes the critical factors that must be part of an effective multipathing solution, and concludes with a brief overview of the driver-level details of the Microsoft MPIO solution.
The Microsoft MPIO driver development kit is not available as part of the Windows Driver Development Kit; it can be licensed by hardware and software vendors following execution of a Program Agreement. Organizations that seek Windows-based multipathing solutions should check with their vendors to ensure that their storage devices are Microsoft MPIO-compatible.
Contents
Introduction 1
High Availability Solutions 2
Application Availability through Server Clustering 2
Storage Redundancy through RAID 2
Storage Availability through Multipathing 2
How Data Becomes Unavailable 3
Failure Modes 3
Points of Failure 3
Multipathing Solutions 6
Third Party Solutions 6
Operating System-Based Solutions 6
A Joint Solution: MPIO and Device-Specific Modules 6
Making Multipathing Solutions Work 8
Device Discovery and Enumeration 8
Dynamic Load Balancing 9
Error Handling, Failover and Recovery 9
The Windows Storage Stack and Drivers 10
Storage Stack 10
Device Drivers 11
MPIO Drivers 11
MPIO DSM 12
Conclusion 13
Resources 14
The most commonly implemented high availability storage solution is to provide disk redundancy through using a redundant array of independent disks (RAID). While RAID-1 (mirroring) provides excellent fault tolerance, it only protects the disks. If there is only a single path from the server to the storage device, however, and one of the components in that path fails, no amount of disk redundancy can keep the data available.
Multipathing solutions, in contrast, are designed to provide failover through the use of redundant physical path components—adapters, cables, and switches—between the server and storage device. In the event that one or more of these components fails, applications can still access their data. Fault tolerance is not the only benefit of multipathing solutions; multipathing software also serves to redistribute the read/write load among multiple paths between the server and storage, thereby helping to remove bottlenecks and to balance work loads.
Previously, the Windows operating system did not have a native Windows architecture to provide failover or load balancing features. Without third party software, if more than one path to a storage device existed, there was no way for the operating system to correctly interpret that multiple storage devices were in fact a single unit. Instead, the operating system “saw” as many devices as there were paths to it. However, even with third party software, multipathing solutions in the Windows environment often failed. Developing multipath solutions that work effectively with the operating system is a complex undertaking. It is even more challenging when an organization needs to support storage devices from multiple vendors.
The release of Microsoft’s multipathing drivers (Microsoft MPIO) allows software and hardware vendors to develop solutions that are not only specific to their storage devices, but also work effectively with the Microsoft® Windows® 2000 Server and Microsoft® Windows Server™ 2003 platforms—including Microsoft® Windows® Storage Server 2003—to provide high availability and high performance solutions. Microsoft MPIO (multipath input/output) drivers are written and maintained by Microsoft to ensure close interoperability with the Windows operating system. Moreover, Microsoft MPIO solutions are designed to work properly when installed on the same server as an existing Microsoft MPIO solution from another vendor.
Keeping mission-critical data continuously available has long been critical in the enterprise environment, where the goal for many organizations is continuous availability—the so-called “five nines” standard of server availability—99.999 percent uptime or only a few minutes of unplanned downtime in a year of 24X7 access. Even for enterprises, this level of availability is very difficult and very costly to achieve. High availability, defined as 99.9% uptime (which translates into several hours downtime per year), is a more attainable goal. Both standards require that redundancy be built in at all levels: disk redundancy, backups to separate recovery servers, server clustering, and redundancy of the path components.
Clustering is the use of multiple computers, interconnects, and storage devices that work together to provide users with high application availability from what looks like a single system. If a computer system goes down, or is temporarily unavailable, end users can still access their data following failover to an active system. Clustering solutions require software that enables transparent failover between systems. Microsoft Cluster Service (MSCS) is one such solution and it is included with Windows Server 2003, Enterprise Edition, and Windows Server 2003, Datacenter Edition.
High availability solutions based on redundant arrays of independent disks (RAID) have been in use with mainframe equipment for several decades, and have been supported in the Windows operating system since the first release of Windows NT. RAID solutions provide protection through the use of redundant disks, which can be configured for both fault tolerance and improved performance. (For a discussion of how RAID solutions work in Windows Server 2003 and Windows Storage Server 2003, see the white paper “Storage Management using Windows Virtual Disk Service and Volume Shadow Copy Service,” issued September 2003.)
Although both multipathing and clustering result in high availability and improved performance, they are not equivalent concepts. While clustering provides high application availability and can be implemented using a pathing solution, multipathing provides high storage availability and does not require a clustering solution.
Multipathing is redundancy of the storage network components only—the cabling, adapters, and switches—that transfer data across the path between the server and storage. Multipathing solutions manage these redundant connections so that read/write requests can be instantaneously rerouted in case a path fails.
Because consolidated resources are inherently at higher risk, implementation of multipathing solutions is especially important for storage administrators when storage resources are consolidated on network attached storage (NAS) devices or storage area networks (SANs).
To design highly available server, network, and storage infrastructures, it is necessary to understand the ways in which applications can fail to access storage. Failure to access to data can be ascribed to two sources:
· The root underlying cause—the failure mode—whether software or hardware
· The component where the point of failure occurs
Failure modes are the ways in which a system (or a system component or process) can potentially fail. Servers can become unavailable for a variety of reasons, ranging from component failures, configuration problems, application and system errors, and excessive network traffic, to just plain human error, such as accidentally unplugging a cable.
Whether or not a failure mode is fatal depends both on the type of failure (some failures can be re-tried and the read/write request will then succeed) and whether or not the hardware of a specific vendor is designed to help with error recovery. In the case of fatal errors where there is no chance that the I/O request can succeed, a multipathing solution ensures that the I/O request can be attempted and completed on a different path.
· Data corruption. Often caused by hardware faults or by incorrect software or firmware, incorrect data is read from or written to media, usually without a recorded indication of an error. This constitutes the most serious server failure mode, since it is almost impossible to troubleshoot; these types of errors are usually fatal.
· File system corruption. Typically due to a failed I/O that results in a partial update of data, this type of fault leaves the file system in an inconsistent state. With a journaled file system, such as NTFS, this error is usually detected; however, it is possible that the application state cannot be rolled back, and the error can result in loss of data. In some cases, the error can be severe enough to result in loss of the entire volume.
· Specific hardware errors. An I/O fails due to a recognized hardware error. Whether or not these errors are fatal depends on the specific hardware error. Since these faults can be tracked, the risk for this type of error can be minimized.
· I/O Problems. Read/write requests remain active, yet fail to reach completion. These difficulties can be caused by improperly coded internal retry queues within different components of the driver stack, such as vendor-supplied multipathing drivers, class drivers, mini-ports or port drivers. These errors usually require manual intervention and can result in loss of data. I/O requests can also fail completely. It is not generally possible to determine the exact nature of these failures beyond general device I/O failure. These errors are usually fatal.
· Soft errors. These are correctable (non-fatal) errors, such as disk block read problems, which might either be retried or reconstructed automatically. For long-term reliability, it is desirable to keep track of these errors. In many cases, if frequent soft errors are encountered, the application may not be able to keep up with the workload and then performance is lowered to unacceptable levels.
Each component in the I/O path is a potential point of failure. Such components include the server (host) components, storage path components—such as HBAs, cable and switches—and the storage array or its components. In addition, each component, while it may not completely or permanently fail itself, still has the potential to impact layers upstream in such a way that I/O failure, improper path failover or catastrophic faults may result.
Server failure can result from a number of common failure modes, including hardware failure and software failure (I/O timeouts or application induced errors). Servers can be protected against such failures by providing redundancy through clustering.
· Bus. Most hardware platforms now have multiple buses. Consequently, redundant hardware (from interface down) should be segregated to different buses wherever possible. While bus failure is not common, it is common for a device on a bus to cause problems for other interfaces (such as improper PCI behavior or bad driver behavior). Except in the case of hardware failure, it is generally difficult to determine exactly why a single bus device is malfunctioning (whether because of I/Os timeouts or failed I/O); however, disabling one or more devices on a bus can allow the server to continue to function. Most bus failures will be seen as a server failure except on the highest end server lines.
· Interface. The main goal of multipathing is to allow multiple interfaces to the same storage unit (LUN). Here, interface refers to an I/O card in a bus, such as a Fibre Channel adapter. Although fairly reliable, failures—stemming from hardware problems, I/O problems, timeouts, and data corruption—do occur. For highest availability, servers should use redundant HBAs, and possibly Hot-Plug PCI systems with compatible adapters and drivers, allowing these components to be replaced without bringing down the server.
There are numerous components in the storage path that can fail. For this reason, multipathing is a particularly critical solution for ensuring highly available data.
· GBIC and SFP. Common physical interface components in Fibre Channel SANs are the GBIC (gigabit interface converter) or the newer SFP (small form factor package, pluggable transceiver), modules that convert electrical signals to be propagated via fibre optics or copper cables. Historically, GBIC modules have had a high failure rate, although SFPs represent a substantial reliability improvement. Failures are generally manifest as either complete loss of signal on a path or very high error rates. Following a module failure, path failover should occur, either automatically or manually once the bad module has been located. Replacement of the GBIC or SPF does not require server reboots because these devices are inherently hot pluggable (unless there is only a single path to the device behind the GBIC/SFP, in which case reboot is required).
· Cabling. Cable failures are very common for out-of-cabinet storage. Poor contact on large multi-conductor parallel SCSI cables and dirty optics or overstressed fibers on fiber cables are not unusual. Resulting faults may or may not result in path failure, although almost certainly such faults will result in high error rates which can result in diminished performance (throttling). The most likely failure modes are manifested as hardware problems or and I/O timeouts.
· Switches. Fibre Channel SANs can be configured using either hubs or switches. Both switches and hubs can fail because of faulty hardware or network connectivity problems. Enterprises are usually discouraged from using hubs as they have a number of drawbacks, including shared bandwidth, and possible management complexity. Additionally, hubs can create addressing problems if there is a disruption of a Fibre Channel Arbitrated Loop.
Switches, on the other hand, can provide several advantages: Each port generally has full bandwidth (point-to-point); switches are readily managed (and in fact often have an embedded management server that provides management functions). Moreover, depending on the model and configuration, switches can prevent disruption of Fibre Channel arbitrated loop. Much of this functionality is price related: Low-end switches, for instance, act like hubs—they may not provide high bandwidth performance and they still rely on arbitration. Mid-level switches provide higher bandwidth and include management features; some include redundant power supplies as well. High-end switches (also called “Director” class) can provide complete fault tolerance with highly redundant components and automatic path failover within the switch, as well as the ability to update the switch firmware without disruption I/O. Note, however, that a switch failover can result in other components, such as drivers, misbehaving. These switches require the same level of intelligent path management as would high-end RAID storage units.
Storage devices—whether JBODs (just a bunch of disks) or RAID storage arrays—can fail as a result of hardware problems, I/O problems, timeouts, data corruption, or soft errors.
· RAID subsystems. For enterprise discussions, high-end RAID subsystems are the norm. These systems are highly redundant, with multiple ports, processors and power supplies. Failure modes of these units merit very careful attention when designing recovery behaviors. Some troublesome areas are controller failover (either due to error or temporary disruption during updating procedures), RAID performance (particularly during disk error recovery) and spare drive behavior (when a spare becomes active, one or more LUNs may suddenly become read-only in the middle of operations).
· JBOD. For some applications, JBOD arrays are specified, but they too must be constructed with redundant components such as power and cooling. JBODs will generally not have as complicated a failure mechanism or recovery as do the RAID boxes. Fibre Channel disks have two paths (“dual-ported”) which can improve fault tolerance and performance, but multipathing must be used to take advantage of those capabilities.
Windows multipathing solutions can be designed into the operating system or can be designed by third party vendors. In either case, the multipathing solution consists of one or more drivers—software that is designed to manage the hardware components on behalf of the operating system.
One of the greatest complexities in developing effective device drivers is ensuring both that they are compatible with the operating system and that they do not block the functionality of other vendor storage devices or drivers. Third party multipathing drivers are usually provided by storage array manufacturers, although in some cases they are developed by the manufacturers of host bus adapters or by storage software vendors.
The multipathing solutions developed by third party hardware manufacturers have the advantage of expert knowledge of their storage subsystem; read/write requests to the storage device can be optimized, for example. Vendor knowledge, however, does not usually extend to the architecture of the operation system kernel or storage stack. One of the negative consequences of lacking such knowledge is that the drivers they develop can fail to work properly. In the Windows operating system environment, this translates into a number of very serious problems, including operating system crashes, failure to load under all conditions such as “Safe mode” boot, and poor Plug and Play functionality—any of which can result in loss of data or data corruption.
Another serious problem with a number of vendor device drivers is that, depending on how they are designed, interoperability with other vendor storage devices may be prevented. This is a serious problem for storage administrators who are either locked into a single storage vendor, or are faced with serious and often insurmountable configuration problems when attempting to use storage or pathing components from multiple vendors.
There are a number of advantages to an operating system-based (native) multipathing solution. Foremost among them is that a solution that is designed to be integrated with the operating system prevents the problems often associated with third party drivers. Another enormous advantage to the customer is that, through a host-based multipathing solution, it is possible to ensure interoperability of storage devices from multiple vendors.
The drawback to operating system-based multipathing solutions is that intimate knowledge of the specific capabilities of the vendor device is not possible.
The Microsoft MPIO solutions are designed to work in conjunction with device specific modules (DSMs) written by vendors; the MPIO driver package does not, by itself, form a complete solution. Microsoft provides a sample device-specific module, or DSM, designed to provide a software interface between the multipath driver package and the hardware device. Vendors must adapt this generic DSM to the specifics of their device or devices.
This joint solution allows vendors to design hardware solutions that are tightly integrated with the Windows operating system, and also enables Microsoft to correctly accommodate the non-generic characteristics of each vendor’s storage device (such as whether there are multiple active controllers or the controllers have only standby capability), without having to design the MPIO solution in anticipation of each possible difference. Compatibility with both the operating system and other vendor storage devices is ensured through requiring that vendors meet a set of standards (the Microsoft Logo program) designed help ensure proper vendor device functionality. The bottom line for the customer is that a multipathing solution based on Microsoft MPIO provides a high performance solution that keeps data highly available.
The Windows operating system relies on the Plug and Play (PnP) Manager to dynamically detect and configure hardware (such as adapters or disks), including hardware used for high availability/high performance multipathing solutions. PnP devices are automatically detectable, and can be hot plugged into a PnP bus without the need to reboot the system. In contrast, legacy devices (such as those that plug into serial or parallel ports) must be manually configured and the system may need to be rebooted for the operating system to register their presence correctly. Dynamic discovery reduces both management complexity and the potential for configuration errors, both of which are especially detrimental in large or complex storage configurations. The Microsoft MPIO driver development kit is designed to work seamlessly with PnP architecture.
PnP devices are under the control of PnP drivers that understand the plug and play protocols of the Windows operating system. Control of the device itself rests with lower layer (vendor) drivers.
Multipathing solutions cannot work effectively unless the redundant adapters connecting to the same storage device can be discovered, enumerated, and configured correctly—that is, without the operating system erroneously interpreting multiple paths as leading to multiple storage devices. Figure 1 shows what happens with and without multipathing software in place.
Figure 1. What the operating system “sees” with and without multipathing software
Each device must first be identified as belonging to a specific vendor, and then a determination is made as to whether the device ID represents a unique device or the same underlying physical device accounted for through another path.
For dynamic discovery to work correctly, some form of identifier must be identified and obtainable regardless of the path from the host to the storage device. Each logical unit must have a unique hardware identifier. The MPIO driver package does not use “disk” signatures placed in the data area of a disk for identification purposes, but instead uses standard information obtainable from the hardware, such as serial numbers.
Since not all vendors assign their devices a unique hardware serial number, Microsoft includes in its sample generic DSM source code a means of deriving one, using other SCSI INQUIRY data. Alternatively, vendor-specific mechanisms can be implemented in the DSM.
Load balancing, the redistribution of read/write requests for the purpose of maximizing throughput between server and storage device, is especially important in high workload settings or other settings where consistent service levels are critical. Without multipathing software, a server sending I/O requests down several paths may operate with very heavy workloads on some paths while others are underutilized.
The Microsoft MPIO software supports the ability to transparently balance I/O workload, without administrator intervention. MPIO determines which paths to a device are in an active state and can be used for load balancing. Each vendor’s load balancing policy (which may use any of several algorithms, such as round robin, the path with the fewest outstanding commands, or a vendor unique algorithm) is set in the DSM. This policy determines how the I/O requests are actually routed. If the DSM returns a path that is inactive, the failover process is initiated.
The MPIO driver package, in combination with the vendor DSM, supports end-to-end path failover. The process of detecting failed paths and recovering from the failure, like load balancing, is automatic, extremely fast, and completely transparent to the IT organization. The data remains available at all times.
Not all errors result in failover to a new path. Some errors are temporary and can be recovered using a recovery routine in the DSM; if recovery is successful, MPIO is notified and path validity checked to verify that it can be used again to transmit I/O requests.
When a fatal error occurs, the path is invalidated and a new path is selected. To do this correctly, outstanding I/Os must be returned to a higher level in the stack. Once this is done, all devices on that path can be removed and then failed over to the new path.
The last section of this paper presents an overview of the low level details of the Microsoft MPIO driver architecture. This information is more in-depth than is required by readers who only seek an introduction to the Microsoft MPIO architecture.
For the operating system to correctly perform operations that relate to hardware, such as addition or removal of devices or transferring I/O requests from an application to a storage device, the correct device drivers must be associated with the device. All device-related functionality is initiated by the operat...
JolanPOL