

Lustre FAQ

From Obsolete Lustre Wiki
Revision as of 05:58, 26 November 2009 by Adilger (Talk | contribs)
Jump to: navigation, search



  • ACL: Access Control List
  • DLM: Distributed Lock Manager
  • EA: Extended Attribute
  • FC: Fibrechannel
  • HPC: High-Performance Computing
  • IB: InfiniBand
  • MDS: Metadata Server
  • OSS: Object Storage Server
  • OST: Object Storage Target (what's the difference?)


Can you describe the data caching and cache coherency method?


Does Lustre separate metadata and file data?

Yes. The entire namespace is stored on Lustre Metadata Servers (MDSs); file data is stored on Lustre Object Storage Servers (OSSs). Note that unlike many block-based clustered filesystems where the MDS is still in charge of block allocation, the Lustre MDS is not involved in file IO in any manner and is not a source of contention for file IO.


What is the difference between an OST and an OSS?





Does Lustre perform high-level I/O load balancing?



Is there a common synchronized namespace for files and directories?




Do Lustre clients use NFS to reach the servers?


Does Lustre use/provide a single security domain?


Does Lustre support the standard POSIX file system APIs?


Is Lustre "POSIX compliant"? Are there any exceptions?




  • 1. atime updates

It is not practical to maintain fully coherent atime updates in a high-performance cluster file system. Lustre will update the atime of files lazily -- if an inode needs to be changed on disk anyways, we will piggy-back an atime update if needed -- and when files are closed. Clients will refresh a file's atime whenever they read or write objects from that file from the OST(s), but will only do local atime updates for reads from cache.

  • 2. flock/lockf

POSIX and BSD flock/lockf system calls will be completely coherent across the cluster, using the Lustre lock manager, but are not enabled by default today. It is possible to enable client-local flock locking with the "-o localflock" mount option, or cluster-wide locking with the "-o flock" mount option. If/when this becomes the default, it is also possible to disable flock for a client with the "-o noflock" mount option.


Can you grow/shrink file systems online?



Lustre includes a patched version of the ext3 file system, called ldiskfs, with additional features such as extents, an efficient multi-block allocator, htree directories, large inodes, extended attributes, transaction optimizations, fine-grained locking, and CPU affinity for critical operations. In newer kernels that support the ext4 file system, this will be used instead of ext3. Work is underway to use the Solaris ZFS filesystem to increase the scalability and robustness of the back-end filesystem.


There are many reasons to choose ext3. One is size; at about 15,000 lines of code, it is extremely understandable, maintainable, and modifiable. Another is reliability; ext3 is proven stable by millions of users, with an excellent file system repair tool.

When we began, there was a big difference between the various Linux file systems, particularly with respect to performance. In the last few years, however, the Lustre team has carried ext3 very substantially forward, and it is now extremely competitive with other Linux file systems. Most of the changes made to ext3 for improving Lustre performance have been included into the upstream ext4 filesystem, reducing the number and size of patches in ldiskfs dramatically.

Nevertheless, we had originally planned to support multiple file systems, so Lustre does contain a file system abstraction. In future Lustre releases we will support ZFS as the backing filesystem for both OSTs and MDTs.

Why didn't you use IBM's distributed lock manager?








Each backend OST file system is restricted to a maximum of 8 TB on Linux 2.6 (imposed by ext3). Filesystems that are based on ext4 (SLES11) will soon be able to handle single OSTs of up to 16TB. Of course, it is possible to have multiple OST backends on a single OSS, and to aggregate multiple OSSs within a single Lustre filesystem. Running tests with almost 4000 smaller OSTs has been tried - hence 32PB or 64PB file systems could be achieved today.

Lustre users already run single production filesystems of over 10PB, using over 1300 8TB OSTs.

What is the maximum file system block size?

The basic ext3 block size is 4096 bytes, although this could in principle be easily changed to as large as PAGE_SIZE (on IA64 or PPC, for example) in ext4. It is not clear, however, that this is necessary.

Some people confuse block size with extent size or I/O request size -- they are not the same thing. The block size is the basic unit of disk allocation, and for our purposes it seems that 4kB is as good as any. The size of a single file extent, by definition, is almost always larger than 4kB, and ldiskfs extents and mballoc features used by Lustre do a good job of allocating I/O aligned and contiguous disk blocks whenever possible. The I/O request size (the amount of data that we try to read or write in a single request) is usually 1MB (aligned to the start of the LUN) and can be further aggregated by the disk elevator or RAID controller.

What is the maximum single-file size?

On 32-bit clients, the page cache makes it quite difficult to read or write a single file larger than 8 TB. On 64 bit clients, the maximum file size is 2^63. A current Lustre limit for allocated file space arises from a maximum of 160 stripes and 2TB per file on current ldiskfs filesystems, leading to a limit of 320TB per file.


We use the ext3 hashed directory code, which has a theoretical limit of about 15 million files per directory, at which point the directory grows to more than 2 GB. The maximum number of subdirectories in a single directory is the same as the file limit, about 15 million.

We regularly run tests with ten million files in a single directory. On a properly-configured quad-socket MDS with 32 GB of ram, it is possible to do random lookups in this directory at a rate of 20,000/second.

A single MDS imposes an upper limit of 4 billion inodes, but the default limit is slightly less than the device size / 4kB, so about 2 billion inodes for a 8 TB MDS file system. This can be increased at initial file system creation time by specifying mkfs options to increase the number of inodes. Production file systems containing over 300 million files exist.

With the introduction of clustered metadata servers (Lustre 2.x) and with ZFS-based MDTs, these limits will disappear.

How many OSSs do I need?




What is the largest possible I/O request?

When most people ask this question, they are asking what is the maximum buffer size that can be safely passed to a read() or write() system call. In principle this is limited only by the address space on the client, and we have tested single read() and write() system calls up to 1GB in size, so it has not been an issue in reality.


Typically, Lustre client nodes will do their best to aggregate I/O into 1 MB chunks on the wire, and to keep between 5 and 10 I/O requests "in flight" at a time, per server. There is still a per-syscall overhead for locking and such, so using 1MB or larger read/write requests will minimize this overhead.

On the OSS, we have gone to significant effort to ensure that these large 1 MB buffers do not get unnecessarily broken up by lower kernel layers. In Linux 2.4, modifications were required to the SCSI layer, block devices, and the QLogic fibrechannel driver. In newer kernels there is less need to modify the kernel, though the block device tunables are still set for low-latency desktop workloads, and need to be tuned for high-bandwidth IO. Details are documented in the Lustre manual.

How many nodes can connect to a single Lustre file system?

The largest single production Lustre installation is approximately 26,000 nodes today (2009). This is the site-wide filesystem for a handful of different supercomputers.

Although these are the largest clusters available to us today, we believe that the architecture is fundamentally capable of supporting many more clients.


Which operating systems are supported as clients and servers?

Please see OS Support.

Can you use NFS or CIFS to reach a Lustre volume?

Yes. Any native Lustre client (running Linux today, by definition) can export a volume using NFS or Samba. Some people have even built small clusters of these export nodes, to improve overall performance to their non-native clients.

Although NFS export works today, it is slower than native Lustre access, and does not have cache coherent access that some applications depend on.


What is the typical MDS node configuration?

10,000-node clusters with moderate metadata loads are commonly supported with a 4-socket quad-core node with 32GB of RAM, providing sustained throughput of over 5,000 creates or deletes per second, and up to 20,000 getattr per second. It is common for these systems to have roughly 200 million files. Even in 10,000-client clusters, the single MDS has been shown not to be a significant bottleneck under typical HPC loads. Having a low-latency network like InfiniBand plays a significant role in improved MDS performance.

High throughput with very large directories is possible with extra RAM and, optionally, solid-state disks. Typically, write I/O is low, but seek latency is very important, hence RAID-0+1 mirrored storage (RAID-0 striping of multiple RAID-1 mirrored disks) is strongly recommended for the MDS. Access to metadata from RAM can be 10x faster, so RAM more RAM is usually beneficial.

MDS storage requirements should be sized at approximately 4kB per file, except in unusual circumstances.

What is the typical OSS node configuration?

Multi-core 64-bit servers with good buses are capable of saturating multiple interconnects of any type. These nodes are often dual- or quad- socket and support up to 4 fibrechannel interfaces. RAM is used for locks and metadata caching, and in 1.8 and later extra server RAM will be used for file data caching, though is not strictly required. Aproximately 2 GB/OST is recommended for OSS nodes in 1.8, and twice that if OST failover will also run backup OSTs on the node.

Which architectures are interoperable?


Which storage devices are supported, on MDS and OSS nodes?

Servers support all block storage: SCSI, SATA, SAS, FC and exotic storage (SSD, NVRAM) are supported.

Which storage interconnects are supported?



Are fibrechannel switches necessary? How does HA shared storage work?

Typically, fibrechannel switches are not necessary. Multi-port shared storage for failover is normally configured to be shared between two server nodes on a FC-AL. Shared SCSI and SAS devices will also work.

Backend storage is expected to be cache-coherent between multiple channels reaching the devices. Servers in an OSS failover pair are normally both active in the file system, and can be configured to take over partitions for each other in the case of a failure. MDS failover pairs can also both be active, but only if they serve multiple separate file systems.

Can you put the file system journal on a separate device?




Can you describe the installation process?

The current installation process is straightforward, but manual:

1. Install the provided kernel and Lustre RPMs. 2. A configuration tool assistant can generate a configuration file for simple configurations, or you can build more complex configurations with relatively simple shell scripts. 3. Format and mount the OST and MDT filesystems. The command is usually identical on all nodes, so it's easy to use a utility like pdsh/prun to execute it. 4. Start the clients with "mount", similar to how NFS is mounted.

What is the estimated installation time per compute node?

Assuming that node doesn't require special drivers or kernel configuration, 5 minutes. Install the RPMs, add a line for the filesystem to /etc/fstab. Compute nodes can be installed and started in parallel.

What is the estimated installation time per I/O node?

5-30 minutes, plus formatting time, which can also be done in parallel. The per-node time depends on how easily the commands can be run in parallel on the I/O nodes. Install the RPMs, format the devices, add a line to /etc/fstab for the filesystem, mount.


Which interconnects and protocols are currently supported?

Today, Lustre supports TCP/IP (commonly over gigabit or 10-gigabit ethernet), OFED, Myrinet MX, and Cray's Seastar networks. Other older networking technologies were also supported, but are virtually unused today and will be dropped in future releases.

The up-to-date versions of each network type are at the beginning of the lnet/ChangeLog file and each release announcement.


Yes, with Lustre 1.4.6 and later.



Can I use TCP offload cards?



Does Lustre support crazy heterogeneous network topologies?



Lustre uses its own implementation of the Portals message passing API, upon which we have implemented Gateway nodes, to route between two native protocols. These are commodity nodes with, for example, both gigabit ethernet and InfiniBand interfaces. The gateway software translates the Portals packets between the interfaces to bridge the two networks.

These routers are in use today, and may become more popular as more enterprises connect multiple clusters with special interconnects to a single global Lustre file system. On the other hand, TCP/IP on GigE is the interconnect of choice for most organizations, which requires no additional Portals routing.

Metadata Servers

How many metadata servers does Lustre support?




How will clustered metadata work?




Isn't the single metadata server a bottleneck?

Not so far. We regularly perform tests with single directories containing millions of files, and we have several customers with 10,000-node clusters (or larger) and a single metadata server.


The Lustre metadata server software is extremely multithreaded, and we have made substantial efforts to reduce the lock contention on the MDS so that many clients can work concurrently. These are the kinds of optimizations which make it possible to do random creations or lookups in a single 10-million-file directory at a rate of more than 5,000 per second. Future versions of Lustre will continue to improve the metadata performance.

If there is a customer need for massive metadata servers prior to the release of clustered metadata, it should be possible to scale quite far using large SMP systems (such as the SunFire x4600 or Bull NovaScale) with large memory spaces and solid-state disks which can utilize large caches to speed MDS operations.

What is the typical MDS node configuration?

Please see Installation.

How do I automate failover of my MDSs?

Please see Recovery.

Do you plan to support MDS failover without shared storage?


How is metadata allocated on disk?


What is intent locking?








How does the metadata locking protocol work?



Does the MDS do any pre-allocation?


Object Servers and I/O Throughput

What levels of throughput should I expect?



  • TCP/IP
    • Single-connected GigE: 115 MB/s
    • Dual-NIC GigE on a 32-bit OSS: 180 MB/s
    • Dual-NIC GigE on a 64-bit OSS: 220 MB/s
    • Single-connected 10GigE on a 64-bit OSS: 550 MB/s, 1GB/s on woodcrest
  • Unoptimized InfiniBand
    • Single-port SDR InfiniBand on a 64-bit OSS: 700-900 MB/s
    • DDR InfiniBand on a 64-bit OSS: 1500 MB/s

How fast can a single OSS be?



How well does Lustre scale as OSSs are added?


How many clients can each OSS support?


What is a typical OSS node configuration?

Please see Installation.

How do I automate failover of my OSSs?

Please see Recovery .

Do you plan to support OSS failover without shared storage?

Yes, Server Network Striping will allow RAID-1 and RAID-5 file I/O features. They will provide redundancy and recoverability in the Lustre object protocol rather than requiring shared storage. In the meantime, some users have been trying DRBD to replicate the OST block device to a backup OSS instead of having shared storage, though this is not an officially supported configuration.

How is file data allocated on disk?



How does the object locking protocol work?








Does Lustre support Direct I/O?

Yes. It locks the data to guarantee cluster-wide consistency, just like normal POSIX I/O, but does not cache the data on the client or server. With an RDMA-capable network (anything other than TCP) there is only a single data copy directly from the client RAM to the server RAM and straight to the disk.

Can these locks be disabled?

Yes, but:


Do you plan to support T-10 object devices?


Does Lustre support/require special parallel I/O libraries?

Lustre supports them, but by no means requires them. We have found equal performance when using standard POSIX I/O calls, the POSIX ADIO driver, or the MPI/IO libraries, when the IO size is sufficient.

A Lustre-specific MPI/IO ADIO driver has been developed to allow an application to provide hints about how it would like its output files to be striped, and to optimize the IO pattern when many clients are doing small read or write operations to a single file.


How do I configure failover services?


This does not typically require a fibrechannel switch.

How do I automate failover of my MDSs/OSSs?



How necessary is failover, really?

The answer depends on how close to 100% uptime you need to achieve. Failover doesn't protect against the failure of individual disks -- that is handled by software or hardware RAID at the OST and MDT level. Lustre failover is to handle the failure of an MDS or OSS node as a whole which, in our experience, is not very common.

We would suggest that simple RAID-5 or RAID-6 storage is sufficient for most users, with manual restart of failed OSS and MDS nodes, but that the most important production systems should consider failover.







These are supported to the extent supported by the HBA drivers. If arrays with multiple ports are shared by multiple I/O nodes, Lustre offers 100% transparent failover for I/O and metadata nodes. Applications will see a delay while failover and recovery is in progress, but system calls complete without errors.

Can you describe an example failure scenario, and its resolution?





OS Support

Which operating systems are/will be supported?


Today, native kernel drivers exist only for Linux. Other native ports, such as to Windows and Solaris are underway.

liblustre is not yet as widely used as Lustre in the Linux kernel, but it is fundamentally almost entirely the same code. The Cray Red Storm supercomputer runs liblustre on the Catamount operating system on 22000 nodes to provide all client and server storage, including root FS and swap space, so this is a very serious use of liblustre. It could in principle be used on any Unix-like operating system, or even Windows.


We currently support only Linux as Lustre server nodes. Support for other operating systems such as Solaris may be available at a later date.

With Lustre 1.6.0 Linux 2.4 clients and servers will no longer be supported. See the lustre/ChangeLog and release notes for specific kernel and vendor distributions that are supported.

Why have you decided to patch the Linux kernel?

Lustre's goals are extremely ambitious; there are few, if any, other file systems which attempt such scalability, performance, and consistency guarantees in a single package. Because this hadn't been done before, the infrastructure was not present in the kernel for such a file system.


In 1.6 releases Lustre runs without patching the kernel, for Linux kernels 2.6.16 and newer. There are still a small number of kernel patches needed for servers, and to the copy of the ext3 filesystem used by Lustre (ldiskfs), but the majority of the ext3 changes have been incorporated into ext4 in the 2.6.27 kernel.



With 2.6.16 and later kernels enough of the Lustre kernel patches have been incorporated into the upstream kernel that clients can run without patches. There is a shrinking number of patches needed on the server.

Currently, there are no definite plans to include the Lustre client or server code itself into the kernel. The requirements for getting the Lustre and LNET code into the kernel would consume too much engineering resources that could better be spent on improving Lustre itself.

Can I run Lustre without patching my kernel?

The SuSE Linux Enterprise Server 9 kernel contains the Lustre VFS patches needed to run both clients and servers, though additional patches are needed to address some minor defects found after that kernel was released.

It is possible to run Lustre clients on all kernels newer than 2.6.16, and the latest RHEL4 2.6.9 kernel without patching.

Which Linux kernels are supported?

We currently support a number of kernels for Lustre 1.4: the Red Hat Enterprise Linux 3 kernel (based on 2.4.21), SuSE Linux Enterprise Server 9 (based on 2.6.5), SuSE Linux Enterprise Server 10 (based on 2.6.16), and the Red Hat Enterprise Linux 4 kernel (based on 2.6.9).

For Lustre 1.6 the Red Hat Enterprise Linux 5 kernel (based on 2.6.16) is also supported, but RHEL3 is removed.

An up-to-date list of exact kernel versions supported by each release is at the start lustre/ChangeLog and the release notes.

Which Linux distributions are supported?



What if I don't run one of those kernels?

If you can't or won't run one of the supported kernels, you can also build Lustre for the particular kernel on your client. For servers we still strongly recommend one of the supported kernels, as it's not a matter of "just building from source". There are patches for each kernel, and given the differences between most kernels, these are fairly non-trivial to port. Until there is a customer demand for a given kernel series, the Lustre group simply does not have the resources to maintain those patches.


Do you support Lustre on an SGI Altix?


When was Lustre for Linux first used in production?


Release Testing and Upgrading

How does the Lustre group fix issues?

The Lustre group approaches bug tracking and fixing seriously and methodically:

  • Architecture and Design: Depending on the severity and invasiveness, an update to the architecture description may be written and reviewed by senior architects. A detailed design description for the patch is written and reviewed by principal engineers.
  • Testing: The developer runs a small suite of tests before the code leaves his or her desk, then it's added to a branch for regression testing, and final release testing.

What testing does each version undergo prior to release?




Before Lustre 1.4.2, the same care was not taken for wire structures, because we've been adding features so quickly that it hasn't been practical. Since 1.4.2 each client and server negotiates a set of supported protocol features between them, allowing new network protocols to be added while not breaking older clients or servers.

Support for running with older protocols are removed on every second major release, so the 1.4.x release will be incompatible with 1.8.x release. Similarly, the 1.6.x release will not be able to interoperate with 2.0.x release.

The Lustre group always tests release 1.x.y with 1.x.(y-1) and 1.(x-2).latest when making a release. It isn't possible to exhaustively test all release combinations, and as a result we only recommend using the most recent version of two successive releases at one time, even though in theory any 1.x release will work with any 1.(x-2) release.

Do you have to reboot to upgrade?



Licensing and Support


The Lustre file system for Linux is an Open Source product.


As we develop ports for operating systems other than Linux, it is highly likely that these will be proprietary. We may also decide to develop some future features as proprietary add-ons. These are easily separable, non-core features primarily of interest to enterprise customers.







If Lustre stops getting professional support, the taxpayers' money will have been wasted. To that end, our government sponsors strongly encourage us to build a sustainable business model.

How does the commercial distribution of Lustre work?


For more information about Lustre support contracts, please contact lustre-solutions@sun.com.

Will Sun develop custom features for a proprietary product?





What Lustre support services are available?



Contract development of new features, or acceleration of our development roadmap features to a guaranteed delivery date, are possible.

Sun also provides public and private on-site training services for your system administrators or support staff. Please check our upcoming course schedule here.

Personal tools