December 22, 2006 ... 评论

Google机器和存储的传说

以前,我只听说那系统强韧到能把硬盘直接摆在地上,坏了就拔,想加就加,于是 Gmail 的容量才能不停地成长。今天,连主机都变成那样了:简化过的主机板裸片,用魔鬼毡黏在机架上,一格就可以多放好几片。那一片的指示灯有问题就下架换新,也不必检查了。而且现在听说还能用机器人自动化地检查抽换。

听说那系统的设计很强悍,每个数据都有数个备份,一个机器坏了,系统会自动把它原来的负担分配给其他的机器,所以管理员的工作很简单,换机器就可以了。

The Google File System
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung

Abstract

We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients.

While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points.

The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients.

In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use.

Appeared in:
19th ACM Symposium on Operating Systems Principles,
Lake George, NY, October, 2003.

http://labs.google.com/papers/gfs.html
http://blog.cathayan.org/item/1438

评论