Tuesday, October 31, 2006

Paper: "Ceph: A Scalable, High-Performance Distributed File System"

Ceph: A Scalable, High-Performance Distributed File System
Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn, University of California, Santa Cruz

Abstract

We have developed Ceph, a distributed file system that provides excellent performance, reliability, and scalability. Ceph maximizes the separation between data and metadata management by replacing allocation tables with a pseudo-random data distribution function (CRUSH) designed for heterogeneous and dynamic clusters of unreliable object storage devices (OSDs). We leverage device intelligence by distributing data replication, failure detection and recovery to semi-autonomous OSDs running a specialized local object file system. A dynamic distributed metadata cluster provides extremely efficient metadata management and seamlessly adapts to a wide range of general purpose and scientific computing file system workloads. Performance measurements under a variety of workloads show that Ceph has excellent I/O performance and scalable metadata management, supporting more than 250,000 metadata operations per second.

6 Comments:

Blogger sage said...

You can find more information, including related papers, on the sourceforge web site:
http://ceph.sourceforge.net/

11:28 AM  
Blogger Leonid Ryzhyk said...

Talk summary
============

Speaker: Sage A. Weil

Ceph is a high-performance distributed file system designed to accommodate hundreds of petabytes of data. The main principles underlying the Ceph architecture are separation of data and metadata, use of intelligent storage devices, and adaptable dynamic metadata management.

In his talk, Sage described the two main components of Ceph -- the metadata store called MDS and the distributed object store called RADOS. MDS achieves excellent scalability by dynamically partitioning the directory tree among metadata servers based on metadata access frequencies. In addition, MDS simplifies the metadata structure by not storing file allocation data, and instead computing it using a hashing function called CRUSH.

The RADOS object store achieves scalability and reliability through replication, failure detection and recovery. In order to enable the use of Ceph for efficient communication, while guaranteeing data safety, RADOS implements a two-phase write acknowledgement protocol. The first acknowledgement confirms that the update has been propagated to all replicas, while the second acknowledgement confirms that the update has been physically committed to disk. RADOS uses the EBOFS object file system for local data storage. EBOFS implements a non-POSIX interface that supports features like atomic transactions and asynchronous commit notifications.

Q&A
===

Q: Does Ceph support directory move/rename?
A: Yes. The only thing that moves is the inode, not the directory.

Q: What is the effect of network latency on performance numbers?
A: One of the assumptions is that we are deploying in a data centre environment. In principle, however, you could deploy it in a wider scenario if you have sufficient bandwidth.

Q: In your model metadata lookups are done on the server. What are the characteristics of your workload that make this more appropriate for scalability than doing the lookup on the client?
A: High-performance workloads, high contention for metadata require consistency to be managed within the metadata server.

Q: What is the recovery strategy for metadata?
A: The short-term log doubles as a journal, so if a metadata server fails, another node can rescan its journal.

5:31 PM  
Blogger Leonid Ryzhyk said...

This comment has been removed by a blog administrator.

2:33 PM  
Blogger Leonid Ryzhyk said...

Talk summary (Revised based on authors' comments)
=========================================

Speaker: Sage A. Weil

Ceph is a high-performance distributed file system designed to accommodate hundreds of petabytes of data. The main principles underlying the Ceph architecture are separation of data and metadata, use of intelligent storage devices, and adaptable dynamic metadata management.

In his talk, Sage described the two main components of Ceph -- the metadata store called MDS and the distributed object store called RADOS. MDS achieves excellent scalability by dynamically partitioning the directory tree among metadata servers based on metadata access frequencies.
In addition, MDS simplifies the metadata structure by replacing file allocation data with seeds to a system-wide well-known hashing function called CRUSH that is very stable in the face of storage device failures and other storage capacity changes.

The RADOS object store provides scalability and reliability through
replication, failure detection and recovery. RADOS achieves scalability by using intelligent object store nodes that take advantage of a very compact representation of the cluster state (made possible by CRUSH) enabling them to efficiently communicate with local peers to quickly recover from failures or any other changes to the cluster. In order to enable the use of Ceph for efficient communication, while guaranteeing data safety, RADOS implements a two-phase write acknowledgement protocol. The first acknowledgement confirms that the update has been propagated to all replicas, while the second acknowledgement confirms that the update has been physically committed to disk. RADOS uses the EBOFS object file system for local data storage. EBOFS implements a non-POSIX interface that supports features like atomic transactions and asynchronous commit notifications.

Q&A
===

Q: Does Ceph support directory move/rename?
A: Yes. The only thing that moves is the inode, not the directory.

Q: What is the effect of network latency on performance numbers?
A: One of the assumptions is that we are deploying in a data centre environment. In principle, however, you could deploy it in a wider scenario if you have sufficient bandwidth.

Q: In your model metadata lookups are done on the server. What are the characteristics of your workload that make this more appropriate for scalability than doing the lookup on the client?
A: High-performance workloads, high contention for metadata require consistency to be managed within the metadata server.

Q: What is the recovery strategy for metadata?
A: The short-term log doubles as a journal, so if a metadata server fails, another node can rescan its journal.

2:39 PM  
Blogger Wendi said...

A typical dictionary definition of hypnosis states that it is: a state that resembles sleep but that is induced by suggestion. However, anyone who has tried hypnosis (and any self respecting hypnotist) will tell you that this is a very simplistic view of the subject!

A much better description comes from the Free Online Dictionary which states that hypnosis is: an artificially induced state of consciousness, characterised by heightened suggestibility and receptivity to direction. So what does this mean and how can it be used to your advantage?

Well, the subject of hypnosis has been discussed and pondered since the late 1700s. Many explanations and theories have come and gone though science, however, has yet to supply a valid and well-established definition of how it actually happens. It's fairly unlikely that the scientific community will arrive at a definitive explanation for hypnosis in the near future either, as the untapped resources of our 'mostly' uncharted mind still remain something of a mystery.

However, the general characteristics of hypnosis are well documented. It is a trance state characterized by extreme suggestibility, deep relaxation and heightened imaginative functioning. It's not really like sleep at all, because the subject is alert the whole time. It is most often compared to daydreaming, or the feeling you get when you watch a movie or read a captivating book. You are fully conscious, but you tune out most of the outside world. Your focus is concentrated intensely on the mental processes you are experiencing - if movies didn't provide such disassociation with everyday life and put a person in a very receptive state then they would not be as popular (nor would TV advertising be as effective!). Have you ever stated that a film wasn't great because you just couldn't 'get into it'???

This works very simply; while daydream or watching a movie, an imaginary world becomes almost real to you because it fully engages your emotional responses. Such mental pursuits will on most occasions cause real emotional responses such as fear, sadness or happiness (have you ever cried at a sad movie, felt excited by a future event not yet taken place or shivered at the thought of your worst fear?).

It is widely accepted that these states are all forms of self-hypnosis. If you take this view you can easily see that you go into and out of mild hypnotic states on a daily basis - when driving home from work, washing the dishes, or even listening to a boring conversation. Although these situations produce a mental state that is very receptive to suggestion the most powerful time for self-change occurs in the trance state brought on by intentional relaxation and focusing exercises. This deep hypnosis is often compared to the relaxed mental state between wakefulness and sleep.

In this mental state, people feel uninhibited and relaxed and they release all worries and doubts that normally occupy their mind. A similar experience occurs while you are daydreaming or watching the TV. You become so involved in the onscreen antics that worries and everyday cares fade away, until all you're focused on is the TV. In this state, you are also highly suggestible. That is why when a hypnotist tells you do something under trance; you'll probably embrace the idea completely.

However, your sense of safety and morality remain entrenched throughout the experience and should either of these be threatened you immediately wake!

A hypnotist can not get you to do anything you don't want to do. So while in such a state, when we are highly suggestible and open to new beliefs, a skillful hypnotist, whether in person or via a recording, can alter life-long behaviours and even give us new ones! erotic hypnosis

2:54 AM  
Blogger cylon said...

I am sure you have heard the song "Karma Chameleon" by Culture Club but have you ever given much thought to its meaning? While on Earth, you are living in a world of reincarnation which is governed by the law of karma. Karma begins to propel you as Soul on a personal journey through the universe. Karma ends when you have reached enlightenment and fully realise that this physical reality and the Universe itself is just an illusion. When you reach a state of knowingness that there is but One all pervading essence and that essence or consciousness is You!
So what is Karma and how does it work? While in the illusion you have a soul. This soul lives past, present, and future lives. To grow in love, joy, and awareness, you reincarnate into a series of physical bodies to experience different existences. This road leads to the experiences of being both sexes, all races, religions, and ethnic types throughout many lifetimes.
Karma in its simplicist terms can be described by the biblical statement "as you sow, so also shall you reap". Karma is the principle of cause and effect, action and reaction, total cosmic justice and personal responsibility. It brings 'good' experiences as well as 'bad' - a debt must be repaid and a blessing rewarded.
A more indepth esoteric look at karma gives us the following distinctions: Sanchita Karma: the accumulated result of all your actions from all your past lifetimes. This is your total cosmic debt. Every moment of every day either you are adding to it or you are reducing this cosmic debt. Prarabdha Karma: the portion of your "sanchita" karma being worked on in the present life. If you work down your agreed upon debt in this lifetime, then more past debts surface to be worked on. Agami Karma: the portion of actions in the present life that add to your "sanchita" karma. If you fail to work off your debt, then more debts are added to "sanchita" karma and are sent to future lives. Kriyamana Karma: daily, instant karma created in this life that is worked off immediately. These are debts that are created and worked off - ie. you do wrong, you get caught and you spend time in jail.
As a soul, you experience a constant cycle of births and deaths with a series of bodies for the purpose of experiencing this illusionary world gaining spiritual insights into your own true nature until the totality of all experiences show you Who you really are - the I AM! Until you have learned, you will find that pretending that the rules of karma do not exist or trying to escape the consequences of your actions is futile.
Although it may often "feel" like punishment, the purpose of karma is to teach not to punish. Often the way we learn is to endure the same type of suffering that we have inflicted on others and also rexperience circumstances until we learn to change our thinking and attitudes.
We are all here to learn lessons as spiritual beings in human form. These lessons are designed to help us grow into greater levels of love, joy, and awareness. They teach us our true nature of love. Where we do not choose love, show forgiveness, teach tolerance, or display compassion, karma intervenes to put us back on the path of these lessons. Quite simply, the only way to achieve a state of karmic balance is to be love.
Before you incarnated into your present personality, you agreed to put yourself in the path of all that is you need to learn. Once you got here, you agreed to forget this. Karma is impersonal and has the same effect for everyone. It is completely fair in its workings and it is predictable - "do onto others as you would have them do unto you" is a way to ensure peace and tranquillity in your own life as well as the lives of those you come into contact with. The law of karma is predictable - "as you sow, so shall you reap" what is done to you is the net result of what you have done to others!
Karma gives you the opportunity at every moment to become a better person than you are and to open up to the realization that you are the master of your own fate.
The goal of karma is to give you all the experiences that you need to evolve into greater levels of love, joy, awareness, and responsibility. Karma teaches that you are totally responsible for the circumstances of your life. They keep you on the straight and narrow until you have mastered your vehicle and can ride freely on your own. Once you understand that you are the master of your own circumstances and that everything you experience is a direct result of your past actions due to your thinking and emotional responses you can overcome its seeming negative effects by creating only 'good' karma.
Karma forces us to look beyond ourselves (oneness) so that we can see ourselves as we truly are Whole, Complete, at One with everything. Once we truly understand ourselves, we can see our divinity and our unity with all life.
Karma drives us to service. Love means service. Once you accept total responsibility for your life, you see yourself as a soul in service to God. Once you do, you become a fully realized being, allowing God to experience the illusion through you.
Belief in karma and an understanding of its workings will lead you to a life of bliss. Only your own deeds can hinder you. Until the time comes when we release ourselves from our own self-imposed shackles of limitation and fully understand who and what we are we will live under the mantle of karma. So until that day why not create some wonderful experiences for ourselves by "doing onto others, as we would have them do unto us". lucid dreaming

12:09 PM  

Post a Comment

<< Home