The Walus overview I am working on is taking shape. At first I had a large voume of unorganized facts about Walrus. I was uncertain as to which manner to present them in. Upon reviewing the data, I managed to group the information I have into some main categories: Walrus’ two manin funcions, Access and security, Buckets and data storage, VM Manipulation, and object version control.

Here is what I have so far:

Walrus Architecture Overview Dan Adams

Walrus is the component of Eucalyptus that allows you to store your data in the cloud. You can store your data as objects or buckets, which are collections of objects. These data stores are secure, and you as the administrator decides who has access to the data, and what privileges they have (read, write, etc.). Walrus serves two purposes:
Walrus provides three types of functionality:
• Users that have access to EUCALYPTUS can use Walrus to stream data into/out of the cloud as well as from instances that they have started on nodes.

• In addition, Walrus acts as a storage service for VM images. Root filesystem as well as kernel and ramdisk images used to instantiate VMs on nodes can be uploaded to Walrus and accessed from nodes. Walrus is a put/get storage service.

• Walrus is also used to store snapshots of VMs for easy restoration and data protection

Accessing the Cloud
Access to Walrus and other components of Eucalyptus is accomplished through a pair of ssh keys. Both are alphanumeric strings. One is a public key, used to validate the software and other settings. Eucalyptus shares user’s credentials with the Cloud Controller’s user database.

The model of Walrus is similar in many ways to the Amazon S3 service. Both store collections of objects in buckets. Both the Walrus component and S3 are accessed using a pair of ssh keys, one private and one public. Amazon, however, requires a 20 and 40 character string, respectively, while the Eucalyptus keys are much longer.
Once the user is authenticated, interaction can take place via a web interface or command line.

Requests to read/write storage can be made via the Amazon S3 Curl Service or through or through the Eucalyptus web interface. Interactions with the service are either authenticated or anonymous. Buckets and objects are owned by individual accounts. To share resources, you must grant permissions as an administrator.

Walrus can use standard web services technologies like Axis and Mule , as well as being interface compatible with S3. It implements the REST or query interface via HTTP, as well as the SOAP interface.

Walrus is accessible to end-users, whether they are running a client outside the cloud or a virtual machine instance inside the cloud.
You can Eucatools commands to put or get data from Walrus, or one of the standard S3 tools. Example of these are S3 Curl , s3cmd
, and s3fs
. s3fs allows users to access S3 buckets as local directories.

Buckets and Data Storage via Walrus
Create a Bucket → Add an object to a bucket → View/move/delete object.

Objects can be grouped into folders. Folders can, in turn, be grouped into other folders. Objects can be public or private, with specific rights given to different users.
Bucket names need to be unique within the individual cloud. A good naming convention is to start with the name of your group or department.
Think of a bucket as analogous to a folder on a Windows system. Bucket storage primarily holds machine images and machine snapshots. The maximum size of a Walrus bucket is 5GB. Walrus is a file level storage system, as compared to
the block level storage system of Storage Controller.

Virtual Machine Image Manipulation via Walrus
Walrus also serves as a VM image storage and management service. Root file system images, kernel images, and ramdisk image can be uploaded to via the Walrus service, and then will be accessible from the different nodes. Images can be compressed, encrypted using user ssh keys, and split into multiple parts. These parts are described in an image description file, sometimes called the manifest.

When a node controller (NC) asks for an image from Walrus before starting it on a node, the node transmits an image download request. The request is authenticated with an internal set of credentials. Walrus then verifies and decrypts images that the users have uploaded, and makes the transfer to the proper directory. Walrus supports parallel and serial transfers of data transfers. Here is a sample write from a Walrus log:
[Fri April 13 04:21:21 2012][001283][EUCADEBUG ] walrus_request(): wrote 5242880000 bytes in 3421570 writes
[ Fri April 13 04:21:21 2012][001283][EUCAINFO ] walrus_request(): saved image in /var/lib/eucalyptus/instances//admin/i-515C08DF/disk

Image instances need to be on the same subnet as Walrus.

Snapshot Storage in Walrus

The volumes that are created with storage controller can be the basis of point-in-time snapshots that are stored on Walrus. The snapshots can, in turn, be used to create volumes if needed.

Snapshots are created using the euca-create-snapshot command:
uecadmin@client1: ̃$ euca−create−snapshot vol−333C04B8

The euca-describe-snapshots lists available snapshots:
uecadmin@client1: ̃$ euca−describe−snapshots
SNAPSHOT snap−32A804A2 vol−333C04B8 completed
2010−04−15T13:48:32.01Z 100%

Volumes from these snapshots can be created by using the snapshot ID:
uecadmin@client1: ̃$ euca−create−volume −s 10 −−snapshot snap−32A804A2
−−zone mycloud
A volume can only be created after the snapshot status
completes. Snapshots from a volume can only be created after
volume status is created, not during the creating stage.

To delete
a snapshot:

uecadmin@client1: ̃$ euca−delete−snapshot snap−32A804A2

Object Version Control in Walrus
Walrus doesn’t provide write-locking for object writes. Users are, however, guaranteed that a consistent copy of the object will be saved even if there are concurrent writes to the same object. If a write to an object is encountered while there is a previous write to the same object in progress, the previous write is invalidated. Walrus responds with the MD5 checksum of the object that was stored. Once a request has been verified, the user has been authenticated as a valid EUCALYPTUS user and checked against access control lists for the object that has been requested, writes and reads are streamed over HTTP.

To increase performance, and due to the fact that VM images can be very big,
Walrus keeps a repository of images that have already been decrypted. Cache invalidation happens when an image manifest is overwritten.

Troubleshooting
Walrus provides some good clues for solving problems in its log files. Two particularly useful logs are walrus-stats, walrus-digest, and registration.log
.

Problems with SSH keypairs commonly cause Walrus errors. Verify your credentials.

From the blog danspc.net Blog » wsucs by danspc.net Blog » wsucs and used with permission of the author. All other rights reserved by the author.

This week Brian, Nadia and I met and discussed the format in which we will deliver the Eucalyptus architecture overview that we are working on. Karl is getting us the contact information for whom to send it to. Untill then we will compile it into a regular text document. I am using Libre Office.

My part of the project is an overview of Walrus. Although the website is scarce on Walrus specifics, I have found a treasure trove of information in the Presentations & Publications section, specifically “Cluster Computing and the Grid”, Shanghai, China, 2009.

From the blog danspc.net Blog » wsucs by danspc.net Blog » wsucs and used with permission of the author. All other rights reserved by the author.