Category Archives: WSU CS

EUTester Documentation

This week a new project was created based on my previous post for documenting EUTester. I was finally able to get in touch with Vic Iglesias – who is the main contributor to EUTester. He was thrilled to hear that we were willing to help with documentation. He joked that documenting code wasn’t his strong suite. Thankfully Clark Mattew – another contributor – had already laid out a document on the GitHub project on how to document in code which Vic sent me to (https://github.com/eucalyptus/eutester/blob/master/eutester/euproperties.py). Vic told me that he would try to be available for the EUTester group if we needed any help with documenting or using the tool.

Since I will be away this weekend, I won’t be able to do any further digging in the code unfortunately, but hope to pick up the project next week. I am also going to look into forking the GitHub project for EUTester so that Vic can easily review and merge any of our changes. The only request that Vic had (which I posted on our EUTester wiki page) is that we make individual commits for each method that you are document so that it will be easier for him to go through. I know Matt said he would be interested in joining this project, so we will also have to figure out how we are going to break up the code.

On another note this is the first time that I have used IRC (outside of class) to communicate with people of the project. I have watched the channel before, but it was cool to jump in there and talk to people. The group was very helpful – especially when I tried to talk to Vic at 10:30AM EST, they pointed out he was on the west coast and wouldn’t be awake for a while most likely.

From the blog nathandoe » WSU CS by nathandoe and used with permission of the author. All other rights reserved by the author.

Learning Eutester

So since euca2ools did not have enough work for me to continue at it for the rest of the semester, I decided I ll simply jump on our main project the Eutester project. It has been a good day so far, most things went well I heard some people had problems with version mismatches so I downloaded and installed everything from source code their most up-to-date versions of “all the things” (boto/m2crypto/paramiko/eutester). Reading through the wiki that the eutester group of guys has set up on cs.worcester.edu, and following  some links to blogg posts from the developers, I managed to run some tests on our newly installed Eucalyptus 3 cloud which is still not able to keep a VM instance running.

>>> eucaops.get_emi()
Image:emi-238B37D9
>>> emi = eucaops.get_emi()
>>> reservations = eucaops.run_instance(emi)
Attempting to run instance-store image Image:emi-238B37D9 in group default
Beginning poll loop for the 1 found in Reservation:r-860340D3
Beginning poll loop for instance Instance:i-0696428B to go to running
Instance(i-0696428B) State(pending), sleeping 10s
Instance(i-0696428B) State(pending), sleeping 10s
Instance(i-0696428B) State(pending), sleeping 10s
Instance(i-0696428B) State(pending), sleeping 10s
Instance(i-0696428B) State(pending), sleeping 10s
Instance(i-0696428B) State(terminated) Poll(5) time elapsed (51)
[CRITICAL]: [TEST_REPORT] FAILED: Instance:i-0696428B did not enter the proper state and was left in terminated

 

After some looking around found out about the existence of nc.log file, and here is the portion of the fail runs:

[EUCAFATAL] Failed to connect to qemu:///system
[i-16F93EC9] could not contact the hypervisor, abandoning the instance
[i-16F93EC9] state change for instance: Staging -> Shutoff (Pending)
doDescribeResource: cores=1/2 mem=1604/1732 disk=11/13 iqn=iqn.1994-05.com.redhat:262aa1ddbe2
[i-16F93EC9] Pending pub=0.0.0.0 priv=0.0.0.0 mac=d0:0d:16:F9:3E:C9 vlan=-24 net=-1 plat=linux vols=
[EUCAFATAL ] Failed to connect to qemu:///system
[i-16F93EC9] cleaning up state for instance
[i-16F93EC9] stopping the network (vlan=-24)
[EUCAWARN  ] vnetStopNetworkManaged(): supplied vlan '-24' is out of range (0 - 4096), nothing to do
[i-16F93EC9] state change for instance: Shutoff -> Teardown (Teardown)
[EUCAFATAL ] Failed to connect to qemu:///system

After trying to work my way through an old bug report and fix on the above problem, I decided I should check if Hardware Virtualization, which turned out it was the right idea, because in this case it is turned off in the BIOS.

From the blog Live and Code » WSU CS by dhimitraq and used with permission of the author. All other rights reserved by the author.

Eutester Documentation

This week we really learned about what goes into the eutester code to make things happen. There were several pieces that we could not figure out right away, but we spent this week doing research and we discovered what each missing piece actually was. For example, we had no idea what “paramiko” meant at first, but we discovered that it is an ssh module that allows Python code to be compiled and run remotely. Paramiko needs to be installed in order to run the eutester package. There were a few other small modules that we did not understand at first, but we eventually figured it all out and posted links on the wiki about everything we came across.

Now, rather than linking a Eucalyptus instance to a kernel like we wanted to do last week, we are able to actually run Eucalyptus images and ssh into them, which will save a lot of time. So, in moving forward, the Eucalyptus team decided that it would be very beneficial if we went through the eutester package and documented all of the code. Most of it is undocumented, so it can be tedious to read through, but we have a fairly strong understanding of it now, so we should be able to document it and contribute our changes to the project. This coming week will involve a lot of code reading and documenting, but it should be very helpful in the end.

From the blog trevorhodde by Trevor and used with permission of the author. All other rights reserved by the author.

Our Eucalyptus Cluster

It is finally getting there but still with struggles, right now for example while I am writing this I wish I could instead be connecting to our own cluster and uploading images but port 8773 still seams to be closed when trying to look in from the outside, so I can’t connect to try and work on an instance. While earlier today Long was able to upload kernel images to the server it looks like there is a problem when trying to actually start up an instance on the server which most likely proves right our fears of hardware support for hypervisor being turned off on the BIOS. One might think “big deal” just turn it on; however these are stoke dell machines we are using the BIOS options are basic at best, so we will most-likely have to install things from scratch trying to install an other BIOS on the mother board and finally unlock the hardware support.

And finally we have our definite proof that kvm support is disabled on the BIOS, and here goes an other week without having a working cloud.

From the blog Live and Code » WSU CS by dhimitraq and used with permission of the author. All other rights reserved by the author.

euca2ools install from source(with dependencies)

So some time during week 4 of the project I had some down time, and I decided to learn how euca2ools worked. Simple command-line tools yet still very powerful as well. However recently we run to some problems with the version mismatches; so here it goes: “how to install everything from source with the latest versions”:

Very easy to install from source:
NOTE: make sure to remove older versions of the software.

Download newest boto: https://github.com/downloads/boto/boto/boto-2.3.0.tar.gz

cd /download/location
tar zxvf boto-2.3.0n.tar.gz
cd boto-2.3.0
sudo python setup.py install

Download newest M2Crypto: http://pypi.python.org/packages/source/M/M2Crypto/M2Crypto-0.21.1.tar.gz

cd /download/location 
tar zxvf M2Crypto-0.21.1.tar.gz 
cd M2Crypto-0.21.1 
sudo python setup.py install

Download euca2ools from: https://github.com/eucalyptus/euca2ools

cd /download/location
cd euca2ools
sudo make

And that should be it.

From the blog WSU CS – Live and Code by dhimitraq and used with permission of the author. All other rights reserved by the author.

EuTester Issue Tracker

Following spring break we discussed the current status of our cluster which appears to be really close to complete. We are now nearing the latter half of the semester and it is time to switch into gear. While I have been attempting to install EuTester and connect to the ECC cloud I have also been checking the EuTester Issue Tracker on occassion to see if there are any new updates.

As of right now there are 5 listed issues and or requests:

  • Define Licensing:

    Need to figure out what license we’re offering Eutester under. I presume GPLv3 to be compatible with the Eucalyptus product itself, but we need to be explicit. Also need to make it clear that it’s copyright Eucalyptus.

  • Create a standard set of debug tools available when tests fail:

    When a eutester test fails, we need to provide a standard set of debugging tools back to the user. This debug info should include as much as is necessary to find the possible cause of the behavior. Currently, the Eutester class provides a very basic framework for this, but we’d like to expand it.

  • Create a standardized Eutester log/output format:

    We need to standardize the way we present info about a running test to a user – essentially, output formatting.

  • Testing capability for Jenkins + EC2 Plugin:

    Would like to look at test cases for Jenkins + EC2 plugin and whether this is possible to script. Jenkins has a CLI client but whether this works with EC2 Plugin is another thing: https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+CLI

  • Goals, purpose, and guidelines for contributionL

    Need to produce better documentation:

    1) Readme file should be bolsterd
    2) Add more sample scripts
    3) Ensure sane defaults (ie requests go to ECC by default)

Hopefully after some more research I will have a working EuTester build and can connect to a cloud in order to further research some of these issues. I am also going to be looking into the Python language with a little more detail, I am not very familiar with Python and learning it will better help me understand how to configure and run the scripts for each test case. There is an IRC meeting next week that discusses the basics of EuTester hosted by Vic Iglesias who is very involved in EuTester. I would love to attend to learn more because I am not very familiar with command line installs and configuration. Hopefully that classroom will give me a better understanding on how and what these arguments are doing rather than just typing text and hoping for the best. As of now I have gotten both boto and paramiko fully installed and have downloaded all of the most recent EuTester tools, I will update after I figure out how to perform the eucaops import in order to actually start the process. I have a feeling it has to do with where the files are located and the folder permissions, but I have not found a lot of documentation on setting these up correctly.

From the blog michaelkenny2 » WSU CS by michaelkenny2 and used with permission of the author. All other rights reserved by the author.

Learning Python

Since EU Tester is written almost completely in Python, I was tasked to learning more about the language for future script writing/modification. I came across a really good website that has lessons that goes through the basics of Python: http://www.sthurlow.com/python/. It was very helpful and I would definitely recommend it to others in the class that want to pick up the language. I haven’t done much with scripting languages in the past, so Python was a new world for me. However, I got through the lessons quickly and it was very easy to pick up.

After I got the hand of the basics of Python, I started looking into the EU Tester Python code. There were a lot of functions that I did not know what they did, but using the API Documentation on the EU Tester GitHub project (https://github.com/eucalyptus/eutester/wiki/API-Documentation) and the Python Docs (http://docs.python.org/library/index.html) I was able to get the general idea of what each script was doing. While going through this code, I did start documenting what each script does (although I probably should have posted it onto the CS-Dept Wiki). I plan on continuing to go through these Python scripts and try to document what each one does even more (which is now posted on the wiki ;-) ) http://cstest.worcester.edu/wiki/pmwiki.php?n=Main.EUTesterPythonScripts

 

From the blog nathandoe » WSU CS by nathandoe and used with permission of the author. All other rights reserved by the author.

3/12-3/25

Progress is being made with our class’ cluster.  Hopefully we will soon be able to add images to it, which will allow us to progress a bit futher in using the Eutester program.  For my part, I have been trying to figure out why I could not ssh into any of the instances started from a Ecalyptus image.

This was a vexing issue, made more so because others in the class were able to make it work.  I ended up installing Euca2ools onto 3 different computers in order to compare the results I got on each.  The problem came down to not being authorized to establish an ssh connection to an instance I had started.  I tried multiple times deleting and adding keypairs, and combing through the available images, thinking that somehow it was the images fault I could not establish a connection.

I finally had the choice to add an instances public key to a known_hosts file in the .ssh folder (I assumed I was doing everything up to this point correctly), but still received an error and no connection was made.  On a whim, I went into the .ssh folder and deleted the files I found there.   My next attempt was successful.

Contemplating why this would occur, I realized that the IP address used for each session remained the same, no matter if multiple instances had been created, either from the same or different images.  The problem was that ssh was uncomfortable with this and so refused to connect.

I performed an experiment, to see if I could duplicate this error.  I established ssh connections to 2 different instances of 2 different images.  I then terminated those instance and restarted them in reverse order.  This established those instances with different IP addresses.  I then attempted to ssh into the instances and got errors stating that the host keys had changed.  I feel as if this error should have been caught a while ago, but my ignorance of using ssh got in the way.

Fortunately, I can now start an instance at will and can resume becoming familiar with Eutester, with the goal of using and modifying this tool.

From the blog sflynn1976 » wsu-cs by sflynn1976 and used with permission of the author. All other rights reserved by the author.

Week of March 19th 2012

We had spring break this week at WSU so I tried to squeeze in some Eucalyptus time. This week the Eucalyptus web interface was made available and we were able to sign up for an account on our cloud. This was some pretty exciting progress. I’ve also been, once again, continuing to research and document for the “Architecture Overview” portion of the Eucalyptus documentation.

From the blog nzahid » WSU CS by nzahid and used with permission of the author. All other rights reserved by the author.

Week of March 12th 2012

This week I’ve been continuing to research and document for the “Architecture Overview” portion of the Eucalyptus documentation project. I also have been browsing and reading posts on http://planet.eucalyptus.com/. I found some great and interesting information on the site about Eucalyptus. I have also been trying out some of the commands that are documented in the Euca2ools documentation on the WiKi. Just another week of familiarizing myself further with Eucalyptus has been helpful.

From the blog nzahid » WSU CS by nzahid and used with permission of the author. All other rights reserved by the author.