Code Break: Making My Grading Easier

Downloading student assignment files from Blackboard as a single zip file saves a lot of time — you don’t have to individually open each “attempt”, download the file (renaming it in the process, so you don’t keep overwriting the previous file, since they are all named “Homework1.pdf” ;) ), and then move on to the next one. Instead you get one convenient .zip file that contains all of the assignment files.

Unfortunately, Blackboard does some other things that make your life a bit more difficult. Once you unzip the file, you will find:

  1. The student files are renamed from filename.ext to assignmentname_username_attempt_datetime_filename.ext
  2. A text file is created for each student named assignmentname_username_attempt_datetime.txt even if the student has not entered any text data or comments.

Checking all of the text files to see if they really contain a comment and deleting those that don’t, and renaming all of the assignment files to username.ext so that I can start grading them 1 This process takes 15 minutes or more per assignment, which certainly lowers my enthusiasm for grading.

Today, I decided that I should write some code to automate this task. The time it would take to write the script would be recouped in only a few assignments. I decided to write the script in Python because I could easily see how to do the string manipulations. My shell scripting string manipulations are not as good. I would have to learn how to do the file system manipulations in Python, but I figured that would be relatively simple.

The first step is getting a list of all the files in the directory (leaving out all of the subdirectories)2:

onlyfiles = [ f for f in os.listdir(dir) if os.path.isfile(os.path.join(os.curdir,f)) ]

The next step is filtering that list to get just the .txt files:

txtfiles = [ f for f in onlyfiles if '.txt' in f ]

Then you can search the contents of the textfiles. You’ll notice that there are two characteristic phrases that indicate no text data and no comments. You can just delete the files that contain both of those:

for f in txtfiles:
    file = open(f)
    contents = file.read()
    file.close()
    if 'There are no student comments for this assignment' in contents and \
       'There is no student submission text data for this assignment.' in contents:
        os.remove(f)
        print('Deleted', f)

After refreshing the list of files to be just the remaining files, you can go about renaming the files. They all have _attempt_ embedded in their filename. Then you want to strip off everything up-to-and-including the first underscore, and from the second underscore up to the file extension. Then rename the file.

for f in onlyfiles:
    if '_attempt_' in f:
        first = f.find('_') # location of first underscore
        second = f.find('_',first+1) # location of second underscore
        extension = f[f.rfind('.'):] # get file extension
        newf = f[first+1:second] + extension
        os.rename(f, newf)
        print('Renamed', f, 'to', newf)

There are probably other features I can add, but this works well enough for now. Back to grading…

Full code is on GitHub here.

  1. I may still have to convert some of them to PDFs, if the students have not followed instructions, since I grade them by marking up the PDFs on my iPad. But that’s something I’ll tackle later. For my programming classes, I do that with my grading scripts which are still a work-in-progress. ↩
  2. http://stackoverflow.com/a/3207973 ↩

From the blog On becoming an Eccentric Professor... » CS@Worcester by Karl R. Wurst and used with permission of the author. All other rights reserved by the author.

CS-140 Lab 1 Using Our New GitLab Server

Chad Day recently completed the installation of our new GitLab server (read about it here and here.) This project was precipitated by some issues I had been having in trying to teach the use of git earlier in our curriculum. I had been having the CS-401 Software Development Process students use git and github in their FOSS projects, but it was difficult for them seeing git for the first time and expecting them to use it intensively in a project in the same semester. They had asked a number of times, “Why don’t you teach this in an earlier course?”

So, I decided to try using it in the first programming course – CS-140 Introduction to Programming. While they don’t do any large projects in CS-140, they do work on their weekly labs as Pair Programming. Using git for the collaboration aspect (so they don’t have to keep emailing versions back-and-forth to each other) and as a way to submit their completed lab assignments to me (so I only have to receive one copy of the assignment per pair) seemed to make a lot of sense. In addition, I had attended a workshop at CCSCNE 2013 entitled “Git on the Cloud” which provided a methodology to do just that, which had encouraged me further.

The “Git on the Cloud” workshop suggested using Bitbucket, since it allows an unlimited number of private repositories.1 I’m very willing (in fact, I often require) students to make their code for senior-level projects public with an open source license. But private is important for coursework at the freshman level.

On the other hand, the “Git on the Cloud” methodology involved using a single repository per student, and a different branch for each assignment(!)  In other words, whenever you change branches/assignments, all of your other code goes away, and is replaced with the code for the current assignment.

After discussing this with Chad and Dillon Murphy, we decided that this was too confusing, and gives students an incorrect idea of how git should be used. Also, it would only work in pairs if the students worked in the same pair throughout the semester, and I like to have my students switch partners for each lab. So, I wrote my lab instructions using Bitbucket, but one repository per assignment, per pair.

I tried it out during my summer 2013 section of CS-140. It was a nice testbed — with only 6 students it was not too difficult to work out the bugs in the procedures. In another post I’ll explain how I had the students use the repository, and how I processed the repositories for grading (including the scripts that I wrote with some help from Stoney Jackson on a train ride from Providence to Philadelphia.)

The problem came when I decided to try it with my CS-135 Programming for Non-CS Majors class in Fall 2014. Soon after we started the first lab — the git lab — the student who had also been in the summer class could not add her lab partner as a collaborator to her Bitbucket repository. After a bit of investigation, we determined that while Bitbucket allows unlimited private repositories, you can have at most 5 collaborators — per account, not per repository — without paying. That had not been a constraint with a class of 6, but it certainly was with a class of 24, and it would just get worse as the students progressed to other courses.

At this point, I gave up on git for the semester and started looking for alternatives. I had used Gitolite in the past, which had worked well but had no web interface. I wanted something more like Github, and came across GitLab. I added installing GitLab to Chad and Dillon’s project of building a number of new servers for the department (see here and here.)

Once Chad had finished the GitLab install and worked out the kinks, we decided to test it by running through the CS-140 Lab 1 using the new server. I quickly updated the lab assignment to refer to a repository on our GitLab server, set up the repository on the new server, and had Dillon and Chad work their way through the lab to look for problems. They found a few typos, and a number of places where I had not replaced all the references to Bitbucket with GitLab, but otherwise it worked.

We had only one puzzling issue — Dillon was able to push changes to a repository he should not have had sufficient privileges to modify. It turns out that (not surprisingly) if you are a GitLab adminstrator, you have the ability to push to any repository on the server.

I’m looking forward to testing the server on a larger scale with 48 students in CS-140 starting in January.

  1. I’m aware that students can get multiple private repositories from GitHub, but that requires contacting GitHub and asking. The Bitbucket option just seemed easier. ↩

From the blog On becoming an Eccentric Professor... » CS@Worcester by Karl R. Wurst and used with permission of the author. All other rights reserved by the author.

Finding Documentation Becomes a Pain

Each time you go about setting up software you may encounter a series of issues or complications that can make the task much more difficult. In order to counteract these issues or complications, one can usually find some useful and comprehensive documentation relating to the problem and solutions on how to fix them. Perhaps the worst time someone can experience when trying to deploy a software is having trouble finding initial documentation in the first place.

Installing (or attempting to install Hadoop) this semester, has allowed me to gain a greater appreciation for documentation and complete step-by-step instructions. Looking through the documentation for Hadoop, I have noticed they have recently updated the software, and describe some of the advantages the newer version has. However, the install instructions they have seem to be for one of the original versions, and searching for other install instructions hasn’t yielded much help either. I’m sure most of you reading this blog can understand the frustration that must be associated with it. Finding the correct way to set up the base system so the different parts of Hadoop works has proven to be anything but a simple task.

At this point the strategy is to continue to look for any documentation that might be useful, use the pieces that I’ve found from different areas and put them together to see if something works, and look at contacting people who have set up the new system before. By following this plan, and devoting the necessary time to the project, should get results before the end of the semester.

At this point, my understanding of Hadoop is slowly growing and as I do certain things, I continue to update my own documentation which hopefully can be used to redeploy the system if necessary in the future.

From the blog WSU Comp. Sci. Servers » wsucs by dillonmurphy10 and used with permission of the author. All other rights reserved by the author.

Working as a Team but on Individual Projects

Although this is an independent study, at times I worked with Dillon Murphy. For the most part we worked on separate tasks, but there were a few we worked together on.

We did work together on all of the virtual server installs, also working with Dr. Wurst. Dillon and I were pretty familiar with server installs, but since the new CS server was being used for the Wiki (which I was focusing on), and the blogs (which Dillon focused on), it made sense to plan out and figure out what we needed. As I said in a previous post, MediaWiki relied on an Apache/PHP/MySQL setup, and luckily for Dillon WordPress relied on that as well.

During our individual projects, we would shoot ideas off of each other if we ran into issues. For example, for moving over WordPress, Dillon wasn’t too sure how to approach it but since I was familiar with PHP/MySQL applications I suggested that it was likely all he really needed to do was dump the database and restore it on the new server and then move over the code. There were a few issues with configuration files pointing to old URLs, but other than that everything worked out.

We worked together on getting the new CS server up, which worked out thanks to some guesswork and brainstorming to figure out the best way to do it.

Getting the cluster with fresh installs of CentOS was another collaborative effort seeing as we planned to use them for our next projects, Dillon focusing on Hadoop and myself focusing on Eucalyptus. We had a few ideas on how to utilize the cluster for both platforms. One suggestion involved installing a virtual machine hypervisor on each machine and having two Virtual Machines on each, one for Hadoop and one for Eucalyptus. Another suggestion was splitting the machines between each project. The suggestion we went with, after Dillon said it looked like it would work out, was having one CentOS install on each machine and installing both on each machine.

Once we figured out what we wanted to do with the network and resolved the issues with it, Dillon focused on getting the servers up. We finally got that up and running, so next is focusing on the final solo projects.

While Dillon focused on that and a bit before, I focused on getting GitLab working to usable conditions. Dillon and Dr. Wurst helped test and ensure that everything was working as it should.

It’s really helpful to work in a team, even if each member is focusing on a separate individual project. Being able to bounce ideas off of each other and resolve things more quickly. Working on your own can be productive for its own reasons, but it never hurts to have a helping hand available.

From the blog Five mvs of Doom by Chad Wade Day, Jr. and used with permission of the author. All other rights reserved by the author.

GitLab Part 2

I missed a few things I wanted to go over with GitLab, and I have an update on trying out CACerts.

I didn’t really go over GitLab or git itself and why it’s important.

To understand GitLab, first you have to understand git. Git is a revision control software for plain text files. What this means is, you edit the code, commit your changes, and edit it again, and your code changes are tracked. Linus Torvalds, the creator of the Linux kernel created it when he wasn’t happy with the other software that was available. Git is one of my personal favourite tools to use, and has helped me with development a ton. I’ve definitely made poor choices in improving code or deleting code I didn’t like and want to undo that- with proper use of git this is trivial. One really advantageous feature of git is that it’s decentralized- when you clone a project with git, all its contents are self contained on your computer and if you want to update with a remote server you can pull the changes from the server or push your changes to the server. You can even have multiple remote repositories to push and pull from, though in practice you won’t use that feature too much. Another useful feature is you can set the remote repository to be another location on your local machine- this can be useful if you’re away from the internet and worried about accidentally ruining your working directory.

GitLab is probably the best free software alternative to Github. Github is a web front-end for remote git repositories, bring a lot of additional features to git such as editing your code from the web, forking other projects and pushing changes back to main projects (useful for feature development), easy account and group management, and of course the advantage of keeping your code stored on a remote server in case of issues with your local machine. For an example of a Github project, you can check out the CS-401 project from Spring 2013 that I was a member of. GitLab offers pretty much all of Github’s main features and a familiar interface. The big advantage of GitLab over Github is that we can store the data on WSU’s location, as well as private repositories, which are a paid feature of Github.

So as far as our GitLab install goes, last night I looked into using a certificate from CACert. Turns out, to apply for a CACert you need to own the domain, so that was quickly scrapped. I don’t think we can get HTTPS working at this point. I tried a few things with the configuration but it *seems* like the issue is the self-signed certificate and obtaining a legitimate certificate I don’t think is possible. This isn’t a huge issue, though with SSH you have to install keys on the server so it requires a bit more documentation.

What needs to be done with the server from here? I think all we need to do is resize the partition and virtual hard drive the server is on. I believe initially we set it to a 16gb install, but if students are going to use it for many assignments in their classes I feel like that’d fill up pretty quickly.

From the blog Five mvs of Doom by Chad Wade Day, Jr. and used with permission of the author. All other rights reserved by the author.

Code Break: Data File Manipulations in Python

In my CS-135 Programming for Non-CS Majors class, one of the primary objectives for the students is to learn to work with collections of data in files. I’m always happy when this requires manipulations that can’t be performed with other tools that the students are comfortable with — thus motivating the need to learn to code.

This afternoon in class, students were working in groups on their final projects. Two groups came up against some problems in getting their data into a format that could be easily processed in Python. Both cases involved data that was only available in the form of PDF files.

The old standby of selecting text and pasting it into Excel did not provide nice columns of information. Our second attempt was to export the data as text.

Case 1

In the first case, we got text data that looked like:

Biology 306 N/A 306
Biotechnology 80 26 106
Business Administration 748 N/A 748
Chemistry 141 N/A 141
Communication 245 N/A 245
Communication Sciences & Disorders 218 N/A 218
Community Health 158 N/A 158
Computer Science 116 N/A 116
Criminal Justice 445 N/A 445
Early Childhood Education 80 19 99
Early Childhood Education, Non-Licensure 26 N/A 26

This looked promising – we’ve dealt with one-record-per-line-space-delimited data files in class before. You just need to read a line at a time, and use Python’s string split method to turn it into a list… But — wait! — the first item  is a variable number of words separated by spaces. That will make for some messy lists — they’ll all be of different lengths:

['Communication', '245', 'N/A', '245']
['Communication', 'Sciences', '&', 'Disorders', '218', 'N/A', '218']
['Community', 'Health', '158', 'N/A', '158']

Here’s the solution: Python lists can be indexed from the end using negative indices. So, we can definitely get at the last three values (numbers of majors — undergraduate, graduate, and total). Assuming a list in a variable department, they are at positions department[-3], department[-2], and department[-1] respectively.

But, what about the department name, which may be in multiple list items? Well, we can get it as a sub-list, using list slicing: department[:-3] yields:

['Communication']
['Communication', 'Sciences', '&', 'Disorders']
['Community', 'Health']

All that’s left is to concatenate them together into a single string:

name = ''
for item in department[:-3]:
    name = name + item + ' '

Full code is here: https://gist.github.com/kwurst/7761789

Case 2

In the second case, we got text data that looked like:

Boston    00350000    4368    65.9    15.2    0.8    2.1    15.9    0.1
Boston Collegiate Charter (District)    04490000    34    67.6    32.4    0.0    0.0    0.0    0.0
Boston Day and Evening Academy Charter (District)    04240000    162    13.0    55.6    0.0    6.8    24.7    0.0
Boston Green Academy
Horace Mann Charter School
(District)    04110000    72    70.8    26.4    0.0    1.4    1.4    0.0
Boston Preparatory Charter Public (District)    04160000    27    74.1    11.1    0.0    3.7    11.1    0.0
Bourne    00360000    145    90.3    4.8    0.0    2.1    2.8    0.0
Braintree    00400000    369    95.1    3.3    0.3    0.3    1.1    0.0

Which could be fixed the same way, except for the fact that some of the district names ended up broken across multiple lines. (I’m not sure why this happened, and it turned out that exporting the data in a different way fixed the problem. But I’d already found a solution, so I’m going to document it here…)

Working from the assumption that the district org code always starts with a zero (I know — not a good assumption, but it works in this case…), the solution involves checking for lines with no zero in them and concatenating them together. Then you can treat the lines as in Case 1.

for line in f:
    while line.find('0') == -1:
        line = line + f.readline()

Full code is here: https://gist.github.com/kwurst/7761789

From the blog On becoming an Eccentric Professor... » CS@Worcester by Karl R. Wurst and used with permission of the author. All other rights reserved by the author.

GitLab

Where do I start with GitLab?

We had to install 3 servers to finally get this up and running and I’m still not completely satisfied with it. It’s frustrating to set up. It’s great software when you get it up and running, there’s no doubt about that, but it sure is a pain to get up and running.

Before I had done any work on setting up a GitLab server for WSU’s CS department, I had a bit of experience with it. I had tried to set up GitLab on some distro of Linux before for playing around with at home following the install guide but didn’t get anything working with it. We had a GitLab server running at my work at one point using the BitNami installer, but there were a few issues with it so we scraped it for something else.

The first attempt was a GitLab server install with the BitNami installer. We set up a CentOS and I ran the BitNami installer and got something up and running pretty easily. However, there was the most bizarre issue I had seen- for some strange reason pages would just show up as blank in Firefox unless you refreshed. I couldn’t find anything covering the same issue at all. I didn’t think something with an issue like that was worth keeping around, so we scrapped that and set up another CentOS server.

Here I decided to find a guide for installing GitLab on CentOS instead of relying on an installer or anything like that. In hindsight this probably wasn’t the best idea, the official install guide recommends Ubuntu and we probably would have had a much easier time just going with Ubuntu server. However, we had CentOS on hand, and I much prefer using CentOS for a server as it has some pretty good default configuration files for a lot of applications.

I didn’t have any issues with setting GitLab up initially using this guide, but afterwards a few problems became apparent. First off, neither SSH or HTTPS cloning worked. After a bit of hassle we got SSH working and were fine with that and decided that things were good to go. We sent in the request to UTS and got a public IP and subdomain for the server. However, we quickly discovered that sending emails didn’t work. It turned out that it was trying to send emails from gitlab@<localip> which get rejected. Unfortunately, I couldn’t find anywhere to change where it was sending emails from. I changed configuration files all over the place but had no success at all in fixing that. It got to the point where I just settled on doing a complete reinstall with the subdomain in mind which would hopefully fix all the issues. I eventually decided that uninstalling GitLab would be too much of a hassle on its own, so I would have to make a completely new server install. After setting my mind on doing that I spent a bit more time trying to get the current one installed and totally broke it, and didn’t care enough to fix it.

So finally, the last install, the one we have up and running. Using the same guide as before but documenting any oddities I found so we’d have a reasonable installation guide, I got it running and working to a pretty reasonable state, but with a few bumps along the way. First off, the guide recommends running a test to make sure everything’s running before you set up the webserver. It gives an error if the webserver isn’t up and running, but I had no indication this was caused by the webserver not being set-up. I decided to just install the webserver and hope it fixed it, and thankfully it did. Secondly, I tried Apache as the webserver, as the guide had the option of using nginx and apache. nginx seems like what GitLab was developed for so I used that for the previous installs, but for this one I decided to go with what I was familiar with. Didn’t get anywhere with it, and just went back to nginx and go that working pretty easily. Thirdly, the SSH issue we had before came back. This issue had two parts: First off the SELinux permissions aren’t set correctly for the ssh keys file. I’m not sure the exact cause for this, but I think it’s because GitLab creates the file itself if one doesn’t exist, so the correct permissions don’t exist for it. It’s an easy fix if you can figure out that’s the issue, but there’s really no clear indication that’s the issue so I was stumped for a while until I decided to stop looking for GitLab related issues with SSH and instead CentOS related issues with SSH. It was a really simple command, restorecon -R -v /home/git/.ssh We didn’t have this issue with the other server installs, but I think in those cases I disabled SELinux altogether and this time I was trying to get it to work with it. The other issue was that CentOS by default locks accounts without passwords. The GitLab docs recommend editing the /etc/shadow file manually, but I found a command that accomplishes the same thing without editing important configurations: usermod -U git. Email was working fine too.

I still need to get HTTPS cloning working. I believe it’s an issue with using a self-signed certificate, so I’m going to try out using a CACert certificate and seeing if I can get it working from there.

From the blog Five mvs of Doom by Chad Wade Day, Jr. and used with permission of the author. All other rights reserved by the author.

MediaWiki + Going Live with the new CS server

In my first post of the semester, we set up a CentOS server that we planned to have be the new CS server. The old one was running a dated version of Redhat, and with the success of the server that was set up for CS401, the latest CentOS seemed like the way to go.

The tasks were split up between myself and Dillon Murphy. Dillon worked on moving over the blogs and I focused on getting a new wiki up.

The old wiki had been plagued by spambots and used a different software so Dr. Wurst figured just setting up MediaWiki and moving over what he could would be a good direction to go in.

MediaWiki is a PHP/MySQL wiki software built and designed for Wikipedia. Although it was built for Wikipedia, it’s a free software application and as such can be freely used by anyone. I’m not sure if it’s the most popular wiki software, but it’s definitely the one I’ve seen used around the most. Aside from Wikipedia, one notable site that uses the MediaWiki software is Wikia where users can easily create MediaWiki installs for arbitrary uses. It’s also the wiki software of choice we use at my work, which we mainly use for documenting various important information such as networking info, what’s running on servers, troubleshooting tips, and that sort of content.

A PHP/MySQL application relies on PHP for the code and MySQL for the database. These applications are usually pretty trivial to install, so long as you have php, mysql, and a webserver running. I’ve only ever really worked with Apache as a webserver, so we have the common Apache/PHP/MySQL set-up for this server. This is also the set-up one would use for WordPress, so it worked well for our multi-purpose server.

I’m pretty familiar with MediaWiki in a general sense, I remember installing it and playing around with it years ago because I found wikis interesting. I remember years ago it wasn’t as easy to install as it is now, with its main focus being to make Wikipedia better, but nowadays its really trivial to set up and get running. (It could have been that I wasn’t as used to installing web applications as I am now though, haha)

So there shouldn’t be any issues setting this up right?

Luckily there weren’t any issues getting this up and running. I had also recently set-up Wikipedia styled URLs for pages at work, so I was able to get that working with no problem either. After setting everything up, I created a user for Dr. Wurst and a user for Dillon to test that everything was working. Everything seemed to be running with no issues, so mission success!

The next step was to figure out how we were going to handle user authentication. Ideally, we don’t want it possible for spam bots to sign up at all due to what happened to the older wiki. Initially I set it up so users can not register at all until we figured things out. We decided against using LDAP authentication since we didn’t want to bother UTS too much. Originally we were thinking of editing the code to only accept registrations from the @worcester.edu domain- this probably wouldn’t be too difficult to implement, but I found that the Google Apps Authentication plugin accomplished pretty much the same thing, restricting logins to Google Apps users from a specific domain using OpenID authentication. Users would be created automatically on login if they didn’t exist, so it was an ideal solution. Unfortunately I found that this authentication method wasn’t enabled for the worcester.edu domain, so we sent in a request to UTS to see if they could enable it.

While we waited for an update on if it was a good idea to enabled and if its enabled, Dr. Wurst asked about setting up subpages. Subpages is a feature not enabled by default, but allows you to make pages with the title PAGE/Subpage which are pages by the name of “Subpage” that link to the original page “PAGE” right at the top for easy navigation. This was pretty trivial to setup, and it’s a pretty nice feature to have.

We also moved the server from testing to production as Dillon had also finished moving over the blogs. All we really needed to do for this was take the old server of its current local IP and move the new server to that IP and the domain and public IP would point to it. We weren’t 100% sure it would work out this way, but it seemed like that was the way it would work to me and we could always undo the changes if there were problems and send in a request to UTS to figure things out, but luckily everything worked out and the server was live.

Earlier today I got word that the OpenID authentication feature was enabled for the worcester.edu domain, so I enabled the plugin again and everything worked out as it should. This lead to two issues though: First of all it wouldn’t let me edit permissions of users with an @ in the name as it treats editing those users as editing users from another wiki for some reason. I found a pretty easy fix for that by changing what delimiter is used for that, from @ to #, allowing editing the @worcester.edu users as normal. The second issue was that the old accounts couldn’t be used any more, so I just redirected their user pages to the new accounts. Not the cleanest fix, but I looked into changing the users who made each contribution over and it seemed like a lot of work for an easy task, but it made sense for MediaWiki to do that as having it possible for people to “steal” contributions on a big site like Wikipedia wouldn’t be good for anyone.

As far as the Wiki goes, I’m pretty satisfied in how it turned out.

From the blog Five mvs of Doom by Chad Wade Day, Jr. and used with permission of the author. All other rights reserved by the author.

Semester Update (24 November 2013)

Moving to Thanksgiving week and nearing the end of classes, here is a semester update of where I stand on the current independent study moving and setting up servers, and where I plan to go from here.

I’ll start by covering some of the tasks we have accomplished with the computer science servers.

  1. We have successfully installed and setup a new server for computer science
    1. Running on a newer version of VMWare
    2. Server installed on CentOS 6.4
    3. Used to host computer science wiki and blog
  2. Moved over computer science blog and plugins
  3. Changed IP for machines
  4. Git server setup (with Chad)
  5. Install base systems on computer science cluster

Although this list appears to be in a good place, there is still a good amount of work that needs to be done with the time we have left this semester. Setting up the base system on the machines in the computer science server room was a good start to building a Hadoop cluster, but the real work lies in setting up the cluster itself and learning how it operates. After installing the cluster on the host machines, I plan on looking into how faculty can use the cluster for examples in class as well as some good assignment ideas, but that will be dealt with after the cluster itself is setup. The remaining list and order looks something like this:

  1. Present on current status of Independent Study
  2. Installing Hadoop cluster on machines
    1. Write and update documentation of the install
  3. Testing for any conflicts with Eucalyptus Cluster (with Chad)
  4. Testing and developing practical understanding of Hadoop cluster
  5. Developing examples/assignments professor’s can use with the Hadoop cluster

The first item on the list is to present what I’ve been doing this semester at the presentation computer science is holding to go over independent studies and internships (3 December 2013, ST-107). This is a good time to share with other students and faculty what’s been going on this semester and what plans I have for continuing the work.

It may be a bit difficult to finish the planned material before the end of finals (19 December 2013), but I will be continuing to work until the work is complete. More updates will continue as tasks on the list are completed.

From the blog WSU Comp. Sci. Servers » wsucs by dillonmurphy10 and used with permission of the author. All other rights reserved by the author.

What is Hadoop and Why are we Interested

Recently the computer science department at Worcester State has chosen to add two tracks to the major. One track is software development, which is similar to the current single track, and the other is Big Data Analytics. Because of the new track system, courses offered had to be changed which meant a change in a few course materials. Part of my independent study this semester has been to implement a Hadoop cluster, which is a tool that can be used for data analytics.

Part of big data analytics is dealing with very large sets of data. Many times, especially when we think of companies like Google or Amazon, it becomes evident that a single machine, or a couple of machines won’t get the job done. This is when distributing tasks throughout a series of machines works much more efficiently. One software platform that has been designed for this job is Apache Hadoop, which is installed on a cluster of machines that can handle large sets of data and jobs associated with this data much more efficiently.

From the Hadoop website (hadoop.apache.org):

“The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.”

In this independent study, I am looking to implement a cluster running Hadoop and testing some applications of this. By the end of the semester, the goal is to have a fully functioning cluster that can be used in the courses that deal with big data analytics, and have a complete set of install instructions that can be used to rebuild the Hadoop cluster at Worcester State in the event something goes wrong (something which is currently tough to obtain).

Keep checking for updates about the install, setup, and use of Hadoop here at Worcester State University.

From the blog WSU Comp. Sci. Servers » wsucs by dillonmurphy10 and used with permission of the author. All other rights reserved by the author.