Category Archives: CS-499

Audio Feature Extraction

This week I’ve continued working on the application part of my Android application, but I’ve also started to “dig deeper” (apprenticeship pattern blog post coming soon) and learn a bit more about audio processing.

I started with general concepts of feature extraction a couple of months ago now. Understanding how to use Python libraries to extract features is simple enough, but actually understanding how they work and why they work is another story. This week’s research has revealed an underlying elegance to the concept of signal processing and helped me reach a higher level of understanding and excitement for this project.

The single best explanation has been a video by Youtuber 3Blue1Brown on Fourier Series. I recommend all of his channel because he has an elegant way of describing and visualizing every topic he speaks about. He helped me understand the beauty of Calculus, and in my process of digging deeper I wound up watching his video on the uncertainty principle which was surprisingly relevant to signal processing. Understanding the specifics of the math behind signals and waves, and knowing the fact that mathematical equations are a language used to describe straightforward physical phenomena is key. This knowledge makes daunting concepts easier to break down. Seeing the same concepts used in different contexts also helps solidify them in your mind. And if you’re implementing this in code, it will make it much easier to remember the necessary logical steps required to extract a feature.

This entire tangent (and however useful, it was an unexpected tangent) started with trying to better understand the types of feature extraction that are used in speech recognition. By the way, you know you’re digging deeper when an article with an estimated read time of 11 minutes takes you a few hours to get through with all the additional research.

And universally, as far as I can tell, the first step in signal processing and feature extraction is the Fourier transform, which is simply turning a raw audio signal into separate sine and cosine signals. I say simply, but as the 3Blue1Brown video states, this seems a bit like figuring out which colors make up a mixed up can of paint. It turns out, however, that clever math makes it quite obvious which summation of sine and cosine signals make up a complex signal. I encourage you to watch the video to understand why.

The summation of cosine and sine waves is considered the frequency domain, while the original signal is in the time domain. From the resulting frequency domain, the individual signals can be normalized by taking the log magnitude of the signals and performing an inverse Fourier transform.

This is a new concept called a cepstrum, and it is one of many possible transformations you can make on a signal to begin to analyze the data. Its usefulness comes from the ability to see changes in individual waves. Additional operations can be performed to reveal new insights into patterns in a signal. Determining which if these works best is part of the process.

These individual transformations would be very interesting to implement in code. I may not get a chance to do so for this project, but the understanding of the underlying operations will help in using existing libraries.

From the blog CS@Worcester – Inquiries and Queries by James Young and used with permission of the author. All other rights reserved by the author.

Android Audio Recording and Playback, and an Animation False Start

This week, I spent a lot of time with animations in Android. And yet, I still could not get transitions to work to my liking. After reading Android’s docs, I resorted to a few different videos and tutorials, some of which seemed straightforward but were using methods of Android past.

The goal was to animate the title of a single audio file in a list, to move to the top of the screen and become a heading for details about that file. Animations are easy enough if you’re transitioning between Activities. But it appears that having a RecyclerView in a Fragment to list audio files adds some complications. I did successfully animate between the screens, but the first item in the RecyclerView’s list was animated, and it abruptly changed to the correct title when the motion ended. The issue is that this transition animation requires two elements to have a shared “transitionName”. Because I am using Card objects to display each of the audio files, only one Card can use that transitionName and be animated.

The solution is to set the name in the RecyclerView’s ViewHolder, so that when objects are bound to a specific card, they can get a unique transitionName. This can then be applied to the Fragment’s View before the animation begins. Attempts to do this caused some problems due to Android’s Lifecycle. Many a blog post have been written on this subject, but I’d like to discuss it in the near future to gain a stronger understanding.

All of this is to say: I want to efficiently getting from point A to point B by realizing which subjects and features are most important. Understanding the Android Lifecycle is clearly more important than an animation, and apparently prerequisite knowledge. And recording and playback are at the heart of the app itself. So my progress in animation is stashed in Git and ready for me to continue once I accomplish these other tasks.

Luckily, getting audio to record and play back was a much more enjoyable process. This might be due to my greater interest in the feature, but I was able to break down the problem and troubleshoot issues much easier.

I erroneously believed that my simple spike project would easily translate to my app. Android’s guide to MediaRecorder and MediaPlayer made it simple to get something quickly up and running. However, using their code directly would create a nightmare of an Activity, which neither properly separate concerns nor follow basic OOP principles. Furthermore, I needed the recording to begin immediately upon opening a new “RecordActivity”. This caused some issues with Android’s lifecycle, so I took to opportunity to explore that. The problem came from trying to start recording in onCreate(), which did not provide enough time to load the MediaRecorder into memory. The solution was to start recording on the onResume() event. However, this may be called more than once in the life of an Activity, so I simply check if the MediaRecorder is currently recording, and start recording if it isn’t.

I spent a bit too much time trying to find the recorded audio files in phone’s physical storage. It seems they do not appear, and I haven’t found a good explanation for this. Luckily, Android Studio’s Device File Explorer did reveal that the files were saved and properly recorded audio.

From there, implementing audio playback as a Service (which in Android is essentially an Activity without a UI) was rather smooth. This also allows playback to be initiated from anywhere in the app by passing the file name in a single line of code.

I have always been a function-over-form kind of guy. I made significant progress on the app this week in the function department. Hopefully the form will come in time.

From the blog CS@Worcester – Inquiries and Queries by James Young and used with permission of the author. All other rights reserved by the author.

Android Activities and Fragments: Getting Them Straight

AndroidX has brought a few changes to the Android framework, but the general architecture remains the same. Likewise, the few years since I’ve first learned Android has completely changed how I feel about Android. At this point, I am developing the basic architecture of my independent study app.

There are a lot of conflicting opinions about Activities and Fragments in the Android Developer community. A few years ago with my limited Android experience, I did not completely understand. I likely don’t completely understand now. However, I have better tools and more programming experience to see how they should be used, as well as to make my own decisions on how to use them.

At first, I found myself paralyzed with confusion on how Google wants its developers to use Activities and Fragments. As an example of how programming concepts translate well to other technologies, learning Angular helped me understand the difference. Activities are a single “thing” that a user does, and can be thought of as a web page. A Fragment should be used for a modular UI component, and function as Components do in Angular.

This isn’t a perfect analogy, as the frameworks are very different, but this is a good way to proceed when deciding how to structure your app. Google’s Introduction to App Architecture guide is a great explanation, and the most important thing to remember is to maintain a separation of concerns. In the end, Activities and Fragments aren’t a significant part of your app. They contain your app. They are something your app uses to work within the Android framework, and your business logic should be elsewhere because Android will pause or stop any Fragments or Activities it needs to if, for example, memory is running low. There is no guarantee they will maintain state unless you take additional steps to ensure it does so.

In researching opinions on how to use them, some people mentioned that they’ve seen developers decide on having a single Activity, and adding all features with Fragments. This is a tempting solution, but that might result in complex logic to control navigation. Furthermore, Fragments are meant to communicate through their parent Activity. In a large app, this would likely result in many implemented interfaces and complicated callbacks. Bloated Activities a big NO.

Likewise, others mentioned only using Activities and not adding complexity with Fragments. This seems a bit more reasonable, but restricts reusability of the Fragments in the UI. Only one Activity can run at a time. The beauty of Fragments is they can easily be dropped into a layout and reused. If they are designed to interact through their parent Activity, two Fragments can be shown at the same time on a larger tablet, even if they must be shown on different screens on a phone. Creating Activities only would mean either creating new Activities with repeated code for a tablet, or reusing the phone UI at the expense of user experience.

I’ll reiterate: separation of concerns. In Android, or any framework, understand the philosophy behind a class and component before deciding to try to simplify things. It’s likely that they were designed to prevent the problems you will run into.

From the blog CS@Worcester – Inquiries and Queries by James Young and used with permission of the author. All other rights reserved by the author.

Machine Learning Problem Framing

Last week, I gave an overview of my planned independent study project. This week, I’ll give a bit more detail on what I’ve done.

I have a habit of preparing for classes before they start. I buy the text books, get an overview of the material, and prepare to apply learning techniques throughout the semester. This helps me identify problems before the semester starts and address them in class as I go. Likewise with this project, I had hoped to get as much as I could done with machine learning before this semester started to hit the ground running with the software portion. Naturally, my ignorance led me to assume the problem was easier than it actually is. After a great conversation with a communications professor, I realized the problems I was trying to solve had to be broken down.

Counter-intuitively, the broken down problem is more difficult. To recap, my project involves machine learning and audio signal processing. Although great leaps have been made in this field and many problems have been solved, they mostly use clever tricks to achieve the results they get. Take speech recognition for example: your text-to-speech software transcribes nearly 100% correctly. Machine learning models can use huge datasets of audio, as well as commonly-spoken phrases to decide which words you’re most likely to say. The result is that mumbling, stuttering, or ambient noise is a bit more forgivable. On the contrary, transcribing each and every syllable is not nearly as easy, and in fact it’s a problem that has yet to be solved. That’s a shame, considering I was hoping to transcribe an audio sample into phones as part of my process and I somehow doubt I can do it without first getting a PhD.

This realization has led me to take a quick course on machine learning problem framing. It teaches the process of developing an hypothesis and developing a model to prove it, as well as resisting the temptation to shoehorn a problem into machine learning when a heuristic solution is as good or better. I did manage to find examples of using machine learning to solve some of the problems I wanted to (dating back 20+ years, even), but unfortunately they were each limited in scope and would be difficult to use to make a cohesive app. My goal isn’t a Frankenstein project.

In an effort to dig deeper, I’ve also done an introduction to Pandas tutorial, and started on a Tensorflow tutorial. These are surface-level in and of themselves, but help in understanding the higher-level frameworks. My hope is that understanding the basics will allow me to create a model that has an exciting application. In the meantime, I can implement the prerequisite features in software: audio recording and signal processing.

I’m going to dive into more specific problems as the semester goes on and I get more comfortable with the topics. I’ve already come up with a list of ideas and find myself wanting to write posts on small things like Android project flavors and build configuration. Posting thoughts and lessons in a public place has been great for accountability, and my goal is to know these technologies inside and out 4 months from now.

From the blog CS@Worcester – Inquiries and Queries by James Young and used with permission of the author. All other rights reserved by the author.

CS-499: Independent Study Introduction

This semester, I’m building an Android app for an independent study.

The Proposal

After building a breadboard computer and beginning to understand electronics, I started to learn about audio electronics. This sparked (or reignited) a latent interest in audio processing. Working in a call center selling audio equipment is actually the reason I was motivated to return to school to study computer science, so I feel there is good reason to pursue it in my final semester.

I also began with Python, and moved to Android apps in my early programming learning process, so I’d like to refresh these skills and dig deeper. This project will serve as a constant reminder to how far I’ve come from those early struggling days.

So the app will use Python machine learning libraries to analyze user audio data and provide the user feedback based on this data. I am purposely being vague; not because I think I have the next big idea on my hands, but because I expect many changes as I struggle with the machine learning model.

Regardless of where the model winds up, this is a software development independent study. I will have a working, professional app within the next 4 months, using the technologies I have proposed.

The Motivation

Why, though? As an independent study, with an already-busy schedule, I’m going to have to set aside time each week when I work on this project, no matter what. Originally, I wanted to take Robotics this semester and I was signed up for it originally, but unfortunately there is not enough time in my schedule. On Tuesdays I’m sure I will find my mind wandering, dreaming of playing with robots instead of struggling through machine learning and Android Studio.

But that is part of my reasoning. I want to find the motivation to do things with a self-imposed deadline. These are tools I want to learn, to create and potentially sell a product. At the end of this degree, I want to be able to show a project to future employers that say, “this is what I did. Not because I had to, but because I enjoy it”. I want to be able to have users who give me unfiltered feedback. I want to fail, figure out why I failed, and eventually succeed.

Of course, I have done all of these to some extent already. But this is my following my current interests and goals.

The Progress

I have made a couple small spike projects to begin relearning Android and get started with Tensorflow. I have already built the back-end and gotten an app to communicate with it. I’ve also done basic user authentication.

When I first proposed this project, I set a schedule of features and tasks to complete. Due to other projects which used the same technologies and flashes of motivation I’ve already worked ahead a bit, but I still plan to complete each portion according to the schedule, as best as I can. The machine learning model will be concurrent work as I adjust it.

Next week, I will go into more detail on the tasks I’ve completed so far.

From the blog CS@Worcester – Inquiries and Queries by James Young and used with permission of the author. All other rights reserved by the author.

Preparing to Present

Over the past semester, I’ve been working with Dr. Vallejos to build a website for Massachusetts HOSA. At the conclusion of my independent study project, I will be presenting my project to the Computer Science faculty and other CS students. In preparing for this presentation, I came to a couple of realizations about what I’ve learned from this experience.

While I certainly think that I have improved upon my technical skills in CSS and PHP, I think that what is perhaps more valuable is the immense amount of real-world project management experience that I have gained. This experience has already allowed me to build a better understanding of project requirements at work and for the software development capstone project with AMPATH Informatics. Being able to understand the requirements of stakeholders is essential to delivering a product that meets their expectations. Asking the right questions the first time will prevent having to reach out again and again for clarification of the requirements. People are generally very busy and they will not be available to answer your questions or provide you with information. Whether it is a customer, manager, or product owner, it is best not to waste other people’s time with comeback questions because of your own failure to fully consider the project’s requirements.

I also believe that I greatly improved my personal software development process throughout this project. Although it took a couple of mistakes for me to learn, I am thankful that I made these mistakes in a safe environment and lost nothing but a few hours of my time. I was initially pretty careless, making customization changes to the theme files directly on the web server itself, not backing up, and not tracking any of my changes. After losing all of my theme customizations by updating the theme, I decided to make some changes to this process. I implemented Git version control, allowing me to make and test changes locally before pushing to the actual website as well as tracking changes incrementally and allowing me to rollback to any revision, as desired. I also implemented automatic offsite backup in Google Drive, which runs weekly to ensure that even if I do mess something up, there’s always a working copy safely stored elsewhere.

I have always been an avid believer in learning through experience, and the MassHOSA website project has been a fantastic opportunity to learn through my experiences. Not only have I had the chance to both sharpen my technical skills and widen my skill set, I have gained invaluable experience managing a project and working with stakeholders on bringing an idea from the conceptual phase through to a working product.

From the blog CS@Worcester – ~/GeorgeMatthew/etc by gmatthew and used with permission of the author. All other rights reserved by the author.

Thinking Like an End User

I am getting ready to deliver a website product that I have been working on for Massachusetts HOSA. Because I’ve been working on the development of the website for the past couple of months, I am familiar with where to find everything. Once the product is delivered, however, it will be updated and maintained by Massachusetts HOSA. While I would be perfectly happy to continue helping out with the website as needed, I would like to minimize the need for my involvement by making the website as self-sustaining as possible.

In previous blog posts, I already outlined the setup of automatic backups. This makes me feel much better about enabling automatic updates for the WordPress installation. With automatic upgrades enabled, the site will be kept secure and up to date as WordPress and plugin or theme developers release new versions. I would be weary of allowing automatic updates if I was unsure of whether or not there were current backups because of the possibility of an update breaking the site. Occasionally there are incompatibilities between different plugin/WordPress version combinations, or simply bugs in a release that could make the site unstable. In case of such a scenario, having a recent backup that can quickly be rolled back to is essential.

The second part of making the site self-sustaining is to write documentation for the use of this specific WordPress installation. While WordPress is already extremely well documented, this vast documentation can sometimes be difficult to navigate efficiently. I would like to pick and choose the essentials to include in a slimmed-down version of documentation to provide to MassHOSA as a guide for the maintenance and updating of the website. This documentation will include guides for use of the WordPress platform, use of the various plugins that are installed, and also references to the locations of various resources such as backups and styling files.

I am extremely thankful for the opportunities that working on this project has granted me. While I may have had some prior experience building WordPress websites, this was quite different. I got a much better idea of the various stages of a design project and experience working directly with stakeholders to turn specifications into a working, real-world implementation.

From the blog CS@Worcester – ~/GeorgeMatthew/etc by gmatthew and used with permission of the author. All other rights reserved by the author.

Transitioning

After meeting to discuss the current status of the website, there are only a few tasks that remain. Although I am still waiting on some of the design content such as images and social media links to be provided, I think that the website design will soon be wrapped up. Once this happens, the next step will be education and training on the use and maintenance of the site. This should not be too intense because of how intuitive WordPress is to use.

Part of this training will likely involve the transfer of the hosting off of my personal virtual server to a permanent host. While I do not mind hosting the site for the time being, I do not want to be responsible in the case that my server goes down. When using a well known hosting provider, you are paying for someone to take on this responsibility. I have prepared the site to be migrated, and I do not anticipate any issues with migration. WordPress is rather portable, not requiring much more than a few directories and a small database.

One item that still needed to be addressed was the size of the font used on the website. Although it appeared appropriate on my screen, it was difficult to read from a distance on higher resolution monitors. While I had tested the website in a few different browsers and even on my mobile phone, none of these allowed me to view the site as if I were using a higher resolution monitor. During the meeting, when viewing the site at a higher resolution, the text appeared to be “zoomed out” and was difficult to read in some of the lower contrast areas of the page.

The next thing that I will be looking at for the MassHOSA project is QuickBase. I am familiar with the platform because of an internship where I am currently auditing and validating user access to QuickBase. Despite this familiarity, there may be a few obstacles to making the desired changes. After a quick inspection of the application, many of the features required to make the desired changes to the application are blocked due to the QuickBase tier in use. I will be looking for workarounds and discussing the potential solutions during my next meeting.

From the blog CS@Worcester – ~/GeorgeMatthew/etc by gmatthew and used with permission of the author. All other rights reserved by the author.

Preparing to Migrate a WordPress Site

Now that I’ve got a functional website built for the MassHOSA project, it is time to start preparing to move it the website to its permanent home. Development has been straightforward partly because it has taken place while the server has been living on my personal virtual private server. With full SSH access to the development server, it was much easier to make server-side tweaks to various environment settings. Many of these tweaks had more to do with my server being misconfigured than with WordPress, however. I am hopeful that the permanent hosting environment that is selected will require minimal modifications. Many of the hosts that we’ve looked at, for example, have environments tailored specifically for WordPress hosting.

To begin preparing, I copied the entire WordPress directory to my local machine using SCP. While this took some time, I wanted to be sure that everything was transferred and remained intact. I did not necessarily trust that FTP was up to the task, as I have had some problems with file integrity after using FTP for large-scale file transfers. While there may have been many other contributing factors, I thought I would try SCP instead this time, at least for the downloading of the website files to my local computer. FTP may be the only option for uploading the files to the new host, as many shared hosts do not allow SSH access.

The next step of the preparation process was to export and download the contents of the database associated with the installation. Choosing how I export the tables is important, because of the limited privileges that may be available for importing the data on the new host. To ensure that I would be able to import the tables on the new host, I used the account used by WordPress to access the database, and exported all of the tables in the database. This way, even if the new host allows only one database, I will be able to migrate all of the necessary tables and simply update the wp-config file to point to the correct database.

Thankfully, if anything goes wrong during the setup of the site on the new host, I have the working installation on my virtual server to fall back on while working things out. I hope that I have not overlooked anything and that the migration will be straightforward and painless.

From the blog CS@Worcester – ~/GeorgeMatthew/etc by gmatthew and used with permission of the author. All other rights reserved by the author.

Using Git with WordPress

As part of my continued efforts to not lose all of my hard work, I’m implementing tools to help me track changes and have decided to use version control to do it. I’ve chosen to use Git because of my relative familiarity with the tool.

For a bit of background, my web server is running Ubuntu 16.04.3 LTS and the latest version of WordPress at the time of this writing, version 4.9.4. Because GitLab allows for free private repositories and the nature of the project makes a public repository undesirable, it was chosen over GitHub. One thing to note about this setup is that I have full shell access to the server, allowing me to install programs and edit properties as necessary to get things setup. When the website is eventually migrated to its permanent hosting location, some changes may be necessary to the following setup to accommodate the server implementation. Many shared hosting providers do not allow shell access, and a new strategy would need to be considered in this case.

I started the setup by performing a bit of housekeeping with

sudo apt-get update

and then performed the initial Git installation with

sudo apt-get install git

I then performed the usual Git setup, uploading my SSH user’s key to GitLab and setting my username/email with

git config –global user.name “Your Name

git config –global user.email “youremail@domain.com

After cd’ing to the directory of the website files, I issued the command

git remote add origin git@gitlab.com:MassHOSA/masshosa-website.git

An important step here is to make sure that no sensitive files are tracked by Git. I did this by adding a .gitignore with the following:

#————————

#  Main ignored items

#————————

/../wp-config.php

/wp-config.php

.maintenance

versionpress.maintenance

/.htaccess

/web.config

/wp-content/*

!/wp-content/db.php

!/wp-content/index.php

!/wp-content/plugins/

/wp-content/plugins/versionpress/

!/wp-content/mu-plugins/

!/wp-content/themes/

!/wp-content/languages/

!/wp-content/uploads/

!/wp-content/vpdb/

#————————

#  Log files

#————————

*.log

error_log

access_log

#————————

#  OS Files

#————————

.DS_Store

.DS_Store?

._*

.Spotlight-V100

.Trashes

ehthumbs.db

*[Tt]humbs.db

*.Trashes

at this point it was safe to issue a

git add .

and commit with

git commit -m “Initial commit”

and finally push changes with

git push –set-upstream origin master

And that’s all there was to it. I’m now tracking all of the changes that I’m making to theme and plugin files. These are the only files that I really care about reverting and recovering changes that I’ve made. Everything else is backed up regularly using Updraft.

From the blog CS@Worcester – ~/GeorgeMatthew/etc by gmatthew and used with permission of the author. All other rights reserved by the author.