Category Archives: Week 9

What is a REST API?

This week we began to discuss REST APIs. While it was explained in class, the concept kind of went over my head so I wanted to look more into what they are. While looking around I found a helpful article from sitepoint.com

The author explains it with an analogy. A REST API is a way for two systems to communicate over HTTP. This is like when you get car insurance. The insurance agency must get data from the government, credit agencies, banks and other systems.

He goes on to give an example of a REST API that just shows a bad joke.

He also talks about the other data communication standards like CORPA and SOAP. They are much stricter than REST which is not a standard but a set of constraints that must be satisfied for an API to be RESTful.

The next section discussed was something we covered in class, requests and responses. He also includes an example for the reader to try at home.

REST APIs also have some challenges associated with them. One is that there are multiple endpoints that can be used to fetch the same data like:

  • /user/123
  • /user/id/123
  • /user/?id=123

This can make is hard to be consistent on a large project with many developers.

It is also important to keep your application secure and a REST API provides another level that needs to be secured.

The author ends with additional articles and links to learn more about REST APIs.

A lot of what this article covered was also covered in class, specifically the part about requests and responses. I feel like what we discussed in class was clearer and more made more sense. This is probably because we were talking about it and could ask questions if necessary.

What was useful from this article was the REST API example. I did try the example myself and it worked with no issues, though this is to be expected as it was just copying and pasting the code. I would have liked if there were more explanation, but this was a simple example so not much more could be said.

The author here assumes that the reader is familiar with JavaScript. I am new to JavaScript, but this example was simple enough that I could understand what was going on. I don’t anticipate us doing an example this simple in class, but it was helpful, nonetheless.

https://www.sitepoint.com/developers-rest-api/

From the blog CS@Worcester – Half-Cooked Coding by alexmle1999 and used with permission of the author. All other rights reserved by the author.

JavaScript/Node.js

This week on my CS Journey, I want to look more into JavaScript and how it is used in docker. Although we did a few activities on JavaScript, I was still confused so I decided to read and research more into it. JavaScript is a text-based programming language used both on the client-side and server-side which is mainly for the web. Many of the websites use JavaScript on all browsers making JavaScript the most-deployed programming language in history. The name JavaScript is quote misleading due to the resemblance of java programming language however, JavaScript is completely different from the Java programming language. Both Java and JavaScript are written, assembled and executed differently, and each has dramatic differences when it comes to what it can do. JavaScript is mainly used for: Adding interactive behavior to web pages like Change the color of a button when the mouse hovers over it, displaying animations, creating web and mobile apps, Game development, and  building web servers.

Now let’s get into Node.js which is what is used in docker. Over the decade Node.js has enabled JavaScript programming outside of web browsers, which has a dramatic success of Node means that JavaScript is now also the most-used programming language among software developers. Node.js is an open-source, cross-platform runtime environment for developing server-side and networking applications. Node.js applications are written in JavaScript and can be run within the Node.js runtime or any other operating systems. some of the features I learned from the blog is that, Node.js  can be  Asynchronous and Event-Driven meaning that All APIs of the Node.js library are asynchronous, it essentially means a Node.js based server never waits for an API to return data. It can be very fast since it is being built on the Google Chrome’s V8 JavaScript Engine, Node.js library is very fast in code execution, and lastly, No Buffering- Node.js applications never buffer any data. These applications simply output the data in chunks. 

The diagram below also helped to understand the concept well.

I highly recommend that everyone read the assigned book on JavaScript because it is very helpful in terms of understanding the concepts and the book covers a wide variety of interesting topics. Also, I suggest that everyone follow this tutorial it is an example that shows how to get a Node.js application into a Docker container. The guide also helps to understand basics of how a Node.js application is structured.

Here is the tutorial: https://www.digitalocean.com/community/tutorials/how-to-build-a-node-js-application-with-docker-quickstart

Sources : https://learning.oreilly.com/library/view/javascript-the-definitive/9781491952016/ch16.html

https://www.tutorialspoint.com/nodejs/nodejs_introduction.htm

From the blog Derin's CS Journey by and used with permission of the author. All other rights reserved by the author.

RESTful API Basics

            As we start our learning for our final project in our class, we have started to learn about and work with RESTful API architecture style (REST API), so I have started some research and done further learning myself. I found a great article from by Margaret Rouse with an wide view of the application program interface, called RESTful API. She breaks down what it is, how it works, its uses, its constraints, challenges, and some background information. REST uses the HTTP requests of GET, PUT, POST, and DELETE for data use and access. This API is based on representational state transfer, which is what the acronym stands for. It works by, “[Breaking] down a transaction to create a series of small modules. Each module addresses an underlying part of the transaction.” Commands are used to obtain resources, and resource’s states is called a resource representation. As previously mentioned, the commands are: GET to retrieve a resource, PUT to modify/updated a resource, POST to create a resource, and DELETE to remove a resource. The idea is for users to access networked components with stateless calls, no information being retained by the REST services. This statelessness makes REST “useful in cloud applications.” They are scalable and can be “redeployed if something fails.” It is very useful for efficient internet usage and is very compatible for distributed environments, and is used by the likes of Amazon, Google, LinkedIn, and Twitter. With cloud and microservices growing in popularity and usage, REST is sure to grow alongside. REST adheres strictly to 6 key concepts. There must be a uniform interface under a single URL. It must be client-server based, dividing request-gathering concerns on the client and data access, workload management, and security on the server. There must be stateless operations, as mentioned previously. Any state information must be managed on the client. Caching is essential for resources unless “explicitly indicated that caching is not possible. It must be a layered system, like a hierarchy to the architecture. There must be code on demand, meaning that “a server will send back static representations of resources,” or “when necessary, servers can send executable code to the client.” Sticking to these requirements can be difficult, which presents the challenges of endpoint consistency, response time and data size, defining paths and locations, security, authentication, request scope, testing, and error handling. Since REST is useful for the aforementioned reasons, this is worthwhile and understandable though. I expect to use this information in my final project, which should be dealing in a REST API, so this will be very useful to me in the near future. It is very interesting and exciting, and is growing to the top of the computer science world.

Source: https://searchapparchitecture.techtarget.com/definition/RESTful-API

From the blog CS@Worcester – Marcos Felipe's CS Blog by mfelipe98 and used with permission of the author. All other rights reserved by the author.

A Strange Way to Use Get Requests

Strap in because this post is going to get very anecdotal.

Backstory

I have a history of running Minecraft servers for my friends and I. A few years ago, I learned how to port-forward a locally hosted server and use it to play on. While it’s not the most secure thing in the world technically, it works really well and it’s produced a lot of fun. For whatever reason, I was thinking about it again recently.

When I’m out of school, that’s usually when we play. I’ve slowly been learning more and more. I’m currently at the point where I can make intermediate datapacks for the game and use shell/bash/batch scripts to run the server. Over the summer for instance I created a shell script to run the server, back it up, and reload it if it crashes.

There’s still one major problem with my servers, however. I need to manually turn on one of my computers and run it. Then that computer has to stay on until I want the server off. It might waste power if no one is using it (also increasing the risk of the world becoming corrupted) and it’s just slowly going to wear out my hardware.

Recently, I remembered that I had two old laptops. Unfortunately, the newer one didn’t have a power cord. So I broke my way in and took out the ram, hard drive, and wifi card. Then I swapped those components into the other laptop. Lastly, I installed Lubuntu onto it. It works surprisingly well. I then set up NoMachine so I can remote into the machine without having it in front of me. The BIOS supports wake on LAN so I took a few hours to get that working. Now, I can leave that laptop plugged in next to my router and send a signal over the internet to wake it up. While the router blocks the magic packet required on ports 7 and 9, a simple port forward allows wake on LAN to work from anywhere. I would definitely not recommend that for most scenarios, but until I find myself the victim of attacks on my router, I think I’ll take my chances.

Now, my goal has been to create programs that run all the time that laptop is running to be able to receive requests to launch Minecraft servers. I’ll work out a script later for handling crashes and such, as well as a datapack for automatically stopping the server when no one has been online in 30 minutes or so. So I started work on a node server.

Node.js

While I am learning about node in my CS-343 course, I have already taken the Web Developer Bootcamp by Colt Steele on Udemy and I highly recommend it. This is the basis of my server so far:

const express = require("express");
const app = express();

const server = 
app.listen(process.env.PORT, process.env.IP, function() {
     console.log("The server has started.");
});

var minecraft_server1_on = false;

app.get("/minecraft/server1/start", function(req, res) {
     res.send("Turning on the server…\n");
     minecraft_server1_on = true;
});

app.get("/minecraft/server1/ping", function(req, res) {
     res.send(minecraft_server1_on);
});

app.get("/minecraft/server1/declare_off", function(req, res) {
     res.send("The server is off.\n");
     minecraft_server1_on = false;
});

app.get("/exit", function(req, res) {
     res.send("Exiting…\n");
     server.close();
});

app.get("*", function(req, res) {
     res.send("The server is running.\n");
});

Now, I’m certain that I’m making mistakes and doing things in a strange way. It definitely is not in a state I would use for anything other than a private server, and even then I’m still working on it.

Here’s the important parts. It’s an express app with a few get routes. I’m running this server from a bash script using nohup so that it can simply run in the background while the pc runs. However, I want to be able to stop the server from another bash script without using process ID’s so I don’t risk closing the wrong thing. While I was considering my options, I thought of something really interesting. I can use the curl command to perform a basic get request in a bash script. So I created a route /exit that when requested, it shuts down the server.

It’s incredibly simple and there are definitely ways around it, but I never even thought of using get requests in this way. There is information in the requesting of a web page, excluding the normal stuff. Just the fact that a request occurred can be useful. What this means is I can check if the server is running by pinging most routes. I can also both send and receive boolean data without any real complexity. Once again, I’m sure this violates some design principle. However, for something like this that is meant to be mostly casual and private, why should I need to set up proper post routes and databases when I can use this.

My method of turning on a Minecraft server now; I store the state in a variable in the node server. This is fine because when this server stops, all other servers should have stopped so I don’t need a database. Then, by pinging certain routes, I can turn a Minecraft server on, check if it’s on, and declare that I have turned it off somewhere else. The node server can maintain the state of Minecraft servers (I can possibly run multiple on different ports) as well as handle inside and outside tracking.

Keep in mind again, I’m looking for the easiest way I know how to make this work right now, not the best way to do it. So from here, I had no idea how to make the node server actually start a Minecraft server. I know how to run one from a bash command though. So I created a C++ program that will also run all the time. It can periodically check the status of the node server. For instance, if I send a request to the node server to turn on a Minecraft server, the C++ program can detect that change by running system("curl http://localhost:PORT/minecraft/server1/ping"); Once again, I’m using a UNIX command in C++ rather than using a wrapper library for curl because it was an easier solution for me. The node server can then return true or false. In fact, the C++ server won’t directly run the command. It will run a bash script that runs the command and stores output into a file. C++ can then read the file and get the result.

I’m currently still in the process of making this work. After this, I’ll make another node server hosted on Heroku that has a nice front end to allow myself and other people to request the laptop to wake on LAN, and then directly interact with those local node server. I may even make a Discord bot to allow people to simply message in chat to request the server to turn on.

Conclusion

Once again, I do not recommend anyone actually does this the way I have. However, the whole point of coding is to make a computer do what you want it to. If you can hide the get requests behind authorization (which I will probably do) as well as fix any other issues, this could be useful. It’s not even specifically about using get requests in this way. Abstract out and realize that it’s possible to do something in a way you never thought of, and it’s possible to use something in a way you never have. Consider what you know and explore what’s possible. Figure out whether or not it’s a good practice based on what problems you run into. I think that’s one of the best ways to learn and if you can find a functional example to fixate on, the way I have, you can find yourself learning new things incredibly quickly.

From the blog CS@Worcester – The Introspective Thinker by David MacDonald and used with permission of the author. All other rights reserved by the author.

Models

https://www.lucidchart.com/blog/types-of-UML-diagrams

https://c4model.com/

For this week we’ll be covering modeling which is an important aspect in not only software design, but also design in general. Modeling is a very simple concept; it is when one creates a visual representation of a project, program, or whatever someone is working on. It is a visual tool used to streamline and simplify the process of development. In software design, there are two specific types of models which are called UML and C4 diagrams. UML, which stands for Unified Modeling Language, is used to give a visual representation to the architecture, design, and implementation of a software system. For example, a software system by a group of interconnected boxes which symbolize each class in that system. A class is represented in UML as a box which is separated into three sections which hold the name of the class, its attributes, and its operations. C4 meanwhile is used to give a visual representation to the architecture at different levels of detail. For example, a software system is represented by an interconnected group of boxes symbolizing each element of that system. As one zooms closer, each element breaks down into smaller parts that make up of that component. This process spans four levels from the system context level, to the container level, to the component, to the code or UML level. The key difference between C4 and UML diagrams is that C4 uses the general framework of UML to visualize a software system at greater scales.

The general importance of these models is that before creating software, you have to form a plan. If one were to skip this step, they would likely run into multiple problems like forgetting to implement a part of the software, making a mistake that isn’t noticed until much later, or having no clear direction in development. Often times when these mistakes are made, developers would have to devote time to correct them when they could have created a model allowing them to develop more of the software in a shorter amount of time. Models can also be useful with how they can make peer review easier. If a developer wants a peer’s feedback on the general structure of their work, they can give them an easy to understand model rather than the code for the classes that make up an entire software system. Models are a very simple but also very important aspect to software development which is also the main reason why I chose to write about this topic.

From the blog CS@Worcester – Rainiery's Blog by rainiery and used with permission of the author. All other rights reserved by the author.

What is UML and the differences with OOM

The difference between OOM and UML by definitions.

Object-oriented modeling (OOM) is the construction of objects using a collection of objects that contain stored values of the instance variables found within an object. Unlike models that are record-oriented, object-oriented values are solely objects.

UML stands for “Unified Modeling Language. The purpose of UML is visually representing a system along with its main actors, roles, actions, artifacts or classes, in order to better understand, alter, maintain, or document information about the system. We can say that UML is the new approach for documenting and modeling the software

Object-oriented modeling is the process of construction of objects. Objects contain stored values of the instance variables. It creates the union of the application and database development. The union is then transformed into a unified data model. This approach makes object identification and communication easy. It supports data abstraction, inheritances, and encapsulation. Modeling techniques are implemented using OOP supporting languages. OOM consists of the following three cases: Analysis, Design, and Implementation

UML, short for Unified Modeling Language, is a standardized modeling language consisting of a set of diagrams. UML is used to help system developers clarify, demonstrate, build, and document the output of a software system. UML represents a set of practices that have proven successful in modeling large and complex systems and is a very important part of the development of object-oriented software and software development processes. UML primarily USES graphical notations to represent the design of software projects. Using UML helps project teams communicate, explore potential designs, and validate the architectural design of the software. Below we will walk you through what UML is, the history of UML, and a description of each UML diagram type, supplemented by UML examples.

Why we like UML? What is the benefit of UML?

As the value of software products increases, companies are looking for technologies to improve software production processes, improve quality, reduce costs, and shorten time to market. These technologies include component technologies, visual programming, and the application of patterns and frameworks. Companies are also looking for technology that can manage the complexity of systems as they grow in scope and scale. They also recognize the need to address periodic architectural issues such as physical distribution, concurrency, replication, security, load balancing, and fault tolerance.

In the end, UML provides users with an off-the-shelf, expressive visual modeling language so that they can develop and exchange meaningful models. It provides Extensibility and Specialization mechanisms for core concepts. It Independents of the specific programming language and development process. Provides a formal basis for understanding the modeling language. It encourages the development of the market for object-oriented tools. It supports higher-level development concepts such as collaboration, frameworks, patterns, and components. It integrates Best Practices.

Sources:

https://www.techopedia.com/definition/28584/object-oriented-modeling-oom

http://www.agiledata.org/essays/objectOrientation101.html

From the blog haorusong by and used with permission of the author. All other rights reserved by the author.

White Belt

The pattern “The White Belt” is all about becoming a student once again and not letting your knowledge or expertise in a certain subject get in the way of learning new things. The pattern also talks about how you should put aside past experience and unlearn things so that you can understand what you were struggling to learn. The pattern also wants you to use a new language or style to solve a problem so that you don’t fall into the same hole that you have been when it comes to learning. I often find myself in a similar problem when it comes to programming because I often use similar code in future projects or rely heavily on what I know works versus trying new things that may work better. This pattern is similar to other patterns but focuses on relearning or learning new things and setting aside past experience. I felt that this pattern is important to remember for my professional career cause I could see myself facing a similar problem to the one described in the pattern.
I think that an extremely important part of the pattern is that you should try to learn from other people. This part of the pattern is discussed and is the key focus of the pattern “Be the Worst”. I do not like the fact that they continue to blend patterns together since it makes it harder to distinguish between the different patterns. I think focusing on learning new things without allowing old knowledge to corrupt the new knowledge is an important part of learning anything and especially coding. With the way people can easily connect on common interests and topics on platforms like Discord, Twitter, FaceBook, and more, I can see myself joining different coding communities and interacting with other people that may have more knowledge on a topic than me, in order to learn more in different areas of coding. This pattern has given me a new way to avoid getting stuck in a rut that is common in the computer science field. Using different languages is important to better understand the inner works of what a function does and how it can be implemented in different ways.

From the blog CS@Worcester – Tyler Quist’s CS Blog by Tyler Quist and used with permission of the author. All other rights reserved by the author.

Kindred Spirits

The pattern “Kindred Spirits” in Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman, by Dave Hoover and Adewale Oshineye, addresses the need for social connections to other software developers. Fostering relationships with kindred spirits will create an environment that promotes curiosity and avoids stagnation. Groups share knowledge which allows members to learn from each other. The pattern warns of groups that will resist change and shun those who present change. Groupthink is also warned of in this pattern.  Conforming to some standards is necessary when participating in a group but that conformity should not discourage growth.

I have never been very good with making new connections. Most, if not all, of my computer science connections have been through school and, with graduation nearing, I will need to find new kindred spirits. “Kindred Spirits” suggests finding groups that meet in person, making this currently unadvisable (Social Distancing) but good to remember for down the road. While maybe not as effective as in person meetings, there are plenty of online groups to check out. The open-source communities seem like a good potential place to start. Also, many in person groups will have ways of communicating remotely so it still may be worth looking into.

“This pattern is simple in principle, and for some people (our extroverted cohorts) it is simple in practice. For others, however, it can be difficult.”

This line makes it clear that the difficulty I find with this pattern is common. My innate aversion to large groups and strangers has caused me to avoid attending events in the past, but it is becoming increasingly apparent that I will need to change that behavior.

While I find groups stressful, I agree with the pattern about the benefits of being part of one. Software development communities are great sources of knowledge and inspiration.

The pattern, “Kindred Spirits,” is one I will need to practice. Since I still have a great deal to learn, I think it will be imperative to find a group that will teach and inspire me. Other software developers will always have new things for me to learn if I seek them out.

From the blog CS@Worcester – D’s Comp Sci Blog by dlivengood and used with permission of the author. All other rights reserved by the author.

A Machine Learning Model That Recognizes Spoken Digits (Introduction)

This week, I managed to prove (to myself, at least) the power of MFCCs in speech recognition. I was quite skeptical that I could get anything to actually recognize speech, despite many sources saying how vital they are to DSP and speech recognition.

A tutorial on Tensorflow I found a couple of months ago sparked the idea: if 2-dimensional images can be represented as a 1-dimensional array and used to train the model, perhaps the same could be done with features extracted from an audio file. After all, the extracted features are nothing but an array of coefficients. So this week, armed with weeks of knowledge of basic Tensorflow and signal processing, I finally tried to get it to work. And of course, many problems arose.

After hours of struggling with mismatches in the shape of the data, waiting for the huge dataset to reload when I made a mistake, and getting no results, I finally put together the last piece of code that made it run correctly, and immediately second-guessed the accuracy of the model (“0.99 out of 100, right???”).

Of course, when training a model, a result this good could be a case of overfitting. And indeed it is, because it is only 95% accurate when using separate test data. And even this percentage isn’t the whole story. The test data comes from the same dataset, which has a lot of recordings of each digit, but using only 4 voices. It’s quite possible that there are patterns found in the voices that would not exist in other voices. This would make it great using a random sample from the original dataset, but possibly useless for someone else. There’s also the problem of noise, which MFCC is strongly affected by. So naturally, I recorded my own voice speaking digits and ran it with the model. Unfortunately, I could only manage approximately 50% accuracy, although it is consistently accurate with digits 0, 1, 2, 4 and 6. Much better than chance, at least!

This is a very simple model, which allows you to extract only MFCCs from an audio recording of a spoken digit (0 through 9) and plug it into the model to get an answer. But MFCCs may not tell the whole story, so the next step will be to use additional extracted features to get this model to perform better. There is also much more tweaking I can do with the model to see if I can obtain better results.

I’d like to step through the actual code next week and describe the steps taken to achieve this result. In the meantime, I have a lot more tweaking and refactoring to do.

I would like to mention a very important concept that I studied this week in the context of DSP: convolution. With the help of Allen Downey’s ThinkDSP and related lecture, I learned a bit more detail on filtering of signals. Convolution is essentially sweeping one signal over another to get a new signal. In DSP, this is used for things such as low-pass filters and adding echo to audio.

Think of an impulse as an instantaneous tone consisting of many (or all) frequencies. If you record this noise in a room, you will get a recording of the “impulse response”. That is, how all of the frequencies are affected by the room over time. The discrete Fourier transform of this response is essentially a filter, because it gives the amplitude of each frequency in the impulse response, including all echos and any muffling. Multiplying these amplitudes by the DFT of an entirely different audio signal will modify each frequency in the exact same way. And thus, to the human hear, this different audio signal will sound like it does in the same room. If this concept is interesting, I encourage you to watch the lecture and work through the examples in the book.

I think these topics may come in handy if I need to pre-process recordings, in the event that noise is in fact causing errors in the above model.

From the blog CS@Worcester – Inquiries and Queries by James Young and used with permission of the author. All other rights reserved by the author.

Learn How You Fail

All the big CEO’s and entrepreneurs will give you trite advice on their LinkedIn pages: you need to fail a lot in order to succeed a lot.

I can’t be too hard on anyone who wants to get this message out, because I had to hear it at some point too. Failure meant incompetence to me, not part of the path to success. But what if failures don’t stop? The “Learn How You Fail” apprenticeship pattern attempts to solve the problem of lingering failures despite increased success.

This is likely due to something you are repeatedly doing that causes the failures. Hone in on these behaviors, because changing them is the only way to stop failing. Furthermore, you may simply not be good at some things, and finding these things can prevent wasted time and effort.

A potential problem with this pattern is that a lot of failure is likely due to blind spots in our perception of ourselves, so it’s likely not easy to solve the problem of uncovering the cause of our failures. But as software developers, our skills in pattern recognition should be top notch. Carefully documenting your failures and how they connect should reveal the patterns.

A coworker asked me years ago what my chosen superpower would be. I answered that I would want to know everything. Omniscience. Besides briefly earning me the nickname “know-it-all”, this reflects my own desire to fix my unfixable flaws. Now, I do think significant effort should be put into things that you are especially bad at. But this pattern is a good reminder to let go of the things that you’ve tried to do and simply haven’t worked out.

Success is many cases is figuring out what you are good at, not about fixing all of your weaknesses. This is the reasons the US President has a cabinet, Congress has subcommittees, companies have divisions, and software team members have roles. We can’t all do everything.

My biggest issue with this pattern going forward will be striking a balance. There will always be a desire to learn as much as possible. Gaining more skill and broadening horizons may be tempting, but knowing when to give up in a specific area is the challenge. The CEOs who credit their success to past failures knew what to give up on within their chosen field. Almost always, this doesn’t mean choosing a brand new career; it means finding what you’re doing wrong in the current one.

From the blog CS@Worcester – Inquiries and Queries by James Young and used with permission of the author. All other rights reserved by the author.