https://www.dossier-andreas.net/software_architecture/pipe_and_filter.html
The Pipe-And-Filter architecture is conceptually very simple. It essentially consists of breaking down one operation into a sequence of smaller operations, in which the input of each is the output of the previous. These operations are called “filters” and the connectors linking them are called “pipes”. Sometimes the terms “pump” and “sink” are used to refer to the initial input and final output, respectively. I think making up those last two terms is a little excessive, but overall I like the metaphor – it makes me think of an actual physical machine, which is similar to how I prefer to think about computing in general.
The most well-known example of this pattern is seen in Unix and Unix-like operating systems, which are also the most obvious example of this pattern’s utility. There is no need for any program to have word counting functionality, because its output can be piped into wc, a program that only does that. Similar functionality for pattern matching is provided by grep. With this ecosystem of programs in place, a skilled user of a Unix shell has a great deal of functionality available to them through composing these programs in different ways, as opposed to creating new programs from scratch that do slightly different things than what the programs on their machine already do. Another common example is compilers, which also function in a similar way in order to streamline and simplify the process of translating between languages. The OpenGL rendering pipeline has similar motivations, only for processing graphics instead of programs.
One drawback of a pipe-and-filter system is that it has the potential to draw too much overhead. Being able to pipe something into wc rather than counting words yourself is a more flexible solution, but it does involve running a separate program. At small scales (i.e. most use cases) this isn’t an issue, but if your data is large enough you may need to abandon this system.
Before looking into this, I wasn’t really aware of the pipe-and-filter architecture as a distinct pattern. I was aware of how the Unix ecosystem worked, but I thought the practice of piping together small programs in sophisticated ways was just a quirk of that operating system. I didn’t connect the dots that it was also the same basic concept being used in graphics pipelines, even though I was aware of them. I also have always regarded compilers as a little magical, and seeing that their workflow can be decomposed like this makes them seem a little more approachable.
From the blog CS@Worcester – Tom's Blog by Thomas Clifford and used with permission of the author. All other rights reserved by the author.


