The PowerShell Pipeline, explained

So, my previous post on PowerShell has prompted some responses, internally and externally. Sufficient that I did actually re-word some parts of it, and sufficient that I feel a need to be positive and offer something to take away the burn.

So let’s have a go at explaining the pipeline, shall we?

To do this, I’m going to give an example of doing something without the pipeline. I hope that by the end of this post, the value of showing the other way first will be clear. But I’ll say up front, if you have written code like I’m about to show, don’t fret. It still works. There’s just a better way.

The example I’ve chosen is this:

You’re deploying an IIS Web Application using PowerShell, and as part of your deployment process, you want to delete the previous web site(s) from IIS.

So, let’s dig in. I’m going to be quite thorough, and it’s fine to follow along in the PowerShell prompt. You will, of course, need IIS installed if you do, but don’t worry, at the end there’s an example or two that should work for everyone.

First, let’s look at Get-Website. Get-Website comes with the WebAdministration module.

Great. So, using this, we can get a reference to, say, the Default Web Site. So we can do this:

Now, in that variable is a reference to the Default Web Site. You can echo it back to the screen by typing $website. And you get something like this:

Good stuff. Now, there’s a Remove-Website cmdlet. Let’s look at that.

Brilliant. OK, so we can pass that the Name property of our website, and it’ll delete the website.

It works, but it’s got some redundancy in there. Let’s remove the redundancy

Yeah, that’ll work. Once. The second time you run it, it’ll return this.

This is no good. This is an operation that should be idempotent, that is it should behave exactly the same way every time. So let’s throw that in a try/catch block, so it’ll just quietly go away if we have a problem

Now, what if the site you want to remove is not the default web site, but the previous version of the app you’re deploying? And the name of the site you’re deploying is “$sitename”.

So you don’t really know if what you’ll have to delete is “Default Web Site” or $sitename.

No problem, try and delete both

Wait, that’s not going to work. If it doesn’t find “Default Web Site” it’s going to fall into catch, and never remove $sitename.

Hmmm… ugly. There has to be a better way, right?

No problem. If you don’t supply a name to Get-Website, it returns all the sites on the box. Let’s do this

OK, good. That’ll remove all the websites on a box. Nice. Unfortunately for you, this box hosts multiple websites. You’re going to have to filter down for sites that have $sitename in them. OK.

Awesome. So this works. It looks like the kind of things most devs are used to. But there’s a lot of boiler plate code in there. It’s very wordy.

Also, there’s very little chance of using that as an ad-hoc command on, say, a misconfigured Windows Server Core machine during a 2am incident.

If only there was a better way.

Enter… The Pipeline

The pipe character – | – is your friend. On my keyboard, it’s off to the right. It’s shift-backslash. Char 124 in ASCII. A humble character, but so powerful.

What the pipe allows you to do in PowerShell is this:

It allows you to funnel the output of one expression into another. Consider:

This is saying “Take this list of services and pipe it to the Out-File cmdlet”.

Or maybe this

This is saying “Open this file, convert the contents from CSV into a native powershell object, then take that object and convert it into a JSON string, then put that in a file called file.json”.

Or this

Which means “Oh no iTunes has hung again I don’t know why I still bother, really I don’t”.

The pipe character is glue that allows you to assemble small components into a larger, but still compact, machines for doing more complex stuff.

So, going back to our website example, we can remove every IIS website on the server with a simple command pipeline

Extending this somewhat, we can remove every website that matches a name pattern, by using Where-Object

It doesn’t matter how many sites are returned out of this, they’ll just all be removed. And the beautiful thing about this is that if the previous command in a pipeline doesn’t return output, processing just stops.

Let’s say Get-Website returns nothing. We don’t even move on to Where-Object. Let’s imagine Where-Object doesn’t return anything… well, nothing is deleted. It’s safe.  We’ve achieved idempotency just by throwing in a little pipe character.

No need for those ugly try/catch blocks from earlier, either, and no need for those if statements inside foreach loops. The loops are done for you implicitly by the receiving cmdlet.

It’s as if the cmdlets are little machines in a factory line, and the pipeline is a conveyor belt running between each, shuttling a widget from the furnace to the extruder to the lathe to the polisher and finally packaging the widget in a box and sending it out into the world. Or in our case, destroying it, never to be seen again.

I like this metaphor, since DevOps is all about manufacturing and assembly-line metaphors for us.

But what if, out of curiosity, you want to see what sites were returned by Where-Object in this process and maybe do something with that information? It’s a little opaque as it stands.

Well, you could do this

or better

Or, better still

You can even squish that down a bit, if you use aliases, positional variables, and the implied Write-Output

And if all you want is information about what the cmdlets were doing, and you don’t care about having it in a variable, you don’t even need the Tee-Object or the second line. You can just chuck in a -verbose switch, bringing us back to the one-liner

Now, not all cmdlets can receive input from all other cmdlets, but that’s the nature of machines. You can’t take a widget extruder and glue it onto a doohickey polisher and expect the widget to come out as a perfectly polished doohickey, but there are all manner of cmdlets that can handle pipeline input in all manner of ways.Get-Help is your friend on this one. Try this:

You’ll see this output

As you can see, Remove-Website accepts objects with a property called Name across the pipeline and it will dutifully try to remove websites matching that name. So you could even pass it, say, the contents of a CSV with a “name” column

This all looks kinda magical, until you start to think of the pipeline as a conveyor belt moving objects from place to place, or as literally a pipeline moving a fluid from process to process in a giant chemical factory. And all this magic is built-in to PowerShell.

You can even build your own Cmdlets to accept pipeline input, but I feel that may be an exercise for a later post.

Anyway, I hope that goes some way to explaining what the pipeline is and why it’s wonderful. Now go away and enjoy yourself, please.


Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">