The PowerShell Pipeline, explained

So, my previous post on PowerShell has prompted some responses, internally and externally. Sufficient that I did actually re-word some parts of it, and sufficient that I feel a need to be positive and offer something to take away the burn.

So let’s have a go at explaining the pipeline, shall we?

To do this, I’m going to give an example of doing something without the pipeline. I hope that by the end of this post, the value of showing the other way first will be clear. But I’ll say up front, if you have written code like I’m about to show, don’t fret. It still works. There’s just a better way.

The example I’ve chosen is this:

You’re deploying an IIS Web Application using PowerShell, and as part of your deployment process, you want to delete the previous web site(s) from IIS.

So, let’s dig in. I’m going to be quite thorough, and it’s fine to follow along in the PowerShell prompt. You will, of course, need IIS installed if you do, but don’t worry, at the end there’s an example or two that should work for everyone.

First, let’s look at Get-Website. Get-Website comes with the WebAdministration module.

C:\users\jasbro> Import-Module WebAdministration
C:\Users\jasbro> Get-Help Get-Website

NAME
    Get-Website

SYNOPSIS
    Gets configuration information for an IIS Web site.


SYNTAX
    Get-Website [[-Name] <String>] [<CommonParameters>]


DESCRIPTION
    Gets configuration information for an IIS Web site.


RELATED LINKS
    Online Version: http://go.microsoft.com/fwlink/p/?linkid=287868

REMARKS
    To see the examples, type: "get-help Get-Website -examples".
    For more information, type: "get-help Get-Website -detailed".
    For technical information, type: "get-help Get-Website -full".
    For online help, type: "get-help Get-Website -online"

Great. So, using this, we can get a reference to, say, the Default Web Site. So we can do this:

$website = Get-Website -name "Default Web Site"

Now, in that variable is a reference to the Default Web Site. You can echo it back to the screen by typing $website. And you get something like this:

Name             ID   State      Physical Path                  Bindings
----             --   -----      -------------                  --------
Default Web Site 1    Stopped    C:\inetpub\wwwroot             net.msmq localhost
                                                                msmq.formatname localhost
                                                                net.pipe *

Good stuff. Now, there’s a Remove-Website cmdlet. Let’s look at that.

NAME
    Remove-Website

SYNOPSIS
    Removes an IIS Web site.


SYNTAX
    Remove-Website [-Name] <String> [-Confirm] [-WhatIf] [<CommonParameters>]


DESCRIPTION
    Removes a Web site from an IIS server.


RELATED LINKS
    Online Version: http://go.microsoft.com/fwlink/p/?linkid=287891

REMARKS
    To see the examples, type: "get-help Remove-Website -examples".
    For more information, type: "get-help Remove-Website -detailed".
    For technical information, type: "get-help Remove-Website -full".
    For online help, type: "get-help Remove-Website -online"

Brilliant. OK, so we can pass that the Name property of our website, and it’ll delete the website.

$site = Get-Website -Name "Default Web Site"
Remove-Website -name $site.name

It works, but it’s got some redundancy in there. Let’s remove the redundancy

Remove-Website -Name "Default Web Site"

Yeah, that’ll work. Once. The second time you run it, it’ll return this.

Remove-Website : Cannot find path 'IIS:\Sites\Default Web Site' because it does not exist.
At line:1 char:1
+ remove-website -name "I don't exist"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (IIS:\Sites\Default Web Site:String) [Remove-Website], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.IIs.PowerShell.Provider.RemoveWebsiteCommand

This is no good. This is an operation that should be idempotent, that is it should behave exactly the same way every time. So let’s throw that in a try/catch block, so it’ll just quietly go away if we have a problem

try
{
   Remove-Website -name "Default Web Site"
}
catch
{
   # fail silently
}

Now, what if the site you want to remove is not the default web site, but the previous version of the app you’re deploying? And the name of the site you’re deploying is “$sitename”.

So you don’t really know if what you’ll have to delete is “Default Web Site” or $sitename.

No problem, try and delete both

try
{
   Remove-Website -name "Default Web Site"
   Remove-Website -name $sitename
}
catch
{
   # fail silently
}

Wait, that’s not going to work. If it doesn’t find “Default Web Site” it’s going to fall into catch, and never remove $sitename.

try
{
   Remove-Website -name "Default Web Site"
}
catch
{
   # fail silently
}
try 
{
   Remove-Website -name $sitename
}
catch
{
}

Hmmm… ugly. There has to be a better way, right?

No problem. If you don’t supply a name to Get-Website, it returns all the sites on the box. Let’s do this

$websites = Get-Website
foreach($site in $websites) {
    Remove-Website -name $site.name
}

OK, good. That’ll remove all the websites on a box. Nice. Unfortunately for you, this box hosts multiple websites. You’re going to have to filter down for sites that have $sitename in them. OK.

$websites = Get-Website
foreach($site in $websites) 
{
    if($site.name -like "$sitename*" -or $site.name -eq "Default Web Site")  
    {
        Remove-Website -name $site.name
        Write-host "Removed" $site.name
    }
}

Awesome. So this works. It looks like the kind of things most devs are used to. But there’s a lot of boiler plate code in there. It’s very wordy.

Also, there’s very little chance of using that as an ad-hoc command on, say, a misconfigured Windows Server Core machine during a 2am incident.

If only there was a better way.

Enter… The Pipeline

The pipe character – | – is your friend. On my keyboard, it’s off to the right. It’s shift-backslash. Char 124 in ASCII. A humble character, but so powerful.

What the pipe allows you to do in PowerShell is this:

It allows you to funnel the output of one expression into another. Consider:

Get-Service | Out-File "C:\temp\services.txt"

This is saying “Take this list of services and pipe it to the Out-File cmdlet”.

Or maybe this

Get-Content "file.csv" | ConvertFrom-Csv | ConvertTo-Json | Out-File "file.json"

This is saying “Open this file, convert the contents from CSV into a native powershell object, then take that object and convert it into a JSON string, then put that in a file called file.json”.

Or this

Get-Process | Where-Object {$_.Name -eq "itunes" } | Stop-Process

Which means “Oh no iTunes has hung again I don’t know why I still bother, really I don’t”.

The pipe character is glue that allows you to assemble small components into a larger, but still compact, machines for doing more complex stuff.

So, going back to our website example, we can remove every IIS website on the server with a simple command pipeline

Get-Website | Remove-Website

Extending this somewhat, we can remove every website that matches a name pattern, by using Where-Object

Get-Website | Where-Object {$_.name -like "$sitename*" } | Remove-Website

It doesn’t matter how many sites are returned out of this, they’ll just all be removed. And the beautiful thing about this is that if the previous command in a pipeline doesn’t return output, processing just stops.

Let’s say Get-Website returns nothing. We don’t even move on to Where-Object. Let’s imagine Where-Object doesn’t return anything… well, nothing is deleted. It’s safe.  We’ve achieved idempotency just by throwing in a little pipe character.

No need for those ugly try/catch blocks from earlier, either, and no need for those if statements inside foreach loops. The loops are done for you implicitly by the receiving cmdlet.

It’s as if the cmdlets are little machines in a factory line, and the pipeline is a conveyor belt running between each, shuttling a widget from the furnace to the extruder to the lathe to the polisher and finally packaging the widget in a box and sending it out into the world. Or in our case, destroying it, never to be seen again.

I like this metaphor, since DevOps is all about manufacturing and assembly-line metaphors for us.

But what if, out of curiosity, you want to see what sites were returned by Where-Object in this process and maybe do something with that information? It’s a little opaque as it stands.

Well, you could do this

Get-Website | Where-Object {$_.Name -like "$sitename*"} | Write-Output
Get-Website | Where-Object {$_.Name -like "$sitename*"} | Remove-Website

or better

$sites = Get-Website | Where-Object {$_.Name -like "$sitename*"}
$sites | Remove-Website
Write-Output $sites

Or, better still

Get-Website | Where-Object {$_.Name -like "$sitename*"} | Tee-Object -variable sites |  Remove-Website
Write-Output $sites

You can even squish that down a bit, if you use aliases, positional variables, and the implied Write-Output

Get-Website | ? {$_.Name -like "$sitename*"} | tee sites |  Remove-Website
$sites

And if all you want is information about what the cmdlets were doing, and you don’t care about having it in a variable, you don’t even need the Tee-Object or the second line. You can just chuck in a -verbose switch, bringing us back to the one-liner

Get-Website | ? {$_.Name -like "$sitename*"} | Remove-Website -verbose

Now, not all cmdlets can receive input from all other cmdlets, but that’s the nature of machines. You can’t take a widget extruder and glue it onto a doohickey polisher and expect the widget to come out as a perfectly polished doohickey, but there are all manner of cmdlets that can handle pipeline input in all manner of ways.Get-Help is your friend on this one. Try this:

Get-Help -name Remove-Website -Parameter Name

You’ll see this output

-Name <String>
    The name of the Web site to remove.

    Required?                    true
    Position?                    1
    Default value
    Accept pipeline input?       true (ByPropertyName)
    Accept wildcard characters?  false

As you can see, Remove-Website accepts objects with a property called Name across the pipeline and it will dutifully try to remove websites matching that name. So you could even pass it, say, the contents of a CSV with a “name” column

gc .\sites.csv | ConvertFrom-CSV -header ("name", "id", "purpose") | Remove-Website

This all looks kinda magical, until you start to think of the pipeline as a conveyor belt moving objects from place to place, or as literally a pipeline moving a fluid from process to process in a giant chemical factory. And all this magic is built-in to PowerShell.

You can even build your own Cmdlets to accept pipeline input, but I feel that may be an exercise for a later post.

Anyway, I hope that goes some way to explaining what the pipeline is and why it’s wonderful. Now go away and enjoy yourself, please.

 

Leave a Reply

Your email address will not be published. Required fields are marked *