Tag Archives: PowerShell

The PowerShell automation language

Blog Update 12/11/15

Sorry I haven’t been posting a lot lately. I’ve been moving house – well, moving two houses – and things have been rather hectic. Hopefully I’ll be properly set up soon and can get on to regular content creation, including some screencast material.

Upcoming talks from Me:

Sydney DevOps Meetup Nov 19th 2015What DevOps Means To Domain. Well, it’s what DevOps means at Domain as well as what DevOps means to Domain. I’ll run through how we Define the DevOps Ethos and some of the results we’ve produced.

This is a short-form talk and will be kind-of ad-hoc, with an Ask-Me-Anything at the end

PowerShell Australia Meetup 26th Nov 2015Unit Testing PowerShell with Pester. A rapid introduction to using Pester to automagically test your PowerShell code, and why you should be doing this, NOW.

This one will be accompanied by Ben Hodge talking about DSC, Kirk Brady telling us why we should be using git and how to do that, and then me blathering about Pester for probably far too long once everyone is tired. Beer and Pizza are, I believe, sponsored.

 

Reliable File Downloads with BITS

Every so often, one of my favourite cycle training video vendors releases a new video or two. These videos are generally multi-gigabyte files and downloading them through a browser, especially over a possibly-flaky wireless network, can be an exercise in frustration. Browser crashes happen, network blips happen, sometimes you even exit the browser session without thinking and terminate a nearly-complete download. That’s why I generally use BITS to download them, in PowerShell. How? Pretty simple, really. Just use the Start-BITSTransfer cmdlet, specifying source and destination, and you’re away.

Start-BITSTransfer http://www.myawesomevideosite.com/files/somebigfile $home\Downloads\somebigfile.zip

Running that will start your download, fire up a progress bar and some time later, you’ll have a usable file in your downloads folder. Of course, doing it this way will take over your PowerShell session for the duration of the download. Which is rubbish. Who wants to clutter up their desktop session with PowerShell windows? That’s why I do it asyncronously

Start-BITSTransfer -source http://www.myawesomevideosite.com/files/somebigfile -destination $home\Downloads\somebigfile.zip -asyncronous

Which is great. I can carry on using my PowerShell session in the foreground, or even close it, without interrupting the download process. I can even fire up another download next to the first one and just let them run in the background.

But how do I check on how the download is going?

I can use Get-BITSTransfer in any PowerShell session, and the BITS service will report the status of any currently running BITS jobs, like so

C:\> Get-BitsTransfer | Format-List

JobId               : d3c1a9a0-68f0-4831-939b-95ab0122476c
DisplayName         : BITS Transfer
TransferType        : Download
JobState            : Transferring
OwnerAccount        : DOMAIN\jason.brown
Priority            : Foreground
TransferPolicy      : Always
FilesTransferred    : 0
FilesTotal          : 1
BytesTransferred    : 208207360
BytesTotal          : 2430734370
CreationTime        : 27/10/2015 12:56:17 PM
ModificationTime    : 27/10/2015 1:09:08 PM
MinimumRetryDelay   :
NoProgressTimeout   :
TransientErrorCount : 1
ProxyUsage          : SystemDefault
ProxyList           :
ProxyBypassList     :

JobId               : 1d0a4b78-7b9c-4977-9b32-b962c754e8f6
DisplayName         : BITS Transfer
TransferType        : Download
JobState            : Transferring
OwnerAccount        : DOMAIN\jason.brown
Priority            : Foreground
TransferPolicy      : Always
FilesTransferred    : 0
FilesTotal          : 1
BytesTransferred    : 15883778
BytesTotal          : 2394848910
CreationTime        : 27/10/2015 1:08:02 PM
ModificationTime    : 27/10/2015 1:09:08 PM
MinimumRetryDelay   :
NoProgressTimeout   :
TransientErrorCount : 1
ProxyUsage          : SystemDefault
ProxyList           :
ProxyBypassList     :

You could even pick out the BytesTransferred and BytesTotal properties and do some quick math on them to see the percentage of download complete. There’s a whole load of stuff you can do with BITS to make your downloads complete more reliably.

Once you see your downloads are done, use the Complete-BitsTransfer cmdlet to save the file from its temporary location to your target.

Get-BitsTransfer | Complete-BitsTransfer

I’d recommend checking out the Get-Help and Get-Command output for these cmdlets to find out more if you want to get more advanced, or I might do a future blog post with some more advanced stuff like changing priorities, or downloading a list of files from a CSV or database. You can even use this system to do reliable uploads. It’s really a very handy set of cmdlets.

 

PoSHServer for Fun and Profit (and open-source props)

At my workplace, we use a lot of PowerShell. I mean a lot. I just did a quick script to count lines of PowerShell in my local git repos – admittedly very quick and dirty – and came up with going on for a quarter of a million. 221668 to be exact.

$x=0; gci -recurse -Filter "*.ps*1" | % { $x += (gc $_.FullName).count}; $x

And, of course, we’re a company that’s heavily bought into the API-first philosophy. Or at least the “while we’re rearchitecting, we’re doing the APIs first” philosophy. We deal with incoming and outgoing hooks and API calls a lot.

So it was kind of inevitable that eventually, the DevOps team would buy in to PoSHServer. Continue reading →

Notes to self: How do you know if a Redis cache is contactable?

I stood up a new Elasticache Redis cluster today for a colleague, and he was having trouble connecting. Often in AWS this means there’s a screwed up security group, but after checking the groups, he was still unable to connect.

So I logged into the staging server in question, raised my fingers to the keyboard and…

Realised I had no idea how to talk to Redis.

Continue reading →

Filtering resources by tag in AWS PowerShell

If you’ve worked with AWS PowerShell for any length of time, you’re probably well used to filtering resources based on attributes. For instance, grabbing any Autoscaling groups with a name that matches a given filter, like this.

Get-ASAutoScalingGroup | ? { $_.AutoScalingGroupName -like "production-*" }

Easy, isn’t it? Just uses the Where-Object cmdlet, with the filter parameter set to a simple -like match

And that’s about as far as many people go with Where-Object. Simple, first level matching. However when you’re dealing with AWS tags, you’ve got to do a bit more work. Tags are not exposed as first-level properties on your object. Instead, the Tags[] object is a first-level property, and the tags themselves are objects, with Key and Value properties. So you have to have a filter in your filter so you can filter while you filter.

With EC2, you can use the -Filter parameter on Get-EC2Instance, but Get-ASAutoScalingGroup doesn’t have this parameter. So you have to get smarter with Where-Object.

Luckily, the filter you pass into Where-Object is in fact a script block. You can do as much work as you like in there. It’s much more versatile than a simple string match. Let’s look, for example, at filtering AutoScaling Groups based on a tag named “Sleepytime” with value of “Yes”. I’ve expanded the properties a bit and added some formatting, to make it easier to read:

Get-ASAutoScalingGroup | Where-Object -FilterScript {
    $_.Tags | Where-Object {
        $_.Key -eq "Sleepytime" -and $_.Value -eq "Yes" 
    }
}

Or, as I’d have it in my own script

Get-ASAutoScalingGroup | ? { $_.Tags | ? { $_.Key -eq "Sleepytime" -and $_.Value -eq "Yes" }}

Taking this to its logical extent, you could take a huge object structure and zoom right in to a property many branches deep into the object tree, with a relatively readable filter structure. If you’ve read a big XML or JSON document into memory, for instance, this will allow you to filter by attributes buried far into the tree.

Of course, if your objects are extremely complex, there may be better, faster ways to query them, but in the case of AWS tags, this is a quick, simple and effective way of getting it done.

Friday, Friday. Gotta get down on Friday

A few of you might have spotted this via the Octopus Deploy June newsletter.

Friday

You might be wondering how it’s done, as at least one person has requested via Twitter. Well wait no longer. Here’s how it’s done.

What you’re looking at is a Slack notification from Octopus Deploy, featuring a little image that’s been doing the rounds on Twitter. It exists already as a Capistrano template, and I figured it wouldn’t be too hard to do in Octopus. Continue reading →

My new favourite feature

As our cloud server fleet grows, the DevOps team is increasingly nickel-and-dimed with small requests and troubleshooting tasks, for which we sometimes needed to RDP into instances. Indeed it’s for this reason that my Connect-RobotArmyv2Group and Connect-EC2Instance scripts were put together. But it’s far from ideal. For one thing, RDPing into individual instances is very Anticloud indeed.

But, we think we’ve got it under control now, and as per usual, it’s down to our trusty multi-limbed friend, OctopusDeploy.

Continue reading →

PowerShell quickie: Setting IIS App Pool IdleTimeout

I just did this and was frankly awash in a sea of misinformation, so here’s how it’s done.

$ValueToSet = 40
$AppPoolPrefix = "*"


$appPools = gci IIS:\AppPools | ? {$_.Name -like $AppPoolPrefix}

$appPools | % {
    $currentTimeout = $_.processModel.IdleTimeOut
    if($ValueToSet -ne $CurrentTimeout.Minutes)
    {
        $_.processModel.IdleTimeout = [TimeSpan]::FromMinutes($ValueToSet)
        $_ | Set-Item 
        Write-Host "Set" $_.Name "to" $_.processModel.IdleTimeout 
    }

}

What are the gotchas here?

Well one, just calling Set-ItemProperty does nothing but apply your change to an in-memory copy of the object. Not the object itself. Forget that solution

Second, you need to pass in a timespan object, not merely a straight number, or you’ll get unexpected results when it tries to set the timeout in ticks

Third: Do not forget the Set-Item call, or you’re utterly lost.

And that’s about it.

[update: I forgot to say, include:

Import-Module WebAdministration

At the start. I don’t have to, because my servers ipmo that module in their PowerShell profiles. But you’ll need to.]

 

Snippet: Extending Connect-MSTSC for fun and profit

If you’ve ever come across Jaap Brasser’s Connect-MSTSC script, you’ll know it’s a pretty cool script. If you have a cloud service such as AWS, and you haven’t leveraged this script in conjunction with Get-EC2PasswordData, then frankly you’re doing something wrong.

I’ve had several variations around this script in our AWS utility script for a while now. There’s Connect-EC2Instance, Connect-RobotArmyv2Group and a few other variations. And there’s the one I put together today, which demonstrates a couple of things, so I thought I’d share

  1. Using Filters to find EC2 instances with particular extended properties
  2. Leveraging open-source code to make your life exponentially easier.

First of all, filters. If you do a Get-Help on Get-EC2Instance, you’ll see the filter property prominently advertised, but what you won’t immediately see is how to use it. What -Filter expects to be handed is an object array of type Amazon.EC2.Model.Filter.

So let’s have a look at what that is

New-Object -typeName Amazon.EC2.Model.Filter | Get-Member 

   TypeName: Amazon.EC2.Model.Filter

Name        MemberType    Definition
----        ----------    ----------
Value       AliasProperty Value = Values
Equals      Method        bool Equals(System.Object obj)
GetHashCode Method        int GetHashCode()
GetType     Method        type GetType()
ToString    Method        string ToString()
Name        Property      string Name {get;set;}
Values      Property      System.Collections.Generic.List[string] Values {get;set;}

OK, so it expects a Name/Value or Name/Values structure. That’s pretty easy to create in an ad-hoc fashion, like so

@{ Name = "private-ip-address"; Values = "10.123.26.144" }

So, we can use this to find an instance with a given IP address, or a given tag, or a given keypair. There’s a whole list of filter properties in the detailed Get-Help output for Get-EC2Instance. So let’s put this together with Get-EC2PasswordData to give us a simple script that will find the Admin password and connect to an instance immediately

Function Connect-ByIp
{
param
(
  $ip
)

  $instance = Get-EC2Instance -filter @{ Name = "private-ip-address"; Values = $ip }
  $instanceid = $instance.RunningInstance.InstanceId
  $password = Get-EC2PasswordData -InstanceId $instanceid -Decrypt -PemFile \\tsclient\c\pemfiles\keypair.pem 
  Connect-Mstsc -ComputerName $ip -User Administrator -Password $password
}

And there you have it. A valuable script now has added value as an AWS tool. Win.

In Just One Hour I will make you a PowerShell Ninja

I just delivered the first session of a series here at work, running our C# and mobile devs through the basics of PowerShell, which is a pivotal component of our DevOps strategy. Here’s the slide deck, via SlideShare.

The real meat of the content is outside of the slides themselves, in the presenter notes and in the live code we type as we go through, but the slides will give a feel for what the session is like.