Author Archives: Jason

HOWTO: Whitelist Pingdom Probe IPs into AWS Security groups

This is something I’ve been meaning to write about for a while.

If you use pingdom for your monitoring, and you have a requirement to lock down your endpoints to a specific set of clients, you may have a painful job on your hands.

Some engineers I’ve spoken to have implemented a kind of proxy to forward pingdom requests through to their locked-down endpoints. Others rely on User-Agent detection to allow Pingdom probes through while denying other traffic.

In my case, I’ve implemented a powershell script that runs at intervals, checking Pingdom’s published Probe IP List and syncing it to my target Security Group. here’s how it’s done.

The very first thing you’ll need to do, if you haven’t already, is contact AWS Support and get your rules-per-group limit increased. By default, you get 50 (at the time of writing), and that’s not enough for this.

Then the code.

First up, you need a list of the IPs you want to whitelist other than pingdom. Not much use only opening your endpoint to the monitoring service, is it?

$whitelist = @(
   "123.123.123.123/32",
   "124.124.124.124/32",
   "52.52.52.0/24"
)

And so on. You may want to store this differently, but for me it’s just straight in the script. For now.

When you have those, you need to grab Pingdom’s probe IPs from their API

$probeIPs = Invoke-RestMethod https://my.pingdom.com/probes/feed

Excellent. Now, the pingdom addresses aren’t in CIDR format, so you need to convert them to CIDR and add them to the $whitelist array you set up earlier. For that, you need a function that does pipeline input.

Function Join-IPs # pipeline function to append to an incoming array of strings
{
    param
    (
        [Parameter(ValueFromPipeline=$true)]
        [string]
        $In,
        [string]
        $JoinTo = "/32"
    )
    process
    {
        return ("" + $_ + "" + $jointo + "")
    }
}

And then you just stick that in your pipeline and get back an array of al the IP ranges that are meant to be in your security group.

$ranges = $whitelist += ($probeIps | select -expand ip | Join-Ips -JoinTo "/32" )

And there you have a list of all the CIDR ranges that are meant to be in your security group’s ingress rule.

My rule literally only opens one port – 443 – so if you have multiple ports, you may want to do this differently. It also does nothing to try and compress down multiple adjacent addresses into a single CIDR, so if you need that, you’re going to need to do a little extra work.

Now, we compare the sec group’s existing rules, and the array we just obtained, like so

$targetGroup = Get-EC2SecurityGroup -Region $region | `
               ? {$_.GroupName -eq "s-fictional-svc-WebElbGroup-H97BD3LE36JI"}


$currentranges = $targetgroup.IpPermissions |`
               ? {$_.FromPort -eq 443} | select -expand IpRanges
$groupID = $targetgroup.GroupId

$diff = Compare-Object $currentranges $ranges 

$diff | % {
    # If the side indicator is =>, we add it
    # if the side indicator is <=, we remove it
    if($_.SideIndicator -eq "=>")
    {
        Write-Host "Granting Ingress perms to" $_.InputObject 
        Grant-EC2SecurityGroupIngress -GroupId $groupID `
                        -IpPermission @{
                                 FromPort = 443; 
                                 IpProtocol = "tcp"; 
                                 IPRanges = $_.InputObject; 
                                 ToPort = 443
                         }
    }

    if($_.SideIndicator -eq "<=")
    {
        Write-Host "Revoking Ingress perms from" $_.InputObject 
        Revoke-EC2SecurityGroupEgress -GroupId $groupId `
                        -IpPermission @{
                                 FromPort = 443; 
                                 IpProtocol = "tcp"; 
                                 IPRanges = $_.InputObject; 
                                 ToPort = 443
                         }
    } 
}

As you can see, we use Compare-Object to determine what needs to be added and what needs to be removed, and push just that rule up – or rip it out of – to the Security Group.

This technique can be used to whitelist any service that publishes its IPs in an API – in fact, if you’re whitelisting a client, you could get your client to publish their IP list to you and literally just put a script like this in place. Why do this crap manually? Let a script do it for you.

Getting Docker up and running on Windows (the long way round)

OK, So Docker Native is available for Windows, and in the wake of NDC Sydney I figured it was about time I got properly started with it. But getting it up and running wasn’t easy.

The installation, actually, was a breeze, with a reboot or two to get Hyper-V working on my laptop. But come time to run hello-world, there were problems.

Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/hello-world/images. You may want to check your internet connection or if you are behind a proxy.

OK, that’s bad. It suggests networking is broken. So, after a cursory inspection of my Hyper-V settings (everything looked OK), it was off to the internet. I found this article, which talks through setting up the beta version. Pretty good, but no dice.

Continue reading →

Extending Pester for fun and profit

Of late, I’ve been working on a little side project to test a rather complex Akamai Property. We wanted to be confident, after making changes, that the important bits were still working as we expected them to, and for some reason there was no easy, automated solution to test this.

Obviously I decided I’d write my testing project in Pester, and so it was that I began writing a whole bunch of tests to see what URLs returned what status code, which ones redirected, which ones were cache hits and cache misses and what headers were coming back.

First up, I wrote a generic function called “Invoke-AkamaiRequest”. This function would know whether we were testing against Staging or production, and would catch and correct PowerShell’s error behaviour – which I found undesirable – and allow us to send optional Akamai pragma headers (I’ll share this function in a later post).

With that up and running, I could start writing my tests. Here’s a set of simple examples

Describe "An example test, to establish things" {
    Context "Hit up the homepage." {
        It "Should return 200" {
            (Invoke-AkamaiRequest -uristem /).StatusCode | Should Be 200
        }
    }

    Context "Hit up a non-existent page" {
        It "Should return 404" {
            (Invoke-AkamaiRequest -uristem /nonexistent.html).StatusCode | Should be 404
        }
    }

    Context "A redirect works" {
        It "Should gimme 301" {
            (Invoke-AkamaiRequest -uristem /redirectedfolder/nonexist).StatusCode | Should Be 301
        }
    }
}

Now, that last one, testing a 301, is interesting. Not only do you need to test that a 301 or 302 status code is coming back, you also need to test where the redirect is sending you. So I started to write tests like this

It "Should redirect /blog/ to /advice/" {
    $blog = Invoke-AkamaiRequest -path /blog/
    ($blog | select -expand statuscode | Should Be 301) -and 
    ($blog.headers.location | Should Be http://$tld/advice/)
}

And this worked fine. But it was a bit clunky. If only Pester had a RedirectTo assertion I could just throw in there, like so

It "Should redirect /blog/ to /advice/ " {
    Invoke-AkamaiRequest -path /blog/ | Should RedirectTo http://$tld/advice/ 
}

If. Only.

Oh, but it can!

Yes, you can write custom assertions for Pester. They’re pretty easy to do, too. What you need is a trio of functions describing the logic of the test, and what to return if it fails in some way. They are named PesterAssertion, PesterAssertionFailureMessage and NotPesterAssertionFailureMessage, where Assertion is the assertion name, in my case “RedirectTo”

For my particular case, the logic was to take in an HTTP response object, and check that the status was 301 (or 302), and match the Location: header to a specified value. Pretty simple really. Here’s the basic code:

function PesterRedirectTo($value, $expected)
{
    return [bool](($value.statuscode -eq 301 -or $value.statuscode -eq 302) -and 
                    $value.headers.location -eq $expected)
}

function PesterRedirectToFailureMessage($value,$expected)
{
    return "Expected to redirect to {$expected}"
}

function NotPesterRedirectToFailureMessage($value,$expected)
{
    return "Expected not to redirect to {$expected}"
}

I put these into my supporting module (not into the Pester module) and ran my tests. Happy happy days, it worked perfectly. Throwing different URLs at it resulted in exactly the behaviour I wanted.

All that remained was to make the failure messages a little smarter and make the Not assertion more useful, but I figured before I did that I should write this little post with the nice clean code before the extra logic goes in and makes everything unreadable.

You can probably think of several ways you could streamline your tests with assertions right now. I’ve also written RedirectPermanently and ReturnStatus assertions, and I’m looking into HaveHeaders and BeCompressed. I may even release these as an add-on module at some point soon.

You can probably think of things that should go right back into the Pester codebase, too. And there are a number of other ways you can customise and extend pester to fit your own use cases.

To summarise: Pester is not just a flexible and powerful BDD framework for PowerShell. It’s also easily extensible, adding even more power to your PowerShell toolbox.

Now get out there and test stuff.

Using PowerShell to extract and report on AWS CloudWatch Metrics

Here at Planet Domain we have a lot of day to day challenges in performance and cost management.

We built a fantastic CD pipeline that allows our developers to rapidly spin up new infrastructure, enabling them to quickly build, test and deploy new features with minimal assistance from the Ops team.

Unfortunately, it also allows our developers to rapidly spin up new infrastructure than then lays idle for the greater part of the day, sucking money out of budget that could otherwise be used on beer, pizza and Xbox accessories.

Now, Cloudwatch is great, but it does tend to involve a lot of clicky-clicky mouse-movey, clicky-clicky stuff in the AWS console. And once you’ve got a lot of points to monitor, the graphs become unreadable. And the filtering options are sometimes finnicky, and it’s not easy to automate things like CloudWatch Dashboards using CloudFormation.

So to get a report on average CPU utilization across our AWS Autoscaling Groups, I turned to PowerShell. Continue reading →

AWS Sydney Summit Roundup: Serverless is here

This past week saw AWS’s Sydney Summit at the historic Hordern Pavilion. While many saw the “coming soon” announcements as a slight disappointment (Lambda coming soon. API gateway coming soon. Everything else: coming soon), I – and am not the only one – saw it as a bit of a wakeup call.

In fact, this is really my only major takeaway from the summit:

Serverless architecture is coming. Get on board.

Lambda was launched to beta in late 2014 and to general availability in April 2015 to much fanfare, but as always with AWS, only in a subset of regions. Since then, it’s expanded out to encompass Virginia, Oregon, Ireland, Frankfurt and Tokyo.

It’s been much talked of in Australian Cloud circles – so much so that I freely admit to feeling a degree of Lambda Fatigue at meetups and conferences over the last year or so.

But the actual release of Lambda in ap-southeast-2 is imminent. And with it will come a fundamental shift in the way many of us in Australian Cloud circles need to think about Cloud infrastructure, because the potential advanatages in cost, development time and scalability are just too big to ignore. If you’re going all-in on cloud and your plans don’t include Lambda somewhere, then you’re probably doing something badly wrong. Continue reading →

NDC Sydney and other various things

I almost forgot to mention. I’ll be speaking at NDC Sydney in August 2016 on the topic of Windows Automation, Continuous Delivery, PowerShell, Robots and World Domination.

The speaker line-up is kind of stellar and this will, I guess, be the highest profile tech conference at which I’ve spoken thus far. I promise to rehearse deeply and not get distracted about trivia such as which idiom in which one should code.

I will be sharing the stage with Fabien Ruffin, who is also speaking at the Sydney AWS Summit on the topic of New Relic. I will be an wandering around that conference and asking people awkward questions while probably sporting a purple hair-dye job and generally making a nuisance of myself while people try to extract sensible answers about Windows workloads in the AWS Cloud.

I will also be around the Sydney DevOps and Cloud meetup scene a bit in the next few months, discussing Windows, Cloud and Automation stuff in-between moments of objectionable surliness and plugging my NDC session.

You have been warned.

A tedious and probably totally wrong post about idiomatic approaches to PowerShell

In PowerShell, there are many ways to do stuff.

This is a good thing. It’s what made Perl so attractive to me back when I wrote in unreadable languages. There was even an acronym. TIMTOWTDI. There Is More Than One Way To Do It.

This is good. It’s a great thing.

Which is what I thought of today, when browsing around the interwebs, I stumbled*, not for the first time, over a not-that-common but still sometimes-encountered PowerShell idiom for function declaration.

$func = {
    param($input)
    write "I am a function. Your input was $input"
}

&$func

Continue reading →

Stricter DSC processing under WMF 5.0

If you’re all being good little Windows opscoders, you’ll be using DSC. And if, like me, you’ve upgraded to WMF 5.0 (PowerShell 5.0), you may have noticed a few new warning messages popping up in your logs

You do look at your logs, right?

Good.

Well, you may have been seeing this little warning of late

WARNING: The configuration ‘ExampleDSCConfig’ is loading one or more built-in resources without explicitly importing associated modules. Add Import-DscResource –ModuleName ‘PSDesiredStateConfiguration’ to your configuration to avoid this message.

I’ve certainly been seeing it. It hasn’t done any harm. Nothing breaks, it’s just a warning. I’ll get round to fixing it eventually.

Well, in the last day or so, I rolled WMF 5.0 into my production server fleet – a couple of hundred servers all up – and today I’ve had developers asking me why their Octopus Deploy projects are generating warnings now.  Continue reading →