What isn’t DevOps?

I was sitting on twitter today and a stream of tweets rolled past that made me… a little annoyed at first, then sort of intrigued because of the amount of discussion and brain wrangling they triggered.

realdevops

To be fair to Chris, I agree with a lot of what he was saying in his Real DevOps stream. But I fear that among the good stuff were a few tweets were either worded badly or wildly off the mark. I’ll admit I was initially tempted to dissect individual tweets but I think it’s more interesting – and less churlish – to get the Yet Another What Is DevOps post out of the way.

So, let’s get one thing clear, and then I’ll get started. A lot of people don’t know what “Devops” actually means. Here’s what it is, and here’s an opportunity for you to get your “no it’s not” ready.

Devops is the application of lean manufacturing principles to the IT Lifecycle

Continue reading →

Filtering resources by tag in AWS PowerShell

If you’ve worked with AWS PowerShell for any length of time, you’re probably well used to filtering resources based on attributes. For instance, grabbing any Autoscaling groups with a name that matches a given filter, like this.

Get-ASAutoScalingGroup | ? { $_.AutoScalingGroupName -like "production-*" }

Easy, isn’t it? Just uses the Where-Object cmdlet, with the filter parameter set to a simple -like match

And that’s about as far as many people go with Where-Object. Simple, first level matching. However when you’re dealing with AWS tags, you’ve got to do a bit more work. Tags are not exposed as first-level properties on your object. Instead, the Tags[] object is a first-level property, and the tags themselves are objects, with Key and Value properties. So you have to have a filter in your filter so you can filter while you filter.

With EC2, you can use the -Filter parameter on Get-EC2Instance, but Get-ASAutoScalingGroup doesn’t have this parameter. So you have to get smarter with Where-Object.

Luckily, the filter you pass into Where-Object is in fact a script block. You can do as much work as you like in there. It’s much more versatile than a simple string match. Let’s look, for example, at filtering AutoScaling Groups based on a tag named “Sleepytime” with value of “Yes”. I’ve expanded the properties a bit and added some formatting, to make it easier to read:

Get-ASAutoScalingGroup | Where-Object -FilterScript {
    $_.Tags | Where-Object {
        $_.Key -eq "Sleepytime" -and $_.Value -eq "Yes" 
    }
}

Or, as I’d have it in my own script

Get-ASAutoScalingGroup | ? { $_.Tags | ? { $_.Key -eq "Sleepytime" -and $_.Value -eq "Yes" }}

Taking this to its logical extent, you could take a huge object structure and zoom right in to a property many branches deep into the object tree, with a relatively readable filter structure. If you’ve read a big XML or JSON document into memory, for instance, this will allow you to filter by attributes buried far into the tree.

Of course, if your objects are extremely complex, there may be better, faster ways to query them, but in the case of AWS tags, this is a quick, simple and effective way of getting it done.

Friday, Friday. Gotta get down on Friday

A few of you might have spotted this via the Octopus Deploy June newsletter.

Friday

You might be wondering how it’s done, as at least one person has requested via Twitter. Well wait no longer. Here’s how it’s done.

What you’re looking at is a Slack notification from Octopus Deploy, featuring a little image that’s been doing the rounds on Twitter. It exists already as a Capistrano template, and I figured it wouldn’t be too hard to do in Octopus. Continue reading →

Using Octopus Deploy? Push out code on Fridays?

I made a “Friday Deploy?” step template just for you, based on this famous snippet

{
  "Id": "ActionTemplates-610",
  "Name": "Friday Deploy?",
  "Description": "Is this a Friday Deployment?",
  "ActionType": "Octopus.Script",
  "Version": 1,
  "Properties": {
    "Octopus.Action.Script.ScriptBody": "$ascii = @\"\r\n┓┏┓┏┓┃\r\n┛┗┛┗┛┃ \\○/\r\n┓┏┓┏┓┃  ∕        Friday\r\n┛┗┛┗┛┃ノ)\r\n┓┏┓┏┓┃          deploy,\r\n┛┗┛┗┛┃\r\n┓┏┓┏┓┃          good\r\n┛┗┛┗┛┃\r\n┓┏┓┏┓┃          luck!\r\n┃┃┃┃┃┃\r\n┻┻┻┻┻┻\r\n\"@\r\n\r\nif((Get-Date).DayOfWeek -eq \"Friday\")\r\n{\r\n    Write-Host $ascii\r\n}\r\n"
  },
  "SensitiveProperties": {},
  "Parameters": [],
  "LastModifiedOn": "2015-06-24T03:08:23.003+00:00",
  "LastModifiedBy": "jason.brown",
  "$Meta": {
    "ExportedAt": "2015-06-24T03:08:30.685Z",
    "OctopusVersion": "2.6.3.886",
    "Type": "ActionTemplate"
  }
}

DrillSergeant – a simple UserData manager for provisioning EC2 Windows instances

There’s a more full announcement due over at the Domain Tech Blog, however I figured I’d drop one here too.

Ever tried to automatically provision Windows EC2 instances in AWS and run into the “Windows needs a reboot” problem? Needed to run some more code after the reboot and been stuck for a decent solution? Had to Join to AD and then carry on installing things? Rename a server? Reload the registry after a change before continuing? Tearing your hair out working with Scheduled tasks and the RunOnce registry key?

Worry no more.

DrillSergeant, a PowerShell UserData manager derived from Domain’s Robot Army, is now available on GitHub.

Yes, I went through all the pain and came out with something that works, and works well, so we released it into the wild for everyone to use. Do feel free to grab a copy and try it out, and send in a Pull request if you have useful changes to add.

Future releases from the Robot Army will include DSC resources to manage S3 and EC2 resources, and some CloudFormation tools, and maybe, eventually, the Robot Army itself. If I can make it a tad more generic.

 

Uploading an SSL Certificate to IAM for use with ELB or Cloudfront using PowerShell

Because I only do this once or twice a year, I can never quite remember how to get it done. So consider this post an aide-memoire of sorts.

Generally, your SSL cert will turn up as one or (preferably) more *.crt files – one the actual certificate, and the other the verification chain – and you should also have a private key, probably as a *.key file.

Step one: convert these to PEM format using openssl.exe at the PowerShell prompt. Don’t have openssl? Download it here or use chocolatey, as follows

cinst openssl -y

Now, down to the business of conversion. Hopefully, you have an x509 format cert and chain. Convert them into the AWS-friendly PEM format like so:

openssl x509 -in mycertificate.crt -outform pem | out-file mycertificate.pem
openssl x509 -in mychain.crt -outform pem | out-file mychain.pem

Next, we do the same to the Private Key

openssl rsa -in mykey.key -outform pem | out-file mykey.pem

Now, how do we get these into AWS?

We use the Publish-IAMServerCertificate cmdlet, of course

For use in ELBs, we use this

Publish-IAMServerCertificate `
       -ServerCertificateName my-certificate-name `
       -Path / `
       -CertificateBody (gc .\mycertificate.pem -raw) `
       -CertificateChain (gc .\mychain.pem -raw) `
       -PrivateKey (gc .\mykey.pem -raw)

And to do the same for AWS CloudFront, well you need the /cloudfront/ path, like so

Publish-IAMServerCertificate `
       -ServerCertificateName my-certificate-name `
       -Path /cloudfront/ `
       -CertificateBody (gc .\mycertificate.pem -raw) `
       -CertificateChain (gc .\mychain.pem -raw) `
       -PrivateKey (gc .\mykey.pem -raw)

And your certificates will be uploaded to IAM and squirrelled away in the certificate store. You can check that it’s there by using

Get-IAMServerCertificates

Or

Get-IAMServerCertificate -ServerCertificateName <your new cert name>

Side Note: There’s a command called Get-IAMServerCertificate and another called Get-IAMServerCertificates – note the ‘s’. This breaks the powershell convention of a single command to operate on singular or multiple values. I would hope AWS would eventually fix this flaw and bring the SDK into line with published PowerShell conventions, but we will see.

Now, when you go to your CloudFront distribution or your ELB’s listeners tab, you should be presented your new Certificate in the dropdown for available certs, and you’ll have your content nicely secured.

On Warmup Scripts and HTTP Methods

So I’m rewriting our venerable GuyGarvie IIS warmup script at the moment. This script started life as a way to warm up a very big .NET application on IIS6 using a list of URLs. Guy would simply hit each URL in turn with a GET request, checking the status thereof and returning an error if a significant number returned a non-200 status.

But Guy is terribly slow. And his list of URLs is full of legacy pages, broken links, duplications and general horror. And the application he warms up has moved on.

For this reason, I’m in the throes of rewriting him, converting his URL list from XML to JSON, cleaning it up and trying to tweak his speed, maybe parallelising him a bit too. And I have called him LiamFray, for reasons explained in the footnote. Continue reading →

My new favourite feature

As our cloud server fleet grows, the DevOps team is increasingly nickel-and-dimed with small requests and troubleshooting tasks, for which we sometimes needed to RDP into instances. Indeed it’s for this reason that my Connect-RobotArmyv2Group and Connect-EC2Instance scripts were put together. But it’s far from ideal. For one thing, RDPing into individual instances is very Anticloud indeed.

But, we think we’ve got it under control now, and as per usual, it’s down to our trusty multi-limbed friend, OctopusDeploy.

Continue reading →

Capturing The Penguin

I recently observed, in one of my more lucid moments, that sometimes doing DevOps feels like hurtling down a railway, building the track as you go. Things get faster and faster and you’re constantly getting everything in a row just-in-time.

Later I realised that Wallace and Gromit, featuring those automation-obsessed northerners – is an end-to-end DevOps metaphor. Observe.