Tag Archives: AWS

Amazon Web Services

You might be paying too much for S3

Actually… you almost definitely are. There’s almost a hoarder’s instinct with S3 and related cloud storage. You keep things in there that you might not need any more, because it can be hard to identify what’s needed. However that’s not what I’m talking about

I’m talking about Aborted multipart uploads.

S3 includes options for multipart uploads on large files. This makes uploads more reliable, by breaking them up into smaller chunks, and makes them resumable, by starting again from a failed chunk. A lot of S3 clients have multipart built-in, so you might well be doing a multipart upload without even knowing it.

However when a multipart upload aborts and does not complete, the slot can be held open – there is literally no timeout – and it’s an object in your account, for which you’ll be charged.

Luckily, AWS provide ways to deal with this. You just have to search them out

If you’re using PowerShell, as I am, you can use the Remove-S3MultipartUploads cmdlet. If you’re using, say, node.js, you can use s3-upload-cleaner. There’s a way to clean these up in your chosen SDK, you just need to know about it and act on it.

There’s even a way to do this with an S3 bucket lifecycle policy, as explained by Jeff Barr here.

Now go forth, and stop paying too much for S3. Also, clean out the attic while you’re in there. You don’t need most of those files, do you? Hoarder.

HOWTO: Whitelist Pingdom Probe IPs into AWS Security groups

This is something I’ve been meaning to write about for a while.

If you use pingdom for your monitoring, and you have a requirement to lock down your endpoints to a specific set of clients, you may have a painful job on your hands.

Some engineers I’ve spoken to have implemented a kind of proxy to forward pingdom requests through to their locked-down endpoints. Others rely on User-Agent detection to allow Pingdom probes through while denying other traffic.

In my case, I’ve implemented a powershell script that runs at intervals, checking Pingdom’s published Probe IP List and syncing it to my target Security Group. here’s how it’s done.

The very first thing you’ll need to do, if you haven’t already, is contact AWS Support and get your rules-per-group limit increased. By default, you get 50 (at the time of writing), and that’s not enough for this.

Then the code.

First up, you need a list of the IPs you want to whitelist other than pingdom. Not much use only opening your endpoint to the monitoring service, is it?

$whitelist = @(
   "123.123.123.123/32",
   "124.124.124.124/32",
   "52.52.52.0/24"
)

And so on. You may want to store this differently, but for me it’s just straight in the script. For now.

When you have those, you need to grab Pingdom’s probe IPs from their API

$probeIPs = Invoke-RestMethod https://my.pingdom.com/probes/feed

Excellent. Now, the pingdom addresses aren’t in CIDR format, so you need to convert them to CIDR and add them to the $whitelist array you set up earlier. For that, you need a function that does pipeline input.

Function Join-IPs # pipeline function to append to an incoming array of strings
{
    param
    (
        [Parameter(ValueFromPipeline=$true)]
        [string]
        $In,
        [string]
        $JoinTo = "/32"
    )
    process
    {
        return ("" + $_ + "" + $jointo + "")
    }
}

And then you just stick that in your pipeline and get back an array of al the IP ranges that are meant to be in your security group.

$ranges = $whitelist += ($probeIps | select -expand ip | Join-Ips -JoinTo "/32" )

And there you have a list of all the CIDR ranges that are meant to be in your security group’s ingress rule.

My rule literally only opens one port – 443 – so if you have multiple ports, you may want to do this differently. It also does nothing to try and compress down multiple adjacent addresses into a single CIDR, so if you need that, you’re going to need to do a little extra work.

Now, we compare the sec group’s existing rules, and the array we just obtained, like so

$targetGroup = Get-EC2SecurityGroup -Region $region | `
               ? {$_.GroupName -eq "s-fictional-svc-WebElbGroup-H97BD3LE36JI"}


$currentranges = $targetgroup.IpPermissions |`
               ? {$_.FromPort -eq 443} | select -expand IpRanges
$groupID = $targetgroup.GroupId

$diff = Compare-Object $currentranges $ranges 

$diff | % {
    # If the side indicator is =>, we add it
    # if the side indicator is <=, we remove it
    if($_.SideIndicator -eq "=>")
    {
        Write-Host "Granting Ingress perms to" $_.InputObject 
        Grant-EC2SecurityGroupIngress -GroupId $groupID `
                        -IpPermission @{
                                 FromPort = 443; 
                                 IpProtocol = "tcp"; 
                                 IPRanges = $_.InputObject; 
                                 ToPort = 443
                         }
    }

    if($_.SideIndicator -eq "<=")
    {
        Write-Host "Revoking Ingress perms from" $_.InputObject 
        Revoke-EC2SecurityGroupEgress -GroupId $groupId `
                        -IpPermission @{
                                 FromPort = 443; 
                                 IpProtocol = "tcp"; 
                                 IPRanges = $_.InputObject; 
                                 ToPort = 443
                         }
    } 
}

As you can see, we use Compare-Object to determine what needs to be added and what needs to be removed, and push just that rule up – or rip it out of – to the Security Group.

This technique can be used to whitelist any service that publishes its IPs in an API – in fact, if you’re whitelisting a client, you could get your client to publish their IP list to you and literally just put a script like this in place. Why do this crap manually? Let a script do it for you.

My favourite Cmdlet of the moment: Get-IAMAccountAlias

In common with many AWS Users at large organisations, I work across a number of AWS Accounts. We silo our “sandbox” experimental environments from our gold production environment by putting them in different accounts. We silo different parts of the business into separate accounts. I also have my own personal account registered in Powershell using AWS Stored Credentials, and we just opened two new accounts for a new business unit we’re about to onboard and migrate.

So I have a lot of accounts to work with. So it can be useful to know which one you’re authenticating to at any given time. And that’s why my favourite Cmdlet this week is

Get-IAMAccountAlias

All our accounts have easy to remember aliases, and I use them when I store credentials using Set-AWSCredentials. If I happen to forget which powershell session is running against what acccount, I just hit up Get-IAMAccountAlias, and doubt is dispelled (and, occasionally, I save myself from provisioning or deleting resources in the wrong account).

If you have multiple AWS accounts, you need to know about this one.

Notes to self: How do you know if a Redis cache is contactable?

I stood up a new Elasticache Redis cluster today for a colleague, and he was having trouble connecting. Often in AWS this means there’s a screwed up security group, but after checking the groups, he was still unable to connect.

So I logged into the staging server in question, raised my fingers to the keyboard and…

Realised I had no idea how to talk to Redis.

Continue reading →

Filtering resources by tag in AWS PowerShell

If you’ve worked with AWS PowerShell for any length of time, you’re probably well used to filtering resources based on attributes. For instance, grabbing any Autoscaling groups with a name that matches a given filter, like this.

Get-ASAutoScalingGroup | ? { $_.AutoScalingGroupName -like "production-*" }

Easy, isn’t it? Just uses the Where-Object cmdlet, with the filter parameter set to a simple -like match

And that’s about as far as many people go with Where-Object. Simple, first level matching. However when you’re dealing with AWS tags, you’ve got to do a bit more work. Tags are not exposed as first-level properties on your object. Instead, the Tags[] object is a first-level property, and the tags themselves are objects, with Key and Value properties. So you have to have a filter in your filter so you can filter while you filter.

With EC2, you can use the -Filter parameter on Get-EC2Instance, but Get-ASAutoScalingGroup doesn’t have this parameter. So you have to get smarter with Where-Object.

Luckily, the filter you pass into Where-Object is in fact a script block. You can do as much work as you like in there. It’s much more versatile than a simple string match. Let’s look, for example, at filtering AutoScaling Groups based on a tag named “Sleepytime” with value of “Yes”. I’ve expanded the properties a bit and added some formatting, to make it easier to read:

Get-ASAutoScalingGroup | Where-Object -FilterScript {
    $_.Tags | Where-Object {
        $_.Key -eq "Sleepytime" -and $_.Value -eq "Yes" 
    }
}

Or, as I’d have it in my own script

Get-ASAutoScalingGroup | ? { $_.Tags | ? { $_.Key -eq "Sleepytime" -and $_.Value -eq "Yes" }}

Taking this to its logical extent, you could take a huge object structure and zoom right in to a property many branches deep into the object tree, with a relatively readable filter structure. If you’ve read a big XML or JSON document into memory, for instance, this will allow you to filter by attributes buried far into the tree.

Of course, if your objects are extremely complex, there may be better, faster ways to query them, but in the case of AWS tags, this is a quick, simple and effective way of getting it done.

Web-enabled S3 bucket migration using OctopusDeploy and PowerShell

I recently had to move a fairly large, fairly heavily trafficked web-enabled S3 bucket between two different AWS accounts. This turned out to be ever so slightly more than a simple drag-and-drop or copy operation.

Why? Well because it was web-enabled, mostly. Web-enabled S3 buckets have some restrictions on what they can be called, and buckets must be given a unique name. So you can’t just create a new webenabledbucket.com.au bucket in account B, copy the data across, flip the DNS and delete the bucket from account A. AWS won’t let you.

You have to have an intermediate stage where your files can live while the old bucket is deleted and the new one is provisioned. Which is where OctopusDeploy and the mighty Robot Army came to the rescue. Continue reading →

Snippet: Extending Connect-MSTSC for fun and profit

If you’ve ever come across Jaap Brasser’s Connect-MSTSC script, you’ll know it’s a pretty cool script. If you have a cloud service such as AWS, and you haven’t leveraged this script in conjunction with Get-EC2PasswordData, then frankly you’re doing something wrong.

I’ve had several variations around this script in our AWS utility script for a while now. There’s Connect-EC2Instance, Connect-RobotArmyv2Group and a few other variations. And there’s the one I put together today, which demonstrates a couple of things, so I thought I’d share

  1. Using Filters to find EC2 instances with particular extended properties
  2. Leveraging open-source code to make your life exponentially easier.

First of all, filters. If you do a Get-Help on Get-EC2Instance, you’ll see the filter property prominently advertised, but what you won’t immediately see is how to use it. What -Filter expects to be handed is an object array of type Amazon.EC2.Model.Filter.

So let’s have a look at what that is

New-Object -typeName Amazon.EC2.Model.Filter | Get-Member 

   TypeName: Amazon.EC2.Model.Filter

Name        MemberType    Definition
----        ----------    ----------
Value       AliasProperty Value = Values
Equals      Method        bool Equals(System.Object obj)
GetHashCode Method        int GetHashCode()
GetType     Method        type GetType()
ToString    Method        string ToString()
Name        Property      string Name {get;set;}
Values      Property      System.Collections.Generic.List[string] Values {get;set;}

OK, so it expects a Name/Value or Name/Values structure. That’s pretty easy to create in an ad-hoc fashion, like so

@{ Name = "private-ip-address"; Values = "10.123.26.144" }

So, we can use this to find an instance with a given IP address, or a given tag, or a given keypair. There’s a whole list of filter properties in the detailed Get-Help output for Get-EC2Instance. So let’s put this together with Get-EC2PasswordData to give us a simple script that will find the Admin password and connect to an instance immediately

Function Connect-ByIp
{
param
(
  $ip
)

  $instance = Get-EC2Instance -filter @{ Name = "private-ip-address"; Values = $ip }
  $instanceid = $instance.RunningInstance.InstanceId
  $password = Get-EC2PasswordData -InstanceId $instanceid -Decrypt -PemFile \\tsclient\c\pemfiles\keypair.pem 
  Connect-Mstsc -ComputerName $ip -User Administrator -Password $password
}

And there you have it. A valuable script now has added value as an AWS tool. Win.

The case of the non-updateable DSC Resource

I’ve been working on something pretty cool here at Domain Tech. Well, I think it’s pretty cool. It’s a Powershell DSC wrapper around our CloudFormation-based server fleet, The Robot Army.

The idea behind doing it this way was to significantly streamline the config management, which in Robot Army v1 was a bit clunky and hard to manage, as well as being based on multiple templates, with multiple parameters specified in multiple places, with a concomitant risk of variance creeping in over time.

Anyway, there’s a full blog post in draft over at Domain Tech explaining Robot Army v2, and that’s not what I want to talk about today. What I want to talk about today is DSC Resource Schemas that don’t update. Continue reading →

Quickie: Need to quickly reset your IAM user password?

We try to keep logins to the AWS console to a minimum here. Automate as much as you can with PowerShell and CloudFormation, we say. Stay away from the UI, we say. Use the Visual Studio Toolkit if you must, using your access keys, we opine.

Well inevitably sometimes we end up forgetting our passwords when we do need to log into the AWS web console. And we have to reset our passwords.

There are a couple of ways round this. One, you can go grovelling to a colleague who has admin rights and ask them to help, or you can use your valid access keys.

Initialize-AWSDefaults -StoredCredentials storedcrednamehere 
Update-IAMLoginProfile -UserName <username> -Password <password>

This of course opens a further can of worms over whether you’re handling your keys properly if you’re stupid enough to forget your password (as I was this morning), but let’s leave that for another day, because this is just a snippet. Let’s likewise leave alone the high-security practice of only granting yourself a password when you actually need it and then taking it away later. Because this is just a snippet.

Cheers!