Luke Waite

NVM Default Global Packages

2018-01-17T10:45:00+00:00

Did you know that nvm supports a list of default global packages to include when installing a new node version?

Default global packages from file while installing

If you have a list of default packages you want installed every time you install a new version we support that too. You can add anything npm would accept as a package argument on the command line.

 # $NVM_DIR/default-packages
 
 rimraf
 object-inspect@1.0.2
 stevemao/left-pad

Personally, I use this to make sure gulp-cli is installed in new node versions.

Viewing the Logstash Dead Letter Queue in Kibana

2017-10-13T09:00:00+00:00

At Intouch Insight our logging infrastructure is our holy grail. The engineering team relies on it every day, so we need to keep it up to snuff. I was lucky enough to be able to update our ELK cluster this week to 5.6 - a huge upgrade from our previous stack running ES 2.3 and Kibana 4.

One of the features I was most looking forward to was the dead letter queue that was introduced in Logstash 5.5.

The Problem

The documentation surrounding the usage of the dead letter queue mostly revolves around re-processing rejected events. I wasn’t particularly interested in that use case; I just wanted to be able to easily see when events were rejected.

The Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
input {
    dead_letter_queue {
        path => "/usr/share/logstash/data/dead_letter_queue"
    }
}

filter {
    # First, we must capture the entire event, and write it to a new
    # field; we'll call that field `failed_message`
    ruby {
        code => "event.set('failed_message', event.to_json())"
    }

    # Next, we prune every field off the event except for the one we've
    # just created. Note that this does not prune event metadata.
    prune {
        whitelist_names => [ "^failed_message$" ]
    }

    # Next, convert the metadata timestamp to one we can parse with a
    # date filter. Before conversion, this field is a Logstash::Timestamp.
    # http://www.rubydoc.info/gems/logstash-core/LogStash/Timestamp
    ruby {
        code => "event.set('timestamp', event.get('[@metadata][dead_letter_queue][entry_time]').toString())"
    }

    # Apply the date filter.
    date {
        match => [ "timestamp", "ISO8601" ]
    }

    # Pull useful information out of the event metadata provided by the dead
    # letter queue, and add it to the new event.
    mutate {
        add_field => {
            "message" => "%{[@metadata][dead_letter_queue][reason]}"
            "plugin_id" => "%{[@metadata][dead_letter_queue][plugin_id]}"
            "plugin_type" => "%{[@metadata][dead_letter_queue][plugin_type]}"
        }
    }
}

How it looks

Dead Letter Queue - Logs

Dead Letter Queue - Detail

Next steps

I’d love to rewrite this as a codec that could be applied to the input, but that’s currently not possible as the dead_letter_queue input plugin doesn’t make any calls to the codec specified in it’s config.

PHPStorm Tips - Go to Implementation

2017-08-21T22:00:00+00:00

Laravel is a great framework; it’s easy to use and extend, and makes liberal use of Interfaces so that you can write your own implementations and provide them to the IoC Container.

Unfortunately, the downside to this is navigability. Many times, your IDE detects that a variable has either been type hinted, or has code comments indicating that it is storing a particular Interface, rather than a concrete class, and this sometimes makes navigation difficult.

Allow me to introduce you to Go to Implementation (CMD-OPT-B) in PHPStorm - just one additional key press away from the most commonly used Go to Declaration (CMD-B) It’ll allow you to select a specific implementation of whatever abstract method you’ve got a reference to.

Sample

The following screenshot shows the utility of being able to drill down past the AdapterInterface into a specific implementation of write inside of the Filesystem::put() method.

Go to Implementation - Filesystem.php

AWS Aurora & CloudWatch Alarms

2017-08-21T22:00:00+00:00

Today I learned an interesting lesson. If you’re using AWS Aurora, don’t set your alarms (CPU, memory, etc) on an individual instance. The reason? Aurora instances, by their very nature, are somewhat transient:

If you have an issue with a given primary, Aurora may promote another instance to your WRITER role.
If you need to perform an in-place update, you can sometimes do so by creating a new set of instances, and promoting those manually.

In either case, you’ll either lose your alarms completely, if you forget to re-create them, or your alarms for your primary instance will no longer be on your primary.

It turns out that the metrics Aurora publishes also include DBClusterIdentifier and Role dimensions.

If you choose that dimension group, then you can view metrics and set alarms based on the WRITER or READER roles, for whatever cluster you desire. This is much more fool-proof than setting alarms per-instance.

Metrics by DBClusterIdentifier and Role

Bash One-Liner: Retry Laravel Failed Jobs, Filtered by Date

2017-08-17T22:00:00+00:00

We all have those moments when Queue jobs fail. Sometimes it’s a bad deploy, others it’s an upstream service that’s taken a poop. Sometimes, we need to retry failed jobs, but can’t just artisan queue retry:all, because maybe we haven’t done a cleanup of failed jobs lately.

Oops…

worker@5986f48e91d6:/var/www/app# php artisan queue:failed | grep -v 2017-08-17
+------+------------+--------------------------------------------------+-------------------------+---------------------+
| ID   | Connection | Queue                                            | Class                   | Failed At           |
+------+------------+--------------------------------------------------+-------------------------+---------------------+
| 3094 | sqs        | https://sqs.us-east-1.amazonaws.com/123/my-queue | App\Jobs\MyImportantJob | 2017-08-17 21:14:07 |
...
| 851  | sqs        | https://sqs.us-east-1.amazonaws.com/123/my-queue | App\Jobs\MyImportantJob | 2017-08-17 03:35:08 |
| 850  | sqs        | https://sqs.us-east-1.amazonaws.com/123/my-queue | App\Jobs\MyImportantJob | 2017-07-28 11:35:08 |
| 849  | sqs        | https://sqs.us-east-1.amazonaws.com/123/my-queue | App\Jobs\MyImportantJob | 2017-07-28 11:27:09 |
| 848  | sqs        | https://sqs.us-east-1.amazonaws.com/123/my-queue | App\Jobs\MyImportantJob | 2017-07-28 11:26:37 |
| 847  | sqs        | https://sqs.us-east-1.amazonaws.com/123/my-queue | App\Jobs\MyImportantJob | 2017-07-28 11:20:38 |
...
| 262  | sqs        | https://sqs.us-east-1.amazonaws.com/123/my-queue | App\Jobs\MyImportantJob | 2017-03-21 10:33:19 |
+------+------------+--------------------------------------------------+-------------------------+---------------------+

In my case today, I was running some maintenance jobs overnight, when I overworked a legacy database and caused about ~2500 of 1m+ jobs to timeout. Not a big problem, but the jobs needed to be retried.

A colleague watching me do it, informed me that last time they did something like this (I sincerely hope it wasn’t 2500 failed jobs) they typed all the numbers in manually!

So, down to the point, the one-liner:

$ php artisan queue:retry $(php artisan queue:failed | grep 2017-08-17 | awk '{print $2}')

Let’s break that down

Basics

$( ... )
- Command substitution allows you to insert the results of the contents into your command
|
- The classic bash pipe sends the output of one command to the input of the next

The commands used

php artisan queue:retry $( ... )
- Run artisan queue:retry, we’ll end up giving it a list of IDs for the jobs to retry
php artisan queue:failed
- Get the listing of failed jobs
grep 2017-08-17
- Search for the date we want to filter by in each line
awk '{print $2}'
- awk will by default split on spaces, so print id which is in the second captured group (the | from the beginning of the table will be in the first position)

Other uses

Change the grep, and you can filter by a lot more than date.

grep -v will inverse filter, so you can run everything except a given date
grep <job name> - filter only specific classes
grep <queue name> - filter specific queues, or connections

Finding Files in S3 (without a known prefix)

2017-08-03T21:00:00+00:00

S3 is a fantastic storage service. We use it all over the place, but sometimes it can be hard to find what you’re looking for in buckets with massive data sets. Consider the following questions:

What happens when you know the file name, but perhaps not the full prefix (path) of the file?

I hit this in production today, which is the motive of the blog post. The next question is what I thought might be a useful example, when I wanted to extrapolate what I’d learned to other use cases.

How do you find files modified on specific dates, regardless of prefix?

Typically, this is when things start to get difficult. As a starting point, I use Transmit - it’s a fantastic tool. However, it starts to fall down when you need to either deal with “folders” that contain many tens of thousands of documents, or when you need to look for things that could be in multiple folders.

Both of the above questions can be answered relatively easily by using the --query parameter of the aws cli. You can pass to --query any JMESPath query.

Find file by partial name

For this example, we will search for a file name containing 1018441. I have slightly redacted some bits of the path as this is a query of production data.

$ aws s3api list-objects --bucket <bucket-name> --query "Contents[?contains(Key, '1018441')]"

[
    {
        "LastModified": "2017-05-31T20:36:28.000Z",
        "ETag": "\"b861daa5cc3775f38519f5de6566cbe7\"",
        "StorageClass": "STANDARD",
        "Key": "clients/<client name>/programs/0ced4d20939fe16978df9e6d8f8985ad/1018441-synth.json",
        "Owner": {
            "DisplayName": "owner",
            "ID": "123"
        },
        "Size": 27032
    },
    {
        "LastModified": "2017-07-27T19:38:47.000Z",
        "ETag": "\"829ba43ee2bca9e20f7165ef7e7f055b\"",
        "StorageClass": "STANDARD",
        "Key": "clients/<client-name>/programs/8c1bb9a738f64c9219d8eda292f90394/1018441-synth.json",
        "Owner": {
            "DisplayName": "owner",
            "ID": "123"
        },
        "Size": 29381
    },
    {
        "LastModified": "2017-07-27T19:05:55.000Z",
        "ETag": "\"70f187a8bbbd2d5c2c54844c1b6da3a6\"",
        "StorageClass": "STANDARD",
        "Key": "cache/1018441.json",
        "Owner": {
            "DisplayName": "owner",
            "ID": "123"
        },
        "Size": 12718
    }
]

In this case, we can quickly see that when we were expecting only two files by this name (a cache, and a processed file) we have two different processed files! Here’s our problem.

Find files modified on a given date

$ aws s3api list-objects --bucket <bucket name> --query "Contents[?contains(LastModified) > '2017-08-03')]"

Find files modified between given times

$ aws s3api list-objects --bucket <bucket name> --query "Contents[?LastModified > '2017-08-03T23' && LastModified < '2017-08-03T23:15']"

Technically, the ability to perform a comparison on a date is not part of the official spec for JMESPath, but I found a GitHub issue which describes that many users rely on the ability to do just this.

That issue was closed by this commit which mentions that it will be added explicitly to the spec:

The spec doesn’t officially support string types yet, but enough people are relying on this behavior that it’s been added back. This should eventually become part of the official spec.

Caveats

When you use the --query option, it’s all done as post-processing, on your local system. Nothing is done remotely by s3, it’s essentially the same as piping the output to jq, jp or another filtering program. This means if you have many 100s of thousands, or millions of objects, or a slow connection it will not be a fast process.

NewRelic Instrumentation of the Elasticsearch PHP SDK

2017-07-15T09:00:00+00:00

We run Elasticsearch in production, fronted by an API which abstracts away complex queries and presents our APIs which consume the data a consistent interface. It came to my attention recently that we had no visibility in NewRelic on external transaction time going to ES.

In a nutshell, the problem turned out to be that the elasticsearch-php-sdk uses RingPHP as a transport, which NewRelic doesn’t support.

I’ve written a RingPHP Guzzle Adapter which the user can pass to elasticsearch and quickly gain instrumentation.

$guzzle = new GuzzleHttp\Client();
$guzzleHandler  = new LukeWaite\RingPhpGuzzleHandler\GuzzleHandler($guzzle);

$client = Elasticsearch\ClientBuilder::create()
            ->setHandler($guzzleHandler)
            ->build();

NewRelic Before and After Instrumentation

Laravel Caching for AWS Credentials

2017-04-29T16:00:00+00:00

We’ve seen occasionally poor performance on the AWS EC2 Metadata API when using IAM roles at Intouch which got me thinking. Why does the aws-pdp-sdk need to hit the EC2 Metadata API during every request? Well, it turns out, it’s simple. If you don’t explicitly give the sdk a cache interface, then it won’t use one!

I’ve just published the initial release of my laravel-aws-cache-adapter plugin on packagist.

If you pass an instance of the CacheInterface class through as your credentials setting when you instantiate the aws-sdk, it’ll use that interface to cache STS tokens returned via the metadata API when granting your application permissions via either EC2 IAM Roles or ECS Task IAM Roles. Simple!

Feedback

How has this worked for you? Found a problem? Let me know and open an issue!

AWS Batch as a Laravel Queue - Stable Release

2017-04-09T20:47:00+00:00

Following up my previous post about the release of my laravel-queue-aws-batch plugin, I am pleased to announce that I have tagged two stable versions for release.

Version 1.0 has been tagged supporting Laravel 5.1, 5.2, and 5.3.

Version 2.0 has been tagged supporting Laravel 5.4.

Feedback

How has this worked for you? Found a problem? Let me know and open an [issue][issues]!

AWS Batch as a Laravel Queue

2017-04-01T11:50:00+00:00

We use Laravel for all of our APIs at Intouch Insight, so when AWS Batch was released, I started wondering about backing our Laravel Queues with AWS Batch. This seemed like the perfect opportunity to give back, since I’m sure others are looking at Batch with the same interest that I am. A few evenings of playing around, and here we are.

I’ve just published the initial release of my laravel-queue-aws-batch plugin on packagist.

Initially the plugin has been built for Laravel 5.1, since we use the LTS releases. ~~Support for 5.4 will be coming, or quite possibly could already be working, but I need to review what has changed in the underlying queue classes before making that promise.~~

If you review the implementation, you’ll see that I based my work off the DatabaseQueue class, as we need a place to store jobs before batch can process them. Serializing the entire job payload and passing that as a string through the job submission to batch seemed like it could be error prone, to say the least.

~~Tests are needed, and will be coming.~~ I’d love to hear feedback, or receive pull requests for improvements.

Update - Stable Releases are now available

Stable releases are now available for Laravel 5.1 through to Laravel 5.4.

Feedback

How has this worked for you? Found a problem? Let me know and open an issue!