Blog | Philipp Rieber

Creating a PSR-7 Guzzle Request object with form-data

Sun, 21 Feb 2016 00:00:00 +0000

Uploading files with Guzzle 6 using a multipart/form-data request is pretty straightforward. But creating a request object with a file upload independently from the actual sending process turns out to not be that obvious.

When wrapping everything within one step, performing a file upload with Guzzle 6 using multipart/form-data is as simple as:

<?php

use GuzzleHttp\Client;

$client = new Client();
$response = $client->request(
    'POST', 
    'https://xyz.acme.dev/api/upload', 
    [
        'multipart' => [
            [
                'name' => 'upload_file',
                'contents' => fopen('path/to/file', 'r')
            ],
            // ... more fields
        ]
    ]
);

But recently I stumbled upon the requirement to create the request object decoupled from actually sending it with the client. Having only a simple string payload in the message body sending as a POST request would still be quite easy:

<?php

use GuzzleHttp\Psr7\Request;

$payload = 'My payload';
$request = new Request(
    'POST', 
    'https://xyz.acme.dev/api/upload', 
    [], 
    $payload
);
$response = $client->send($request);

But how to a create multipart/form-data payload? Because files can get huge they need to be fed into the request by a stream. Guzzle provides a lot of different stream types. Unfortunately the one we need is currently not linked from the docs, but you can find it in the supporting guzzle/psr7 library. Using a MultipartStream, the request object can now be created decoupled from the client:

<?php

use GuzzleHttp\Psr7\Request;
use GuzzleHttp\Psr7\MultipartStream;

$multipart = new MultipartStream([
    [
        'name' => 'upload_file',
        'contents' => fopen('path/to/file', 'r')
    ],
    // ... more fields
]);
$request = new Request(
    'POST', 
    'https://xyz.acme.dev/api/upload', 
    [], 
    $multipart
);
$response = $client->send($request);

If required, you can now pass around this $request object as needed before actually sending it with the Guzzle client.

Using Doctrine Naming Strategies in Symfony

Sat, 20 Feb 2016 00:00:00 +0000

Frequently, I’m stumbling upon very verbose Doctrine metadata definitions that translate each and every table and column definition from camelCase names to names separated by underscores, e.g. firstName to first_name. But did you know that can simply get rid of all this noise by implementing a Doctrine naming strategy?

Let’s consider such a verbose sample entity definition for a person:

Acme\MyService\Domain\Person:
    type: entity
    table: person
    # ...
    fields:
        firstName:
            column: first_name
            type: string
        lastName:
            column: last_name
            type: string
        createdAt:
            column: created_at
            type: datetime

Or, using annotations:

<?php

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 * @ORM\Table(name="person")
 */
class Person
{
    // ...
    
    /**
     * @ORM\Column(name="first_name", type="string")
     */
    private $firstName;
    
    /**
     * @ORM\Column(name="last_name", type="string")
     */
    private $lastName;
    
    /**
     * @ORM\Column(name="created_at", type="datetime")
     */
    private $created_at;
    
    // ...
}    

It seems quite verbose to translate the camelCase entity table name and properties names to under_score table and column names, e.g. Person to person, firstName to first_name etc. Furthermore, it is quite prone to become inconsistent if someone forgets it for a column that’s added later. So why not automate the translation?

Doctrine Naming Strategies to the rescue! Doctrine naming strategies allow to define how entity class names and properties are translated to table and column names if they are not provided by the metadata. By default, Doctrine does not change anything and in the example above you’d end up with a Person table and the column names firstName, lastName etc. But by configuring the Underscore Naming Strategy you’ll get exactly what you want – without all the table and column config noise in your metadata:

Acme\MyService\Domain\Person:
    type: entity
    # ...
    fields:
        firstName:
            type: string
        lastName:
            type: string
        createdAt:
            type: datetime

Or, using annotations:

<?php

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 */
class Person
{
    // ...
    
    /**
     * @ORM\Column(type="string")
     */
    private $firstName;
    
    /**
     * @ORM\Column(type="string")
     */
    private $lastName;
    
    /**
     * @ORM\Column(type="datetime")
     */
    private $created_at;
    
    // ...
}    

How to set this up? If using Symfony, it’s as simple as adding a configuration parameter doctrine.orm.entity_managers.default.naming_strategy to your config.yml:

doctrine:
    dbal:
        # ...
    orm:
        # ...
        entity_managers:
            default:
                naming_strategy: doctrine.orm.naming_strategy.underscore

If using plain PHP, you can check the Doctrine docs for the few lines you need to add.

By providing a custom implementation of Doctrine\ORM\Mapping\NamingStrategy, you can also easily define your own naming strategy. This comes in handy if for example you need to follow naming guidelines like prefixing every table name, e.g. by tbl_.

By the way, as string is always the default data type, so you can even skip this config for simple columns by only putting a tilde ~ character:

Acme\MyService\Domain\Person:
    type: entity
    # ...
    fields:
        firstName: ~
        lastName: ~
        createdAt:
            type: datetime

When using annotations, just leave out the type definition:

<?php

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 */
class Person
{
    // ...
    
    /**
     * @ORM\Column(type="string")
     */
    private $firstName;
    
    /**
     * @ORM\Column(type="string")
     */
    private $lastName;
    
    /**
     * @ORM\Column(type="datetime")
     */
    private $created_at;
    
    // ...
}

This will finally result in the following SQL definition, which is the same as for the verbose metadata definition from the beginning:

CREATE TABLE `person` (
/* ... */
`first_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`last_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`created_at` datetime NOT NULL,
PRIMARY KEY (`person_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Integrating PHP Coding Standards Fixer with PhpStorm

Fri, 19 Feb 2016 00:00:00 +0000

Fixing coding standard issue individually is very tedious, e.g after running a static code analysis tool or after being declined by a code sniffer pre-commit hook. To ease these kind of tasks you may consider using SensioLabs’s awesome little CLI tool called PHP Coding Standards Fixer.

You can simply run the PHP Coding Standards Fixer recursively on a directory or on a single file and it’ll make your PHP files mostly compliant with PSR-1/PSR-2 or Symfony coding standards, respectively. This vastly unburdens you from doing mindless and repetitive tasks. In addition, your coding style then relies on accepted standards, so there’s no pointless dispute about details.

Using Homebrew, installing the PHP CS Fixer is as simple as:

$ brew install homebrew/php/php-cs-fixer

For other platforms and installation options check out the project page.

To fix code styles recursively in all PHP files of a directory:

$ php-cs-fixer fix /path/to/dir

To fix code styles in a single PHP file:

$ php-cs-fixer fix /path/to/file

There are very fine-grained options to define the fixes to apply, but using the default symfony level should usually be a good choice to be as close as possible to the Symfony defaults. This already saves much time when applied from time to time or before a new commit to your VCS. Of course, you can also do a dry run first and display a diff of all the changes to be made. Check out php-cs-fixer help fix for all the details.

You can also integrate the tool into PhpStorm so that you can apply style fixes by executing a custom shortcut. This is possible by setting up an External Tool definition in PhpStorm: Got to Preferences > Tools > External Tools > + (Create Tool) and fill in the details about how to execute the PHP CS Fixer (for convenience, you can simply copy&paste the field values given below the screenshot):

Name: php-cs-fixer
Program: /usr/local/bin/php-cs-fixer
Parameters: --level=symfony --verbose --config=sf23 fix "$FileDir$/$FileName$"
Working directory: $ProjectFileDir$

Then navigate to Preferences > Keymap, search for php-cs-fixer and assign a custom shortcut, e.g. Ctrl+Alt+P. Now, when you open up a PHP file in your editor and hit your assigned shortcut, the code gets formatted according to your external tool definition of php-cs-fixer.

Another nice feature of the PHP CS Fixer is to add a .php_cs file at the project’s root to improve the tool’s integration by giving sensible default options. This way it’ll be quite easy for example to additionally enforce PHP’s short array syntax or to give a default list of directories to be analysed.

Of course, it’s should also possible to integrate the PHP CS Fixer into an automated process.

Fixing code styles might also be possible by using PhpStorm’s Reformat Code action, and I also use it from time to time to clean things up a bit, but from my experience this tool is a bit greedy and tries to fix things you may not want to be fixed, like chained calls of a fluent interface. Moreover, sharing code style definitions across teams becomes much easier when they are not tied to an IDE and can even be stored within the code repository (.php_cs file). On the contrary what’s quite useful I think is PhpStorm’s Optimize imports action which cleans up and orders all the use statements – at least since PhpStorm is smart enough to not remove imports that are only used within annotations :-)

Project website: http://cs.sensiolabs.org/
GitHub repository: https://github.com/FriendsOfPHP/PHP-CS-Fixer

Optimize Debugging in a Symfony application

Tue, 11 Aug 2015 00:00:00 +0000

When debugging a Symfony application with Xdebug you might not only want to step through your own custom code, but also through the Symfony code itself. As a lot of Symfony’s core code is aggregated and optimized into a single file at app/bootstrap.php.cache for performance reasons, you will not feel very comfortable reading non-highlighted, non-indented code of classes from multiple namespaces all assembled in the same file. And this is even true for the dev environment, where debugging usually happens. But some simple tweaks in the front controller will allow you to step through the original Symfony code if and only if a debug session was initialized.

For this article we’ll use Symfony’s development front controller located at web/app_dev.php. Usually it looks like this:

// ...

$loader = require_once __DIR__.'/../app/bootstrap.php.cache';
Debug::enable();

require_once __DIR__.'/../app/AppKernel.php';

$kernel = new AppKernel('dev', true);
$kernel->loadClassCache();
$request = Request::createFromGlobals();
$response = $kernel->handle($request);
$response->send();
$kernel->terminate($request, $response);

The result in a debugging session is similar to the following screenshot when it comes to Symfony code:

A dedicated cookbook in the Symfony documentation already describes how to disable the bootstrap.php.cache file and the class cache:

// ...

// $loader = require_once __DIR__.'/../app/bootstrap.php.cache';
$loader = require_once __DIR__.'/../app/autoload.php';
Debug::enable();

require_once __DIR__.'/../app/AppKernel.php';

$kernel = new AppKernel('dev', true);
// $kernel->loadClassCache();
$request = Request::createFromGlobals();
$response = $kernel->handle($request);
$response->send();
$kernel->terminate($request, $response);

But in the end it recommends to revert the code changes after each debugging session. In practice, this would mean to toggle the changes all the time. So why not change the code only once and figure out at runtime if the current request is a Xdebug debugging session? Let’s introduce a boolean flag called $enableCaching to switch between enabled and disabled caches. The following code corresponds to the default:

// ...

$disableCaching = false;
if ($disableCaching) {
    $loader = require_once __DIR__ . '/../vendor/autoload.php';
} else {
    $loader = require_once __DIR__ . '/../app/bootstrap.php.cache';
}
Debug::enable();

require_once __DIR__.'/../app/AppKernel.php';

$kernel = new AppKernel('dev', true);
if (!$disableCaching) {
    $kernel->loadClassCache();
}
$request = Request::createFromGlobals();
$response = $kernel->handle($request);
$response->send();
$kernel->terminate($request, $response);

An initialized Xdebug debugging session can be identified by checking for these things:

xdebug extension must be loaded AND
one of these must be true:
- GET parameter XDEBUG_SESSION_START is set OR
- Cookie XDEBUG_SESSION is set OR
- php.ini setting xdebug.remote_autostart is enabled

This results in a slightly adapted development front controller:

// ...

$disableCaching = extension_loaded('xdebug') && (
        isset($_REQUEST['XDEBUG_SESSION_START']) ||
        isset($_COOKIE['XDEBUG_SESSION']) ||
        ini_get('xdebug.remote_autostart')
    );
if ($disableCaching) {
    $loader = require_once __DIR__ . '/../vendor/autoload.php';
} else {
    $loader = require_once __DIR__ . '/../app/bootstrap.php.cache';
}
Debug::enable();

require_once __DIR__.'/../app/AppKernel.php';

$kernel = new AppKernel('dev', true);
if (!$disableCaching) {
    $kernel->loadClassCache();
}
$request = Request::createFromGlobals();
$response = $kernel->handle($request);
$response->send();
$kernel->terminate($request, $response);

Contrary to the previous screenshot you’ll now see a pretty useful result when it comes to Symfony code:

Happy debugging!

If you want learn more about initializing Xdebug debugging session, check out my blog post about Connecting Xdebug to PhpStorm.

Run phpspec when saving files with gulp and PhpStorm

Sat, 01 Aug 2015 00:00:00 +0000

In our weekly Code Kata at work, we’re exercising TDD using phpspec. TDD means switching between coding and running the tests all the time, may causing cramps by abuse of Cmd+Tab. So why not automate the test runs whenever a source file changes and show a decent notification about success or failure?

This article shows two different approaches to automatically run phpspec whenever a source or spec file was changed in your codebase.

Gulp File Watcher

Gulp is a Node.js based task runner. It allows you to create and run lightweight customized tasks on the command line and provides tons of plugins, e.g. for desktop notifications or running phpspec with specific options – exactly what we need for our purposes. After installing Node.js, gulp first needs to be installed globally by using the Node Package Manager npm:

$ npm install -g gulp

Gulp also needs to be installed locally for each project that makes use of it – just as well as the desired gulp plugins. To install local Node packages, just type a node install command right in the project’s root directory:

$ npm install gulp gulp-notify gulp-phpspec --save-dev

Packages and their dependencies are installed into the node_modules/ directory by default, which should be ignored by your version control system. If you have a package.json file in place (which in contrast should be under version control) to describe your Node dependencies, the --save-dev option appends the installed packages and their respective versions to the development dependencies section:

{
  "devDependencies": {
    "gulp": "^3.8.7",
    "gulp-notify": "^1.5.0",
    "gulp-phpspec": "^0.2.5"
  }
}

With package.json, you or your fellow teammates just need to run npm install without any argument to get the exact same setup of Node module on their machine. Think of it as Composer for Node.

Gulp tasks are defined in a file called gulpfile.js in the project’s root directory. First, we import the plugins and assign them to variables:

var gulp = require('gulp'); // gulp task runner
var phpspec = require('gulp-phpspec'); // for running phpspec
var notify = require('gulp-notify'); // for desktop notifications

Then, we create a stub task that will run the phpspec tests later:

gulp.task('test', function() {
  console.log('Run phpspec ...')
});

You can already try this task on the command line, providing the task name test to the gulp command:

$ gulp test
[12:16:25] Using gulpfile ~/test/gulpfile.js
[12:16:25] Starting 'test'...
Run phpspec ...
[12:16:25] Finished 'test' after 67 μs

Now it’s time to fill the test task with real code:

gulp.task('test', function() {
  var options = {
    verbose: 'v',
    notify: true,
    clear: true,
    formatter: 'pretty'
  };
  gulp.src('spec/**/*.php')
    .pipe(phpspec('./bin/phpspec run', options))
    .on('error', notify.onError({
      title: 'Crap',
      message: 'Your tests failed!',
      icon: __dirname + '/node_modules/gulp-phpspec/assets/file-watcher/test-fail.jpeg'
    }))
    .pipe(notify({
      title: 'Success',
      message: 'All tests have returned green!',
      icon: __dirname + '/node_modules/gulp-phpspec/assets/file-watcher/test-pass.jpeg'
    }));
});

In options, you can specify whatever command line options you want to run phpspec with, e.g. for pretty or verbose output or if you wish to clear the console after every test run. With gulp.src('spec/**/*.php'), gulp finds all the PHP files in the spec/ directory recursively (using a glob pattern) and then pipes them into the phpspec command with the options from above using pipe(phpspec('./bin/phpspec run', options)). You may need to adjust the path to the phpspec executable. Now, when running gulp test in your project’s root folder, every phpspec test gets executed. Depending on test success or failure, the appropriate method from the gulp-notify plugin gets called and shows a nice desktop notification – depending on your operating system.

To implement a watch functionality that is monitoring files for changes and then automatically runs the test task, we implement another task called watch, which makes use of a native gulp feature to watch for file changes:

gulp.task('watch', function() {
  gulp.watch(['spec/**/*.php', 'src/**/*.php'], ['test']);
});

The first parameter of gulp.watch() indicates the files we wish to watch for changes, i.e. in our case all the PHP source and spec files. The second parameter specifies the task(s) to run when a file change is detected, i.e. in our case only the test task. Now you may start the file watcher with gulp watch. But gulp also allows you to specify a default task that gets executed whenever you do not provide a task name to the gulp command:

gulp.task('default', ['test', 'watch']);

In the second parameter, instead of a callback method, we can also provide an array of existing task names that we wish to run. To run the tests once and then start the file watcher, there’s now nothing more to it than just running:

$ gulp

If we put everything together, we end up with a quite straightforward gulpfile.js solution, allowing us to watch for file changes and execute the phpspec tests:

// gulpfile.js
var gulp = require('gulp');
var phpspec = require('gulp-phpspec');
var notify = require('gulp-notify');

gulp.task('test', function() {
  var options = {
    verbose: 'v',
    notify: true,
    clear: true,
    formatter: 'pretty'
  };
  gulp.src('spec/**/*.php')
    .pipe(phpspec('./bin/phpspec run', options))
      .on('error', notify.onError({
        title: 'Crap',
        message: 'Your tests failed!',
        icon: __dirname + '/node_modules/gulp-phpspec/assets/file-watcher/test-fail.jpeg'
      }))
      .pipe(notify({
        title: 'Success',
        message: 'All tests have returned green!',
        icon: __dirname + '/node_modules/gulp-phpspec/assets/file-watcher/test-pass.jpeg'
      }));
});

gulp.task('watch', function() {
  gulp.watch(['spec/**/*.php', 'src/**/*.php'], ['test']);
});

gulp.task('default', ['test', 'watch']);

Now just open a console window (e.g. integrated in PhpStorm: Alt+F12), work on your code and watch the test runs every time you save a file (e.g. in PhpStorm: Cmd+S).

A successful test run in PhpStorm looks like thi:

A failed test run in PhpStorm looks like this:

PhpStorm file watcher

Modern IDEs also provide solutions to create file watchers.

In my IDE of choice – PhpStorm – this feature is simply called File Watchers. There’s even built-in support for popular transcompilers like Sass or CoffeeScript that compile to their CSS or JavaScript counterpart on file save, respectively. For our purposes we can easily add a custom file watcher by importing a ready made solution from Github. Save the raw XML file somewhere and import it from Preferences > File Watcher using the Import button.

All the options are quite self-explanatory, but you may adopt some little things like always showing the console, whether it is an erroneous or a successful test run. That way, you may be more confident that the tests ran smoothly.

A successful test run in PhpStorm looks like this:

A failed test run in PhpStorm looks like this:

As you may have have noticed, there is no desktop notifications. Implementing desktop notifications is possible, but requires additional work that I’ll leave as an exercise to the interested reader.

Gulp file watcher vs. PhpStorm file watcher

Although gulp requires Node.js, I prefer the gulp solution to the IDE File Watcher because:

it works in every IDE or even without
the gulpfile.js can be under version control and easily distributed with a project
it provides desktop notifications out-of-the-box

Moving to the cloud part 7: Enabling Elastic Beanstalk

Sat, 01 Jun 2013 00:00:00 +0000

Finally, after preparing our application during the last few months bit by bit, it was time for the last and most important step: move the application code to an auto-scaling multi-server environment in the cloud and do a smooth DNS switch for the productive application. First we thought doing this with EC2 directly, but after some investigations we found solutions like Scalr, OpsWorks (the former Scalarium) or Elastic Beanstalk doing the heavy work for us. We decided in favor for Elastic Beanstalk - let’s see why and how.

The first time I played around with Amazon’s EC2 service some years ago I was completely excited about all the possibilities: Setting up server instances on demand by simple mouse clicks, choosing between different instance types regarding CPU and RAM size for the specific needs of an application, wiring several instances together with a load balancer in front - and only paying for what’s currently in use. Suddenly, even small companies became able to provide highly scalable multi-server environments without buying resources in advance. No financial risk if the application fails. No sysadmin that needs hours or days to configure fresh root servers. Easily extendable setup. No long running contracts. No setup fees.

Mostly, if you find a solution for an issue, you’ll get new issues when implementing this solution. Ok, with EC2 you can start whatever server instance type you need at any time. You can do this by a mouse click in the Amazon console or by programmatically using the API. You can create a load balancer and add instances to it. You can add more instances if the load grows, you can terminate instances as soon as you no longer need them. You can SSH onto any instance of your cluster and install packages, application code or cronjobs. This is really great - but you’ll want more for a productive application. You’ll want the scaling to be triggered automatically by the current load. You’ll want all instances to be configured the same. You’ll want to roll out new code to all instances in one go. You’ll want some cronjobs to be running only on one instance of the cluster that never gets terminated by autoscaling. You’ll want to have health checks and monitoring.

There are some services on the internet that help you with exactly this kind of issues. In other words, they help you manage your Amazon resources with an abstraction layer on top of the raw Amazon webservices. Those kind of abstraction layers are called Platform-as-a-Service (PaaS) - you get all the single services combined to the whole application platform. For instance, you can create named server farms in a nice browser GUI, configure the autoscaling triggers, deploy application code to all instances at once with no downtime etc. Some of them are third party services like Rightscale or Scalr. You’ll need to pay a service fee on top of the Amazon resources you’re managing with the service. But Amazon itself also provides two free services fo this purpose in the AWS console: OpsWork (formerly Scalarium) and Elastic Beanstalk. Scalarium was acquired by Amazon in the beginning of 2013 and is the foundation of OpsWorks. It makes heavy use of the configuration management software Chef and can therefore be highly customized. Elastic Beanstalk was the first Amazon PaaS and is less complex. It is using several Amazon webservices like EC2, Elastic Load Balancer (ELB), S3 and ties them together with a configuration made by the user.

OpsWorks will probably be the tool of choice for our next cloud migration. But when we started our migration it was freshly released and at first glance Chef cookbooks are not your best friends. So for this time, we put our trust in the fully established and well documented Elastic Beanstalk service which is free of charge.

In Elastic Beanstalk, you can host multiple applications. Each application allows you to have multiple environments, so you can for instance setup a Staging environment with only a single and cheap EC2 instance and an autoscaling Production environment using a powerful but expensive EC2 instance type.

Setup an Elastic Beanstalk application

To create your first PaaS application, login to the Elastic Beanstalk console. At first, adjust the appropriate region for your needs, then click on Create New Application. Enter a name and description for your application:

Then, step-by-step, you will be guided through the process to setup your first environment within your application. First, select your solution stack and if you wish an autoscaling environment. Actual triggers for autoscaling – like CPU or RAM usage – will be setup later.

In the third step, you can upload your application code as a ZIP package or, in the first instance, opt for a sample application. We’ll go with the sample application and install a custom application later on:

Provide some info for your environment name, an unique elasticbeanstalk.com sub domain, e.g. foobar.elasticbeanstalk.com, and a description. Please note that your real application URL is fully customizable using Amazon Route53.

We don’t need any additional resources at this point, so let’s just skip the next step. Regarding the RDS instance, it’s even more convenient to do the setup manually.

In the configuration details, you can specify the EC2 instance type for your environment. Please note: check out the pricing before selecting an instance type. Powerful instance types can get real expensive, so you should go with a micro instance first. Besides, give an EC2 keypair so you can SSH into the managed EC2 machines and an email address for status notifications. You can also specify a health check URL within your application which will is pinged every few minutes to check if your application is still alive.

Review all information and finally Create your platform:

You can follow the launch process of your first application environment by looking at the events log. This may take some time to complete.

After the creation process has completed you’ll see a green checkmark for the health status …

… and you’re able to access the sample application by the elasticbeanstalk.com sub domain you specified before, e.g. http://foobar.elasticbeanstalk.com.

The EC2 console now contains EC2 instances for your new Elastic Beanstalk environment:

A load balancer has been set up automatically in front of your EC2 instance(s):

And there’s also a new security group for your new environment, allowing access to the web server on port 80 and via SSH on port 22:

Looking at your S3 console, you’ll notice a fresh S3 bucket related to your Elastic Beanstalk environment. This is were new release versions of your application code get stored, so they can be deployed and also be roll-backed easily.

To deploy a custom application, go back to the dashboard of your Elastic Beanstalk environment and upload a ZIP package of your application code by clicking on Upload and Deploy:

Again, you can follow the environment update by looking at the Recent Events:

As soon as the deployment is complete you should be able to access your application by entering the chosen sub domain in your browser, e.g. http://foobar.elasticbeanstalk.com.

What actually happens during deployment is that a ZIP archive of your whole application is uploaded to a dedicated S3 bucket first. Then a so-called Application Version is created from your upload and a deployment to the Elastic Beanstalk gets triggered. No matter how many servers are currently running behind the load balancer in your environment, Elastic Beanstalk takes care of the roll-out to all of them and switching everything live in the end.

Customizing

If your application has specific server requirements like additional software packages, a different document root, specific PHP settings or if you want to add custom files (e.g. containing specific parameters for your database connection), it is possible to include an /.ebextensions directory in your zipped application code. All files using the extension *.config in this directory can be used to specify custom configuration. For instance, if you need the lynx package installed, an .htaccess secured area, a custom parameters.yml in a Symfony project, some custom php.ini settings, a cron job to send out what’s in the Swiftmailer file spool and set an appropriate timezone for the EC2 instances, you may use a configuration file /.ebextensions/app.config containing this:

packages:
  yum:
    lynx: []

files:
  "/tmp/htaccess":
    mode: "644"
    owner: webapp
    group: webapp
    content: |
      AuthName "My secured area"
      AuthType Basic
      AuthUserFile /var/app/current/.htpasswd
      require user admin
  "/tmp/parameters.yml":
    mode: "644"
    owner: webapp
    group: webapp
    content: |
      parameters:
        database_driver: pdo_mysql
        database_host: omdb.vnbnlwvfvi3zc.eu-west-1.rds.amazonaws.com
        database_port: ~
        database_name: app
        database_user: app
        database_password: JKIAJ97WSC3L9GFZS7P2
  "/etc/php.d/app.ini":
    mode: "644"
    content: |
      date.timezone = Europe/Berlin
  "/tmp/app-crons":
    mode: "644"
    content: |
      MAILTO=cron@app.com
      * * * * * webapp /var/app/current/app/console swiftmailer:spool:send --env=stage > /dev/null 2>&1

commands:
  10_servertime:
    command: cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime && echo 'ZONE="Europe/Berlin"' | tee /etc/sysconfig/clock

container_commands:
  10_parameters:
    command: mv /tmp/parameters.yml app/config/
  20_protection:
    command: 'mv /tmp/htaccess .htaccess && htpasswd -cb .htpasswd admin "B0DEK/TK2k_p0FgPxK1" && chmod 444 .htpasswd'
  30_crons:
    command: mv /tmp/app-crons /etc/cron.d/

There’s lot of documentation how you can customize to your specific needs and how to execute custom commands during the deployment process.

Setup a deployment script

Since all Amazon web services can be managed by powerful APIs, you can also do a deployment by wiring together some CLI commands in a shell script and by using the AWS Command Line Tools (which give you the aws command):

# Application name on Elastic Beanstalk
APP="foobar"
# Environment name on Elastic Beanstalk
ENV="staging"
# S3 bucket name of Elastic Beanstalk application
BUCKET="elasticbeanstalk-eu-west-1-294798127440"

# Create a unique version string for this release
VERSION="$ENV-v_`date +%Y-%m-%d_%H-%M-%S`"
# Zip the application
zip -r $VERSION.zip /path/to/your/application

# Push zipped application to S3 - this may take a while
aws --profile $APP s3 cp $VERSION.zip s3://$BUCKET/$VERSION.zip
# Create application version
aws --profile $APP elasticbeanstalk create-application-version --application-name $APP --version-label $VERSION --source-bundle S3Bucket=$BUCKET,S3Key=$VERSION.zip
# Trigger environment update
aws --profile $APP elasticbeanstalk update-environment --environment-name $ENV --version-label $VERSION

The remaining question here is how to prepare the application code that gets zipped so that it can run properly and secure on the Elastic Beanstalk environment. You may need to setup a custom process here, e.g. to get rid of unnecessary files like development front controllers or uncompressed asset files. You can do this by by extending the shell script or by using build tools like Ant or Phing. In the end, what you need to build, is a directory with your application code completely stripped down to the bare minimum of what’s really needed to run the application, maybe compressed assets and likely you need to include a configuration file appropriate for your target environment (database settings, API keys etc. for this specific environment). For the latter, you may also the custom configuration file feature shown above or environment variables. On Elastic Beanstalk, you can set environment variables easily in the environment configuration and they are available on every EC2 instance managed by your environment.

One thing to note is that in a PHP application that uses Composer, you do not have to include your /vendor directory. Elastic Beanstalk makes some assumption about your application, and if there’s a composer.json in your application root, it automatically does a composer install during deployment. This will save you a lot of time and bandwidth when transmitting your zipped application code.

Autoscaling

Finally, to enable autoscaling, go to the Configuration page of your Elastic Beanstalk environment and enter a range of servers you’ll want your application to run on. For instance, you may want of minimum of 2 servers to be on the safe side, and a maximum of 4 to absorb peak loads. You can even setup a timetable for autoscaling – e.g. to prepare for your TV commercials – and also spread the EC2 instances across Availability Zones to be independent from regional outages etc.

There are a way more features available on Elastic Beanstalk than be covered in this article. You can do for example far more customizations, there are special options for all the different platform stacks, zero-downtime deployments, SSL, usage of custom domains, integration of other AWS services etc. Just check out the extensive documentation for more information.

Moving to the cloud part 6: Enabling RDS

Sun, 19 May 2013 00:00:00 +0000

In this part of my series we’ll get closer to the fundamentals of our application. In the last article we’ve outsourced our mail server to the cloud. Now we want to do the same with our database server to get rid of all the tasks that come with hosting a database server like setup, configuration, maintenance, backups, security, replication or updates. Amazon provides a really straightforward solution: RDS

Using Amazon RDS (Relational Database Service) is as simple as changing the database connection settings in our application’s parameters.yml to the ones provided in the AWS console.

# parameters.yml
parameters:
    # ...
    database_host: vv6m55fk5jgz8tg.gh4ruwxi8vb4.eu-west-1.rds.amazonaws.com
    database_port: ~
    database_name: database-name
    database_user: database-user
    database_password: VCXJAI9WEV9QZ2HL0BN9C

But how do we setup the database in RDS and how does Amazon handle all that stuff mentioned in the introduction?

We just click on Launch a DB instance in the AWS console and choose our prefered database provider: MySQL

In the next dialog we choose from different options:

MySQL version; in our example it’s the current GA release version 5.5
Instance Class, defining the size of our database engine; in our example it’s the smallest and cheapest: Micro
Multi-AZ Deployments would replicate our database instance in different availability zones providing enhanced availability in case of a failure. This is recommended for production environments
Allow automatic minor version updates, e.g. from 5.5.27 to 5.5.28
Choose identifier and credentials

This frees us from setting up the database server itself, from dealing with replication and from applying patch level updates every few weeks.

In the next step we can enter a database name and select a security group. Security groups are a fundamental concept in AWS and they let us create some kind of different firewalls by specifying what services are available under which ports and for which IP addresses or subnets. For instance, we can restrict RDS instance access to a specific EC2 instance or our company’s static IP address. Security groups have to be setup separately in the RDS console as shown on the second screenshot.

This frees us from digging to deep into security concerns.

In the next step we can define the backup retention period up to 35 days. RDS allows to restore the database to any second during the backup retention period (up to the last five minutes).

This frees us completely from setting up, maintaining and monitoring database backups.

In the last step we can verify all settings and then finally launch the database instance.

The new database instance will be available a few minutes later.

We then just need to copy the endpoint URL and the credentials chosen during the setup process to our parameters.yml and the application is ready to establish a connection to RDS.

Performance It is possible to use an RDS instance from outside the AWS ecosystem, for instance from your existing non-AWS server. This is very useful for testing or development purposes but you don’t want to do that in production. Connecting a database over the internet is just too slow. But as soon as our production move to EC2 or Elastic Beanstalk is complete we’ll also switch the database to RDS.

Moving to the cloud part 5: Enabling SES

Wed, 15 May 2013 00:00:00 +0000

Sending emails from a Symfony2 application is no challenging task. Just configure the Swiftmailer library with a handful of simple parameters, create a message object, trigger the sending process and you are done. Things change slightly if you are responsible of the mail server at the same time. Setting up and maintaining mail server software may become a challenging task, especially if there are complaints about missing emails, security holes or spam issues. Moreover, sending from a cloud server is not very reliable because of its doubtful IP reputation. Amazon’s SES service provides relief.

Amazon’s SES (Simple Email Service) service frees us from setting up and maintaining a mail server by providing an email service with a single SMTP endpoint.

Switching to SES is as simple as updating your Swiftmailer configuration. It should look similar to the following:

// app/config/config.yml
# ...
swiftmailer:
  transport: "%mailer_transport%"
  host: "%mailer_host%"
  port: "%mailer_port%"
  encryption: "%mailer_encryption%"
  username: "%mailer_user%"
  password: "%mailer_password%"

It should be noted that the encryption parameter is not part of the default configuration and no encryption is used by default. Since SES requires TLS encryption we need to add this setting.

If you’ve signed up for AWS you can find your personal SMTP settings in the [SES Management Console] (https://console.aws.amazon.com/ses) and you can complete your parameters.yml with it:

parameters:
  # ...
  mailer_transport:  smtp
  mailer_host: email-smtp.us-east-1.amazonaws.com # to be found in the SES console
  mailer_user: SES-User # to be found in the SES console
  mailer_password: SES-Password # to be found in the SES console
  mailer_encryption: tls # TLS encryption required
  mailer_port: 25 # 25, 465 or 587

That’s all – all your email will be sent through SES now and you can start thinking about shutting down your mail server.

Remember that Amazon insists on high quality emails to prevent abuse. Otherwise your account may be disabled.

Sandbox mode A fresh SES account runs in a sandbox mode. In the sandbox mode Amazon limits the number of messages you can send per day and only allows verified sender and recipient addresses. You can verify addresses from the SES console.

Production mode Once you want to switch to production you can request production access from within the SES console. Production mode raises the message limit per day and only requires verified sender addresses. You can also verify a whole domain as a sender domain.

Moving to the cloud part 4: Enabling S3

Sat, 11 May 2013 00:00:00 +0000

Moving to the cloud mostly means moving to a scalable multi-server environement with a load balancer in front. The load balancer redirects a user to an available web server instance of the cluster. Imagine a form with an image file upload somewhere in your application allowing a user to publish an avatar on his profile page. Handling the uploaded file the old way would mean to store it on the current web server’s file system. But how could this file be accessed by other web servers of the cluster, e.g. to display the avatar in the user’s public profile to users that have been redirected to another instance of the cluster? Moreover, what happens if we want to scale down our multi-server environment – meaning that we may need to shut down a web server that stores uploaded images? One possible solution would be to setup an additional file server for this purpose, not being part of the scaling cluster. All webs servers could access uploaded files at the same central location. But there are several drawbacks with this setup: First, it means setting up and maintaining another server with a different configuration. Second, it means a single point of failure: if our single file server fails then the whole application is concerned – and for the sake of simplicity mirroring the file server is not an option… S3 to the rescue!

Amazon S3 (Simple Storage Service) is a cloud storage: it allows you to store files on Amazon servers. Based on one of the most reliable infrastructures on the internet and with features like permissions and REST/SOAP APIs on top it should perfectly suit our needs for a centralized file storage. There are some nice S3 management tools which allow file handling similar to all the good old FTP clients, e.g. S3 Browser or CloudBerry. But remember the file upload in our application: we will need to handle files programatically. The S3 documentation describes all the low level commands for file operations: how to upload or download files, how to setup permissions or how to list your existing files. If you do not want to reinvent the wheel you should use the appropriate standard development kit (SDK), abstracting away all the tedious work of crafting restful URLs and looking up all the parameters. Amazon provides SDKs for all major languages, e.g. the AWS SDK for PHP.

But we didn’t wanted to tie our application completely to S3 by spreading SDK code everywhere, so we went one step further by using the fabulous Gaufrette library and its Symfony2 bundle. Gaufrette provides a filesystem abstraction layer allowing us to handle files independently from the underlying storage system. In other words: our application will use Gaufrette commands to save and retrieve files without knowing the underlying filesystem: Local, FTP, S3, database, Dropbox, APC …

Every installation of our application can be configured separately which filesystem to use. This allows us to continue working with the local filesystem in our development, staging or test environment without being charged by Amazon and to still be able to develop offline.

Installing the AWS S3 SDK for PHP is as simply as adding a new line to your composer.json:

{
    require: {
      "amazonwebservices/aws-sdk-for-php": "1.x"
    }
}

Then update your vendors:

$ composer udpate amazonwebservices/aws-sdk-for-php

Configure the AmazonS3 class from the AWS SDK as a service. Usually you’ll have to pass an options array with the S3 credentials as constructor arguments:

<!-- ... -->
<service id="my.storage.s3" class="AmazonS3">
  <argument type="collection">
    <argument key="key">%uploads_s3_key%</argument>
    <argument key="secret">%uploads_s3_secret%</argument>
  </argument>
</service>
<!-- ... -->

Add the credentials to your parameters.yml:

parameters:
  # ...
  uploads_s3_key: MyS3Key
  uploads_s3_secret: MyS3Secret

Now you could grab the service from the container and just use it:

$s3 = $this->container->get('my.storage.s3');
$s3->create_object('my-bucket-name', 'my-file');

As mentioned above we can get rid of this dependency on the S3 service by using Gaufrette. Install the Gaufrette Symfony2 bundle by adding another line to your composer.json:

{
    require: {
        "knplabs/knp-gaufrette-bundle": "0.1.x"
    }
}

Update your vendors again:

$ composer udpate knplabs/knp-gaufrette-bundle

// app/AppKernel.php
// ...
public function registerBundles()
{
    $bundles = array(
        // ...
        new Knp\Bundle\GaufretteBundle\KnpGaufretteBundle(),
        // ...
    );

    return $bundles;
}
// ...

The container configuration of the Gaufrette bundle allows to register different virtual filesystems in a so called filesystem map. Every filesystem requires an adapter for defining and configuring the underlying real storage. In our example we register a filesystem named uploads and make its adapter configurable:

# app/config/config.yml
# ...
knp_gaufrette:
  adapters:
    # ...
  filesystems:
    uploads:
      adapter: "%uploads_adapter%"

Usage of a filesystem is easy. Just retrieve the desired filesystem from the knp_gaufrette.filesystem_map service and use its self-explanatory methods to handle files:

// Get filesystem map
$filesystemMap = $this->container->get('knp_gaufrette.filesystem_map');

// Get filesystem
$filesystem = $filesystemMap->get('uploads');

// Write file
$filesystem->write($path, $content);

// Read file
$filesystem->read($path);

// Delete file
$filesystem->delete($path);

But how to define and switch the adapters? This also happens in the container configuration:

# app/config/config.yml
# ...
knp_gaufrette:
  adapters:
    uploads_local:
      local:
        directory: "%uploads_local_directory%"
    uploads_s3:
      amazon_s3:
        amazon_s3_id: my.storage.s3 # Service ID for AmazonS3 class, see above
        bucket_name: "%uploads_s3_bucket_name%"
        options:
          region: "%uploads_s3_region%"
  filesystems:
    # ...

The first adapter uploads_local defines a simple local filesystem with only one parameter: the directory where the files should be stored. The second adapter uploads_s3 defines what we were waiting for: S3 storage. We have to specify the service key of the above configured AmazonS3 service and additionally pass a bucket name and the appropriate AWS region.

Now we can complete our parameters.yml:

parameters:
  # ...
  uploads_adapter: uploads_local # uploads_local|uploads_s3
  uploads_local_directory: "%kernel.root_dir%/../web/uploads"
  uploads_s3_key: MyS3Key
  uploads_s3_secret: MyS3Secret
  uploads_s3_region: s3-eu-west-1.amazonaws.com
  uploads_s3_bucket_name: my-bucket-name

By switching the uploads_adapter value between uploads_local and uploads_s3 we can now switch between local file storage and S3 storage.

Moving to the cloud part 3: Enabling Route 53

Mon, 18 Mar 2013 00:00:00 +0000

This is part 3 of my series of articles about our first application move to the Amazon Cloud. As we are not the owner of the productive domain of the application, all communication about DNS changes has always been quite tedious, time consuming and prone to errors in the past – especially when your contact person lives in another time zone. In preparation to the final application move to EC2, which involves some DNS changes to switch to the Amazon load balancer, we wanted to gain some flexibility. Enter Route 53.

Route 53 allows you to manage all the DNS records of a given domain – even if you are not the owner of the domain. For instance, you can route a domain or any subdomain at any time to any server of your choice. This is quite cool because whenever we’ll be ready with the setup of our EC2 server cluster we ourselves will be able to flip the switch. No need to contact someone, no need to wait impatiently for the changes to happen. If something goes wrong – let’s do a rollback to the previous setup.

To let Route 53 take over control your domain’s DNS we just have to change the nameserver records to Amazon’s nameservers. Go to the Route 53 console, create a new hosted zone, enter the domain name and save. After saving, the console lists the names of four Amazon nameservers.

You, your registrar or your client’s registrar will have to update your domain NS records with these nameservers. In our case we had to ask our client’s registrar to do this. But wait! As we are dealing with a productive application we should put the existing DNS configuration in place before requesting the nameserver switch – otherwise our application would not be accessible any more. Having our application running at myapp.onemedia.de and the code be hosted at 192.51.100.234 we would just have to create an appropriate A record:

Now we have some kind of shadow configuration, i.e. we’ve created a clone of the existing live DNS configuration which is not yet in use. So we are prepared to switch the nameservers to Amazon and everything will just work as before – but then we are in control of the domain’s DNS records.

Unfortunately something unexpected happend when we finally requested the nameserver switch at our client’s contact person. Along with replacing the nameservers the registrar also flushed the existing configuration – more precisely the A records we have also setup in Route 53. During the next hours we had lots of complaints about the application being offline and this was quite a painful situation. Moreover, reachability seemed to be quite arbitrary: some users had no issues while others were able to reach some of the subdomains only or even nothing at all. After thinking a while about what had happend we came to the conclusion that all the networks that requested the domain the day before the nameserver switch were not able to reach our application any more. The reason was the DNS cache of the network routers or the computers respectively. We could only wait for the time to live (TTL) of the nameserver records to expire and ask for understanding. Nonetheless, we learned our lesson for the next time and there are two possibilities what needs to be communicated to the registrar:

Replace the nameserver records but leave the current configuration in place for at least one day
Reduce the TTL for the nameserver records one or two days in advance – then change the nameserver records and flush the configuration

After about 18 hours everything was back to normal and we were happy to announce our first Amazon webservice to be live: Route 53