Deploying an Application With a Plan

Difficulty: Advanced

Time: Approximately 20 minutes

In this exercise you will further explore Bolt Plans by writing a multi-stage plan to deploy a sample application.

The sample application for this lesson consists of four targets. It has a single database target, two application servers, and a single load balancer in front of them. When a new version of the application is released the following steps must be carried out:

1. Install application code on the application and database servers. my_app::install
2. Run migrations on the database server. my_app::migrate
3. For each application server:
   1. Check connections on the load balancer. my_app::lb
   2. Drain connections on the load balancer. my_app::lb
   3. Restart with the new code version. my_app::deploy
   4. Run a health check. my_app_healthcheck
   5. Add back to the load balancer. my_app::lb
4. Clean up the old version of the application.

Prerequisites

For the following exercises you should have Bolt installed and have four Linux targets available. The following guides will help:

• Acquiring Targets
• Writing Tasks
• Writing Advanced Plans

Acquire targets

This lesson requires four targets. If you set up targets in Lesson 2, you need to provision an extra target.

Provision Extra Targets on Vagrant

If you set up targets in Vagrant in Lesson 2, run:

TARGETS=4 vagrant up

Note: Vagrantfile looks for environment variable TARGETS for number of centos targets to provision. Set environment variable based on your shell, for example in bash: export TARGETS=4.

Update your SSH config to include the new target.

vagrant ssh-config --host target4 | sed /StrictHostKeyChecking/d | sed /UserKnownHostsFile/d >> ~/.ssh/config

Provision Extra Targets on Docker

If you set up targets in Docker in Lesson 2, run:

docker-compose up --scale ssh=4 -d

Update your SSH config to include the new target.

Set up inventory file

Set up an inventory file to more easily map the targets to their role in the application. Save this file in Boltdir’s parent directory.

Assign one target to the load balancer (lb) group, one to the database (db) group and the other two to the application (app) group.

---
version: 2
groups:
  - name: lb
    targets:
      - "0.0.0.0:32771"
  - name: db
    targets:
      - "0.0.0.0:32770"
  - name: app
    targets:
      - "0.0.0.0:32768"
      - "0.0.0.0:32769"
config:
  ssh:
    host-key-check: false
    # These are credentials for Docker. Manage Vagrant with SSH config.
    password: root
    user: root

If you configured your targets using Vagrant, your inventory file may look like this:

---
# Inventory file for Bolt Hands-on Lab Lesson 10
version: 2
groups:
  - name: lb
    targets:
      - target1
  - name: db
    targets:
      - target2
  - name: app
    targets:
      - target3
      - target4
config:
  ssh:
    host-key-check: false

Make sure your inventory is configured correctly and you can connect to all targets. Run:

bolt command run 'echo hi' -n db,app,lb --inventoryfile ./inventory.yaml

Using the --inventoryfile flag, you can specify the source for your inventory file other than the default location Boltdir/inventory.yaml.

Write tasks for each stage of the application deployment

The tasks for this plan are code samples to enable us to focus on the plan itself. Save each of the following files to Boltdir/site-modules/my_app/tasks/.

#!/usr/bin/env python
# my_app/tasks/install.py
# Install application

import json
import sys

params = json.load(sys.stdin)
json.dump(dict(status = "success", previous_version = "1.0.0", new_version = params['version']), sys.stdout)

#!/usr/bin/env python
# my_app/tasks/migrate.py
# Migrate DB schema

import json
import sys

params = json.load(sys.stdin)
json.dump(dict(status = "success"), sys.stdout)

#!/usr/bin/env python
# my_app/tasks/lb.py
# manipulate load_balancer

import json
import sys

from random import randint
from time import sleep

params = json.load(sys.stdin)

def stats():
    return { "connections": randint(0, 10), "status": "ok" }

def drain():
    sleep(3)
    return { "status": "success"}

def add():
    return { "status": "success" }

result_fn  = {
  "stats" : stats,
  "drain": drain,
  "add" : add,
}[params["action"]]

json.dump(result_fn(), sys.stdout)

#!/usr/bin/env python
# my_app/tasks/deploy.py
# Update and restart the application on the new version

import json
import sys

json.dump(dict(status = "success"), sys.stdout)

#!/usr/bin/env python
# my_app/tasks/health_check.py
# perform a healthcheck of a url

import json
import sys

json.dump(dict(status = "success"), sys.stdout)

#!/usr/bin/env python
# my_app/tasks/uninstall.py
# Remove and old version of the aplication

import json
import sys

json.dump(dict(status = "success"), sys.stdout)

Write a plan that uses the tasks

Once you have finished writing the tasks, you can add them to a plan and automate the application deployment. The plan performs some validation, installs the application, migrates the database, makes the new code available on each application server and, finally, cleans up old versions of the application.

Save this file to Boltdir/site-modules/my_app/plans/deploy.pp.

plan my_app::deploy(
  Pattern[/\d+\.\d+\.\d+/] $version,
  TargetSpec $app_servers,
  TargetSpec $db_server,
  TargetSpec $lb_server,
  String[1] $instance = 'my_app',
  Boolean $force = false
) {
  # Validate that there is only a single lb server to check
  if get_targets($lb_server).length > 1 {
    fail_plan("${lb_server} did not resolve to a single target")
  }

  # First query the load balancer and make sure the app isn't under too much load to do a deploy.
  unless $force {
    $conns = run_task('my_app::lb', $lb_server,
       "Check load before starting deploy",
       action => 'stats',
       backend => $instance,
       server => 'FRONTEND',
    ).first['connections']
    if ($conns > 8) {
      fail_plan("The appplication has too many open connections: ${conns}")
    } else {
      # Info messages will be displayed when the --verbose flag is used.
      info("Application has ${conns} open connections.")
    }
  }

  # Install the new version of the application and check what version was previously
  # installed so it can be deleted after the deploy.
  $old_versions = run_task('my_app::install', [$app_servers, $db_server],
    "Install ${version} of the application",
    version => $version
  ).map |$r| { $r['previous_version'] }

  run_task('my_app::migrate', $db_server)

  # Don't log every action on each target only important messages
  without_default_logging() || {
    # Expand group references or globs before iterating
    get_targets($app_servers).each |$server| {

      # Check stats and print a message to the user
      $stats = run_task('my_app::lb', $lb_server,
        action => 'stats',
        backend => $instance,
        server => $server.name,
        _catch_errors => $force
      ).first
      out::message("Deploying to ${server.name}, currently ${stats["status"]} with ${stats["connections"]} open connections.")

      run_task('my_app::lb', $lb_server,
        "Drain connections from ${server.name}",
        action => 'drain',
        backend => $instance,
        server => $server.name,
        _catch_errors => $force
      )

      run_task('my_app::deploy', [$server],
        "Update application for new version",
      )

      # Verify the app server is healthy before returning it to the the load
      # balancer.
      $health = run_task('my_app::health_check', $lb_server,
        "Run Healthcheck for ${server.name}",
        target => "http://${server.name}:5000/",
        '_catch_errors' => true).first

      if $health['status'] == 'success' {
        info("Upgrade Healthy, Returning ${server.name} to load balancer")
      } else {
        # Fail the plan unless the app server is healthy or this is a forced deploy
        unless $force {
          fail_plan("Deploy failed on app server ${server.name}: ${health.result}")
        }
      }

      run_task('my_app::lb', $lb_server,
        action => 'add',
        backend => $instance,
        server => $server.name,
        _catch_errors => $force
      )
      out::message("Deploy complete on ${server}.")
    }
  }

  run_task('my_app::uninstall', [$db_server, $app_servers],
    "Clean up old versions",
    live_versions => $old_versions + $version,
  )
}

Run this plan with the following command. It will randomly fail 10% of the time when the simulated load is high.

bolt plan run my_app::deploy version=1.0.2 app_servers=app db_server=db lb_server=lb --inventoryfile ./inventory.yaml

The result (when simulated load is below threshold)

Starting: plan my_app::deploy
Starting: Check load before starting deploy on target1
Finished: Check load before starting deploy with 0 failures in 0.89 sec
Starting: Install 1.0.2 of the application on target3, target4, target2
Finished: Install 1.0.2 of the application with 0 failures in 0.89 sec
Starting: task my_app::migrate on target2
Finished: task my_app::migrate with 0 failures in 0.85 sec
Deploying to target3, currently ok with 4 open connections.
Deploy complete on Target('target3', {"connect-timeout"=>10, "tty"=>false, "host-key-check"=>false}).
Deploying to target4, currently ok with 10 open connections.
Deploy complete on Target('target4', {"connect-timeout"=>10, "tty"=>false, "host-key-check"=>false}).
Starting: Clean up old versions on target2, target3, target4
Finished: Clean up old versions with 0 failures in 0.88 sec
Finished: plan my_app::deploy in 4.35 sec
Plan completed successfully with no result

The result (when simulated load is above threshold)

Starting: plan my_app::deploy
Starting: Check load before starting deploy on target1
Finished: Check load before starting deploy with 0 failures in 0.89 sec
Finished: plan my_app::deploy in 0.90 sec
{
  "kind": "bolt/plan-failure",
  "msg": "The appplication has too many open connections: 10",
  "details": {
  }
}

Parameters

plan my_app::deploy(
  Pattern[/\d+\.\d+\.\d+/] $version,
  TargetSpec $app_servers,
  TargetSpec $db_server,
  TargetSpec $lb_server,
  String[1] $instance = 'my_app',
  Boolean $force = false
)

• $version is restricted to a string matching and x.y.z version format with the Pattern Type.
• $app_servers, $db_servers, and $lb_server, three TargetSpec parameters for the different tiers of the application. The TargetSpec type any String, Target or Array. By using this type we allow the command line to pass target URL strings, group name strings or allow another plan or JSON parameters to pass arrays of urls or targets.
• $instance the name of the instance of this application in the load balancer.
• $force an option to ignore errors and force the deploy to complete.

Install application

The first step in the plan is to install the new version of the code so all application servers can serve a new version of the assets and to migrate the database in preparation for deploying each app server.

  $old_version = run_task('my_app::install', [$app_servers, $db_server],
    "Install ${version} of the application",
    version => $version
  ).first['previous_version']
  run_task('my_app::migrate', $db_server)

The run_task command accepts a description argument that you can use to provide clearer log messages for the installation step. You can include the version being installed. Use the result of the task to store the old versions of the application for the uninstall step.

Loop over each application server

Now the application is staged loop over each app server and update it to use the new code.

  # Expand group references or globs before iterating
  get_targets($app_servers).each |$server| {

    run_task('my_app::lb', $lb_server,
      "Drain connections from ${server.name}",
      action => 'drain',
      backend => $instance,
      server => $server.name,
    )

    run_task('my_app::deploy', [$server],
      "Update application for new version",
    )

    run_task('my_app::lb', $lb_server,
      action => 'add',
      backend => $instance,
      server => $server.name,
    )
    out::message("Deploy complete on ${server}.")
  }

To loop over targets call get_targets to expand any groups or globs referenced in the $app_servers parameter then loop over each server with each. For each server drain connections from the load balancer, deploy the new version of application and then add the server back to the load balancer. Afterwards log an out::message to inform the user that the deploy is complete on that server.

Perform checks before the deploy

The plan doesn’t support multiple load balancers so validate that the TargetSpec passed for $lb_server resolves to only a single target and fail the plan otherwise. In order to prevent deploys during dangerously high load check how many open connections the app has and fail if load is too high.

if get_targets($lb_server).length > 1 {
  fail_plan("${lb_server} did not resolve to a single target")
}

$conns = run_task('my_app::lb', $lb_server,
   "Check load before starting deploy",
   action => 'stats',
   backend => $instance,
   server => 'FRONTEND',
).first['connections']
if ($conns > 8) {
  fail_plan("The application has too many open connections: ${conns}")
} else {
  # Info messages will be displayed when the --verbose flag is used.
  info("Application has ${conns} open connections.")
}

Use the fail_plan function to stop the plan in both cases. fail_plan will stop the current plan execution and the execution of any calling plan. It accepts a message that will be displayed to the user if the error is not caught.

Check server status before deploy

# Check stats and print a message to the user
$stats = run_task('my_app::lb', $lb_server,
  action => 'stats',
  backend => $instance,
  server => $server.name,
  _catch_errors => $force
).first
out::message("Deploying to ${server.name}, currently ${stats["status"]} with ${stats["connections"]} open connections.")

Before starting to deploy to a server call the my_app::lb task with the stats action and name of the server that is about to be deployed to and save the result the $stats variable. Then use those stats to print an informative notice about which server is going to be deployed to next.

Suppress default log messages

Bolt logs every action on each target, which results in 5 messages for each target. By default these messages are logged at the notice level and can make it hard to see the more useful notice messages that the plan logs directly. You can wrap code in a without_default_logging block to make the automatic messages log at info instead so they only appear on the terminal with the --verbose flag. All code executed inside a without_default_logging block including in functions or subplans will log actions at info instead of notice.

without_default_logging() || {
  # Expand group references or globs before iterating
  get_targets($app_servers).each |$server| {
    # Call deploy actions here
    out::message("Deploy complete on ${server}.")
  }
}

Note: Pay careful attention to the empty () and ||. The without_default_logging block takes no parameters but empty pipes are required for puppet block syntax. The () is required to avoid parser ambiguity with the empty pipes.

Catch errors for force

run_task('my_app::lb', $lb_server,
  "Drain connections from ${server.name}",
  action => 'drain',
  backend => $instance,
  server => $server.name,
  _catch_errors => $force
)

In order to prevent execution from halting after an error when the $force parameter is specified we have to use the _catch_errors metaparam. For each run_task command that should continue when in force mode add _catch_errors => true to the parameters.

Next steps

Now that you have learned about deploying an app you can learn to apply manifest code with bolt.

Applying Manifest Code With Bolt