In this post I will try to guide you through creating an Amazon AMI which we will use to eventually create a Elasticsearch Cluster in AWS.

While writing it I kinda found out that this is too long. I will try to keep the length down in the future.

Table of Contents

Introduction into AWS AMI’s

Using Amazon EC2, you can run virtual servers in the cloud. Servers that do what you need them to do, webservers, database clusters or your crypto mining (Which would be stupid, no seriously, the ROI would be extremely stupid ;-))))

These virtual servers are started with a pre-installed OS based on “AMI’s” or “Amazon Machine Image’s”. These things are basically an OS compressed into a file that gets expanded when you start a virtual server. It contains everything needed to run the OS. Better put: it IS the OS.

Building AMI Building AMI

Amazon’s introduction:

“An Amazon Machine Image (AMI) provides the information required to launch an instance, which is a virtual server in the cloud.”

Yeah, very descriptive. Like all of Amazon’s documentation.

My use of Packer

Packer is one of the programs from the Hashicorp company. According to their website:


Packer Packer

“Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.”

Ofcourse, they should have suggested Ansible as the tool of choice. But we will forgive them this little error in judgement / lapse in sanity.

Basically using packer and ansible you can create ready made machine images in AWS repeatably. The packer template is a json file like this:

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": ""
  },
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "us-east-1",
    "source_ami_filter": {
      "filters": {
      "virtualization-type": "hvm",
      "name": "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*",
      "root-device-type": "ebs"
      },
      "owners": ["099720109477"],
      "most_recent": true
    },
    "instance_type": "t2.micro",
    "ssh_username": "ubuntu",
    "ami_name": "packer-example {{timestamp}}"
  }]
}

You can see that their example uses variables in certain places. First you define and fill them in the variables section, after which you can use them elsewhere, like in the builders section with {{user 'aws_access_key'}}.

image.json

The file we will be using is this one though:

{
  "builders": [
    {
      "type": "amazon-ebs",
      "region": "{{ user `aws_region` }}",
      "instance_type": "{{ user `aws_instance_type` }}",
      "spot_price": "auto",
      "spot_price_auto_product": "Linux/UNIX (Amazon VPC)",
      "source_ami_filter": {
        "filters": {
          "virtualization-type": "{{ user `source_ami_virtualization_type` }}",
          "name": "{{ user `source_ami_name` }}*",
          "root-device-type": "{{ user `source_ami_root_device_type` }}"
        },
        "owners": ["{{ user `source_ami_owner` }}"],
        "most_recent": "{{ user `source_ami_most_recent` }}"
      },
      "ami_name": "{{ user `ami_name` | clean_ami_name }}-{{ user `meta_timestamp` }}",
      "launch_block_device_mappings": [
        {
          "device_name": "/dev/sda1",
          "volume_size": 32,
          "volume_type": "gp2",
          "delete_on_termination": true
        }
      ],
      "ami_description": "{{ user `ami_description` }}",
      "ssh_username": "{{ user `ssh_username` }}",
      "ssh_timeout": "{{ user `ssh_timeout` }}",
      "ssh_pty": false,
      "subnet_id": "{{ user `subnet_id` }}",
      "tags": {
        "Amazon_AMI_Management_Identifier": "{{ user `ami_name` }}",
        "Name": "{{ user `ami_name` }}"
      },
      "user_data_file": "{{ user `user_data_file` }}",
      "vpc_id": "{{ user `vpc_id` }}"
    }
  ],
  "provisioners": [
    {
      "type": "ansible",
      "user": "{{user `ssh_username`}}",
      "sftp_command": "{{ user `sftp_command` }}",
      "playbook_file": "{{user `playbook_file`}}",
      "groups": [
        "packer"
      ],
      "extra_arguments": [
        "-{{ user `ansible_verbosity_level` }}",
        "--extra-vars=clustername={{ user `clustername`}}",
        "--extra-vars=es_version={{ user `es_version`}}",
        "--extra-vars=es_memory_gb={{ user `es_memory_gb`}}",
        "--extra-vars=es_version_family={{ user `es_version_family`}}"
      ],
      "ansible_env_vars": [
        "ANSIBLE_HOST_KEY_CHECKING=False",
        "ANSIBLE_NOCOLOR=True",
        "ANSIBLE_SSH_ARGS='-o ForwardAgent=no -o ControlMaster=auto -o ControlPersist=yes'"
      ]
    }
  ]

}

As you can see we will not be using a variables section. We will provide packer with all the variables from the commandline with make and a Makefile later on.

Besides that, our builders section is somewhat larger. The options for our amazon-ebs builder are all explained here

We made our source_ami_filter a little more extensive, and using launch_block_device_mappings we tell packer to make the root drive of our resulting AMI 32GB instead of the default 8GB. I want options later on. Limiting yourself to 8GB serves no one.

We also make use of spot pricing. That means we can start up our servers at a significantly reduced price. We won’t be needing it long anyway.

Then we add a provisioners section. This section tells packer that it needs to run Ansible after it is able to connect to the Virtual Instance. Packer also takes care of ssh-keys. All you need to do is create an ansible playbook or role (Or both; I won’t judge) and have packer run it.

Introduction into Ansible

Ansible Ansible

The bees-knees of “configuration management”.

Beautiful product. Seriously, this one is the one to learn and use. Forget about puppet, chef, saltstack, bash scripts, CFEngine (Really? You still use this?) or others. Just Ansible. It gets out of your way, is easy to use / adjust / manage.

I should probably make a complete blog series about ansible. There’s so much to explain and show you!

For now I will assume that you know Ansible already, and if not: start here

Using Make to tie it all together

Makefile for the win!!!! Yes I confess, I still use Makefile’s. And why not, they are perfectly capable and don’t need daemons or silly wrappers or the necessity for me to learn their special code. (Looking at you Gradle).

Anyway, this is the Makefile I use:

#!/usr/bin/make -f

# Makefile for maas-martin

.ONESHELL:

###
 # configuration
###

# AMI-specific settings
AMI_NAME_SLUG_CENTOS = mm-centos-6
AMI_USER_CENTOS = centos
AMI_NAME_SLUG_UBUNTU = mm-ubuntu-1804

AMI_USER_UBUNTU = ubuntu
AWS_PROFILE_STRING = maas-martin
AWS_REGION = eu-west-1

INSTANCE_TYPE = m3.medium
KEEP_RELEASES = 5
PLAYBOOKS_DIR = ./files/playbooks

PACKER_PLUGIN_PATH = $(HOME)/.packer.d/plugins
PACKER_POSTPROCESSOR_1 = "github.com/wata727/packer-post-processor-amazon-ami-management"
PACKER_PROVISIONER_1 = "github.com/unifio/packer-provisioner-serverspec"

SFTP_COMMAND = /usr/libexec/openssh/sftp-server -e

SOURCE_AMI_MOST_RECENT = true
SOURCE_AMI_VIRTUALIZATION_TYPE = hvm
SOURCE_AMI_ROOT_DEVICE_TYPE = ebs
SOURCE_AMI_NAME_BASE = CentOS Linux 6 x86_64 HVM EBS 1602
SOURCE_AMI_NAME_BASE_UBUNTU = ubuntu/images/*ubuntu-bionic-18.04-amd64-server-
SOURCE_AMI_OWNER_BASE = 679593333241
SOURCE_AMI_OWNER_BASE_UBUNTU = 099720109477
SSH_TIMEOUT = 5m

SUBNET_ID = subnet-eafae3b3
USER_DATA_FILE = ./files/scripts/user-data.sh
VPC_ID = vpc-ksjdhfskjfhkds

ANSIBLE_PLAYBOOKS = ./files/playbooks
ANSIBLE_TAGS_SKIP = skip
ANSIBLE_VERBOSITY_LEVEL = v

DESC_SUFFIX = image for maas-martin

# bootstrap images
IMAGE_BOOTSTRAP_CENTOS = bootstrap_centos
IMAGE_BOOTSTRAP_UBUNTU = bootstrap_ubuntu
DESC_BOOTSTRAP_CENTOS = CentOS 6.x-based $(DESC_SUFFIX)
DESC_BOOTSTRAP_UBUNTU = Ubuntu 18.04-based $(DESC_SUFFIX)

##
AWS_AMI_ID_BOOTSTRAP_CENTOS = $(strip $(shell sh ./files/scripts/get-ami-id.sh "$(AWS_PROFILE_STRING)" "$(AMI_NAME_SLUG_CENTOS)-$(IMAGE_BOOTSTRAP_CENTOS)"))
AWS_AMI_ID_BOOTSTRAP_UBUNTU = $(strip $(shell sh ./files/scripts/get-ami-id.sh "$(AWS_PROFILE_STRING)" "$(AMI_NAME_SLUG_UBUNTU)-$(IMAGE_BOOTSTRAP_UBUNTU)"))

META_TIMESTAMP = $(strip $(shell date +%s))

# check for availability of Packer
ifeq ($(shell which packer >/dev/null 2>&1; echo $$?), 1)
	PACKER_AVAILABLE = false
else
	PACKER_AVAILABLE = true
	PACKER_PATH = $(shell which packer)
	PACKER_VERSION = $(shell packer version | grep -m 1 -o '[0-9]*\.[0-9]*\.[0-9]')
endif
# end: check for availability of Packer

# check for availability of AWS CLI
ifeq ($(shell which aws >/dev/null 2>&1; echo $$?), 1)
	AWSCLI_AVAILABLE = false
else
	AWSCLI_AVAILABLE = true
	AWSCLI_PATH = $(shell which aws)
endif
# end: check for availability of AWS CLI

# check for availability of jq
ifeq ($(shell which jq >/dev/null 2>&1; echo $$?), 1)
	JQ_AVAILABLE = false
else
	JQ_AVAILABLE = true
	JQ_PATH = $(shell which jq)
endif
# end: check for availability of Packer

.PHONY: debug
debug:
	$(eval PACKER_DEBUG := -debug)
	@echo "Enabling debug mode for Packer"

.PHONY: debug-ansible
debug-ansible:
		$(eval ANSIBLE_DEBUG := -$(ANSIBLE_VERBOSITY_LEVEL))
		@echo "$(SIGN_WARN) Enabling debug mode for Ansible"

PACKER_CENTOS= \
	export AWS_PROFILE="$(AWS_PROFILE_STRING)" && \
	packer \
		build \
			$(PACKER_DEBUG) \
			-var "ansible_verbosity_level=$(ANSIBLE_VERBOSITY_LEVEL)" \
			-var "aws_instance_type=$(INSTANCE_TYPE)" \
			-var "aws_region=$(AWS_REGION)" \
			-var "meta_timestamp=$(META_TIMESTAMP)" \
			-var "sftp_command=$(SFTP_COMMAND)" \
      -var "source_ami_most_recent=$(SOURCE_AMI_MOST_RECENT)" \
      -var "source_ami_virtualization_type=$(SOURCE_AMI_VIRTUALIZATION_TYPE)" \
      -var "source_ami_root_device_type=$(SOURCE_AMI_ROOT_DEVICE_TYPE)" \
			-var "ssh_username=$(AMI_USER_CENTOS)" \
			-var "ssh_timeout=$(SSH_TIMEOUT)" \
			-var "subnet_id=$(SUBNET_ID)" \
			-var "user_data_file=$(USER_DATA_FILE)" \
			-var "vpc_id=$(VPC_ID)"

PACKER_UBUNTU= \
	export AWS_PROFILE="$(AWS_PROFILE_STRING)" && \
	packer \
		build \
			$(PACKER_DEBUG) \
			-var "ansible_verbosity_level=$(ANSIBLE_VERBOSITY_LEVEL)" \
			-var "aws_instance_type=$(INSTANCE_TYPE)" \
  		-var "keep_releases=$(KEEP_RELEASES)" \
			-var "aws_region=$(AWS_REGION)" \
			-var "meta_timestamp=$(META_TIMESTAMP)" \
			-var "sftp_command=$(SFTP_COMMAND)" \
			-var "source_ami_most_recent=$(SOURCE_AMI_MOST_RECENT)" \
			-var "source_ami_virtualization_type=$(SOURCE_AMI_VIRTUALIZATION_TYPE)" \
			-var "source_ami_root_device_type=$(SOURCE_AMI_ROOT_DEVICE_TYPE)" \
			-var "ssh_username=$(AMI_USER_UBUNTU)" \
			-var "ssh_timeout=$(SSH_TIMEOUT)" \
			-var "subnet_id=$(SUBNET_ID)" \
			-var "user_data_file=$(USER_DATA_FILE)" \
			-var "vpc_id=$(VPC_ID)"

.PHONY: check
check:
	@echo
	@echo "Checking local dependencies..."

# BEGIN: check for `packer` availability
	@echo
	@echo "Packer"

ifeq ($(PACKER_AVAILABLE), true)
	@echo "$(SIGN_OK) found binary at \"$(PACKER_PATH)\""
	@echo "$(SIGN_OK) found version \"$(PACKER_VERSION)\""
else
	@echo "$(SIGN_ERR) unable to find \"packer\""
	@EXIT_WITH_ERROR=true
endif
# END: check for `packer` availability

# BEGIN: check for `aws` availability
	@echo
	@echo "AWS CLI"

ifeq ($(AWSCLI_AVAILABLE), true)
	@echo "$(SIGN_OK) found binary at \"$(AWSCLI_PATH)\""
else
	@echo "$(SIGN_ERR) unable to find \"aws\""
	@EXIT_WITH_ERROR=true
endif
# END: check for `aws` availability

# BEGIN: check for `jq` availability
	@echo
	@echo "jq"

ifeq ($(JQ_AVAILABLE), true)
	@echo "$(SIGN_OK) found binary at \"$(JQ_PATH)\""
else
	@echo "$(SIGN_ERR) unable to find \"jq\""
	@EXIT_WITH_ERROR=true
endif
# END: check for `jq` availability

	@echo

ifeq ($(EXIT_WITH_ERROR), true)
	exit 1
endif


.PHONY: all
all: bootstrap_centos bootstrap_ubuntu

.PHONY: bootstrap_centos
bootstrap_centos:
	echo "Building \`bootstrap_centos\` image using base ($(c)) in VPC \`$(VPC_ID)\`" && \
	echo && \
	$(PACKER_CENTOS) \
		-var "source_ami=$(AMI_ID_SOURCE_CENTOS)" \
		-var "ami_name=$(AMI_NAME_SLUG_CENTOS)-$(IMAGE_BOOTSTRAP_CENTOS)" \
		-var "ami_description=$(DESC_BOOTSTRAP_CENTOS)" \
		-var "playbook_file=$(ANSIBLE_PLAYBOOKS)/$(IMAGE_BOOTSTRAP_CENTOS).yml" \
		-var "rspec_target=$(IMAGE_BOOTSTRAP_CENTOS)" \
    -var "source_ami_name=$(SOURCE_AMI_NAME_BASE)" \
    -var "source_ami_owner=$(SOURCE_AMI_OWNER_BASE)" \
		image.json

.PHONY: bootstrap_ubuntu
bootstrap_ubuntu:
	@echo "Building \`bootstrap_ubuntu\` image using base ($(AMI_ID_SOURCE_UBUNTU)) in VPC \`$(VPC_ID)\`" && \
	echo && \
	$(PACKER_UBUNTU) \
		-var "source_ami=$(AMI_ID_SOURCE_UBUNTU)" \
		-var "ami_name=$(AMI_NAME_SLUG_UBUNTU)-$(IMAGE_BOOTSTRAP_UBUNTU)" \
		-var "ami_description=$(DESC_BOOTSTRAP_UBUNTU)" \
		-var "playbook_file=$(ANSIBLE_PLAYBOOKS)/$(IMAGE_BOOTSTRAP_UBUNTU).yml" \
		-var "rspec_target=$(IMAGE_BOOTSTRAP_UBUNTU)" \
    -var "source_ami_name=$(SOURCE_AMI_NAME_BASE_UBUNTU)" \
    -var "source_ami_owner=$(SOURCE_AMI_OWNER_BASE_UBUNTU)" \
		image.json

include es-server.mk

My very warm and special thanks go out to Kerim. He made most of this Makefile I just heavily messed it up after him ;-)

First part

Woof! That’s a lot. Let’s break it down into smaller chunks:

#!/usr/bin/make -f

# Makefile for maas-martin

.ONESHELL:

###
 # configuration
###

# AMI-specific settings
AMI_NAME_SLUG_CENTOS = mm-centos-6
AMI_USER_CENTOS = centos
AMI_NAME_SLUG_UBUNTU = mm-ubuntu-1804

AMI_USER_UBUNTU = ubuntu
AWS_PROFILE_STRING = maas-martin
AWS_REGION = eu-west-1

INSTANCE_TYPE = m3.medium
KEEP_RELEASES = 5
PLAYBOOKS_DIR = ./files/playbooks

PACKER_PLUGIN_PATH = $(HOME)/.packer.d/plugins
PACKER_POSTPROCESSOR_1 = "github.com/wata727/packer-post-processor-amazon-ami-management"
PACKER_PROVISIONER_1 = "github.com/unifio/packer-provisioner-serverspec"

SFTP_COMMAND = /usr/libexec/openssh/sftp-server -e

SOURCE_AMI_MOST_RECENT = true
SOURCE_AMI_VIRTUALIZATION_TYPE = hvm
SOURCE_AMI_ROOT_DEVICE_TYPE = ebs
SOURCE_AMI_NAME_BASE = CentOS Linux 6 x86_64 HVM EBS 1602
SOURCE_AMI_NAME_BASE_UBUNTU = ubuntu/images/*ubuntu-bionic-18.04-amd64-server-
SOURCE_AMI_OWNER_BASE = 679593333241
SOURCE_AMI_OWNER_BASE_UBUNTU = 099720109477
SSH_TIMEOUT = 5m

SUBNET_ID = subnet-eafae3b3
USER_DATA_FILE = ./files/scripts/user-data.sh
VPC_ID = vpc-ksjdhfskjfhkds

ANSIBLE_PLAYBOOKS = ./files/playbooks
ANSIBLE_TAGS_SKIP = skip
ANSIBLE_VERBOSITY_LEVEL = v

DESC_SUFFIX = image for maas-martin

# bootstrap images
IMAGE_BOOTSTRAP_CENTOS = bootstrap_centos
IMAGE_BOOTSTRAP_UBUNTU = bootstrap_ubuntu
DESC_BOOTSTRAP_CENTOS = CentOS 6.x-based $(DESC_SUFFIX)
DESC_BOOTSTRAP_UBUNTU = Ubuntu 18.04-based $(DESC_SUFFIX)

##
AWS_AMI_ID_BOOTSTRAP_CENTOS = $(strip $(shell sh ./files/scripts/get-ami-id.sh "$(AWS_PROFILE_STRING)" "$(AMI_NAME_SLUG_CENTOS)-$(IMAGE_BOOTSTRAP_CENTOS)"))
AWS_AMI_ID_BOOTSTRAP_UBUNTU = $(strip $(shell sh ./files/scripts/get-ami-id.sh "$(AWS_PROFILE_STRING)" "$(AMI_NAME_SLUG_UBUNTU)-$(IMAGE_BOOTSTRAP_UBUNTU)"))

META_TIMESTAMP = $(strip $(shell date +%s))

.ONESHELL: Not many people know this trick. It makes it so that every command in a command block is executed in the same shell instead of a new one for each new command.

That way you can set a variable in a command and still access that variable in the next command.

The next big part is just setting variables.

Second part

Next part:

# check for availability of Packer
ifeq ($(shell which packer >/dev/null 2>&1; echo $$?), 1)
	PACKER_AVAILABLE = false
else
	PACKER_AVAILABLE = true
	PACKER_PATH = $(shell which packer)
	PACKER_VERSION = $(shell packer version | grep -m 1 -o '[0-9]*\.[0-9]*\.[0-9]')
endif
# end: check for availability of Packer

# check for availability of AWS CLI
ifeq ($(shell which aws >/dev/null 2>&1; echo $$?), 1)
	AWSCLI_AVAILABLE = false
else
	AWSCLI_AVAILABLE = true
	AWSCLI_PATH = $(shell which aws)
endif
# end: check for availability of AWS CLI

# check for availability of jq
ifeq ($(shell which jq >/dev/null 2>&1; echo $$?), 1)
	JQ_AVAILABLE = false
else
	JQ_AVAILABLE = true
	JQ_PATH = $(shell which jq)
endif
# end: check for availability of Packer

When you break things down like this, things kinda reveal themselves right?

Here we check to see if certain programs are actually there. And if they are, to set their variables to true and set their paths in a variable. Makes the command blocks later one easier to read and makes this whole thing work on more then one OS.

Third part

Next up:

.PHONY: debug
debug:
	$(eval PACKER_DEBUG := -debug)
	@echo "Enabling debug mode for Packer"

.PHONY: debug-ansible
	debug-ansible:
		$(eval ANSIBLE_DEBUG := -$(ANSIBLE_VERBOSITY_LEVEL))
		@echo "$(SIGN_WARN) Enabling debug mode for Ansible"

PACKER_CENTOS= \
	export AWS_PROFILE="$(AWS_PROFILE_STRING)" && \
	packer \
		build \
			$(PACKER_DEBUG) \
			-var "ansible_verbosity_level=$(ANSIBLE_VERBOSITY_LEVEL)" \
			-var "aws_instance_type=$(INSTANCE_TYPE)" \
			-var "aws_region=$(AWS_REGION)" \
			-var "meta_timestamp=$(META_TIMESTAMP)" \
			-var "sftp_command=$(SFTP_COMMAND)" \
      -var "source_ami_most_recent=$(SOURCE_AMI_MOST_RECENT)" \
      -var "source_ami_virtualization_type=$(SOURCE_AMI_VIRTUALIZATION_TYPE)" \
      -var "source_ami_root_device_type=$(SOURCE_AMI_ROOT_DEVICE_TYPE)" \
			-var "ssh_username=$(AMI_USER_CENTOS)" \
			-var "ssh_timeout=$(SSH_TIMEOUT)" \
			-var "subnet_id=$(SUBNET_ID)" \
			-var "user_data_file=$(USER_DATA_FILE)" \
			-var "vpc_id=$(VPC_ID)"

PACKER_UBUNTU= \
	export AWS_PROFILE="$(AWS_PROFILE_STRING)" && \
	packer \
		build \
			$(PACKER_DEBUG) \
			-var "ansible_verbosity_level=$(ANSIBLE_VERBOSITY_LEVEL)" \
			-var "aws_instance_type=$(INSTANCE_TYPE)" \
  		-var "keep_releases=$(KEEP_RELEASES)" \
			-var "aws_region=$(AWS_REGION)" \
			-var "meta_timestamp=$(META_TIMESTAMP)" \
			-var "sftp_command=$(SFTP_COMMAND)" \
			-var "source_ami_most_recent=$(SOURCE_AMI_MOST_RECENT)" \
			-var "source_ami_virtualization_type=$(SOURCE_AMI_VIRTUALIZATION_TYPE)" \
			-var "source_ami_root_device_type=$(SOURCE_AMI_ROOT_DEVICE_TYPE)" \
			-var "ssh_username=$(AMI_USER_UBUNTU)" \
			-var "ssh_timeout=$(SSH_TIMEOUT)" \
			-var "subnet_id=$(SUBNET_ID)" \
			-var "user_data_file=$(USER_DATA_FILE)" \
			-var "vpc_id=$(VPC_ID)"

The first two command blocks allow us to enable debugging levels for packer and ansible if we need that.

And then basically setting up two variables with the complete shell command for running packer for a Centos machine and a Ubuntu version.

Fourth part

.PHONY: check
check:
	@echo
	@echo "Checking local dependencies..."

# BEGIN: check for `packer` availability
	@echo
	@echo "Packer"

ifeq ($(PACKER_AVAILABLE), true)
	@echo "$(SIGN_OK) found binary at \"$(PACKER_PATH)\""
	@echo "$(SIGN_OK) found version \"$(PACKER_VERSION)\""
else
	@echo "$(SIGN_ERR) unable to find \"packer\""
	@EXIT_WITH_ERROR=true
endif
# END: check for `packer` availability

# BEGIN: check for `aws` availability
	@echo
	@echo "AWS CLI"

ifeq ($(AWSCLI_AVAILABLE), true)
	@echo "$(SIGN_OK) found binary at \"$(AWSCLI_PATH)\""
else
	@echo "$(SIGN_ERR) unable to find \"aws\""
	@EXIT_WITH_ERROR=true
endif
# END: check for `aws` availability

# BEGIN: check for `jq` availability
	@echo
	@echo "jq"

ifeq ($(JQ_AVAILABLE), true)
	@echo "$(SIGN_OK) found binary at \"$(JQ_PATH)\""
else
	@echo "$(SIGN_ERR) unable to find \"jq\""
	@EXIT_WITH_ERROR=true
endif
# END: check for `jq` availability

	@echo

ifeq ($(EXIT_WITH_ERROR), true)
	exit 1
endif

Here we actually show a notification to the user if we are missing programs and need to install those.

Fifth and meaty part

.PHONY: all
all: bootstrap_centos bootstrap_ubuntu

.PHONY: bootstrap_centos
bootstrap_centos:
	echo "Building \`bootstrap_centos\` image using base ($(c)) in VPC \`$(VPC_ID)\`" && \
	echo && \
	$(PACKER_CENTOS) \
		-var "source_ami=$(AMI_ID_SOURCE_CENTOS)" \
		-var "ami_name=$(AMI_NAME_SLUG_CENTOS)-$(IMAGE_BOOTSTRAP_CENTOS)" \
		-var "ami_description=$(DESC_BOOTSTRAP_CENTOS)" \
		-var "playbook_file=$(ANSIBLE_PLAYBOOKS)/$(IMAGE_BOOTSTRAP_CENTOS).yml" \
		-var "rspec_target=$(IMAGE_BOOTSTRAP_CENTOS)" \
    -var "source_ami_name=$(SOURCE_AMI_NAME_BASE)" \
    -var "source_ami_owner=$(SOURCE_AMI_OWNER_BASE)" \
		image.json

.PHONY: bootstrap_ubuntu
bootstrap_ubuntu:
	@echo "Building \`bootstrap_ubuntu\` image using base ($(AMI_ID_SOURCE_UBUNTU)) in VPC \`$(VPC_ID)\`" && \
	echo && \
	$(PACKER_UBUNTU) \
		-var "source_ami=$(AMI_ID_SOURCE_UBUNTU)" \
		-var "ami_name=$(AMI_NAME_SLUG_UBUNTU)-$(IMAGE_BOOTSTRAP_UBUNTU)" \
		-var "ami_description=$(DESC_BOOTSTRAP_UBUNTU)" \
		-var "playbook_file=$(ANSIBLE_PLAYBOOKS)/$(IMAGE_BOOTSTRAP_UBUNTU).yml" \
		-var "rspec_target=$(IMAGE_BOOTSTRAP_UBUNTU)" \
    -var "source_ami_name=$(SOURCE_AMI_NAME_BASE_UBUNTU)" \
    -var "source_ami_owner=$(SOURCE_AMI_OWNER_BASE_UBUNTU)" \
		image.json

include es-server.mk

Aaaaah, the meat of the Makefile.

These command blocks are the same really, so let’s just walk through on of them.

By now all variables are filled with actual data and we can ask packer to run the image.json we discussed earlier and give it all the variables it needs to fill in the parts inside the image.json.

Es servers makefile

Then we end with including es-server.mk which means we are including another Makefile from this one.

Sigh.. Another one? Well don’t worry, it’s a small one.

# Elastic Search
IMAGE_ES_SERVERS = es_server
IMAGE_ES_SERVERS_VERSION = 6.3.0
DESC_ES_SERVERS = Elastic Search Node Server $(DESC_SUFFIX)
SOURCE_AMI_NAME_ES_SERVERS = $(AMI_NAME_SLUG_UBUNTU)-$(IMAGE_BOOTSTRAP_UBUNTU)
SOURCE_AMI_OWNER_ES_SERVERS = self

.PHONY: es_server_acceptance
es_server_acceptance:
	export CLUSTERNAME=acc-es-server && \
	echo "Building \`$(IMAGE_ES_SERVERS)\` image using base in VPC \`$(VPC_ID)\`" && \
	echo && \
	$(PACKER_UBUNTU) \
		-var "ami_name=$(AMI_NAME_SLUG_UBUNTU)-$(IMAGE_ES_SERVERS)-acc-es-server-$(IMAGE_ES_SERVERS_VERSION)" \
		-var "ami_description=$(DESC_ES_SERVERS) - acc-es-server" \
		-var "playbook_file=$(ANSIBLE_PLAYBOOKS)/$(IMAGE_ES_SERVERS).yml" \
		-var "source_ami_name=$(SOURCE_AMI_NAME_ES_SERVERS)" \
		-var "source_ami_owner=$(SOURCE_AMI_OWNER_ES_SERVERS)" \
		-var "clustername=acc-es-server" \
		-var "es_version=$(IMAGE_ES_SERVERS_VERSION)" \
		-var "es_version_family=6.x" \
		-var "es_memory_gb=2" \
		image.json

.PHONY: es_server_production
es_server_production:
	export CLUSTERNAME=pro-es-server && \
	echo "Building \`$(IMAGE_ES_SERVERS)\` image using base in VPC \`$(VPC_ID)\`" && \
	echo && \
	$(PACKER_UBUNTU) \
		-var "ami_name=$(AMI_NAME_SLUG_UBUNTU)-$(IMAGE_ES_SERVERS)-pro-es-server-$(IMAGE_ES_SERVERS_VERSION)" \
		-var "ami_description=$(DESC_ES_SERVERS) - pro-es-server" \
		-var "playbook_file=$(ANSIBLE_PLAYBOOKS)/$(IMAGE_ES_SERVERS).yml" \
		-var "source_ami_name=$(SOURCE_AMI_NAME_ES_SERVERS)" \
		-var "source_ami_owner=$(SOURCE_AMI_OWNER_ES_SERVERS)" \
		-var "clustername=pro-es-server" \
		-var "es_version=$(IMAGE_ES_SERVERS_VERSION)" \
		-var "es_version_family=6.x" \
		-var "es_memory_gb=15" \
		image.json

See? Small one. And something you have see before. But this time we are creating acceptance and production versions of elastic search ami’s (They differ in size of the intended virtual machine. Less memory etc.)

Putting it all together

Assuming you can access your amazon environment with the aws cli command tool (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)

When you run make bootstrap_ubuntu es_server_acceptance packer should first create an ubuntu server, and then continue to make an elastic search server based off of that.

“But ehm Mark. We are missing the ansible playbooks / roles.”

Yep, that blog post is next.

Questions in the comments, I will answer them!