Images as code
Packer standardizes and automates the process of building system images. Create one workflow for images across multiple clouds and private infrastructure.
Sample code Part 1 - Redpanda - Stream Processing Server
HCL Launguage | Filename format: redpanda-aws-dc1.pkr.hcl
Amazon EC2 Services | AMI Linux 2 Instance | t2.micro Server
# Redpanda Image on Amazon
packer {
required_plugins {
amazon = {
version = ">= 0.0.2"
source = "github.com/hashicorp/amazon"
}
}
}
locals {
timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}
source "amazon-ebs" "redpanda-01" {
ami_name = "redpanda-blu-${local.timestamp}"
source_ami_filter {
filters = {
name = "amzn2-ami-kernel-5.10-hvm-2.*.0-x86_64-gp2"
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = ["amazon"]
}
# source_ami = "ami-0022f774911c1d690"
instance_type = "t2.micro"
region = "us-east-1"
ssh_username = "ec2-user"
}
build {
sources = [
"sources.amazon-ebs.redpanda-01"
]
provisioner "shell" {
script = "./redpanda-app.sh"
}
}
Sample code Part 2 - Redpanda - Stream Processing
Bash Shell Script Launguage | Filename format: redpanda-app.sh
Amazon (EC2) AMI Linux 2
#1/bin/bash
sleep 30
sudo yum update -y
curl -1sLf 'https://packages.vectorized.io/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' |
sudo -E bash && sudo yum install redpanda -y && sudo systemctl start redpanda
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
# Install HashiCorp Zero Trust Framework
sudo yum -y install yum-utils consul nomad vault
# Install Other Management Tools
# sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
# sudo mkdir /etc/prometheus.d
# sudo mkdir /etc/grafana.d
Output log information of install
packer build redpanda-aws-dc1.pkr.hcl
amazon-ebs.redpanda-01: output will be in this color.
==> amazon-ebs.redpanda-01: Prevalidating any provided VPC information
==> amazon-ebs.redpanda-01: Prevalidating AMI Name: redpanda-blu-20221220184408
amazon-ebs.redpanda-01: Found Image ID: ami-03d3f57d14bd8b2f5
==> amazon-ebs.redpanda-01: Creating temporary keypair: packer_63a20278-0aaf-8ddf-c771-998067aeace2
==> amazon-ebs.redpanda-01: Creating temporary security group for this instance: packer_63a2027a-d569-33de-dd9a-d8240bd21380
==> amazon-ebs.redpanda-01: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs.redpanda-01: Launching a source AWS instance...
amazon-ebs.redpanda-01: Instance ID: i-02d8eXXXX
==> amazon-ebs.redpanda-01: Waiting for instance (i-02d8eXXXX) to become ready...
==> amazon-ebs.redpanda-01: Using SSH communicator to connect: 3.87.8.245
==> amazon-ebs.redpanda-01: Waiting for SSH to become available...
==> amazon-ebs.redpanda-01: Connected to SSH!
==> amazon-ebs.redpanda-01: Provisioning with shell script: ./redpanda-app.sh
amazon-ebs.redpanda-01: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amazon-ebs.redpanda-01: Resolving Dependencies
amazon-ebs.redpanda-01: --> Running transaction check
amazon-ebs.redpanda-01: ---> Package kernel.x86_64 0:5.10.157-139.675.amzn2 will be installed
amazon-ebs.redpanda-01: ---> Package kernel-tools.x86_64 0:5.10.155-138.670.amzn2 will be updated
amazon-ebs.redpanda-01: ---> Package kernel-tools.x86_64 0:5.10.157-139.675.amzn2 will be an update
amazon-ebs.redpanda-01: --> Finished Dependency Resolution
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Dependencies Resolved
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Package Arch Version Repository Size
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Installing:
amazon-ebs.redpanda-01: kernel x86_64 5.10.157-139.675.amzn2 amzn2extra-kernel-5.10 33 M
amazon-ebs.redpanda-01: Updating:
amazon-ebs.redpanda-01: kernel-tools x86_64 5.10.157-139.675.amzn2 amzn2extra-kernel-5.10 195 k
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Transaction Summary
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Install 1 Package
amazon-ebs.redpanda-01: Upgrade 1 Package
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Total download size: 33 M
amazon-ebs.redpanda-01: Downloading packages:
amazon-ebs.redpanda-01: Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
amazon-ebs.redpanda-01: --------------------------------------------------------------------------------
amazon-ebs.redpanda-01: Total 55 MB/s | 33 MB 00:00
amazon-ebs.redpanda-01: Running transaction check
amazon-ebs.redpanda-01: Running transaction test
amazon-ebs.redpanda-01: Transaction test succeeded
amazon-ebs.redpanda-01: Running transaction
amazon-ebs.redpanda-01: Updating : kernel-tools-5.10.157-139.675.amzn2.x86_64 1/3
amazon-ebs.redpanda-01: Installing : kernel-5.10.157-139.675.amzn2.x86_64 2/3
amazon-ebs.redpanda-01: Cleanup : kernel-tools-5.10.155-138.670.amzn2.x86_64 3/3
amazon-ebs.redpanda-01: Verifying : kernel-5.10.157-139.675.amzn2.x86_64 1/3
amazon-ebs.redpanda-01: Verifying : kernel-tools-5.10.157-139.675.amzn2.x86_64 2/3
amazon-ebs.redpanda-01: Verifying : kernel-tools-5.10.155-138.670.amzn2.x86_64 3/3
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Installed:
amazon-ebs.redpanda-01: kernel.x86_64 0:5.10.157-139.675.amzn2
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Updated:
amazon-ebs.redpanda-01: kernel-tools.x86_64 0:5.10.157-139.675.amzn2
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Complete!
==> amazon-ebs.redpanda-01: Executing the setup script for the 'redpanda/redpanda' repository ...
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: OK: Checking for required executable 'curl' ...
amazon-ebs.redpanda-01: OK: Checking for required executable 'rpm' ...
amazon-ebs.redpanda-01: OK: Detecting your OS distribution and release using system methods ...
amazon-ebs.redpanda-01: ^^^^: ... Detected/provided for your OS/distribution, version and architecture:
amazon-ebs.redpanda-01: >>>>:
amazon-ebs.redpanda-01: >>>>: ... distro=amzn version=2 codename=2 arch=x86_64
amazon-ebs.redpanda-01: >>>>:
amazon-ebs.redpanda-01: OK: Importing 'redpanda/redpanda' repository GPG keys into rpm ...
amazon-ebs.redpanda-01: OK: Checking for available package manager (DNF/Microdnf/YUM/Zypper) ...
amazon-ebs.redpanda-01: ^^^^: ... Detected package manager as 'yum'
amazon-ebs.redpanda-01: OK: Checking for yum dependency 'yum-utils' ...
amazon-ebs.redpanda-01: OK: Checking if upstream install config is OK ...
amazon-ebs.redpanda-01: OK: Fetching 'redpanda/redpanda' repository configuration ...
amazon-ebs.redpanda-01: OK: Installing 'redpanda/redpanda' repository via yum ...
==> amazon-ebs.redpanda-01: Existing lock /var/run/yum.pid: another copy is running as pid 6285.
==> amazon-ebs.redpanda-01: Another app is currently holding the yum lock; waiting for it to exit...
==> amazon-ebs.redpanda-01: The other application is: yum
==> amazon-ebs.redpanda-01: Memory : 171 M RSS (390 MB VSZ)
==> amazon-ebs.redpanda-01: Started: Tue Dec 20 18:45:33 2022 - 00:07 ago
==> amazon-ebs.redpanda-01: State : Running, pid: 6285
==> amazon-ebs.redpanda-01: Importing GPG key 0x4918BC85:
==> amazon-ebs.redpanda-01: Userid : "Cloudsmith Package (vectorized/redpanda) <support@cloudsmith.io>"
==> amazon-ebs.redpanda-01: Fingerprint: ce4d 72b4 50d2 598d e73c b533 988a 7b0a 4918 bc85
==> amazon-ebs.redpanda-01: From : https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/gpg.988A7B0A4918BC85.key
==> amazon-ebs.redpanda-01: Importing GPG key 0x4918BC85:
==> amazon-ebs.redpanda-01: Userid : "Cloudsmith Package (vectorized/redpanda) <support@cloudsmith.io>"
==> amazon-ebs.redpanda-01: Fingerprint: ce4d 72b4 50d2 598d e73c b533 988a 7b0a 4918 bc85
==> amazon-ebs.redpanda-01: From : https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/gpg.988A7B0A4918BC85.key
==> amazon-ebs.redpanda-01: Importing GPG key 0x4918BC85:
==> amazon-ebs.redpanda-01: Userid : "Cloudsmith Package (vectorized/redpanda) <support@cloudsmith.io>"
==> amazon-ebs.redpanda-01: Fingerprint: ce4d 72b4 50d2 598d e73c b533 988a 7b0a 4918 bc85
==> amazon-ebs.redpanda-01: From : https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/gpg.988A7B0A4918BC85.key
amazon-ebs.redpanda-01: OK: Updating the yum cache to fetch the new repository metadata ...
amazon-ebs.redpanda-01: OK: The repository has been installed successfully - You're ready to rock!
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amazon-ebs.redpanda-01: Resolving Dependencies
amazon-ebs.redpanda-01: --> Running transaction check
amazon-ebs.redpanda-01: ---> Package redpanda.x86_64 0:22.3.9-1 will be installed
amazon-ebs.redpanda-01: --> Finished Dependency Resolution
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Dependencies Resolved
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Package Arch Version Repository Size
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Installing:
amazon-ebs.redpanda-01: redpanda x86_64 22.3.9-1 redpanda-redpanda 50 M
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Transaction Summary
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Install 1 Package
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Total download size: 50 M
amazon-ebs.redpanda-01: Installed size: 182 M
amazon-ebs.redpanda-01: Downloading packages:
amazon-ebs.redpanda-01: Running transaction check
amazon-ebs.redpanda-01: Running transaction test
amazon-ebs.redpanda-01: Transaction test succeeded
amazon-ebs.redpanda-01: Running transaction
amazon-ebs.redpanda-01: Installing : redpanda-22.3.9-1.x86_64 1/1
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Redpanda is set to run in development mode. To get the most out of the
amazon-ebs.redpanda-01: fastest queue in the west, enable production mode by running the following:
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: sudo rpk redpanda mode production
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: followed by:
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: sudo rpk tune all
amazon-ebs.redpanda-01: sudo systemctl start redpanda
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: This will autotune your system to give you the best performance from Redpanda.
amazon-ebs.redpanda-01: You can get more information on the tunning parameters here:
amazon-ebs.redpanda-01: https://docs.redpanda.com/docs/introduction/autotune/
amazon-ebs.redpanda-01: Verifying : redpanda-22.3.9-1.x86_64 1/1
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Installed:
amazon-ebs.redpanda-01: redpanda.x86_64 0:22.3.9-1
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Complete!
amazon-ebs.redpanda-01: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amazon-ebs.redpanda-01: adding repo from: https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
amazon-ebs.redpanda-01: grabbing file https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo to /etc/yum.repos.d/hashicorp.repo
amazon-ebs.redpanda-01: repo saved to /etc/yum.repos.d/hashicorp.repo
amazon-ebs.redpanda-01: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
==> amazon-ebs.redpanda-01: Existing lock /var/run/yum.pid: another copy is running as pid 6522.
==> amazon-ebs.redpanda-01: Another app is currently holding the yum lock; waiting for it to exit...
==> amazon-ebs.redpanda-01: The other application is: yum
==> amazon-ebs.redpanda-01: Memory : 126 M RSS (344 MB VSZ)
==> amazon-ebs.redpanda-01: Started: Tue Dec 20 18:46:00 2022 - 00:04 ago
==> amazon-ebs.redpanda-01: State : Uninterruptible, pid: 6522
==> amazon-ebs.redpanda-01: Another app is currently holding the yum lock; waiting for it to exit...
==> amazon-ebs.redpanda-01: The other application is: yum
==> amazon-ebs.redpanda-01: Memory : 165 M RSS (383 MB VSZ)
==> amazon-ebs.redpanda-01: Started: Tue Dec 20 18:46:00 2022 - 00:06 ago
==> amazon-ebs.redpanda-01: State : Running, pid: 6522
amazon-ebs.redpanda-01: Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
amazon-ebs.redpanda-01: Resolving Dependencies
amazon-ebs.redpanda-01: --> Running transaction check
amazon-ebs.redpanda-01: ---> Package consul.x86_64 0:1.14.3-1 will be installed
amazon-ebs.redpanda-01: ---> Package htop.x86_64 0:2.0.2-1.amzn2.0.2 will be installed
amazon-ebs.redpanda-01: ---> Package nomad.x86_64 0:1.4.3-1 will be installed
amazon-ebs.redpanda-01: ---> Package vault.x86_64 0:1.12.2-1 will be installed
amazon-ebs.redpanda-01: --> Finished Dependency Resolution
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Dependencies Resolved
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Package Arch Version Repository Size
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Installing:
amazon-ebs.redpanda-01: consul x86_64 1.14.3-1 hashicorp 50 M
amazon-ebs.redpanda-01: htop x86_64 2.0.2-1.amzn2.0.2 amzn2-core 98 k
amazon-ebs.redpanda-01: nomad x86_64 1.4.3-1 hashicorp 40 M
amazon-ebs.redpanda-01: vault x86_64 1.12.2-1 hashicorp 81 M
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Transaction Summary
amazon-ebs.redpanda-01: ================================================================================
amazon-ebs.redpanda-01: Install 4 Packages
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Total download size: 171 M
amazon-ebs.redpanda-01: Installed size: 426 M
amazon-ebs.redpanda-01: Downloading packages:
amazon-ebs.redpanda-01: Public key for nomad-1.4.3-1.x86_64.rpm is not installed
==> amazon-ebs.redpanda-01: warning: /var/cache/yum/x86_64/2/hashicorp/packages/nomad-1.4.3-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID a3219f7b: NOKEY
amazon-ebs.redpanda-01: --------------------------------------------------------------------------------
amazon-ebs.redpanda-01: Total 110 MB/s | 171 MB 00:01
amazon-ebs.redpanda-01: Retrieving key from https://rpm.releases.hashicorp.com/gpg
==> amazon-ebs.redpanda-01: Importing GPG key 0xA3219F7B:
==> amazon-ebs.redpanda-01: Userid : "HashiCorp Security (HashiCorp Package Signing) <security+packaging@hashicorp.com>"
==> amazon-ebs.redpanda-01: Fingerprint: e8a0 32e0 94d8 eb4e a189 d270 da41 8c88 a321 9f7b
==> amazon-ebs.redpanda-01: From : https://rpm.releases.hashicorp.com/gpg
amazon-ebs.redpanda-01: Running transaction check
amazon-ebs.redpanda-01: Running transaction test
amazon-ebs.redpanda-01: Transaction test succeeded
amazon-ebs.redpanda-01: Running transaction
amazon-ebs.redpanda-01: Installing : htop-2.0.2-1.amzn2.0.2.x86_64 1/4
amazon-ebs.redpanda-01: Generating Vault TLS key and self-signed certificate...
amazon-ebs.redpanda-01: Generating a 4096 bit RSA private key
amazon-ebs.redpanda-01: ......++
amazon-ebs.redpanda-01: ....................................................................++
amazon-ebs.redpanda-01: writing new private key to 'tls.key'
amazon-ebs.redpanda-01: -----
amazon-ebs.redpanda-01: Vault TLS key and self-signed certificate have been generated in '/opt/vault/tls'.
amazon-ebs.redpanda-01: Verifying : consul-1.14.3-1.x86_64 1/4
amazon-ebs.redpanda-01: Verifying : nomad-1.4.3-1.x86_64 2/4
amazon-ebs.redpanda-01: Verifying : vault-1.12.2-1.x86_64 3/4
amazon-ebs.redpanda-01: Verifying : htop-2.0.2-1.amzn2.0.2.x86_64 4/4
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Installed:
amazon-ebs.redpanda-01: consul.x86_64 0:1.14.3-1 htop.x86_64 0:2.0.2-1.amzn2.0.2
amazon-ebs.redpanda-01: nomad.x86_64 0:1.4.3-1 vault.x86_64 0:1.12.2-1
amazon-ebs.redpanda-01:
amazon-ebs.redpanda-01: Complete!
==> amazon-ebs.redpanda-01: Stopping the source instance...
amazon-ebs.redpanda-01: Stopping instance
==> amazon-ebs.redpanda-01: Waiting for the instance to stop...
==> amazon-ebs.redpanda-01: Creating AMI redpanda-blu-20221220184408 from instance i-02d8eXX
amazon-ebs.redpanda-01: AMI: ami-0b8e3e6fXXX
==> amazon-ebs.redpanda-01: Waiting for AMI to become ready...
==> amazon-ebs.redpanda-01: Skipping Enable AMI deprecation...
==> amazon-ebs.redpanda-01: Terminating the source AWS instance...
==> amazon-ebs.redpanda-01: Cleaning up any extra volumes...
==> amazon-ebs.redpanda-01: No volumes to clean up, skipping
==> amazon-ebs.redpanda-01: Deleting temporary security group...
==> amazon-ebs.redpanda-01: Deleting temporary keypair...
Build 'amazon-ebs.redpanda-01' finished after 6 minutes 9 seconds.
==> Wait completed after 6 minutes 9 seconds
==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs.redpanda-01: AMIs were created:
us-east-1: ami-XXXX