Bamboo AWS Plugin /

Filename Size Date modified Message
src
665 B
131 B
140 B
95 B
86 B
705 B
11.2 KB
611 B
16.7 KB
5.9 KB

NOTE: This add-on is deprecated and superseded by the commercially supported Tasks for AWS and will remain available in maintenance mode only - please use Tasks for AWS for new projects, it offers all existing features, further improvements and additional tasks.

Overview

The Bamboo AWS Plugin adds several Amazon Web Services (AWS) related Tasks to deploy and operate AWS resources on demand. Furthermore, you can enable various development, testing and disaster recovery scenarios by operating backup schedules for EBS volumes and EC2 instances:

  • AWS CloudFormation Stack - create, update or delete an AWS CloudFormation stack (a collection of related Amazon Web Services resources).
  • Amazon Elastic Compute Cloud (EC2) Instance - start, stop or reboot provisioned EC2 instances.
  • Amazon Elastic Block Store (EBS) Snapshot - create, delete or backup snapshots of EBS volumes.
  • Amazon Elastic Compute Cloud (EC2) Image - create, delete or backup images of EBS backed EC2 instances.

Integrate AWS into your Bamboo builds and DevOps workflows with the following key features:

  • Stack Lifecycle Management - create, update or delete a CloudFormation stack defined by a template provided via URL or inline and specify template parameters and advanced options.
  • Variable Substitution/Definition - substitute Bamboo variables in CloudFormation template parameters (and some other fields) to customize the stack and reuse variables defined by stack outputs in subsequent Tasks.
  • Instance State Transitions - start, stop or reboot provisioned EC2 instances on demand, e.g. only when needed by the build itself (development) or at specific times of the day (operations).
  • Backup Set Management - backup provisioned EBS volumes (via EBS snapshots) or EBS backed EC2 instances (via EC2 images) with retention handling.

Release Notes

1.6.0

Release 1.6.0 adds compatibility with Bamboo 5.0, adds support for CloudFormation stack updates, adds variable definitions for created/affected resources, adds multiple resource handling for EC2 instance operations, enables region selection by variable and fixes AWS API eventually consistency errors, see the Changelog for details.

  • Note: Bamboo 5.0 compatibility does not imply support of Deployment Projects, which will be addressed separately.

Goals / Usage

The short term goal is mirroring the functionality available via the AWS management console in a Bamboo Task, i.e. creating/updating/deleting a stack with parameters. Familiarity with CloudFormation operation is assumed accordingly, thus the documentation currently covers noteworthy specifics regarding the plugin only.

Installation

Simply navigate to your Bamboo Administration -> Plugins -> Find New Add-ons menu, and search for "AWS" under the Install tab.

Sample Templates

The provided CloudFormation sample template URL simply exercises parameter substitution (thus there are no cost implied).

Amazon provides many more AWS CloudFormation Sample Templates - please be aware of the following:

  • You will be billed for the AWS resources used, if you create a stack from a template including non free resources (check the template source, which will indicate this in case).
  • You must reference a template in an S3 bucket in the same region in which you are creating the stack. Each sample template is available in every AWS Region (see 'Using Templates in Different Regions' on the bottom of the referenced URL for details).

The provided CloudFormation sample inline template simply creates a SNS Topic (thus there are no cost implied until you send a significant amount of notifications).

Variable Substitution/Definition

To empower advanced build and automation scenarios, it his highly recommended to become acquainted with Using Global, Plan or Build-specific Variables.

Variable Substitution

Variables are substituted in all Task configuration text fields (e.g. Stack Name, Template URL, Instance ID, Volume ID etc.).

Please note the following feature:

  • if the variable key contains the phrase "password", the value will be masked with "********" in the build logs; e.g. if the key is "password", "awsAccessKeyPassword" or "awsSecretKeyPassword" the build log will show the substituted value as "********"

Variable Definition

Variables are defined by all Tasks for reuse in subsequent Tasks as follows:

  • variables have a dedicated prefix like bamboo.custom.aws.*, with * being a task specific prefix, e.g. bamboo.custom.aws.cfn.stack
  • variables refering to a collection of resources provide their ids in a semicolon separated list (i.e. the same format available on input), e.g. ${bamboo.custom.aws.ec2.image.resources} with values ami-985b21f1;ami-9a5b21f3
  • you can refer to these variables from subsequent tasks via something like ${bamboo.custom.aws.cfn.stack.sampleStackOutputKey}
  • these variables are also available as environment variables in the Script Task for example, albeit named slightly different, e.g. $bamboo_custom_aws_cfn_stack_StringWithRegex (Unix) or %bamboo_custom_aws_cfn_stack_StringWithRegex% (Windows)

Please note the following constraints:

  • variables are only reusable in subsequent tasks and not in other jobs/stages due to the implied concurrency, see the following discussion and workaround

Available variables per Task are as follows:

  • AWS CloudFormation Stack
    • the collection of affected stack resources, e.g. bamboo.custom.aws.cfn.stack.resources=i-3280997f;vol-e1debbcd;test-stack-SecurityGroup-KZWPADIUPCL6
    • the status of each affected stack resource, e.g. bamboo.custom.aws.cfn.stack.i-3280997f=CREATE_COMPLETE
    • the collection of generated stack outputs, e.g. bamboo.custom.aws.cfn.stack.outputs=sampleStackOutputKey;StringWithRegex;...
    • the value of each generated stack output, e.g. bamboo.custom.aws.cfn.stack.StringWithRegex=Hello
      • Note: The variables with legacy naming without prefix remain available for compatibility (e.g. bamboo.StringWithRegex=Hello)
  • AWS EC2 Instance
    • the collection of affected instances, e.g. bamboo.custom.aws.ec2.instance.resources=i-fa7b4596;i-080eec64
    • the status of each affected instance, e.g. bamboo.custom.aws.ec2.instance.i-fa7b4596=started
  • AWS EC2 Image
    • the collection of affected images, e.g. bamboo.custom.aws.ec2.image.resources=ami-985b21f1;ami-9a5b21f3
    • the status of each affected instance, e.g. bamboo.custom.aws.ec2.image.ami-985b21f1=available
  • AWS EBS Snapshots
    • the collection of affected snapshots, e.g. bamboo.custom.aws.ec2.snapshots.resources=snap-f4dc35a1
    • the status of each affected snapshots, e.g. bamboo.custom.aws.ec2.snapshots.snap-f4dc35a1=completed

AWS Credentials

The AWS credentials need to be specified for each task currently, which can be cumbersome quickly. Pending a more generic solution, it is already possible to ease this a bit via variable substitution as follows:

  • configure Access Key and Secret Key as e.g. ${bamboo.awsAccessKeyPassword} and ${bamboo.awsSecretKeyPassword}
  • define plan and/or global variables for the configured variable names (i.e. awsAccessKeyPassword and awsSecretKeyPassword given this example) with the actual credentials, which will then be substituted on task execution accordingly

AWS Client Configuration

The AWS API is eventually consistent only and also exhibits a customer specific dynamic throttling policy, both of which require respective retry logic to be in place. Accordingly the facilitated AWS SDK for Java features an exponential backoff strategy already, but its default retry number of 3 (accumulating to a retry window of up to ~4 seconds) has proven to be too low for the tasks at hand, which has been increased to 7 accordingly (accumulating to a retry window of up to ~1 minute).

This should ideally be sufficient for most scenarios, but the values are adjustable by defining one or both of the following variables if need be:

  • ${custom.aws.maxErrorRetry} - how many retries should the exponential backoff algorithm perform (default: 7)
  • ${custom.aws.awaitTransitionInterval} - how long should the task wait before querying the resource transition state again (default: 15000 milliseconds)

HTTP(S) Proxy Configuration

If your Bamboo instance is running behind a firewall, the add-on will reuse the proxy configuration from Bamboo.

Please note that the AWS API calls use SSL throughout, so the add-on relies on the respective system properties https.proxyHost and https.proxyPort to be available from the Java Virtual Machine (JVM). Depending on your environment, you might need to set these https.* variations explicitly in addition to the usual http.* ones.

Limitations / Roadmap

You should be aware of the following limitations regarding AWS API coverage and integration:

AWS CloudFormation

  • We have focused on API coverage so far and usability is not up to speed yet accordingly, there is obviously room for improvements, e.g.:
    • Some configuration screen polish should get us a long way already.
    • Fetching existing AWS resources for selection during configuration would avoid manual copying of information.
    • Offering several samples/defaults to ease getting started.
    • ...
  • Stack rollback is currently handled as follows, hopefully covering the majority of use cases (please let us know otherwise):
    • A stack rolled back successfully by CloudFormation is treated as a failed build by Bamboo.
    • A stack not rolled back due to rollback being disabled explicitly is still treated as a failed build by Bamboo.

Amazon Elastic Compute Cloud (EC2)

  • We currently consider instance lifecycle goals (i.e. create/terminate) to be sufficiently addressed by the CloudFormation Task, hopefully covering the majority of use cases (please let us know otherwise).

Further down the road we'd like to integrate other AWS services as well.

License

Licensed under the Apache License, Version 2.0, see LICENSE for details.

Changelog

1.6.0

Release 1.6.0 adds compatibility with Bamboo 5.0, adds support for CloudFormation stack updates, adds variable definitions for created/affected resources, adds multiple resource handling for EC2 instance operations, enables region selection by variable and fixes AWS API eventually consistency errors.

  • Note: Bamboo 5.0 compatibility does not imply support of Deployment Projects, which will be addressed separately.

Stories

  • BAWS-11 - As a user, I need a task to update a CloudFormation stack so that I can maintain a collection of related AWS resources
  • BAWS-169 - As a user, I need variables with ids of affected resources so that I can refer to them in subsequent tasks

Improvements

  • BAWS-109 - Add multiple resource handling for EC2 Instances Task
  • BAWS-144 - Allow Bamboo variable usage for AWS region selection

Bugs

  • BAWS-167 - Fix EC2 Image task failing with 'InvalidAMIID.NotFound' or 'InvalidSnapshot.InUse'

1.5.4

Release 1.5.4 fixes variable substitution in resource id fields.

Improvements

  • BAWS-166 - Update documentation regarding reuse of CloudFormation output values.

Bugs

  • BAWS-167 - Fix field validation preventing Bamboo variable substitution.

1.5.3

Release 1.5.3 fixes/adds outbound proxy support.

Bugs

  • BAWS-160 - Fix/Add outbound proxy support.

1.5.2

Release 1.5.2 adds compatibility with Bamboo 4.4, fixes AWS API throttling and related issues and fixes execution errors on remote agents.

Bugs

  • BAWS-116 - Fix EC2 Image task failing with 'InvalidAMIID.NotFound'.
  • BAWS-123 - Fix reported number of transitioned images, when none are matched by the resource filter specification.
  • BAWS-125 - Fix build not failing for specified resources.
  • BAWS-139 - Fix SLF4J version/binding errors, causing build failure with Bamboo 4.4.
  • BAWS-143 - Fix frequent polling of AWS API causing thottle limit errors.
  • BAWS-145 - Fix invalid class loader hierarchy regarding 'org.apache.commons.logging.Log'.
  • BAWS-151 - Fix NullPointerException when executing on remote agents.

1.5.1

Release 1.5.1 fixes image creation failures when Bamboo operates with low latency to the AWS API (e.g. in the same datacenter).

Bugs

  • BAWS-115 - Fix either lack of support for or awareness of instance store backed EC2 Windows instances.
  • BAWS-116 - Fix EC2 Image task failing with 'InvalidAMIID.NotFound'.

1.5.0

Release 1.5.0 adds a Task to create, delete or backup (create/delete with retention) snapshots of EBS volumes and a Task to create, delete or backup (create/delete with retention) images of EBS backed EC2 instances.

Stories

  • BAWS-78 - As a user, I need a task to create/delete EBS snapshots so that I can backup provisioned EBS volumes.
  • BAWS-99 - As a user, I need a task to create/delete EBS-backed AMIs so that I can backup provisioned EBS-backed EC2 instances.

Improvements

  • BAWS-74 - Add logo/icon for Plugin.
  • BAWS-91 - Add multiple resource handling.
    • The EBS Snapshot Task and the EC2 Image Task currently support multiple resources by id, support for full fledged filters is forthcoming.
    • The EC2 Instance Task still supports only one instance at a time, but support for multiple resources is forthcoming as well.

1.4.0

Release 1.4.0 adds a task to start, stop or reboot a provisioned EC2 instance; in addition it adds event reporting during CloudFormation stack transitions, retrieves AWS configuration data remotely and features some minor usability improvements as well.

Stories

  • BAWS-68 - As a user, I need a task to start/stop/reboot an EC2 instance so that I can use provisioned AWS resources on demand.

Improvements

  • BAWS-48 - Add stable AWS artifact retrieval via missingcloud.
    • Many thanks to Mitch Garnaat for starting this indeed missing cloud data collection!
  • BAWS-66 - Add event reporting during stack transition.
  • BAWS-77 - Adjust configuration screen radio button layout.

1.3.0

Release 1.3.0 adds support for variable substitution for template parameters (and some other fields) as well as variable definition from stack outputs for reuse in subsequent Tasks.

Improvements

  • BAWS-50 - Add variable substitution in template parameters.
  • BAWS-59 - Add variable definition from stack outputs.

Bugs

  • BAWS-58 - Fix templates requiring IAM capability not being usable.

1.2.0

Release 1.2.0 adds support for template specification via URL as well as template parameter input and stack output logging.

Improvements

  • BAWS-44 - Add template parameter handling.
  • BAWS-45 - Add stack outputs handling.
  • BAWS-46 - Add URL based template specification.

1.1.0

Release 1.1.0 adds support for advanced options on stack creation and fixes a major bug with tasks not completing when stack creation is successfully rolled back by CloudFormation; see the changelog below for details.

Improvements

  • BAWS-19 - Add disable rollback option.
  • BAWS-20 - Add creation timeout option.
  • BAWS-21 - Add SNS notification option.

Bugs

  • BAWS-31 - Fix not all completed stack transitions triggering task completion and/or proper task result.
  • BAWS-37 - Fix validation errors not being displayed.

1.0.0

Release 1.0.0 offers support to create and delete parameter-less CloudFormation stacks across all AWS regions.

Stories

  • BAWS-9 - As a user, I need a task to create a CloudFormation stack so that I can provision a collection of related AWS resources.
  • BAWS-10 - As a user, I need a task to delete a CloudFormation stack so that I can terminate a collection of related AWS resources.
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.