HTTPS SSH
##############################################################################################################################

22 March 2016 Update:

This project is now archived. It is no longer being maintained by the ABC.

For more information about the ABC and the Digital Network, refer to our blog at http://developers.digital.abc.net.au/

##############################################################################################################################



Introduction
--------------------------

ABCRedback reads enriched Snowplow events off a kinesis stream and sends them to an Amazon Redshift instance via S3 using the AWS Kinesis Connector Library (KCL)

The primary motivations are:

	- to simplify the sample KCL code for this specific use case (i.e - endpoints, buckets & cluster already exist, not using dynamo DB etc)
	- to work around the default in-memory model sample that risks out of memory conditions by streaming to S3 from local temporary files rather than RAM
	- to support switching Snowplow event models as the current Kinesis event model does not match the current Redshift event model. 0.9.0, 0.9.6 and 0.13.1 event models are supported
	- to support configurable options for the COPY into Redshift

Configuration

This is a simplified version of the KCL sample with additional properties for Snowplow and Redshift. 
A sample properties file is included in src/main/resources.

Usage
--------------------------

java -Daws.accessKeyId=<access-key> -Daws.secretKey=<secret-key> -jar target/red-back-1.0.0-jar-with-dependencies.jar -config redback.properties

Change List
--------------------------

	- 22/08/2014 Initial version (Sam.Mason@abc.net.au)
	- 21/10/2014 Updated source to include Apache 2.0 license
	- 20/11/2014 v1.1.0 Added support for optional event massage step during event creation. The default configuration is to use the anonymising data masseur to remove IP addresses and hash the user Ids.
	- 25/11/2014 v1.1.1 Added pipe encoding for string fields
	- 28/11/2014 v1.1.2 added back IP addresses as hashed strings
	- 01/12/2014 v1.1.3 changed pipe encoding to colon character
	- 02/12/2014 v1.1.4 added fixes for 0.9.12 Snowplow release Kinesis processing
	- 10/12/2014 v1.1.6 LRS deployment version
	- 07/01/2015 v1.1.7 Added session token to COPY command as needed by using the built-in AIM role assumuption provided by the SDK
	- 12/03/2015 v2.0.0

		- mobile events will use OpenIDFA field if supplied in context as the network_userid for cross-app correlation
		- added support for dropping events by app_id by supplying a comma separated list in the event.filters.appids property
		- added support for detailed logging for a set of app_ids by supplying a comma separated list in the event.logging.appids property
		- a set of operators is optionally applied to events by specifying implementation classes in the event.operators property. For example:

		event.operators = au.net.abc.innovation.kinesis.snowplow.operators.IPAddressHashOp,au.net.abc.innovation.kinesis.snowplow.operators.UserIdHashOp,au.net.abc.innovation.kinesis.snowplow.operators.MobileContextExtractionOp

	- 11/05/2015 v2.0.1 - patched operator pipeline to handle invalid context processing
	- 22/05/2015 v2.0.2 - added support for new Kinesis schema (0.13.1 - compatible with v65 pipeline)
	- 25/05/2015 v2.0.3 - patched Redshift import for new fields
	- 09/06/2015 v2.0.4 - patched date parsing synchronisation