This is a raw dump of my AWS RI Market project.
I'm not a Dev guy, but an Ops guy.. with bad coding skills.
Expect dirty code, not optimized and lots of full path stuff...
How do I get set up?
*Elasticsearch and Kibana
*Working AWS CLI client - http://docs.aws.amazon.com/cli/latest/userguide/installing.html
*Installed JQ - https://stedolan.github.io/jq/
*Installed EMBulk - http://www.embulk.org/docs/
*Optional MYSQL installed if you want to load the CSV in to that.
*Some json, linux shell and scripting knowledge to make my bugie code work.
Install and config:
Install Kibana with NGinx.
Add the default Elasticsearch template update from the 'elasticsearch-embulk-template' file to make sure you get raw fields also.
If you like to run Kibana exposed to the big bad internet, look at this guide: https://gist.github.com/r0mdau/db31403c0338b057bc7a
The 'instance-market.sh' is the main script. Execute it manually or in CRON job. Example CRON file included.
Install JQ and EMBULK. The 'config.yml' file is the config file for the EMBULK run. Schedule it using CRON.
*The dump is every 24h.
*Example of the Kibana dashboard can be seen here: http://ri-market.janwiersma.com .
Example of the dashboard setup can also be found in 'export.json'.
*Use/edit the line from the 'delete-records.sh' file to delete specific time dumps.
*Example of the AWS CLI dump raw file is in 'instancedumpall.json'. It can be used to play with JQ at https://jqplay.org/
Questions?: jan AT wiersma.com