Windows Azure Setup
In azure_scripts folder in sparseq repository you can find a couple shell scripts for managing of SparkSeq cluster in Windows Azure
In order to you the managing scritps you need to:
1. Have an Windows Azure account up and running.
2. Install azure-cli - a command-line, cross-platform Windows Azure managing tool.
3. Download and import yout publish settings - a quick guide can found here.
4. Generate Windows Azure Compatible Keys - more information here.
5. Windows Azure Virtual Network and Affinity Group configured( a quick guide can found here).
6. Install parallel-ssh package.
1. Edit azureConfig.cfg - a configuration file for create SparkSeq cluster in Windows azure. The most important params are:
- userName - an username used for password-less ssh connection
- privKey - a path to your private key generated in point 4.(prerequisites section)
- intNumber - a number of all instances to be created (i.e. Haadoop name nodes, Hadoop data nodes/Spark WorkerNodes,etc)
- vmSize - a size of all the instances in Windows Azure
- instPreffix - a preffix that together with 3-digits constitutes the instances name, e.g. sparkseq001,sparkseq002, etc. It has to be unique in Windows Azure.
- location - whete to store your instances' drives (don't used in this version of the script-location is configure while creating Affinity Group)
- cer - your certificate created in point 4 (prerequisites section)
- vpnName - your virtual network created in point 5 (has to be unique as well)
- agName - your Affinity Group created in point 5 (prerequisites section, unique as well)
userName=mesos instList="instances.txt" privKey="myPrivateKey.key" instNumber=8 imageName=b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_3-LTS-amd64-server-20131205-en-us-30GB vmSize=a6 instPreffix=sparkseq location="West Europe" endPoints="443:8888,80:80,4040:4040,7077:7077,8080:8080,50010:50010,50070:50070,50075:50075" instanceList="instances.txt" cert="myCert.pem" vpnName=sparkseqvpn agName=sparkseqag
2. Run deployment script:
To shutdown your cluster (with preserving states of all instances without deleting anything, just to release resources) you can run the script:
and to start it again:
To destroy it completely (deleting all instances and their hard drives permanently):