1. Unity Technologies Avatar Unity Technologies
  2. Unity
  3. ML-ImageSynthesis

Wiki

Clone wiki

ML-ImageSynthesis / AWS

View History

How to run on Amazon Cloud

It is very tempting to harness scalable power of Amazon AWS Cloud when rendering lots and lots of images. Below are instructions on how to setup Ubuntu 14.04 based Amazon instances ready for Unity (or any OpenGL application for that matter) running in 'headless' (no monitor attached) mode, rendering images and saving them to disk.

GPU Instances

Following is tested on g2.2xlarge and g2.8xlarge Amazon instances.

Using the AMI image

We have built an AMI based on Ubuntu 14.04 and containing the latest (at the moment of writing) NVIDIA drivers 367.57 that enable hardware acceleration on Amazon GPU instances along with X server that is readily configured with virtual display.

X server launches via init.d upon bootup and is assigned to display :0. DISPLAY environment variable is set to :0 as well.

AMI id is ami-921c40f2. It is designed to work with g2.2xlarge and g2.8xlarge instances. Link: https://console.aws.amazon.com/ec2/v2/home?region=us-west-1#LaunchInstanceWizard:ami=ami-921c40f2

Building the AMI image yourself

Below is the script which was used to create the image. Most of the following complexity is due to NVIDIA driver installation specifics on Ubuntu machine without monitor. This script is almost vanilla copy from: https://askubuntu.com/questions/429596

#!bash
    # Prerequisites
    sudo apt-get update
    sudo apt-get install -y gcc make linux-generic

    # install X server and OpenGL tools
    sudo apt-get install -y xserver-xorg mesa-utils

    # disable Nouveau
    sudo echo 'blacklist nouveau'  | sudo tee -a /etc/modprobe.d/blacklist.conf
    sudo echo 'options nouveau modeset=0'  | sudo tee -a /etc/modprobe.d/blacklist.conf
    echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
    sudo update-initramfs -u

    # reboot to get kernel update and nouveau disabling activated
    sudo reboot now

    # install NVIDIA drivers
    wget http://us.download.nvidia.com/XFree86/Linux-x86_64/367.57/NVIDIA-Linux-x86_64-367.57.run
    sudo /bin/bash ./NVIDIA-Linux-x86_64-367.57.run --accept-license --no-questions --ui=none
    sudo reboot now

    # configure virtual display device for X server - it should writes into xorg.conf file
    sudo nvidia-xconfig -a --use-display-device=None --virtual=1280x1024

    # add missing BusID
    sudo sed -i 's/    BoardName      "GRID K520"/    BoardName      "GRID K520"\n    BusID          "0:3:0"/g' /etc/X11/xorg.conf

    # run X server
    sudo /usr/bin/X :0 &
    export DISPLAY=:0

CPU Instances

CPU instances are considerably cheaper, so you might want to use more instances running significantly slower software rasterisers. Be advised that it might be less stable solution since software versions seem to cover much, but not all functionality of GPU drivers. Software drivers support only OpenGL Core 3.3.

Using the AMI image

We have built an AMI based on Ubuntu 14.04 and containing the latest (at the moment of writing) Mesa 17.1.0 drivers with two high-performance software rasterisers llvmpipe and Intel's OpenSWR. As well as X server that is readily configured with virtual display.

See GPU instances above for details on X server configuration.

AMI id is ami-01287461. It is designed to work with g2.2xlarge and g2.8xlarge instances. Link: https://console.aws.amazon.com/ec2/v2/home?region=us-west-1#LaunchInstanceWizard:ami=ami-01287461

Building the AMI image yourself

Below is the script which was used to create the image. 

#!bash

    # prerequisites
    sudo apt-get update
    sudo apt-get install -y git build-essential mesa-utils python-pip 
    sudo pip install mako
    sudo apt-get build-dep mesa

    # just in case you had them installed before
    sudo apt-get purge llvm-3.4 clang-3.4
    sudo apt-get auto remove

    # install LLVM 3.9 (default one is 3.4, not good enough for OpenRWS)
    sudo echo 'deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main’ | sudo tee -a /etc/apt/sources.list
    sudo echo 'deb-src http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main’ | sudo tee -a /etc/apt/sources.list
    wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
    sudo apt-get update 
    sudo apt-get install -y clang-3.9 lldb-3.9

    ln -s /usr/bin/llvm-config-3.9 /usr/bin/llvm-config

    # now clone and compile mesa
    git clone git://anongit.freedesktop.org/git/mesa/mesa mesa.git

    cd mesa.git

    ./autogen.sh \
      --disable-va --disable-gbm --disable-xvmc --disable-vdpau \
      --disable-dri --with-dri-drivers= \
      --disable-egl --with-egl-platforms= --disable-gles1 --disable-gles2 \
      --enable-texture-float \
      --enable-glx --enable-xlib-glx \
      --enable-gallium-llvm=yes --enable-gallium-osmesa \
      --with-gallium-drivers=swrast,swr \
      --prefix=/home/mesa

    make
    sudo make install

    # most likely not necessary to reboot
    sudo reboot now

    # install and setup X server
    sudo apt-get install -y linux-generic xserver-xorg

    export LIBGL_DRIVERS_PATH=/home/mesa/lib
    export LD_LIBRARY_PATH=/home/mesa/lib

    # switch to OpenSWR rasterizer backend
    export GALLIUM_DRIVER=swr
    # however default llvmpipe seems to be actually faster in the most cases for me. Try what works better for you!
    # export GALLIUM_DRIVER=llvmpipe

    # launch X server
    sudo /usr/bin/X :0 &
    export DISPLAY=:0

Test that it actually works

Good practice is to install mesa-utils package and then run glxinfo and glxgears to check that configuration is working. You will not see any images of course, but you should NOT have any errors in the console either.

#!bash

    sudo apt-get install -y mesa-utils
    DISPLAY=:0 glxinfo
    DISPLAY=:0 glxgears

Next you can try running Unity standalone with extremely simple ImageSynthesis setup. After running you should get bunch of PNG images in your current directory.

#!bash

    sudo apt-get install -y unzip
    wget https://www.dropbox.com/s/abycjktjzmkrmb4/auto.zip .
    unzip auto.zip -d auto
    sudo chmod +x auto/auto.x86_64
    DISPLAY=:0 auto/auto.x86_64

Updated