Raspberri Pi header

This document describes how to initially prepare a low-write image up a Raspberry Pi 2.
Though any embedded Computer would be adequate, this guide concentrates on a specific system, using Raspbian Jessie. Any Debian OS could probably work with the same instructions.

1 Installing Debian Wheezy OS

Download the latest Raspbian Jessie lite image.

Write the .img file to your SD card: e.g. on Windows, by inserting the SD memory card into your computer and flashing it by using the Win32 Disk Imager.

With the prepared SD card now:

Power off your Raspberry pi.
Put the SD card in the micro-SD card slot.
Connect the LAN (first) and power cable, for the pi to automatically power up and connect to the network. This will happen by default over DHCP protocoll, so the IP address needs to be detected(e.g. by doing an IP scan or view the DHCP servers assignment list)

Now, one can connect to the raspberry via SSH, for example using PuTTY.
The username is by default: pi
the users password: raspberry

2 Optimize SD card and r/o file system

Due to the operating system of the Raspberry Pi running on a SD card, some steps should be done to protect the card from unproper shutdown and also against premature end of life.
Simplest, a 16GB+ SD card should be used, as a lot of available writable space reduces the chance of writing several times on the same place. Further, this can be achieved by disabeling some very write intensive processes, migrating others into temp directories located in RAM, and restricting write access to the whole file system, except for variable data.

2.1 Prepare the SD card

First, partition the SD card for a read-only root partition and a read/write /var partition, in which the workspace of future programs will be located.

sudo fdisk /dev/mmcblk0

If entering 'p', you should be shown a print out about the cards partitions like e.g. for a 8GB card

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1        8192  131071  122880   60M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      131072 2658303 2527232  1.2G 83 Linux

Next, enter 'd' to delete a partition (And then enter '2' for partition 2 (/dev/mmcblk0p2) if not deleted automatically because its the only partition to chose) Entering 'p' again will show you the alterations
Now create a new partition by entering 'n' then 'p' for primary and then '2'. Beginning with the next sector of the first partition, as we saw on the partition we deleted, for this example we have to enter 131072. The last sector can be selected by specifying its range: e.g. +3G. Entering 'p' again will show you your newly created partition like this
```
        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1        8192  131071  122880  60M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      131072 6422527 6291456   3G 83 Linux
```

The second partition can again be created by entering 'n' then 'p' and then '3'. Selecting the next sector, here 6422528, as start and last sector by pressing Enter (select the default) will result in something like this

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1         8192   131071   122880   60M  c W95 FAT32 (LBA)
/dev/mmcblk0p2       131072  6422527  6291456    3G 83 Linux
/dev/mmcblk0p3      6422528 15523839  9101311  5.3G 83 Linux

If you are satisfied with your changes at this point you can enter 'w' to commit to your changes. If you want to cancel or restart just enter <CTRL>+Z
Reboot your System
```
sudo reboot
```

Simplified

p
d
n
p
2
131072
+3G
n
p
3
6422528
enter
p
w
reboot

2.2 Relocate /var

From here, the file system will be resized to fill the first partition we created, limited to 3GB in this case. This partition will be made read-only in the next steps, to reduce the cards lifetime and reliability

sudo resize2fs /dev/mmcblk0p2

The second partition will be formatted as ext3, to relocate the /var folder, containing all variable files to be expected to grow like a SQL database

sudo mkfs.ext3 /dev/mmcblk0p3

Next, make a place to temporarily store the variable files from rootfs

sudo mkdir /tmp/var

and mount the newly created partition to the appropriate points and move the data. If you have a new installation, this should be relatively painless, but it might take longer if you have a lot of files in these directories

sudo mount /dev/mmcblk0p3 /tmp/var
sudo mv /var/* /tmp/var/

To complete the relocation, a line has to be added to /etc/fstab, to reflect the changes made, and point to the new location of the file system. e.g. by using the nano editor:

sudo nano /etc/fstab

/dev/mmcblk0p3  /var            ext3    defaults,noatime        0       1

The relocation can be completed by mounting the new lines in fstab

sudo umount /tmp/var
sudo mount -a

2.3 Optimisation

To limit the amount of write operations on the SD card, especially in some fast growing dirs, the most relevant of them can be installed as temporary filesystems. This can be achieved by again altering /etc/fstab: the file should look something like this

proc            /proc           proc    defaults                                0       0
/dev/mmcblk0p1  /boot           vfat    defaults                                0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime                        0       1
/dev/mmcblk0p3  /var            ext3    defaults,noatime                        0       1
tmpfs           /var/log        tmpfs   defaults,noatime,nosuid,mode=0755,size=100M 0   0
tmpfs           /var/tmp        tmpfs   defaults,noatime,nosuid,size=50M        0       0
tmpfs           /tmp            tmpfs   defaults,noatime,nosuid,size=150M       0       0

And after rebooting/remounting, the file system can be checked df -h to look something like this

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.0G  648M  2.3G  22% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M  6.2M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
tmpfs           150M     0  150M   0% /tmp
/dev/mmcblk0p1   60M   20M   41M  34% /boot
/dev/mmcblk0p3  4.2G   79M  3.8G   2% /var
tmpfs           100M     0  100M   0% /var/log
tmpfs            50M     0   50M   0% /var/tmp

Now, swap will be disabled and prevented, to start itself again at boot

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo apt-get remove --purge dphys-swapfile
sudo apt-get autoremove --purge

Swapping is the process of using part of the SD card as volatile memory. This will increase the amount of RAM available, but it will result in a high number of read/writes. It is unlikely to increase performance significantly.

Enable automatic file system fixes at boot by editing /etc/default/rcS

FSCKFIX=yes

2.3.1 Read-only file system

To increase stability and the lifetime of the SD card, the read/write access can be restricted on the system.
This can be done manually, by remounting the root partition

sudo mount -n -o remount,ro /

and reversed again, if something has to be modified later on

sudo mount -n -o remount,rw /

to automise this at startup, /etc/fstab has to be modified again. Here, the root file system "/" and "/boot" will need the additional ,ro

/dev/mmcblk0p1  /boot           vfat    defaults,ro                             0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime,ro                     0       1

Now that we made filesystem read-only, it can be complex to install further programs. To prevent that, we allow apt to automatically remount the root partition as read/write by adding the following lines to /etc/apt/apt.conf.d/70debconf. To avoid the automatic read-only remounting after completion, a variable may be set export DISABLE_REMOUNT=true

DPkg {
    // Auto re-mounting of a read-only system /
    Pre-Invoke {
        "mount -o remount,rw /";
    };
    Post-Invoke {
        "if ! [ "$DISABLE_REMOUNT" = true ]; then mount -o remount,ro /; fi";
    };
};

Additionally, the hourly cron script that saves clock every hour /etc/cron.hourly/fake-hwclock may be edited

#!/bin/sh
#
# Simple cron script - save the current clock periodically in case of
# a power failure or other crash

if (command -v fake-hwclock >/dev/null 2>&1) ; then
  mount -o remount,rw /
  fake-hwclock save
  if ! [ "$DISABLE_REMOUNT" = true ]; then mount -o remount,ro /; fi
fi

as well as /etc/bash.bash_logout, to save session details at logout

sudo mount -o remount,rw /
history -a
sudo fake-hwclock save
sudo mount -o remount,ro /
sudo mount -o remount,ro /boot

Finally, some lines could be added to /etc/bash.bashrc, to avoid the complex syntax of manually remounting the system.
The simple commands rw and ro can be added and graphically indicated to the command line by adding

# set variable identifying the filesystem you work in (used in the prompt below)
set_bash_prompt(){
    fs_mode=$(mount | sed -n -e "s/^\/dev\/.* on \/ .*(\(r[w|o]\).*/\1/p")
    PS1='\[\033[01;32m\]\u@\h \[\033[01;36m\]${fs_mode:+($fs_mode)} \[\033[01;34m\]\w \$\[\033[00m\] '
}

alias ro='sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot'
alias rw='sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot'

# setup fancy prompt"
PROMPT_COMMAND=set_bash_prompt

2.3.2 tmpfs log directory

Now that the log directory was made temporary, to limit SD card write access and keep log files in the RAM, all logfiles and subdirs will be lost at poweroff. Consequential, logfiles which are needed for debugging purposes should be handled with care and use e.g. a logback rotation.

Additionally, some services need specific log dirs at boot to be prepared to function, as apache2 or redis do. This will be handled by systemds' service tmpfiles, which can be configured in /usr/lib/tmpfiles.d/

Caution: Some configurations are necessary, as programs like mysql or redis-server depend on a correctly resolved logging dir to work. Some services install their own files to tmpfiles.d, to which a line can be added, while some files need to be created:

sudo nano /usr/lib/tmpfiles.d/apache2.conf

d /var/log/apache2 0755 - - -
sudo nano /usr/lib/tmpfiles.d/lighttpd.conf

d /var/log/lighttpd 0755 www-data root -
sudo nano /usr/lib/tmpfiles.d/mysql.conf

d /var/log/mysql 0755 mysql root -
sudo nano /usr/lib/tmpfiles.d/redis-server.conf

d /var/log/redis 2775 redis redis -
sudo nano /usr/lib/tmpfiles.d/emoncms.conf

f /var/log/emoncms.log - www-data root -
sudo nano /usr/lib/tmpfiles.d/supervisor.conf

d /var/log/supervisor 0755 - - -
sudo nano /usr/lib/tmpfiles.d/cossmic.conf

d /var/log/cossmic 0755 pi root -

Now to reduce the used RAM, the amount of logging can be reduced by

ensuring logrotate to work properly with the right rights by adding a line in /etc/logrotate.conf
```
su root adm
```
specify a dir for cycled files not in RAM: sudo mkdir /var/log.archive
```
olddir /var/log.archive
```

and copying the config file to /etc/logrotate.d/

sudo cp /etc/logrotate.conf /etc/logrotate.d/

modifying /etc/rsyslog.conf, to move those every-5-minutes cron jobs out of /var/log/syslog. To do that, modify the second line in the Rules paragraph to
```
*.*;auth,authpriv.none,cron.none -/var/log/syslog
```
modifying /etc/pam.d/common-session-noninteractive, to avoid entries like pam_unix(cron:session): session closed in /var/log/auth.log. To do that, add a line to the end of the file, right before the pam_unix.so call
```
session [success=1 default=ignore] pam_succeed_if.so service in cron quiet use_uid
session required        pam_unix.so
# end of pam-auth-update config
```

3 Dependencies

First, you may start by upgrading the system repositories and installing the newest versions of core dependency packages like mercurial, build essentials, containing tools (like the gcc compiler, make tool, etc) for compiling/building software from source the ntp Network Time Protocol synchronization tool and the ftp File Transfer Protocol.

sudo apt-get update && sudo apt-get dist-upgrade -y
sudo apt-get install mercurial build-essential ntp ftp pwgen

3.1 Gradle

The open source project gradle is a advanced build automation tool, based on groovy instead of the more traditional XML form of declaring the project configurations. Gradle was designed for multi-project builds which can grow to be quite large, and was used for the java driver system of this project.

To install gradle, first download and unzip the latest version to the newly created dir. As an example, the download link to version 2.7 can be copied from below

cd /opt
sudo wget https://services.gradle.org/distributions/gradle-2.12-bin.zip
sudo unzip gradle-2.12-bin.zip
sudo rm gradle-2.12-bin.zip

At last, the ~/.profile file has to be altered, to export the gradle bin directory to the PATH variable everytime pi logs in

export GRADLE_HOME=/opt/gradle-2.12
export PATH=$PATH:$GRADLE_HOME/bin

Optionally, a gradle.properties file can be created, to speed gradle up. For this, the properties file needs to be created in the user home directory

mkdir ~/.gradle
nano ~/.gradle/gradle.properties

and add this line to the file

org.gradle.daemon=true

3.2 JSVC

The Apache project commons-daemon allows the user to wrap a multi-threaded java project, to handle it as a UNIX daemon and control it with an init script/service.

Install the UNIX platform tool jsvc to enable this projects driver services to work properly

sudo apt-get install oracle-java8-jdk jsvc

3.3 RXTX

RXTX is a Java native library providing serial and parallel communication for the Java Development Toolkit (JDK). It is a necessary dependency for many communication devices, using e.g. RS485.

To install, download the binaries via debian repository:

sudo apt-get install librxtx-java

Create two soft links, to configure rxtx for future use.

cd /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/ext
sudo ln -s ../../../../../jni/librxtxSerial-2.2pre1.so librxtxSerial.so
sudo ln -s ../../../../../../share/java/RXTXcomm-2.2pre2.jar RXTXcomm.jar

By default, the serial port is configured as a console port for interacting with the Linux OS shell. To use the serial port in a software program, it must be disabled for the OS to use.
To do this, the kernel parameters file /boot/cmdline.txt should be edited to remove all parameters involving the serial port (ttyAMA0), e.g. from

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

console=ttyAMA0,115200 has to be deleted:

dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Last, getty needs to be modified, to stop being set on the serial port

sudo systemctl disable serial-getty@ttyAMA0.service

After a reboot, the serial port will be prepared to use.

3.4 Python libraries

Any serious data analytics effort with Python generally includes to some extent some additional libraries like the pandas.
To download these, first install the Python PIP installer and update it by

sudo apt-get install python-pip python-dev libffi-dev
sudo pip install --upgrade pip

A recommended first step is to update https requests and all installed packages. As pip does not have this functionality yet, this can be achieved with:

sudo pip install requests[security] --upgrade
sudo pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 sudo -H pip install -U

This, and the installation of large libraries like numpy or pandas, may take a while

sudo pip install numexpr numpy pandas --upgrade

Notice: Some packages may need more than 150MB /tmp space to upgrade. To install those, the dir needs to be remounted with more space available.

3.5 Uncomplicated Firewall

ufw is a great lightweight firewall program that can be used to control server access rules. The default set below are fairly standard for a web server but are quite permissive. You may want to only allow connection on a certain ip if you will always be accessing your pi from a fixed ip (Documentation https://help.ubuntu.com/community/UFW).

sudo apt-get install ufw
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 22/tcp
sudo ufw allow 21/tcp
sudo ufw allow 9001/tcp
sudo ufw enable

3.6 PV prediction

The python application pvprediction was initially developed to use weather forecast data from Deutscher Wetterdienst (DWD), in order to predict the daily photovoltaic yield of defined systems.

To calculate important irradiance data, the open-source python project pvlib was used. The toolbox was originally developed in MATLAB at Sandia National Laboratories and implements many of the models and methods developed at the Labs. It can be installed with pip, as well as PyEphem, a library to performing high-precision astronomy computations.

sudo pip install PyEphem pvlib

3.7 Distributed Scheduler

The taskscheduler is a C++ application, using an auctioning-algorithm, to determin a optimal schedule. This sophisticated controlling algorithm depends on a list of libraries to be installed:

sudo apt-get install libboost-dev libboost-system-dev libboost-signals-dev libarmadillo-dev libgsl0-dev libnlopt-dev libswiften2 libswiften-dev python-matplotlib -y

3.8 Multy Agent System

CoSSMics' Multy Agend System uses SPADE, an agent-framework written in python, to build its P2P overlay as an XMPP platform.

All SPADE agent servers and agents are running using the web interface supervisor, that needs to be installed first

sudo apt-get install supervisor

Additionally, make sure pythons http libraries are installed

sudo pip install httplib2 --upgrade

3.9 EmonCMS

The dependencies of emoncms (Energy monitoring Content Management System), an open-source web-app for processing, logging and visualising energy, temperature and other environmental data and part of the OpenEnergyMonitor project, need to be prepared.

To do this, a separate dependencies section of the provided emoncms installation guide can be followed.

Wiki

developer-guide / Raspberry Pi image preparations