1. ROSELINE Avatar ROSELINE
  2. QoT Stack
  3. qot-stack

Wiki

Clone wiki

qot-stack / Home

View History

QoT Stack Setup

Overview

This project is intended for developers, and so it presumes a certain working knowledge of embedded Linux. The general idea is to have the BeagleBones either using standalone microSD card setups or the NFS based setup.

In any kind of setup, there are three key devices in the system:

  1. Controller (Ubuntu 15.04, x86_64-linux-gnu)
  2. Host (Ubuntu 15.04, x86_64-linux-gnu) - Where you do your development
  3. Slaves (Ubuntu 15.04, arm-linux-gnueabihf) - The actual BeagleBones

Since the synchronization algorithm is based on wired PTP, for the qot-stack to work effectively you will need a IEEE 1588v2 compliant network. The slaves have a PTP-compliant Ethernet adapter, and Linux supports hardware time stamping out of the box. Our version of PTP is derived from the linuxptp project.

Robert Nelson's bb-kernel project provides almost everything we need to build a suitable kernel for the BeagleBone Black. The idea will be to check this project out on the controller and build a kernel for the slaves.

When you compile the kernel the kernel build script should automatically download and use the arm-linux-gnueabihf compiler. And when you compile the applications you want to use the same compiler, which can easily be installed on Ubuntu through the gcc-arm-linux-gnueabihf and g++-arm-linux-gnueabihf packages. Make sure the gcc/g++ versions match the version deployed on the slaves (v4.9.2).

#Table Of Contents

Node Setup

SD Card based setup

Step 0: Controller preparation

First install the necessary system applications

$ sudo apt-get install build-essential git cmake cmake-curses-gui gawk flex \
    bison perl doxygen u-boot-tools
$ sudo apt-get install nfs-kernel-server tftpd-hpa isc-dhcp-server ufw

Step 1: Install Debian on SD Card

Get the Debian 7.8 image from the BeagleBoard official site. Install onto your SD Card using dd or whichever disk utility you prefer.

For example, if your SD card is /dev/sdb, the dd command would be

$ xz -d [imgfile].img.xz // To extract the image file
$ sudo dd if=[imgfile].img of=/dev/sdb bs=4K

You may check that the Beaglebone boots successfully with the SD Card at this point.

Step 2: Build the kernel

Here we build the kernel from source using a config file provided in the Downloads section. We assume the working directory is /export, which is the directory for NFS setup, but the directory doesn't really matter for the SD card setup.

Download and extract qot-bundle.tar.bz2. This should extract files needed for the NFS setup, but the qot.config is a kernel config file needed for building.

Checkout the bb-kernel code at the 4.1.12-bone-rt-r16 tag

$ git clone https://github.com/RobertCNelson/bb-kernel.git -b 4.1.12-bone-rt-r16

Next we want to build the kernel using the QoT configuration. Run build_kernel.sh, and when the Linux menu configuration appears, click the LOAD option at the bottom right and choose /export/qot.config as the path to the configuration file. After loading the config file, compile the kernel.

Step 3: Install the kernel

In the tools/ directory, there is a script install_kernel.sh to install the newly compiled kernel onto whatever block device you specify as MMC in system.sh.

Plug in your SD Card, use lsblk to get its device name, and specify it in system.sh. Then run tools/install_kernel.sh to install the kernel.

Step 4: SSH Access

The root password is randomized by default, so in order to SSH into the device you will need to add your RSA public key to /export/rootfs/root/.ssh/authorized_keys. Your public key is in ~/.ssh/id_rsa.pub.

$ cat  ~/.ssh/id_rsa.pub >> /export/rootfs/root/.ssh/authorized_keys

If you get a 'No such file or directory found' error, then you need to first create your key pair with:

$ ssh-keygen -t rsa

At this point you should be able to SSH into the BeagleBone. You can either login through usb or through ethernet. To configure the networking to assign a static IP or through dhcp, change the /etc/network/interfaces file (more detail at this link)

$ ssh root@ipaddress

NFS based setup

Refer to repository Overview for NFS based setup.

Building the QoT Stack

Now that the kernel is built/installed and the BeagleBone Black successfully booted, we want to build the QoT Stack and install it.

Step 1: Check out the QoT stack code

Start by checking out the qot-stack code

$ git clone https://bitbucket.org/rose-line/qot-stack.git
$ cd qot-stack

Intialize the third-party code (OpenSplice and the DTC compiler)

$ git submodule init
$ git submodule update

Step 2: Update the device tree compiler

Then, install an overlay for the device tree compiler. This compiles a new version of dtc which you can use to build overlays for the BeagleBone Black.

$ pushd thirdparty/bb.org-overlays
$ ./dtc-overlay.sh
$ popd

Note the version of dtc you have.

Step 3: Build OpenSplice

Switch to the OpenSplice directory and pull the third party repo for the C++ bindings

$ pushd thirdparty/opensplice
$ git checkout -b v64 OSPL_V6_4_OSS_RELEASE
$ git submodule init
$ git submodule update

Configure and build the OpenSplice DDS library. The configure script searches for third party dependencies. The third party libraries ACE and TAO are only required for Corba, and in my experience introduce compilation errors. So, I would advise that you do not install them. 

$ ./configure

Assuming that you chose the build type to be armv7l.linux-dev, then you will see that a new script envs-armv7l.linux-dev.sh was created in the root of the OpenSplice directory. You need to first source that script and then build. The build products will be put in the ./install directory. 

$ . envs-armv7l.linux-dev.sh
$ make
$ make install

Then, add C++11 support by inserting the -std=c++11 argument to the CPPFLAGS variable in ./install/HDE/armv7l.linux-dev/custom_lib/Makefile.Build_DCPS_ISO_Cpp_Lib file. You will now need to recompile the C++ interface:

$ pushd install/HDE/%build%/custom_lib
$ make -f Makefile.Build_DCPS_ISO_Cpp_Lib
$ popd
$ popd

You now have a working OpenSplice distribution with C++11 support. This basically provides a fully-distributed publish-subscribe messaging middleware with quality of service support. This mechanism will be used to advertise timelines across the network. The slaves have their own arm versions of Java, ROS and OpenSplice 6.4 in the /opt directory of the rootfs, and whenever you SSH into a slave the /etc/profile script initializes all three for you.

Note that whenever you run an OpenSplice-driven app you will need to set an environment variable OSPL_URI that configures the domain for IPC communication. This is described by an XML file, which is usually placed somewhere in your OpenSplice source tree. There are some default files. The slave rootfs is configured by default to find the XML configuration in /mnt/openxplice/ospl.xml, as the configuration needs to be different for each slave -- they have different IPs and thus different<NetworkInterfaceAddress> tag values. Instructions for installing this XML file are in Step 6

Step 4: Build the QoT Stack

Before building, make sure that you have followed the steps here for unit tests.

The entire project is cmake-driven, and so the following should suffice:

$ mkdir -p build % Do this in the top most project directory /qot-stack %
$ pushd build
$ ccmake ..

Make sure that CROSS_COMPILE is ON and that your INSTALL_PREFIX is /export/rootfs/usr/local.

$ make
$ popd

Step 5: Build Device Tree Object

In the top most project directory run

$ make

After installing the kernel modules you might need to run depmod on the nodes.

Step 6: Install Kernel Modules and Device Tree

The installation of the QoT Stack depends on whether you set up with SD cards or NFS.

With the SD card setup, edit the IP address of your node in the Makefile and and run:

$ make install_sd

Configuring the QoT stack

Firstly, SSH into a node of choice:

If you type capes on the command line you should see four slots as empty:

root@arm:~# capes
0: PF----  -1
1: PF----  -1
2: PF----  -1
3: PF----  -1

When you installed the kernel module a DTBO file was copied to /export/rootfs/lib/firmware. This is a device tree overlay file that tells the BeagleBone how to multiplex its I/O pins, and which kernel module to load after it has done so. To apply the overlay use the capes command.

$ capes BBB-AM335X

The output of the capes command should now look like this:

root@arm:~# capes
0: PF----  -1
1: PF----  -1
2: PF----  -1
3: PF----  -1
4: P-O-L-   0 Override Board Name,00A0,Override Manuf,BBB-AM335X

And lsmod should list two new kernel modules, qot and qot_am335x.

Normally modules are automatically loaded from the capes command. However, it is not always the case. You can manually load the modules using the commands:

root@arm:~# insmod qot
root@arm:~# insmod qot_am335x

If you get a path error, try

root@arm:~# cd /lib/modules/4.1.12-bone-rt-r16/kernel/drivers/misc
root@arm:~# insmod qot.ko root@arm:~# insmod qot_am335x.ko

If you still have issues, run dmesg and see the logs for issues.

Running PTP Synchronization on the QoT stack

QoT Stack does synchronization in two steps. It first aligns the local core clock to the network interface clock. We need to enable ptp pin capabilities in order for this to work. You can check pin capabilities of various ptp devices using,

$ testptp -d /dev/ptp0 -c % This is the ethernet controller driver exposed as ptp clock %
$ testptp -d /dev/ptp1 -c % qot_am335x driver exposes the core as a ptp clock %

In the output, see that pin functionalities like external timestamping and interrupt trigger are enabled.

For the onboard ptp device for the ethernet controller (/dev/ptp0), the pin capabilities are not enabled by default. We need to patch a file (cpts.c) in the kernel at (/export/bb-kernel/KERNEL/drivers/net/ethernet/ti/cpts.c) to enable the pins. (You can use the diff and patch utilities to do this easily) Patch the file and commit the changes in the KERNEL repository. The new file can be found here

In /export/bb-kernel/KERNEL: 

$ patch (...)  % patch the cpts.c file
$ git add drivers/net/ethernet/ti/cpts.c
$ git commit -m "patched cpts.c to add eth controller pin functionalities"

Then we need to create a git formatted patch, put it inside the the /bb-kernel/patches/ directory, and point patch.sh to the patch file. This is because the kernel building scripts (build_kernel.sh and rebuild.sh) use patch.sh to apply all required patches before building the kernel.

$ git format-patch HEAD~ % creates a patch file with the changes from the previous commit
$ cd .. % to go back to /export/bb-kernel/
$ cp KERNEL/0001-patched-cpts.c-to-add-eth-controller-pint-functionali.patch \
    patches/beaglebone/phy/

Now edit patch.sh and find where the script enters the /patches/beaglebone/phy/ directory. (Search for 'patches/beaglebone/phy' in the file) and add the following line:

${git} "${DIR}/patches/beaglebone/phy/0001-patched-cpts.c-to-add-eth-controller-pin-functionali.patch"

Now we are ready to rebuild the kernel.

In /export/bb-kernel: 

$ ./tools/rebuild.sh

After the kernel is rebuilt, restart the beaglebone nodes, and run the following in one terminal,

$ phc2phc

Keep this running for the entire synchronization process. Once the on-board Core clock and NIC are properly aligned (follow the logs of phc2phc to verify), we now need to synchronize clocks across different devices.

The qotdaemon application monitors the /dev directory for the creation and destruction of timelineX character devices. When a new device appears, the daemon opens up an ioctl channel to the qot kernel module query metadata, such as name / accuracy / resolution. If it turns out the character device was created by the qot module, a PTP synchronization service is started on a unique domain over eth0. Participating devices use OpenSplice DDS to communicate the timelines they are bound to, and a simple protocol elects the master and slaves. Right now, the node with the highest accuracy requirement is elected as master.

In order to create a timeline first, run helloworld application in a separate terminal,

$ helloworld

And then launch the synchronization daemon in another terminal:

$ qotdaemon -v

Repeat the synchronization step for other nodes as well, and then you will see multiple nodes -- bound to same timeline -- synchronize to each other.

Updated