Overview

A key aim of the ROSELINE project is to provide richer time information exchange across the stack. More specifically, applications running within user-space must have (a) low latency access to local time, (b) the ability to time-stamp external events accurately, (c) the ability to obtain synchronization parameters that allow a local count to be projected into global time, and (d) control the quality of time, ie. the oscillator, synchronization frequency, etc.

To understand how this information flows through the stack, I'll refer to the block diagram below.

At the heart of this block diagram is a kernel module called CORE it is the responsibility of this module to initlialize timers, input capture and clock sources and cache any information relating to synchronization and time capture. In a nutshell it acts as a proxy for information exchange between the hardware timers and application-layer code.

The time synchronization algorithm runs at the application layer and interacts with the CORE kernel module using ioctl calls. The time synchronization algorithm can be started or stopped with the command sudo service timesync stop|start. For those who are interested, this script is defined in /etc/init.d/timesync. Using switches (try timesync --help) can modify various time sync parameters, such as the algorithm, logical clock and time stamping mechanism.

Reading synchronization parameters

The way applications interact with this module is through ioctl calls. Right now there are only five unique ioctl calls that can be made, each of which are described below:

1. ROSELINE_GET_CAPTURE : Get the last input capture info for some timer into a roseline_capt struct
2. ROSELINE_GET_SYNC : Get the synchronization information for some timer into a roseline_sync struct
3. ROSELINE_SET_SYNC : Set the synchronization information for some timer
4. ROSELINE_GET_PARS : Get the synchronization parameters for some timer into a roseline_pars struct
5. ROSELINE_SET_PARS : Set the synchronization parameters for some timer

The general usage pattern is as follows.

#include <sys/ioctl.h>
#include <fcntl.h> 
#include <unistd.h>
#include <stdio.h>

#include "../module/roseline_ioctl.h"

int main ( int argc, char **argv )
{
  const char *file_name = "/dev/roseline";
  int fd = open(file_name, O_RDWR);
  if (fd == -1)
  {
    printf("Error: could not open /dev/roseline\n");
    return 1;
  }

  // Allocate memory to hold synchronization information
  roseline_sync syn; 

  // Get synchronization parameter for TIMER4
  syn.id = TIMER4;
  if (ioctl(fd, ROSELINE_GET_SYNC, &syn)==-1)
    printf("Error: could not make ioctl() call\n");
  else
    printf("TIMER4: Number of times updated =  %u\n",syn.seq);

  // Clean up
  close(fd);
  return 0;
}

This pattern is used by the time synchronization application running in user-space to get and set synchronization information for each of the four timers used in the stack.

Accessing global time

We already know how to read synchronization parameters. However, these parameters aren't really useful until we are able to access a local counter value. In this section we will show how to memory-map the hardware count directly to user space, and project our local count into a global time frame.

The local hardware count for TIMER4 can be read in the following way using memory mapping. The latency on this call is approximately 1 to 2 clock cycles, which is around 100ns.

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>

#include "../module/roseline_ioctl.h"

int main ( int argc, char **argv )
{
  int fd = open("/dev/mem", O_RDWR | O_SYNC);
  volatile uint32_t *treg = (uint32_t *)mmap(NULL, 0x1000, 
      PROT_READ|PROT_WRITE, MAP_SHARED, fd, TIMER4_REG); 
  printf("T4: %lu\n",(long unsigned int)treg[0x3c/4]);
  close(fd);
  return 0;
}

The OMAP timers are represented by unsigned 32 bits integers and therefore overflow after 4,294,967,295 ticks (about 179 seconds at 24MHz). When an overflow occurs, the overflow and seq fields of the roseline_sync structure are updated immediately. For this reason, the sync ioctl() should be polled periodically for overflow updates.

Here is how the overflow and memory mapped count should be merged into a unsigned 64 bit time local and then projected into a fractional double time global.

uint64_t upper = (uint64_t) sync.overflow;
uint64_t lower = (uint64_t) treg[0x3c/4];
uint64_t local = (upper << 32) | lower;
double global = sync.est_skew * ((double)local - sync.est_local) + sync.est_offset

This unsigned 64 bit representation of time will take 24 millenia to overflow, which should be sufficient for any application. The maximum oscillator frequency that can be used with the BeagleBone Black is 25MHz.

Time stamping external events

There are many cases in which is proves useful to precisely time stamp an external event. For example, the wireless time synchronization algorithm implemented in the QoT stack relies on time marking a packet exchange between two radios. In this case the radio generates a highly determinstic 3.3v interrupt on an external pin, which we have connected to the input capture pin of TIMER4 (BeagleBone Black P8.7).

Our default device tree configures input capture TIMER4 to TIMER7 by default:

    timer_pins: pinmux_timer_pins {
         pinctrl-single,pins = <
            0x90  0x22  /* P8.7   MODE2 TIMER4 */
            0x1b4 0x2A  /* P9.41A MODE2 TIMER4 TCLKIN */
            0x1a8 0x0F  /* P9.41B MODE7 TIMER4 INPUT (high-Z, tied to P9.41A) */
            0x98  0x22  /* P8.10  MODE2 TIMER5 */
            0x9C  0x22  /* P8.9   MODE2 TIMER6 */
            0x94  0x22  /* P8.8   MODE2 TIMER7 */
            >;
    };

Later in the device tree we specify arguments to the ROSELINE kernel module that describe how each input capture should be set up. The excerpt below shows that TIMER4 is set up to be clocked by the 24MHz on-board oscillator and captures falling edge events. Similarly, TIMER5 is set up to be clocked by the 24MHz on-board oscillator and captures rising edge events.

/ {
    roseline {
        compatible = "roseline";
        status = "okay";
        timer_4 = <&timer4 1 1>;    /* FORMAT   : <TIMER SOURCE EDGE>, where ...                */
        timer_5 = <&timer5 1 0>;    /* - TIMER  : &timer4 | &timer5 | &timer6 | &timer7         */
        timer_6 = <&timer6 1 3>;    /* - SOURCE : 0|1|2 where 0=TCLKIN, 1=24MHz,   2=32kHz      */
        timer_7 = <&timer7 1 3>;    /* - EDGE   : 0|1|2 where 0:RISING, 1=FALLING, 2=BOTH, 3=NONE   */
        pinctrl-names = "default";
        pinctrl-0 = <&timer_pins>;
    };
};

The interrupt handler for the time capture is actually implemented in the kernel module, and therefore happens some number of microseconds after the event occurs. However, this latency does not have any effect as the counter value is shadow copied at the time of interrupt by the micro-controller and read in the interrupt handler. The resulting time mark therefore has very low error.

The ROSELINE kernel module then caches the capture information for use later. A user-space application can query the time at which the input capture was taken and, in a similar way to the memory-mapped local count, project this local time to a global time frame. Again, ioctl() is used:

#include <sys/ioctl.h>
#include <fcntl.h> 
#include <unistd.h>
#include <stdio.h>

#include "../module/roseline_ioctl.h"

int main ( int argc, char **argv )
{
  const char *file_name = "/dev/roseline";
  int fd = open(file_name, O_RDWR);
  if (fd == -1)
  {
    printf("Error: could not open /dev/roseline\n");
    return 1;
  }

  // Allocate memory to hold capture information
  roseline_capt cap; 

  // Get synchronization parameter for TIMER5
  cap.id = TIMER5;
  if (ioctl(fd, ROSELINE_GET_CAPT, &cap)==-1)
    printf("Error: could not make ioctl() call\n");
  else
    printf("TIMER4: %lu%lu\n",cap.overflow,cap.count_at_capture);

  // Clean up
  close(fd);
  return 0;
}

Controlling quality of time

Work in progress.