Introduction

The firmware for the logger is implemented by assembly of a number of modules and support libraries, including those for NMEA2000 message reception and interpretation, WiFi access and HTTP server, SD card access, and some hardware-specific functionality (such as the Flash File Storage (FFS) file system on ESP32 modules).

The code could be compiled with the Arduino IDE, but is better manipulated with the VSCode/PlatformIO system (see below), and consists of a core driver file in main.cpp which interfaces with the Arduino run-time setup, and a series of C++ modules that provide the majority of the functionality of the system. Setting up the firmware in PlatformIO, and configuring the logger are dealt with here first, and documentation of the various modules follows.

Firmware Versions

The firmware is given a formal version number, both as a tag in the repository, and as a set of variables in the code (see LoggerFirmware/include/Configuration.h and logger::FirmwareVersion(), which returns a string version suitable for output). However, each of the modules internal to the system have their own version numbering so that they can evolve independently of each other; the logger reports all version numbers in response to the version command or the setup command.

Flashing the Firmware and Website

Initial Setup

Using the recommended VSCode/PlatformIO system combination, setting up a new logger is a two stage process: first flash the firmware, then flash the filesystem image for the firmware's management website. In both cases, the hardware preparation is the same:

1. Connect the board to a computer via a USB to UART converter, making sure that the converter has 5V power, but 3.3V data. The Adafruit FTDI Friend or USB-C to UART are recommended.
2. Link the two "PROG" pins with a wire jumper.
3. Link the two "RESET" pins with a wire jumper.
4. Remove the "RESET" jumper.
5. Remove the "PROG" jumper.

The on-board indicator LEDs should go out, and the serial monitor in PlatformIO indicates "waiting for download" from the boot-loader on the ESP32. Note that you must exit the serial monitor before attempting to flash the logger with either firmware or filesystem image, or you'll receive mysterious error messages from the downloader.

After putting the logger into download mode, the usual PlatformIO tool can be used to flash the firmware: If it hasn't been built before, a version of the firmware will be built, and then downloaded to the logger. The default platformio.ini has configuration options for a debug and release version of the firmware (env:debug and env:release in the usual selector). Selecting env:debug provides symbol information that allows the serial monitor to decode stack backtraces if the system panics, and turns up the verbosity of Free RTOS logging so that system services (e.g., web server interactions, file system issues) can be seen. This also makes the firmware a little larger (so it just fits into the current flash slot) and a little slower (since it's generating more debug messages). For production work, the env:release option should obviously be chosen.

After the download process is complete, the filesystem for the firmware management website has to be built, and then uploaded:

1. Ensure that you have a Python environment that includes Beautiful Soup 4 (use pip install bs4 if required).
2. In LoggerFirmware/website, execute python inline_flatten.py. This converts the website to an inlined version (LoggerFirmware/data/website) so that each HTML file contains all of the CSS and JavaScript components that it requires. This works around a limitation of the current web server where multiple simultaneous requests (e.g., getting the supporting CSS for a page) can fail at random. If you have the Google Closure Compiler installed, you can alternatively use LoggerFirmware/website/compile_website.sh to minify the JavaScript, inline it and the CSS into the HTML, and prepare all of the website components (skipping step 3 below).
3. If it does not already exist, copy directory LoggerFirmware/website/images to LoggerFirmware/data/website/images. This provides the WIBL logo.
4. Select the PlatformIO icon on the left side of the VSCode window, then the "Build Filesystem Image" task in the "Platform" menu: and confirm that it builds without errors.
5. Put the logger back into download mode using the procedure above.
6. Select the "Upload Filesystem Image" menu item: and confirm that the system uploads successfully.

At this stage, the logger is ready for operation. It is useful to use the serial monitor in PlatformIO to watch the boot process, so you should start this before the final reboot. Then link, and then remove, the wire on the "RESET" jumper to reboot the logger into operational mode. On first boot, the logger will load the default (compiled in) configuration (see LoggerFirmware/src/boot_config.json and below), and should bring up the logger as an access point on SSID wibl-config, password wibl-config-password (which you should of course change). At this stage, the Desktop GUI or website can be used to further configure the logger (see below for more details).

Over-the-air (OTA) Updates

The firmware has the facility to do "over-the-air" (i.e., WiFi) updates. The ota command sets the logger into this mode (note that once in OTA mode, the logger cannot recover to normal operation short of rebooting, which happens automatically once the OTA update completes).

In PlatformIO, OTA upload of firmware can be configured by setting:

[shell] upload_protocol = espota upload_port = 192.168.4.1

in the platformio.ini file for the LoggerFirmware project. Note that you may have to adjust the IP address if your logger is connected to another network (this is the default IP that the logger uses when booted on its own AP).

Note: It is not clear if OTA updates for the internal website are possible: the platform seems to suggest that this is possible, but attempts have led to corrupted flash images on the logger. This is therefore not currently recommended.

Configuring the Logger

Each logger has some non-volatile memory that stores both configuration parameters (through the logger::Config object in Configuration.cpp) and more detailed information (through the logger::NVMFile base class in NVMFile.cpp). This space is used to maintain information on the current state of the logger, and configuration parameters.

In order to allow the logger to boot immediately after being flashed with the firmware, without having to configure through the serial interface, the code maintains a default boot config (LoggerFirmware/src/boot_config.json) which is converted into a C-style static string using LoggerFirmware/src/json2cstring.py and then copied into the code in LoggerFirmware/src/Configuration.cpp so that it is always available to the logger. If, on boot, the UniqueID for the logger is not set, the code in ConfigJSON::SetStableConfig() loads the static configuration to boot-strap the logger.

The configuration can be set up using the Desktop GUI tool, by first flashing the logger with the current firmware, then rebooting (the web-server is enabled by default on first boot so that the configuration interface is available). See the Desktop GUI documentation for details. You can also use the firmware's web interface by flashing the logger with the current firmware and website image, rebooting, and then attaching to the logger's WiFi (usually SSID wibl-config by default, password wibl-config-password ... which you should of course change!) and then navigating to the main page (typically http://192.168.4.1/index.htm by default). The "configure" button can be used to read the current configuration, and update as required. See the documentation on the firmware's website for more details.

The command interface above can also be used to set the parameters as required, so to hand-configure each new logger:

1. Set up the logger's unique ID: uniqueid logger-id. This should typically be your DCDB provider ID followed by a unique string, e.g., UNHJHC-GulfSurveyor. Often, however, the unique string is a UUID, since it's more likely to be globally unique for all loggers. The Desktop GUI or firmware website can do this for you.
2. Set up the WiFi interface. This is done in three stages:
   1. Set the SSID: ssid ap wifi-id. The WiFi SSID is also used as an identifier for the ship, and therefore should be something unique, like WIBL-GulfSurveyor. In the case where the logger is adding itself to an existing WiFi on board the ship, this should be the SSID for the WiFi, ssid station wifi-id.
   2. Set the WiFi password: password ap pass-string. For stand-alone systems (i.e., the logger is starting its own WiFi access point, see below), this should be something that the mobile app is going to know so that it can connect. For connected systems (i.e., the logger is adding itself to an existing WiFi network onboard the ship), this should be the password for the ship's WiFi, password station pass-string.
   3. Set the WiFi mode: wireless accesspoint (logger starts its own local network) or wireless station (logger attempts to join an existing network).
3. Configure the web-server for start on boot webserver on 20 5 5 or delayed webserver off 20 5 5. The parameters are the station join algorithm configuration (see documentation for the Desktop GUI for details).
4. Configure loggers that should be running: configure on logger-name. The WIBL hardware can optionally record NMEA0183 (nmea0183), NMEA2000 (nmea2000), and the motion sensor (imu); you should enable only the loggers required in order to save processing cycles and memory on the SD card.
5. Configure power monitoring configure on power. This turns on power monitoring so that the system starts a shutdown sequence if the main power gets pulled.
6. Specify any algorithms to request on post-processing. The logger can specify which algorithms need to be run (e.g., depth deduplication), and arbitrary parameters for those algorithms. You can specify any required with algorithm alg-name alg-params and remove any current specified with algorithm none.
7. Specify the metadata to store on the logger and report to the output file for post-processing. The logger can store an arbitrary string for (typically JSON-formatted) metadata that gets passed into the output files and used in post-processing. You can specify the string with metadata arbitrary-string. Note that no CR/LF are allowed.
8. If NMEA0183 data are being recorded, then you can limit which messages get accepted using the accept message-id command. For example, to record only GGA messages, you'd say accept GGA. Note that turning on accept for any message means that only the messages you specify get accepted; use accept all to revert to recording everything that comes across the network.

Driver Module

The driver code in main.cpp implements setup for the components of the library, establishes the appropriate connections between them, and then goes into a processing loop that continually runs the NMEA2000 and NMEA0183 message processing code, the IMU logging code, the command interface for the logger, status LED management, and power monitoring. As required for the NMEA2000 library, it also specifies which messages are to be accepted for processing (although it appears that many others are also passed to the reception code, reasons unknown).

The setup() code establishes the connection to the Serial port (at 115200 baud) and the instantiates a logger::Manager object to handle logging of all of the data from both data streams, a nmea::N2000::Logger object for the NMEA2000 interface, a nmea::N0183::Logger for the NMEA0183 interface, a imu::Logger object for the IMU, a StatusLED object to control the status LEDs, and a SerialCommand object to handle the command interface. Instantiation for the NMEA2000, NMEA0183, and IMU loggers is conditional on the configuration parameters set by the user and stored in flash on the module. Subsequent code brings up the NMEA2000 logger, establishing the device identifier from the serial_number.cpp module. Instantiating the objects during setup() is unconventional (they are usually made global so that they can be readily accessed elsewhere in the code), but is required since some of the instantiation uses the Serial interface to report traffic, which cannot be done until the Serial.begin() call in setup().

NMEA2000 Logger Module

The core of the NMEA2000 logger is implemented in N2kLogger.cpp (within namespace nmea::N2000), which provides a Timestamp object to record times of reception (synthesised from the elapsed time of the microcontroller since boot, and the real time provided by NMEA2000 SystemTime and GNSS packets), and a Logger class to handle messages received on the NMEA2000 bus, including serialising the packets recognised; data is written using the logger::Manager module passed on instantiation.

Timestamp Mechanism

The only NMEA2000 messages with an integral time reference are SystemTime and GNSS; all others need to have a timestamp synthesised for them. Since the logger does not have a real-time clock (that's reliable and available), the only way to do that is to use the elapsed time for the microcontroller to act as a consistent (although not necessarily reliable) sense of time, and match this against the external (reliable) sense of time provided from the GNSS receiver in the system (through either message that has time).

The Timestamp class is intended to provide a first-pass estimate of time by doing a causal estimate of time, while preserving the elapsed time markers so that a better estimate can be generated in post-processing.

A single instance of the Timestamp class can be used to maintain a reference point for the last known real time information available (i.e., from a valid SystemTime or GNSS message); the sense of time can be updated when new messages arrive using the Timestamp::Update() method with the received date/time pair, and the Arduino millis() count for the time datum reception time. (The code will automatically generate a millis() value if one is not given, but the latency in doing so is unknown.)

Once the time reference is set, any number of TimeDatum objects can be generated using the Timestamp::Now() method, which establishes the current elapsed time at TimeDatum construction, and then computes a real time reference based on the difference between the TimeDatum elapsed time and that from the Timestamps last known reference time. The code in Timestamp::Now() checks for whether the elapsed time has looped over 0xFFFFFFFFUL (the maximum value) between the reference time and the current time, and if the difference between reference and current time has gone beyond a day, adjusting the time reference appropriately. The raw elapsed time at TimeDatum construction is always preserved in the output.

Methods are provided to convert both the Timestamp and TimeDatum into strings for printing in human-readable format. A specialisation for interface of TimeDatum to the serialisation.cpp module is also provided.

Logger Mechanism

The N2kLogger::HandleMsg() method is provided as a call-back for the NMEA2000 library during initialisation, and is therefore called by the library during NMEA2000.ParseMessages() invocations in loop() runs. The code first instantiates a Timestamp object for the current time, and then attempts to match the Programme Group Number (PGN) for the NMEA2000 message in the list of known messages, and hands off to a specialised method to translate and serialise the data. Any unknown messages are ignored, with optional reporting if "verbose" mode has been set (see Command Interface description).

The message handlers generically report their packet (in "verbose" mode), and then use the NMEA2000 library to interpret the packet into its component parts. These components are then serialised into the current log file (see Log Manager description below).

The SystemTime and GNSS messages are, in addition, used to set the logger's sense of real time for the Timestamp object since they contain estimates of the GNSS-derived time within the message. SystemTime messages from a local crystal clock (e.g., a standard computer) are ignored, and the GNSS time is only used if there has been no other time reference applied to the logger.

NMEA1083 Logger Module

The core of the NMEA0183 logger is implemented in N0183Logger.cpp (within namespace nmea::N0183), which provides a Sentence class (a sub-class of the logger::IncBuffer base used to accumulate data into a character buffer one character at a time) to encapsulate a single string from the serial RS-422 input, a MessageAssembler object to run for each input channel that established timestamps for the initial $ in the NMEA0183 message, and then accumulates data until an end-of-sentence CR-LF is found. Once fully assembled, the message is passed into a ring buffer until it can be picked up for logging. The Logger object manages the two channels, and provides a user-callable routine that will process any messages (and assemble them piecewise as required).

Timestamp Mechanism

If the system is running solely on NMEA0183, there is no reliable timestamp information. The logger could potentially parse out the timestamps in the $GPGGA messages, or look for $ZDA messages, but parsing these and attempting to correct is unlikely to be either reliable or easy in real-time. Therefore, the code simply record the system elapsed time for the starting $ on each message, and reports this with the sentence when logged. A full timestamp is computed in post-processing using this information and any timing packets in the file. For full details, see wibl-python/wibl/core/timestamping.py.

Logger Mechanism

The Logger::ProcessMessages() method encapsulates the code to drive the MessageAssembler objects for each channel, adding as many characters as are available on the input buffers on each channel into the messages being assembled on each call. At the end of the processing, any fully assembled messages are pulled from each channel and logged using the LogManager object. Verbose output of messages being logged can be turned on using the verbose command (see Command Interface below).

Log-file Manager

In order to allow multiple channels of information to be interlaced into the log files, and to provide a central position for manipulating the log files (e.g., to delete, enumerate, etc.), the LogManager.cpp code provides (in namespace logger) a Manager object to start, close, remove, count, and enumerate files, and to record information into them using anything that can be pushed into a Serialisable object (see Serialisation Module below). The Manager object also specifies the ID numbers used to identify the objects once serialised.

The LogManager.h interface also specifies the number of log files that are allowed to exist on SD card at the same time as a constant. At any time that a Manager::Record() method call is made, the code checks the size of the file being written, and cycles to the next file in sequence if it gets too big. The maximum size is a compile time option (MAX_LOG_FILE_SIZE in LogManager.cpp), but by default is 10MB.

The Manager object also provides a separate console log file, stored as console.log in the root directory of the SD card and accessible through the Manager::Console() method, on which any debugging or error information can be written. This file is accessible through the log command of the command interface so that the user can check on the status of the logger directly.

Parameter Storage

Various parts of the system need to be able to store key-value parameters in the non-volatile memory on the module. Due to differences in where this is available within the different possible modules, the ParamStore.cpp code provides a ParamStore object that provides key set/get, and a specialised pair to set/get a binary value associated with a key (mainly to avoid having to convert results). A static factory call in ParamStoreFactory::Create() allows this to be made dynamically with the correct implementation for the underlying module.

ESP32 Implementation

The ESP32 module has 4MB flash memory, which is shared between the system, the application, space for over-the-air update, and the SPIFFS file system. The SPIFFS section is used to make files with the key names that contain the associated value. Therefore, there is no limit to the number of keys that can be stored beyond the fundamental limits for the file system. Note that no special effort is made to accommodate variations on the known name: only exact matching is performed. So, for example, idstring and idstrings are different things, and it's up to the developer to make sure that problems like this don't cause problems.

Arduino Due Implementation

The Arduino Due has no NVRAM of its own, and therefore relies on the Bluetooth adaptor (if used) to provide. Memory is limited, but can accommodate a few parameters. This is implemented by the BLEParamStore object, a sub-class of ParamStore, as defined in ParamStore.cpp. Note that only a specific number of parameters with known names are allowed, and no special effort is made to accommodate variations on the known names: only exact matching is performed. Current names are:

Configuration Information

Building on top of the ParamStore module, the Configuration.cpp module provides a set of named keys to manage the configuration of the logger. The logger::Config class, and the global instantation logger::LoggerConfig provide a generic interface for binary and string parameters, which the logger::ConfigJSON adapter class allows all of the configuration parameters (and some version information) to be retrieved, and set, using a JSON-format string. This is hard to do by hand, so the most common use case for this is using the Desktop GUI to set up configurations for new loggers.

In order to ensure that loggers boot as expected when first flashed (i.e., before any configuration is set using either the serial interface or with the Desktop GUI), the Configuration.cpp code has a static configuration JSON string embedded which is used to set a "known good" configuration if the logger detects that the current configuration is corrupted and won't boot. Currently (firmware 1.4.0), the check for validity is implemented in logger::Config::IsValid(), and consists of checking that the logger's UUID is set.

This configuration is set at compile time, and cannot be changed without rebuilding the firmware. The LoggerFirmware/src/boot_config.json file is used to set the configuration; the LoggerFirmware/src/json2cstring.py code can be used to convert from the text JSON to a snippet of C++ code that can be pasted into the Configuration.cpp code.

NVM File Storage

Although key-value storage is sufficient for configuration information, the logger also requires more extensive storage for some parameters with special handling. For example, the logger can store a list of algorithms, and parameters, that should be applied to the data in post-processing. To provide this, the NVMFile.cpp module provides a base class (NVMFile) and two specialisations (NVMFileReader and NVMFileWriter) to provide interface adaptors for read/write semantics. Sub-classes are then provided for various features, such as the algorithm store, the list of NMEA0183 sentences that will be accepted for logging, JSON-format metadata, and scaling parameters for the on-board sensors.

Command Interface

In order to allow the user to interact with the logger, either during development and debugging, or during service, the SerialCommand.cpp module implements a very simple ASCII command interface through the SerialCommand object. This automatically monitors both the Serial object (typically the debugging serial port on any Arduino core) and the WiFi interface (when active) and executes commands from all sources. Due to the single-threaded nature of the Arduino development environment, the SerialCommand::ProcessCommand() method must be called at regular intervals in order to ensure that commands continue to be executed. Typically, this can be done through the loop() function.

Commands are implemented in private methods, with the SerialCommand::Execute() method being used to wrangle details. In order to interact with the logger and status LEDs, the SerialCommand object stores pointers to the relevant objects, but does not attempt to deallocate them if it were ever to be deleted.

Note that the SerialCommand::ProcessCommand() method that is used to read data from a command source and execute it attempts to read from the FTDI (UART) serial port, the WiFi webserver (if active), and maintains a record of which interface provided each command. Results from the command are only sent back through the interface used to send the command to the module. Note that some commands are only accepted through a limited number of interfaces, and the format of the response might be different on each interface.

Note, as a design principle, that there should only be a single command to set or display a particular parameter; the command name without parameters should display the value of the parameter being set. Thus, uniqueid reports the unique identifier for the logger, while uniqueid wibl-1 would set the unique identifier to be wibl-1.

Serialisation Module

The serialisation.cpp module provides generic serialisation services for the logger. This is done through a Serialisable object to buffer the data to be serialised in one packet, and a Serialiser object to handle the file interface and writing of the data. Currently, each packet written into the file consists of a header of a payload ID number (32-bit) followed by a length count (32-bit) (that excludes the 8-byte header length), and then a string of bytes corresponding to the payload. The serialisation module mandates no further structure within the packets, which depends on the higher-level code. Primitives to add unsigned integers from 8-bit to 64-bit, single- and double-precision floats, and C-style strings are provided. Other primitives could readily be added if required.

The Serialisable object manages a raw buffer for the data being assembled for serialisation, and will expand the buffer if required. A hint as to the length can be specified on construction, however (default 255 bytes), which will significantly reduce the cost of serialisation is set appropriately - reallocating memory is very expensive.

The serialisation module has an independent version number, which is written to each file as the first packet in addition to the logger version information, using packet ID 0. This packet is always guaranteed to be 30-bytes long, written in format:

so that it can always be read in the same way, independent of serialiser version. The public interface to serialisation, Serialiser::Process(), rejects any attempt to write a packet with this ID; the private interface, Serialiser::rawProcess() does not do this check. Users should therefore read the first packet to understand the version of serialiser and loggers that are being used, and then adapt reading appropriately.

The serialisation module also writes a Metadata (ID 12) packet, followed by a Setup (18) packet, into the log file. See the binary data format description for more details.

To avoid undue complexity (and time), the file is always serialised in the endianness of the host microcontroller. This can be automatically detected if required through the serialiser version packet, where the length word is always set to 22 bytes.

SD Card Interface

ESP32 Implementation

The ESP32 has native support for SD cards using the SD/MMC (high speed) interface. The implementation of the SD card library is not like that of the standard Arduino setup, however. Specifically:

1. Directory entries have the full pathname, rather than the file leaf name. This means that they don't need to have the pathname added before access.

2. The library needs to specify FILE_APPEND for appending onto the end of a file, while the standard library FILE_WRITE does this by default.

This is resolved using conditional compiles in the source of LogManager.cpp.

Arduino Due Implementation

The Arduino Due standard library is used. In the reference implementation, the data logger shield provides a link from the hardware SPI on the ICSP port to the SD card; the chip select GPIO is specified in LoggerSource.ino.

WiFi Support

WiFi is required in order to transfer files on reasonable timescales. The interface to this is implemented in WiFiAdapter.cpp, with object WiFiAdapter as base that can be specialised for the specific adapter being used. Currently, only the built-in ESP32 adapter is implemented. The WiFiAdapterFactor::Create() static method can be used to generate an appropriate adapter for the module in question.

The WiFiAdapter object is intended to be instantiated early, but only starts doing anything useful after a call to WiFiAdapter::Startup() (which can be accessed via the wireless on command, or if the webserver is configure to start at boot time, webserver on ...), which initiates the WiFi access point, finds the address assigned to the module by the access point (generally self-assigned, since it is typically brought up in access point mode, rather than station mode, unless otherwise configured by the wireless station command). The module then goes into server mode and starts a webserver that can be used to pass commands that can be interpreted by the SerialCommand object (above).

The webserver listens on the standard port 80, and provides a number of endpoints that can be used to interact with the logger: * /heartbeat. This endpoint always returns 200 OK, with a body of the results of a "status" command, including system versions, current uptime, and the list of files currently on the logger, their sizes and MD5 hashes. * /command. This accepts ASCII text for commands, as per the description above in SerialCommand, and caches them for execution. The endpoint typically returns 200 OK, but can return a variety of status codes for non-existent services or other errors (this can be set by WiFiAdapter::SetStatusCode()). The body of the response is assembled through SerialCommand::EmitMessage() calls, SerialCommand::EmitJSON(), or otherwise, and are stored by WiFiAdapter::AddMessage() or WiFiAdapter::SetMessage() until WiFiAdapter::TransmitMessages() is called, which completes the HTTP transaction. Note that all normal responses are therefore JSON strings (application/json), although the transfer command returns binary data (application/octet-stream), and sets the Digest header to the MD5 digest of the file. * /index.htm. This is a static website for a management interface for the logger, which uses a combination of HTML, CSS, and JavaScript to provide an equivalent to the Desktop GUI. The implementation of the website, and how to configure it, is described here.

The logger by default starts in AP mode, with SSID wibl-config and password wibl-config-password. You should, of course, change the password as a first order of business (e.g., password ap foo, or through the Desktop GUI). A separate SSID and password is configurable for station mode, so that you can specify the network for the logger to join, but still maintain a "safe mode" as an AP if that network cannot be joined. The station joining algorithm is implemented by the ConnectionStateMachine object in WiFiAdapater.cpp; construct by default, use ConnectionStateMachine::Start() to begin, and then regularly call ConnectionStateMachine::SetpState() to maintain the state.

The station joining algorithm attempts to join the configured network with the given password a configurable number of times, waiting a configurable time between each attempt, and assuming that the connection attempt has failed if it hasn't been handed an IP address after a configurable duration. The idea here is that if you mess up the password or SSID, you don't want to have to crack open the logger to recover: eventually, the system will give up and default back to the "safe mode" of being an AP with known SSID and password so that you can join its network and try again. This also allows the logger to recover if there is a power cycle on the host platform and it restarts before the WiFi AP that it's using for connectivity. The triplet of (inter-attempt delay, retries, timeout) is set by the webserver command, and have to be adapted to the particular setup for the installation on the host platform.

On smaller platforms without installed WiFi APs, it is normal to leave the logger in AP mode permanently.

Note that the single-threaded nature of the Arduino environment means that you need to call WiFiAdapter::RunLoop() regularly to allow it to capture message packets from the client, and to maintain the ConnectionStateMachine system. This also means that calls to the web-server that appear too quickly (generally when the system is still processing the previous request) can fail. This is particularly problematic for JavaScript on the firmware website. See the documentation there for more details.

Auto-Upload for Log Files

For loggers that are attached to WiFi networks with internet access, the firmware can be configured to automatically upload log files when connected. The upload command can be used to enable/disable the effect (upload on|off), and to set the parameters for the access algorithm. Note that modifications to the uploader state are only read at boot time, so the logger needs to be rebooted for any changes (on/off and parameters) to take effect.

The algorithm runs in cycles as part of the main loop, and to avoid taking up too much of the logger's time, the code will only attempt to upload at fixed intervals (by default every 30 min), and then only for a fixed duration (by default 10 s), continuing upload bursts over time until all of the data is transferred. To test whether the destination server exists, the code in AutoUpload.cpp generates a status report (which includes all of the information for all log files on the logger) and HTTPS POSTs this to the "/checkin" endpoint on the server as the first request of the transaction. Assuming that the response is positive (within the timeout limit, by default 5s), the logger then sends files in the list as POSTs to the "/update" endpoint until all files are sent, or the duration expires. For each transaction, if the server confirms transmission, the logger then removes the log file from the SD card. The CA certificate associated with the upload server, and the password for authentication can be set using the auth cert and auto token commands, respectively; see the command list for details. The logger's UUID is used for the username.

The server side of the protocol can be seen in UploadServer/wibl-monitor.go demonstration server, and illustrates re-computing the MD5 hash for the transferred data as a means of confirming that the file has been successfully transmitted, and the handling of the authentication mechanism and TLS configuration. Self-sign certificates for testing can be generated using the UploadServer/cert-gen.sh script (best executed in UploadServer/certs to make sure that the server code looks for the certificate and key in the right place), which requires OpenSSL installed. Note that the CA certificate (usually UploadServer/certs/ca.crt) needs to be loaded into the logger (using the firmware website or desktop GUI tool) in order for uploads to work correctly. Depending on browser, the ca.crt file might not be recognised; copying this to ca-cert.txt is an effective workaround. Note that in practice it would typically be the case that you'd be handling multiple loggers, and therefore would need a database of username/password pairs for the loggers (best practice is a different password for each logger). The demonstration code avoids this implementation detail by providing a fixed username/password pair directly in the code. You will therefore also have to set an upload token (effectively, a password) using the same tools, and update UploadServer/support/middleware.go to match, in order to complete configuration (the default logger name of wibl-logger should work with any logger).

How the automatic upload should be implemented might be a matter for the particular installation. For example, the logger is not (currently) particular about WiFi security, and does not implement any authentication for commands. Consequently, it might not be entirely appropriate to have the system on a live internet connection (unless there is a suitable firewall configured somewhere in the system). In this situation, it might be better to have a separate computer that's dual-homed so that the logger can only talk to the computer, and the computer relays the files to the rest of the world (using the same protocol, or via direct upload of files using the WIBL Python toolset), since it can run a software firewall internally.

Status LEDs

The code implements a set of status LEDs through StatusLED.cpp, which represent a set of logical states, rather than specific LED combinations and/or colours. The basic model is for a single common-cathode RGB LED which minimises the board size required for implementation. However, it is also possible to have separate LEDs if required. Default values for the GPIO pins to use for red, green, and blue channels are specified in StatusLED.h; conditional compilation allows for different implementations.

Flashing the LEDs is a part of the status indication mechanism. To do this, the code must call StatusLED::ProcessFlash() regularly to allow for status to be changed. The default flash rate is 0.5s, so calls significantly more frequently than this are required to ensure that an approximately stable flash rate is maintained. Calling in an Arduino loop() is usually sufficient.

Board Serial Number

The NMEA2000 library needs to have a unique ID for the logger to advertise on the bus. We could derive this from the logger ID that's set by the user, but having something that's derived from the board itself seems like a better idea. This is implemented in the serial_number.cpp code, which is cribbed from an example in the NMEA2000 library implementation. The method differs depending on the type of board being used.

ESP32 Implementation

The ESP32 has a MAC (media access controller) address associated with each module. This is used for the WiFi, and in different forms for the Bluetooth, or Ethernet interfaces. Because this has to be unique for the system to function, this is the target for identifier on ESP32 platforms. This can be read through an ESP32 API call.

Arduino Due Implementation

The Arduino Due is based on an Atmel SAM3X8E microcontroller (ARM Cortex M3 core), in which the Enhanced Embedded Flash Controller (EEFC) maintains a unique identifier for the chip, readable using the "Start Read Unique Identifier" (STUI) command. When the data becomes available, it's the first 128-bit value at 0x400000-0x400003. The example code reads this value, but then returns the lower 32-bit word as the identifier.