1. Tom Roche Avatar Tom Roche
  2. Untitled project
  3. headful_EPA_jumpbox_config

Wiki

Clone wiki

headful_EPA_jumpbox_config / Home

View History

part of the headful_EPA_jumpbox_config project

summary

This project records my efforts to do the following, listed in declining order of effort required:

  • setup a linode to use as a jumpbox for access to a US EPA compute cluster (in order to, e.g., submit jobs).
  • to configure a client to access the cluster via the linode/jumpbox, using VNC and the F5 Network Access Plugin

All code in this project is bash unless otherwise noted. I assume that both the linode and the client run Debian Linux, but little of the following code and instructions is so Linux-specific (much less Debian-specific) that it could not be adapted to other operating systems.

background

glossary

Throughout these docs, I use the following terms:

  • client or client device: this is the hardware you use to access your linode or server (remotely, by SSH): e.g., your laptop. Any environment variable (aka envvar) prefixed CLIENT_ refers to your client device. Unfortunately this usage conflicts with terms referring to software "network clients" (like "a VPN client"), which require a "client device" on which to run. Hence I will try to use "client device" as needed to differentiate from "client software."
  • jumpbox: the remote host (typically a cloud VM) on which you will do most configuration and installation. It is the only device which you will rebuild, at least for this project! Variables prefixed JUMPBOX_ refer to this host.
  • cluster: scientific-computing nodes which are not merely remote, but also behind a firewall. The target cluster and its associated firewall are managed independently (and with almost no input from me) from the client device and the jumpbox.
  • session: doing work on a cluster requires connecting and authenticating to a particular login node via SSH in a terminal. One can then run commands from the resulting shell/commandline (assumed to run bash unless otherwise noted). The term session denotes the period during which the given shell is operative on a given cluster, i.e., from login to the cluster's login node to (possibly involuntary) logout.
  • the environment variable PATH_TO_PROJECT denotes the filepath on your client or jumpbox at which you have copied or git cloned this project on that device. When running manually, you can define that environment variable in a shell and then just cut'n'paste commands from my docs.

motivation

I'm a student working on an open-science thesis project using data and other computing resources on compute clusters provided by the US EPA. (For reasons unknown to me, a student accessing agency resources must be a federal contractor, aka a business partner.) Those compute clusters are behind one or more firewalls, requiring a particular VPN to access. Thus, e.g., attempts to nslookup the FQDN for any cluster node (including login nodes) will fail unless one is "on the VPN."

productive past

For 2 years I was able to SSH from home through the firewall(s) to the clusters, using

  • different 64-bit laptops {owned, admin}ed by me, running various flavors of Debian Linux.
  • different ISPs' connection hardware (first cablemodem, then 4G modem).

  • the agency-designated F5 SSL VPN (aka the F5VPN), the client side of which was implemented as a web-browser plugin (the F5NAP, where NAP == network access plugin). The F5NAP is only available on Linuxes via an ancient 32-bit Firefox plugin: see

  • an agency-provided RSA SecurID.

Note that F5 (the VPN vendor)

  • has all proprietary products
  • does not apparently provide a headless/non-GUI means to run this VPN--at least, not with the F5 versions which my agency apparently runs--per this F5 forum post.
  • seems quite Windows-oriented (as is my agency)
  • seems not (in my experience) much responsive to third-party users (such as myself)
F5VPN-only access

On my home client devices, in "the productive past," I was able to use the F5VPN client software "directly." Running the F5VPN client connected (transparently to me) to a cluster login node:

<------------my control |<-  ISPs ->| agency control-------------------------------------->

(modem's DHCP IP#)                                  (per-connection)
INITIAL_PUBLIC_IPN    modem                         F5VPN_PUBLIC_IPN firewall
     +-----------+     +-+       c     +---------+     +--------+       ||     +---------+
     | laptop +  |     | |       l     | remote  |     | F5VPN  |       ||     | cluster |
     | F5 client |-> <-| |->   <-o-> <-| access  |-> <-| server |->   <-||-> <-| node(s) |
     |           |     | |       u     | website |     |        |       ||     |         |
     +-----------+     +-+       d     +---------+     +--------+       ||     +---------+

Note that

  1. F5 client== F5NAP which is quite opaque (at least to me) and not well supported (at least for me).
  2. F5VPN_PUBLIC_IPN changes with each connection to the F5VPN. This was transparent to me when I was connecting directly, since the F5NAP handled all connection details.
F5VPN-only connection

  1. Start the specially-configured F5NAP'ed Firefox (aka the "F5 client"), scripted to

  2. Login to the remote-access website using a password partly generated by the SecurID.

  3. Upon successful login, click a link on the resulting page causing the F5NAP to put up a second, smaller Firefox window with VPN UI (e.g., for VPN start/stop, displaying VPN status). Only when this second window shows VPN status== Connected was the VPN actually usable.

  4. For each cluster I wanted to access: from a shell/terminal, start a session using the login node's FQDN (which is not DNS-able off the agency's physical LAN without the VPN).

F5VPN-only disconnection
  1. End the session (e.g., enter logout or C-d in its shell/terminal).
  2. Stop the F5VPN via UI in the F5NAP-launched smaller window.
  3. Logout from the F5VPN: hit link= Logout in the F5NAP's large/main window (which auto-whacked the small window). Note that stopping the F5VPN usually resulted in auto-logout after a relatively short period.
  4. Kill the F5NAP'ed Firefox. I noticed that reconnecting seemed to become problematic for an instance after a short period of inactivity. Since the 32-bit firefox was relatively quick to restart from a script--it only ran one tab, for the remote-access website--I got into the habit of killing it after each session.

Aug 2014 policy change

In August 2014 my remote access was terminated without notice (in mid-session!). I later discovered that agency contractors had implemented new security requirements for "remote business partners":

  1. IP#s: one may connect to the clusters only from a registered IP number. While I am not sure whether the agency intends for the connecting IP#s to be static, the registration process seems to be sufficiently slow/painful that one will not want one's cluster-connecting IP# to change.
  2. host security: several requirements, all specified by (literally) XP screenshots.
  3. all the old requirements: notably, to use the F5NAP.

While waiting for approval for a Linux host security proposal (i.e., how to provide the desired protections without running XP on my client device), I began to setup a linode jumpbox with a static IP#.

plan A

My initial intention (in Aug 2014) was to

  1. Register the jumpbox's static public IP# with the agency.
  2. Run the F5VPN (and other security requirements) on the jumpbox.
  3. Forward SSH and X traffic through the jumpbox back to my client device.
plan B

I was strongly advised to run the jumpbox headless: not to run the F5NAP on the jumpbox, but rather to run the F5NAP on my client device and to tunnel the F5VPN through the jumpbox (using an OpenVPN). After many months of work, I learned a lot about networking, but found I (and folks I consulted) simply could not make this design work. The problem was essentially circular:

  1. I have no control over how the F5NAP configures the F5VPN, and particularly its (network) routes: it's a blackbox.

  2. I could setup and run the OpenVPN such that the world (and particularly EPA's remote-access website) saw my client device as having the registered IP# (of the jumpbox).

  3. When the F5VPN started/connected, it imposed its own routes on the client device, which were incompatible with the OpenVPN's routes.

  4. The F5VPN-imposed routes caused the client device to

    1. lose its connection to the OpenVPN, which
    2. caused the client device's public IP# to revert to its normal/unregistered one (as DHCPed from its current ISP), which
    3. caused the F5VPN to disconnect from the client device before I could "fix the routes" (i.e., impose routes on the client device compatible with both the OpenVPN and the F5VPN).
plan C

The good news was, in the interim, my Linux host security proposal was eventually approved. Thus I eventually returned to a plan C ~= plan A:

  1. Register the jumpbox's static public IP# with the agency.
  2. Run the F5VPN (and other security requirements) on the jumpbox.
  3. Run jumpbox applications (notably, the F5NAP) from my client device via a VNC connection.

process

The following procedures should be somewhat similar for any jumpbox, whether managed by the user or a cloud provider (e.g., Digital Ocean, Linode). Essentially, one must perform the following 5 tasks:

  1. Pre-install process:

    1. Prepare your client device for this task:

      1. You are presumed to have already git cloned this project to some path in your client's filespace. I will refer to that path as ${PATH_TO_PROJECT}: for convenience you should probably define that envvar.
      2. Edit your public and private properties files to reflect your intentions and your client's configuration.

    2. Prepare your jumpbox for this task:

      1. If using a jumpbox provider (aka vendor), make necessary arrangements with them (e.g., account setup).
      2. Edit your public and private properties files to reflect your intentions and your jumpbox's configuration.

  2. Setup a "base config" on the jumpbox: starting from some "barebones" configuration provided by the jumpbox provider (which might be you), build the jumpbox up to the point that you can interact with it securely and headlessly (i.e., via ssh) from your client device.

  3. Setup VNC:

    1. ssh-tunnel from the client device (for security)
    2. server on the jumpbox
    3. viewer on the client device

  4. Install/configure the F5NAP on the jumpbox, test connecting to EPA clusters.

  5. Install/configure headless ssh tunnel to EPA clusters.

process properties

These properties files (which are essentially just bash scripts) contain definitions of envvars used in scripted or manual processes that can be easily

  • sourced for either scripted or manual processes
  • managed without needing to edit scripts.

You should edit them to appropriately reflect your situation and intentions.

public properties

Public properties provide information about your situation that I presume you do not wish to keep secret. (ICBW :-) Find the file locally @ ${PATH_TO_PROJECT}/scripts/public.properties and edit it.

private properties

Your "private properties" provide information about your situation that I presume you do wish to keep secret. Accordingly I distribute a template file that you need to copy and fill out. The private properties template files can be found

  • online here
  • locally @ ${PATH_TO_PROJECT}/scripts/private.properties_template

Copy the template file locally to ${PATH_TO_PROJECT}/scripts/private.properties and edit it.

linode

Currently I have only performed this procedure (i.e., configuration of a headful jumpbox for interacting with EPA clusters)

  • on a linode, but the procedure it should be easily transferable to jumpboxes from other vendors.
  • using a Debian linode, but the procedure should be easily transferable to linodes running other Linux OSs.

The procedure is follows the same 5 steps outlined above:

  1. Pre-install process: prepare your client device and jumpbox.

  2. Install a "base config" on your jumpbox.

  3. Setup VNC on jumpbox:

    1. ssh-tunnel from the client device (for security)
    2. VNC server on the jumpbox
    3. VNC viewer on the client device

  4. Setup F5VPN client on jumpbox:

    1. ssh-tunnel from the client device (for security)
    2. VNC server on the jumpbox
    3. VNC viewer on the client device
    4. F5NAP'ed Firefox on the jumpbox

  5. TODO: Install/configure headless ssh tunnel to EPA clusters.

status

The linode-based process is currently fatally blocked by inability to obtain F5NAP binaries for Linux either via the normal mode (automated install via web UI) or from EPA. (Hence this failure mode is probably not linode-specific.)

Updated