1. Tom Roche Avatar Tom Roche
  2. Untitled project
  3. linode_jumpbox_config

Wiki

Clone wiki

linode_jumpbox_config / Home

View History

part of the linode_jumpbox_config project

summary

This project records my efforts to

  • setup a linode to use as a jumpbox for access to a compute cluster (in order to, e.g., submit jobs).
  • to configure a client to access the cluster via the linode/jumpbox, using other required software

Both the linode and the client run Debian Linux. All my code in this project is bash.

background

glossary

Throughout these docs, I use the following terms:

  • client or client device: this is the hardware you use to access your linode or server (remotely, by SSH). E.g., my laptop. Script environment variables prefixed CLIENT_ refer to this. Unfortunately this usage conflicts with terms referring to software "network clients" like the "OpenVPN client" or the "F5VPN client," both of which run on a (single) "client device." Hence I will try to use "client device" as needed to differentiate from "client software."

  • jumpbox or linode or server: the remote host on which you will do most configuration and installation. It is the only device which you will rebuild, at least for this project! Variables prefixed MYNODE_ refer to this (because the 'LINODE_' namespace is reserved in StackScripts). I may occasionally refer to it as "my linode," despite the facts that

    1. I merely rent it.
    2. I share control of it with its vendor.
    3. What I share is merely a virtual instance of something, running on physical hardware which is purely vendor-controlled.
  • cluster: scientific-computing nodes which are not merely remote, but also behind a firewall. The target cluster and its associated firewall are managed independently (and with almost no input from me) from the client device and the jumpbox.
  • Linode Dashboard or dashboard: a page linked from your Linode Manager for the particular linode you are using. The Linode Dashboard provides a web UI used to provide some high-level control of, and information about, your linode. Notably, your dashboard allows you to rebuild your linode from scratch, i.e., from OS install.
  • session: doing work on a cluster requires connecting and authenticating to a particular login node via SSH in a terminal. One can then run commands from the resulting shell/commandline (assumed to run bash unless otherwise noted). The term session denotes the period during which the given shell is operative on a given cluster, i.e., from login to the cluster's login node to (possibly involuntary) logout.
  • ${PATH_TO_PROJECT}/ denotes the filepath on your client at which you have git cloned this project.

motivation

I'm a student working on an open-science thesis project using data and other computing resources on compute clusters provided by the US EPA. (For reasons unknown to me, a student accessing agency resources must be a federal contractor, aka a business partner.) Those compute clusters are behind one or more firewalls, requiring a particular VPN to access. Thus, e.g., attempts to nslookup the FQDN for any cluster node (including login nodes) will fail unless one is "on the VPN."

productive past

For 2 years I was able to SSH from home through the firewall(s) to the clusters, using

  • different 64-bit laptops {owned, admin}ed by me, running various flavors of Debian Linux.
  • different ISPs' connection hardware (first cablemodem, then 4G modem).

  • the agency-designated F5 SSL VPN (aka the F5VPN), the client side of which was implemented as a web-browser plugin (the F5NAP, where NAP == network access plugin). The F5NAP is only available on Linuxes via an ancient 32-bit Firefox plugin: see

  • an agency-provided RSA SecurID.

Note that F5 (the VPN vendor)

  • has all proprietary products
  • does not apparently provide a headless/non-GUI means to run this VPN--at least, not with the F5 versions which my agency apparently runs--per this F5 forum post.
  • seems quite Windows-oriented (as is my agency)
  • seems not (in my experience) much responsive to third-party users (such as myself)
F5VPN-only access

On my home client devices, in "the productive past," I was able to use the F5VPN client software "directly." Running the F5VPN client connected (transparently to me) to a cluster login node:

<------------my control |<-  ISPs ->| agency control-------------------------------------->

(modem's DHCP IP#)                                  (per-connection)
INITIAL_PUBLIC_IPN    modem                         F5VPN_PUBLIC_IPN firewall
     +-----------+     +-+       c     +---------+     +--------+       ||     +---------+
     | laptop +  |     | |       l     | remote  |     | F5VPN  |       ||     | cluster |
     | F5 client |-> <-| |->   <-o-> <-| access  |-> <-| server |->   <-||-> <-| node(s) |
     |           |     | |       u     | website |     |        |       ||     |         |
     +-----------+     +-+       d     +---------+     +--------+       ||     +---------+

Note that

  1. F5 client== F5NAP which is quite opaque (at least to me) and not well supported (at least for me).
  2. F5.VPN.IP.NUM changes with each connection to the F5VPN. This was transparent to me when I was connecting directly, since the F5NAP handled all connection details.
F5VPN-only connection

  1. Start the specially-configured F5NAP'ed Firefox (aka the "F5 client"), scripted to

  2. Login to the remote-access website using a password partly generated by the SecurID.

  3. Upon successful login, click a link on the resulting page causing the F5NAP to put up a second, smaller Firefox window with VPN UI (e.g., for VPN start/stop, displaying VPN status). Only when this second window shows VPN status== Connected was the VPN actually usable.

  4. For each cluster I wanted to access: from a shell/terminal, start a session using the login node's FQDN (which is not DNS-able off the agency's physical LAN without the VPN).

F5VPN-only disconnection
  1. End the session (e.g., enter logout or C-d in its shell/terminal).
  2. Stop the F5VPN via UI in the F5NAP-launched smaller window.
  3. Logout from the F5VPN: hit link= Logout in the F5NAP's large/main window (which auto-whacked the small window). Note that stopping the F5VPN usually resulted in auto-logout after a relatively short period.
  4. Kill the F5NAP'ed Firefox. I noticed that reconnecting seemed to become problematic for an instance after a short period of inactivity. Since the 32-bit firefox was relatively quick to restart from a script--it only ran one tab, for the remote-access website--I got into the habit of killing it after each session.

Aug 2014 policy change

In August 2014 my remote access was terminated without notice (in mid-session!). I later discovered that agency contractors had implemented new security requirements for "remote business partners":

  1. IP#s: one may connect to the clusters only from a registered IP number. While I am not sure whether the agency intends for the connecting IP#s to be static, the registration process seems to be sufficiently slow/painful that one will not want one's cluster-connecting IP# to change.
  2. host security: several requirements, all specified by (literally) XP screenshots.
  3. all the old requirements: notably, to use the F5NAP.
false start

While waiting for approval for a Linux host security proposal (i.e., how to provide the desired protections without running XP on my client device), I began to setup a linode jumpbox, intending to

  1. Run the F5VPN (and other security requirements) on the jumpbox, using its static public IP# (to be registered with the agency).
  2. Forward SSH and X traffic through the jumpbox back to my client device.

However I was strongly advised to run the linode headless, and therefore not to run the F5NAP on the jumpbox.

intended solution

I was advised instead to run the F5NAP on my client device and to tunnel the F5VPN through the jumpbox. I.e.,

  1. Jumpbox provides

    • a static public IP# (to be registered with the agency)
    • an OpenVPN tunnel
    • applications and configurations satisfying the other security requirements

  2. Client device runs one instance each of the OpenVPN client software and the F5VPN client software (i.e., the F5NAP'ed Firefox).

  3. Make the F5VPN tunnel through the jumpbox, so that

    • the remote-access website saw its IP# (which I have since registered)
    • the agency firewall saw me as being on the F5VPN
    • the resulting "system architecture" would be like the following (where F5 client== the F5NAP'ed Firefox) :
<------------my control | <-------------ISPs------------> | agency control-------------------------------------->

(modem's DHCP IP#)                      (static)                           (per-connection)
INITIAL_PUBLIC_IPN    modem        JUMPBOX_PUBLIC_IPN                      F5VPN_PUBLIC_IPN firewall
     +-----------+     +-+           +-----------+            +---------+     +--------+       ||     +---------+
     | laptop +  |     | |     c     | linode  + |     c      | remote  |     | F5VPN  |       ||     | cluster |
     | OpenVPN   |-> <-| |-> <-l-> <-| OpenVPN   |-> <-l->  <-| access  |-> <-| server |->   <-||-> <-| node(s) |
     | client +  |     | |     o     | server  + |     o      | website |     |        |       ||     |         |
     | F5 client |     | |     u     | other     |     u      |         |     |        |       ||     |         |
     |           |     | |     d     | security  |     d      |         |     |        |       ||     |         |
     +-----------+     +-+           +-----------+            +---------+     +--------+       ||     +---------+

To start a session, I would

  1. Start an OpenVPN tunnel:

    1. Ensure the jumpbox was running its OpenVPN server.
    2. Start the OpenVPN client software on my client device to forward traffic to/through the tunnel.

  2. Use the F5VPN to complete session startup.

To end a session, I would

  1. Use the F5VPN to disconnect.

  2. (optionally) Stop the OpenVPN tunnel:

    1. Stop the OpenVPN client on my client device.
    2. (optionally) Stop the OpenVPN server on the jumpbox.
networking problem

The good news was, my Linux host security proposal was eventually approved. The bad news is, I cannot currently (as of November 2014) make my intended solution work, presumably due to misconfiguration. My current linode jumpbox config cannot start a session (particularly, it cannot SSH to any login node) inside the agency firewall (details here). It currently exhibits the following desired abilities/behaviors:

  1. I can start an OpenVPN server on the jumpbox as described here. Server-only startup, without any (known :-) running OpenVPN client, produces this main log ending with Initialization Sequence Completed.

  2. After starting the OpenVPN server on the jumpbox, I can start an OpenVPN client instance on my client device as described here. This seems successful:

    1. Before OpenVPN client startup, normal browser access (i.e., not using the F5NAP'ed Firefox) to http://www.whatismyip.com shows the normal/ISP-provided IP# of my client device's modem.
    2. OpenVPN client startup results in console text== Initialization Sequence Completed on the client device: see this main log and this status log.
    3. After OpenVPN client startup, normal browser access to http://www.whatismyip.com shows the IP# of the jumpbox.

  3. After starting both the OpenVPN server and client, I can (on the client device)

    1. start the F5NAP'ed Firefox
    2. (using it) browse to http://www.whatismyip.com/ (etc) and see "my linode's" IP#

  4. After starting all of {the OpenVPN server, the OpenVPN client, the F5NAPed Firefox), I can browse to the agency's remote-access website and login normally using my SecurIDed password.

  5. After logging-in to the remote-access website, I can start the F5VPN, and see status== Connected in the F5VPN UI.

However, successful connection to the F5VPN results in plan failure:

  1. On my client, I cannot even ping IP#s, e.g.,

    $ ping -c 4 141.101.120.15 # == www.whatismyip.com
PING 141.101.120.15 (141.101.120.15) 56(84) bytes of data.

--- 141.101.120.15 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3022ms

  2. On my client, DNS fails, e.g.,

    $ nslookup www.whatismyip.com
;; connection timed out; no servers could be reached

  3. My OpenVPN tunnel fails fairly quickly after connecting to the F5VPN (presumably due to the DNS failure). Once that fails, I lose my registered IP#, and I get blocked from the F5VPN. (The only good news here is that, once I lose the OpenVPN tunnel, my client regains networking--faster if I kill the client's openvpn process.)

  4. fatal: even before my OpenVPN tunnel fails, I cannot SSH into any cluster login nodes, because I have only their FQDNs, and cluster nodes are not DNS-able off the agency's physical LAN without the VPN.

process

pre-install process

  1. Create account @ linode.com (i.e. the website--not yet on one of their (virtual) boxes).

    1. read/save Terms of Service
    2. read/save Privacy Policy
    3. ... eventually, get a link to a Linode Dashboard, like this one.

  2. From your client,

    1. Checkout (i.e., git clone) this project to some path on your client (to which I will refer as ${PATH_TO_PROJECT}/).

  1. Record public properties for your project, which will be sourced by other code.

    1. Edit ${PATH_TO_PROJECT}/scripts/public.properties (which is bash/text) as directed in the file, and save your values.

  1. Record private properties for your project, including details about your {account, linode, etc} which you do not want to make public. This private.properties (et al) will be sourced by other code.

    1. Copying its template== ${PATH_TO_PROJECT}/scripts/private.properties_template (e.g., this) to ${PATH_TO_PROJECT}/scripts/private.properties
    2. Edit ${PATH_TO_PROJECT}/scripts/private.properties (which is bash/text) as directed in the file, and save your values.

install process

  1. Do a base install on your linode: slightly more sophisticated than the base OS install (done from your linode dashboard).

  2. Install and configure OpenVPN:

    1. Run this client pretest "for sanity": if your client networking does not work at this point, you should fix it before proceeding.
    2. Configure server networking.
    3. Install and configure OpenVPN on your server.
    4. Install and configure OpenVPN on your client.
    5. Test your client/server networking.

client connection process

The current process (and its current problem) is described in detail here.

Updated