Answers

Notes:

• a detailed analysis may be found at https://bitbucket.org/carlpulley/challenge11/wiki
• where possible, we have tried to solely use open-source tools - only when the specific need has arisen (e.g. in analysing the stage 2 payload) have we then switched to using the demo versions of commercial tools (i.e. the excellent IDA Pro)
• all computation for this work was performed on a 16 core, 2.4GHz Intel Xeon PowerMac, with additional support from the University of Huddersfield's HPC grid.

Question 1

10.0.0.1 is running a FreeBSD 8.2 operating system with a telnet service supporting encryption. Unfortunately, the telnet service is vulnerable to exploit CVE-2011-4862, a buffer overflow exploit that allows arbitrary code to be executed.

The attacker appears to use an exploit that is derived from the Metasploit FreeBSD Telnet Service Encryption Key ID Buffer Overflow?

The following sequence diagram provides an overview of the sequence of events that triggers this buffer overflow and then allows the (10.0.0.2) attacker's stage 1 code to execute:

Question 2

By saving an FPU environment record, the stage 1 payload executes a getPC sequence. This then leads into two separate XOR decryption loops that reveal the hard coded password #1090F056. This password is used to ensure that the stage 1 code is engaged with an expected server instance.

A series of exchanges between the server and the stage 1 payload now allow an XOR encrypted copy of the stage 2 payload, along with a starting offset and its decryption key, to be downloaded.

The following sequence diagram provides an overview of the sequence of events that download and execute the stage 2 payload:

Question 3

The stage 2 payload may be separated into two phases:

• phase 1: generates a random session key that is communicated (via an exponentiation calculation) to the server and then, in the server's reply packet (another exponentiation calculation), used to communicate a shared initial state for the Rabbit stream cipher (this is extracted using the multiplicative inverse of the randomly generated session key):

• phase 2: downloads (using the Rabbit stream cipher) and verifies (using an Adler32 checksum) the stage 3 payload, before executing it:

Note: the Rabbit stream cipher is only used to encrypt communications from the server to 10.0.0.1. Other than packet #24, all communications from 10.0.0.1 to the server appear to be in plain text.

Question 4

Without cracking the stage 2 encryption, it has not been possible to answer this question accurately. However, based on known prior behaviour patterns, it is possible to make some reasonable estimations.

In examining the encrypted traffic between 10.0.0.1 and its server, we notice that, in packet #30, 10.0.0.1 communicates a null byte to the server. Based on known prior behaviour, we hypothesis that this packet marks entry into the stage 3 payload by writing back the stage 3 payload's entry offset (after it has been zeroed):

Examining the encrypted traffic between 10.0.0.1 and its server, we notice that, in packet #207, 10.0.0.1 communicates a null byte to the server. Again, based on known prior behaviour, we hypothesis that this packet marks entry into the stage 4 payload by writing back the stage 4 payload's entry offset (after it has been zeroed):

Splitting our network packets using this observation, we infer that the stage 3 payload appears to just download 9862735 bytes (in packet's #32-#199) making up the stage 4 payload. As we hypothesis that an entry byte needs to be specified for the stage 4 payload, we note that this byte could be read from this downloaded data. Additionally, as we have previously witnessed Adler32 being used to perform checksumming of downloaded payloads, we presume that it is used again here. So, we speculate that:

• 4 bytes of the data is used for checksumming information
• and 1 byte is used to specify a starting offset.

Thus, we estimate that the stage 4 payload consists of the first 986270 downloaded bytes of packets #32-#199.

Without being able to reconstruct the random session key (by performing our discrete logarithm calculation), this is as far as we can reasonably go here with reconstructing the stage 4 payload.

Question 5

Unable to answer this question.

Question 6

• Avoid signature-based detection by obfuscating binaries
• Impede automated analysis with multi-stage loading and FPU based getPC instruction sequences
• Impede reversing efforts with:
  • use of randomly generated session keys
  • pre-computed discrete logarithms for an unknown factor base (providing coverage for all messages 10.0.0.2 wishes to send to 10.0.0.1)
• Modular structure/design to reduce information leaks by only loading code modules that are needed
• Secure coding practices (e.g. zeroing and repurposing of sensitive variables) to impede memory analysis
• Complex pointer arithmetic and memory manipulations to impede manual reversing efforts
• Use of standard algorithms and code libraries (e.g. use of a big num package and the Rabbit stream cipher) to simplify the codes development/production and to reduce probability of cryptographic programming/design errors.