Source

Psiphon Circumvention System / EmailResponder /

Filename Size Date modified Message
..
1.5 KB
13.5 KB
3.9 KB
2.4 KB
46 B
2.3 KB
1.7 KB
13.5 KB
5.5 KB
1.1 KB
1.3 KB
6.1 KB
3.8 KB
--------------------------------------------------------------------------------
Psiphon Email Autoresponder README
--------------------------------------------------------------------------------


Setup
--------------------------------------------------------------------------------

OS
----------------------------------------

1. Used Ubuntu 11.10 Server 64-bit. AMI IDs can be found via here:
   https://help.ubuntu.com/community/EC2StartersGuide
   Security Group must allow port 25 (SMTP) through (and SSH, so configuration
   is possible.)

2. OS updates
    > sudo apt-get update
    > sudo apt-get upgrade

3. Create a limited-privilege user that will do most of the mail processing.
   Ref: http://www.cyberciti.biz/tips/howto-linux-shell-restricting-access.html
   Add "/usr/sbin/nologin" (sans quotes) to /etc/shells.
    > sudo useradd -s /usr/sbin/nologin email_responder
   * Also create a home directory for the user:
    > sudo mkdir /home/mail_responder
    > sudo chown mail_responder:mail_responder /home/mail_responder

4. Create directory where the responder configuration file and attachments will 
   go.
    > sudo mkdir /mail_responder_config
    > sudo mkdir /mail_responder_config/attachments
   Change the directory permissions so that the "normal" user (that will be used
   for connecting via ssh) can write and all users can read.
    > sudo chown ubuntu:ubuntu -R /mail_responder_config /mail_responder_config/attachments


SSH, fail2ban
----------------------------------------

For extra security, we'll have SSH listen on a non-standard port and use 
fail2ban to prevent brute-force login attempts on SSH. Alternatively/additionally,
EC2 security groups or OS firewall policies can restrict the incoming IPs 
allowed to access the SSH port.

1. Change SSH port number. 
    > sudo nano /etc/ssh/sshd_config
    Change 'Port' value to something random.
    Make sure EC2 security group allows this port through for TCP.

2. Install fail2ban.
    > sudo apt-get install fail2ban
    
3. Configure fail2ban to use non-standard SSH port. 
    Create or edit /etc/fail2ban/jail.local:
      [ssh]
      port    = ssh,<port#>
      [ssh-ddos]
      port    = ssh,<port#>

4. Edit /etc/fail2ban/filter.d/sshd.conf, and add the following line to the 
   failregex list:
    ^%(__prefix_line)spam_unix\(sshd:auth\): authentication failure; logname=\S* uid=\S* euid=\S* tty=\S* ruser=\S* rhost=<HOST>(?:\s+user=.*)?\s*$
   (We found with the Psiphon 3 servers that fail2ban wasn't detecting all the 
   relevant auth.log entries without adding this regex.)
   
5. Restart ssh and fail2ban:
    > sudo service ssh restart
    > sudo service fail2ban restart


Postfix
----------------------------------------

1. Install postfix
    > sudo apt-get install postfix
   During installation:
   * Choose the "Internet Site option".
   * Set the system mail name to the public DNS name of the instance. When 
     we're ready to go live, this will change to the real domain name.

2. Change aliases.
   * Edit /etc/aliases to map postmaster to whatever the "normal" user is 
     (probably "ubuntu").
   * Reload aliases map: > sudo newaliases

3. Add .forward file for "normal" (probably "ubuntu") user. In the file, put 
   the email address that error mail should go to.

4. Edit /etc/postfix/main.cf
   * Change myhostname to be the public DNS name of the instance, or the domain
     name that's pointing to the instance.
   * Change mydestination to be:
        mydestination = $myhostname localhost.$mydomain localhost $extradomains
   * extradomains is a custom variable that will hold all of the domains that 
     our server accepts mail for (and responds to). So, you will have:
        extradomains = example1.com example2.com
     Note that the variable can be missing or empty and that's okay. (Also note
     that I haven't actually tested it with multiple values, so I don't yet
     know if that works.)
   * Change smtpd_use_tls to 'no'
   * Add these two lines to the end. They cause all email that doesn't 
     correspond to a real address to be forwarded to the mail_responder account
     (which will then do the response processing).
        local_recipient_maps =
        luser_relay = mail_responder+$local@localhost
   * Reduce the maximum message size. Something like:
        message_size_limit = 2048000
   * See the bottom of this README for a sample main.cf file.

(Note: If too much error email is being sent to the postmaster, we can also 
add this line: 
notify_classes = 
)

5. By default, if an uncaught error occurs (which shouldn't occur, but...), 
   postfix responds to the user with a bounce email that gives a lot of internal 
   details about the error. This is undesirable, so we'll disable this in postfix.
    > sudo nano /etc/postfix/master.cf
   Comment out the bounce line, so it looks like this:
    #bounce    unix  -       -       -       -       0       bounce

6. Check the contents of /etc/mailname. It should be the FQDN of the current
   instance's public name. If you're using an EC2 Static IP (which you surely
   are), then the value in /etc/mailname is proably the public name that the
   instance had *before* you assigned the static IP (at which point it got a new
   public name). Make sure this is updated to the correct value. 

7. Reload postfix conf and restart:
    > sudo postfix reload
    > sudo /etc/init.d/postfix restart


Logwatch and Postfix-Logwatch
----------------------------------------

1. Install logwatch build-essential
    > sudo apt-get install logwatch
    
2. Install postfix-logwatch.
    - Download current version from: http://logreporters.sourceforge.net/
    - Extract the archive, enter the new directory, and execute:
        > sudo make install-logwatch


Source files and cron jobs
----------------------------------------

The install.sh script does the following:
    - copy files from the source directory to the mail_responder home directory
    - modify those files, if necessary
    - set the proper ownership on those files
    - create the cron jobs needed for the running of the system

The script requires the following package:
> sudo pip install python-crontab

To run the install script:    
> sh install.sh <stats_mail_addr>


Logging
----------------------------------------

* Copy 20-psiphon-logging.conf to /etc/rsyslog.d/
  * This will prevent potentially sensitive user information from ending up in
    the system logs.
  * It also copies the mail responder logs to a dedicated log file that will be
    processed to get statisitics about use.
    
* Enable RFC 3339 compatible high resolution timestamp logging format (required
  for stats processing).
  In /etc/rsyslog.conf, ensure this line is commented out:
    #$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
    
* Turn off "repeated message reduction" (the syslog feature that results in 
  "last message repeated X times" instead of repeated logs). For our stats, we 
  need a line per event, not a compressed view. 
  In /etc/rsyslog.conf, change this:
    $RepeatedMsgReduction on
  to this:
    $RepeatedMsgReduction off
  (TODO: Can this be turned off for only mail_responder.log?)
    
* Restart the logging service:
    > sudo service rsyslog restart


Amazon SES
----------------------------------------

1. Install boto
    > sudo easy_install boto

2. It's best if the AWS user being used for the email sending is created through
   the AWS IAM interface and has limited privileges.

3. Put AWS credentials into boto config file. Info here:
    http://code.google.com/p/boto/wiki/BotoConfig
   We've found that using ~/.boto doesn't work, so create /etc/boto.cfg and put
   these lines into it:
      [Credentials]
      aws_access_key_id = <your access key>
      aws_secret_access_key = <your secret key>
   Ensure that the file is readable by the mail_responder user.


Stats
----------------------------------------

* Stats will be derived from the contents of /var/log/mail_responder.log

* mail_stats.py can be executed by a cron job (installed by install.sh) to email
  basic statisitics to a desired email address. Edit that file so that 
  RECIPIENT_ADDRESS and SENDER_ADDRESS meet your needs.

* The emailing is done with the same SES code and credentials that the 
  responder itself uses.


Blacklist
----------------------------------------

Users will only receive responses to three requests per day (configurable), 
after which they are "blacklisted". The blacklist is cleared once a day 
(configurable).

The blacklist code requires the following package be installed:
> sudo apt-get install mysql-server
> sudo apt-get install python-mysqldb

The install.sh step above does the rest.


DKIM
----------------------------------------

For information about DKIM (DomainKeys Identified Mail) see dkim.org, RFC-4871,
and do some googling. 

A handy DKIM DNS record generator can be found here:
http://www.dnswatch.info/dkim/create-dns-record

A couple of python packages are required:

> sudo easy_install dnspython pydkim

See the DKIM section of settings.py for more values that must be set/changed.



Email Responder Configuration
--------------------------------------------------------------------------------

* The /mail_responder_config/conf.json file contains the addresses that the
  responder will respond to, and the message bodies that will sent back to 
  those addresses. 
  
* /mail_responder_config/conf.json must be writable by the normal (ubuntu) user
  and readable by mail_responder.
  
* The file is in JSON format, with these fields:
{"email_addr": {"body": [["mimetype", "body"], ...], "attachment_bucket": "bucketname"}, ...}

For example:
{
"multipart-email@example.com":
{
  "body": 
    [
      ["plain", "English - https://example.com/en.html\n\u0641\u0627\u0631\u0633\u06cc - https://example.com/fa.html"], 
      ["html", "<a href=\"https://example.com/en.html\">English - https://example.com/en.html</a><br>\u0641\u0627\u0631\u0633\u06cc - https://example.com/fa.html<br>"]
    ],
  "attachment_bucket": null
}, 
"justtext@example.com":
{
  "body": 
    [
      ["plain", "English - https://example2.com/en.html\n\u0420\u0443\u0441\u0441\u043a\u0438\u0439 - https://example2.com/ru.html"]
    ], 
  "attachment_bucket": null
},
"simplebody@example2.com": { "body": "Just a string", "attachment_bucket": null },
"attachment@example3.com": { "body": "I have an attachment", "attachment_bucket": "aaaa-bbbb-cccc-dddd" }
}

* Things to notice about the format:
  * The email body can be a just a string, which will be interpreted as 'plain'
    mimetype, or an array of one or more tuples which are ["mimetype", "body"].
    Mimetypes can be 'plain' or 'html' (so there's really no reason to specify
    more than two).
  * There can be multiple domains served by the same responder server, so the
    whole email address is important.
  * The attachment can be null.
  * There can only be one attachment specified.
  * The attachment file will have to exist at: {attachment_bucket}/{settings.S3_EXE_NAME}

* NOTE: The *last* mimetype will be the one that's preferred by mail clients,
  so you should put the 'html' body last.


Appendix
--------------------------------------------------------------------------------

In order for our responses to not be flagged as spam, these guidelines should be 
followed:
http://docs.amazonwebservices.com/ses/latest/DeveloperGuide/index.html?SPFSenderIDDKIM.html


Possible Issues
--------------------------------------------------------------------------------

* There is no In-Reply-To header being used, so, for example, Gmail's 
  conversation view is broken.



Sample main.cf
--------------------------------------------------------------------------------

# See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $mail_name ESMTP
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=no
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = psiphon3.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
extradomains = saifeng3.com torimizda.com
mydestination = $myhostname localhost.$mydomain localhost ec2-107-22-236-168.compute-1.amazonaws.com $extradomains
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all


# PSIPHON ADDITIONS

# Notify postmaster of all errors
# Note that if this results in too much pointless mail, we can just remove these values.
#notify_classes = bounce, 2bounce, delay, policy, protocol, resource, software
notify_classes = bounce, 2bounce, delay, policy, resource, software

# Add a catch-all for all addresses
local_recipient_maps =
luser_relay = mail_responder+$local@localhost

### Tarpit those bots/clients/spammers who send errors or scan for accounts
smtpd_error_sleep_time = 20
smtpd_soft_error_limit = 1
smtpd_hard_error_limit = 3
smtpd_junk_command_limit = 2

# Reduce the message size limit. There's no reason for large messages to be coming in.
message_size_limit = 2048000