hbbackend / README.txt

== Requirements ==

* JDK 6 Update 24+
* GlassFish 3.1.1b10
* PostgreSQL 9.0.4

== Compiling ==

* Requirements:
 * Apache Ant 1.8.2
 * Apache Ivy 2.2  

1. Retrieve project dependencies:
ant retrieve

2. If the default compile time configuration properties in conf/default.conf won't suffice, copy the file
(e.g. to conf/local.conf) and set them. Set the config.file property when running build targets to specify the custom
configuration file (e.g. ant -Dconfig.file=conf/local.conf).

3. Build all modules simply by running ant. The modules are placed in target/modules.

== Modules ==

Currently, the functional modules are:

submithttp - submit_http handler
maxmindgeoip - looks up geo IP information for every submit_http submission using the MaxMind GeoLite City database [1]
ipfilter - provides unique IPs for other modules
cymruwhois - Team Cymru bulk mode whois query [2]
shadowserverdns - Shadowserver DNS origin query [3]
virustotal - VirusTotal API file report query

[1] http://www.maxmind.com/app/geolitecity
[2] http://www.team-cymru.org/Services/ip-to-asn.html
[3] http://www.shadowserver.org/wiki/pmwiki.php/Services/IP-BGP

The modules are interconnected by the following schema. All destinations are topics, all consumers are durable
subscribers.

submithttp out: new_attack, new_binary
maxmindgeoip in: new_attack out: new_attack_geoip
ipfilter in: new_attack out: new_ip
cymruwhois in: new_ip
shadowserverdns in: new_ip
virustotal in: new_binary

For simplicity, the modules are configured in compile time by the properties in conf/default.conf. Properties that are
not specific to a particular module are prefixed with "main", otherwise module specific options are prefixed with the
module's name.

== Initial Setup ==

1. Create the directory structure. It is recommended to use a similar to the following one. E.g.:

mkdir /tmp/hbbackend/
mkdir /tmp/hbbackend/main/ # the main storage directory for binaries and corresponding files (main.storagedir)
mkdir /tmp/hbbackend/upload/ # for work in progress file uploads
mkdir /tmp/hbbackend/geoip/ # for geo IP databases
mkdir /tmp/hbbackend/log/ # logs
mkdir /tmp/hbbackend/conf/ # for configurations files run time configuration files (logger configuration)
mkdir /tmp/hbbackend/xadisk/ # XADisk's working directory

2. Create the database by executing schema/hbbackend.sql (tables) and schema/hbbackend_functions.sql (functions and
triggers). E.g.:

psql < schema/hbbackend.sql
psql < schema/hbbackend_functions.sql

3. Download and uncompress the MaxMind GeoLite City database into the appropriate location. E.g.:

cd /tmp/hbbackend/geoip/
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

4. Create GlassFish domain. Throughout the following steps, the name "hbbackend" with a port base of 9990 will be
used. E.g.:

asadmin create-domain --portbase 9900 hbbackend

5. Copy library dependencies into the domain library directory.

cp lib/postgresql-9.0-*.jdbc4.jar \
	lib/xadisk-*.jar \
	lib/slf4j-api-1.6.*.jar \
	lib/logback-core-0.9.*.jar \
	lib/logback-classic-0.9.*.jar\
	lib/concurrentlinkedhashmap-lru-1.2.jar \
<path>/glassfish3/glassfish/domains/hbbackend/lib/

6. Copy the logback configuration file conf/logback.xml to e.g. /tmp/hbbackend/conf/ and customize the log destination
files properties if the defaults won't suffice.

cp conf/logback.xml /tmp/hbbackend/conf/

7. Start asaddmin with the appropriate port (e.g. asadmin --port 9948) and run the following commands to set up the
domain including JMS resources. All following commands assume values given in the above, otherwise default values. The
PostgreSQL role is assumed to be "hbbackend" with the password "hbbackend".

# start
start-domain hbbackend

## postgres
# connection pool
create-jdbc-connection-pool --datasourceclassname org.postgresql.xa.PGXADataSource --restype javax.sql.XADataSource --property user=hbbackend:password=hbbackend:databaseName=hbbackend:serverName=localhost:port=5432 PgPool
ping-connection-pool PgPool
# jdbc resource
create-jdbc-resource --connectionpoolid PgPool jdbc/hbbackend

## xadisk
create-threadpool --minthreadpoolsize=5 --maxthreadpoolsize=50 hbbackend-xadisk-thread-pool
# must restart domain for the thread pool to become available
restart-domain hbbackend 
create-resource-adapter-config --threadpoolid hbbackend-xadisk-thread-pool --property xaDiskHome=/tmp/hbbackend/xadisk:instanceId=hbbackend xadisk
deploy --name xadisk lib/xadisk-1.1.rar
create-connector-connection-pool --raname xadisk --connectiondefinition org.xadisk.connector.outbound.XADiskConnectionFactory --property instanceId=hbbackend --transactionsupport XATransaction xadisk/ConnectionFactory
ping-connection-pool xadisk/ConnectionFactory
create-connector-resource --poolname xadisk/ConnectionFactory xadisk/ConnectionFactory

## jms
# stomp bridge
set configs.config.server-config.jms-service.jms-host.default_JMS_host.property.imq\\.bridge\\.enabled=true
set configs.config.server-config.jms-service.jms-host.default_JMS_host.property.imq\\.bridge\\.activelist=stomp
set configs.config.server-config.jms-service.jms-host.default_JMS_host.property.imq\\.bridge\\.admin\\.user=admin
set configs.config.server-config.jms-service.jms-host.default_JMS_host.property.imq\\.bridge\\.admin\\.password=admin
set configs.config.server-config.jms-service.jms-host.default_JMS_host.property.imq\\.bridge\\.stomp\\.tcp\\.port=9972
# disable autocreate
set configs.config.server-config.jms-service.jms-host.default_JMS_host.property.imq\\.autocreate\\.queue=false
set configs.config.server-config.jms-service.jms-host.default_JMS_host.property.imq\\.autocreate\\.topic=false
# destinations
create-jmsdest --desttype topic new_attack
create-jmsdest --desttype topic new_attack_geoip
create-jmsdest --desttype topic new_binary
create-jmsdest --desttype topic new_ip
create-jms-resource --restype javax.jms.Topic --property Name=new_attack jms/new_attack
create-jms-resource --restype javax.jms.Topic --property Name=new_attack_geoip jms/new_attack_geoip
create-jms-resource --restype javax.jms.Topic --property Name=new_binary jms/new_binary
create-jms-resource --restype javax.jms.Topic --property Name=new_ip jms/new_ip
# connection factories
create-jms-resource --restype javax.jms.ConnectionFactory jms/ConnectionFactory
create-jms-resource --restype javax.jms.ConnectionFactory --property ClientId=testdurable jms/TestDurableConnectionFactory
create-jms-resource --restype javax.jms.ConnectionFactory --property ClientId=ipfilter jms/DurableConsumer/ipfilter
create-jms-resource --restype javax.jms.ConnectionFactory --property ClientId=cymruwhois jms/DurableConsumer/cymruwhois
create-jms-resource --restype javax.jms.ConnectionFactory --property ClientId=shadowserverdns jms/DurableConsumer/shadowserverdns
create-jms-resource --restype javax.jms.ConnectionFactory --property ClientId=maxmindgeoip jms/DurableConsumer/maxmindgeoip
create-jms-resource --restype javax.jms.ConnectionFactory --property ClientId=virustotal jms/DurableConsumer/virustotal
ping-connection-pool jms/ConnectionFactory

# logback config location
create-jvm-options -Dlogback.configurationFile=/tmp/hbbackend/conf/logback.xml

# monitoring
enable-monitoring --modules connector-connection-pool=HIGH:connector-service=HIGH:deployment=HIGH:ejb-container=HIGH:http-service=HIGH:jdbc-connection-pool=HIGH:jms-service=HIGH:jvm=HIGH:thread-pool=HIGH:transaction-service=HIGH:jms-service=HIGH:web-container=HIGH

# finally, restart once again for a clean start
restart-domain hbbackend

== Deploying Modules ==

Generally, the order of deployment should be the reverse of the flow of the system. Consumers of a destination should be
deployed before producers to avoid losing messages. For durable consumers it's only critical at the first deployment,
but following the rule is always a safe bet. Accordingly, submithttp should be the last in the row. When deploying for
the first time, check the broker that all durable subscriptions are created before deploying submithttp.

E.g.:

deploy target/modules/org.honeynet.hbbackend.virustotal.jar
deploy target/modules/org.honeynet.hbbackend.cymruwhois.jar
deploy target/modules/org.honeynet.hbbackend.shadowserverdns.jar
deploy target/modules/org.honeynet.hbbackend.ipfilter.jar
deploy target/modules/org.honeynet.hbbackend.maxmindgeoip.jar
deploy target/modules/org.honeynet.hbbackend.submithttp.war
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.