Nu Data Scraper

Simple script that connects to the nu RPCJson server and gets information from it.
Every 30 seconds the getliquidityinfo call is made.
All other calls are made only when a new block has been received.

Setup

install all dependancies

apt - git, python-pip, postgresql, libpq-devi, python-dev, build-essential, apache2, libapache2-mod-wsg pip - web.py, pygresql
manual - https://github.com/jgarzik/python-bitcoinrpc, nud

copy nud to /usr/bin/

create user nu

This user will run nud and needs the nu-block.py, block-checker.py and nu-liquidity.py files in their home directory. these files need to be chowned to nu

nu.conf:

rpcuser=[user]
rpcpassword=[password]
listen=1
daemon=1
server=1
addnode=98.52.160.71
addnode=198.52.217.4
addnode=198.52.199.75
addnode=198.52.199.46
addnode=162.242.208.43
addnode=192.237.200.146
addnode=119.9.75.189
addnode=119.9.12.63
rpcallowip=127.0.0.1
block-notify=python block-checker.py

git clone nu-data and distribute files.

block-checker.py, nu-liquidity.py and nu-block.py go in /home/nu also run

    chown nu /home/nu/nu-block.py  
    chown nu /home/nu/nu-liquidity.py   
    chown nu /home/nu/block-checker.py

code.py goes in new directory /var/www/nu-historical-data

Configure Apache

add these lines to /etc/apache2/sites-available/000-default.conf after the DocumentRoot declaration:

    WSGIScriptAlias /nu-historical-data /var/www/nu-historical-data/code.py/
    AddType text/html .py

Reload apache config

    service apache2 reload

Set up database

This was useful (http://www.yolinux.com/TUTORIALS/LinuxTutorialPostgreSQL.html)

change to postgres user

su postgres

create database

createdb nu-historical-data

login to database

psql nu-historical-data

create data table

CREATE TABLE data (id SERIAL PRIMARY KEY, timestamp TEXT, command TEXT, value TEXT);

create reader user

CREATE USER nu_reader WITH PASSWORD '[password]';

grant privileges (SELECT) and usages

GRANT SELECT ON data to nu_reader;
GRANT USAGE, SELECT ON SEQUENCE data_id_seq TO nu_reader;

create update user 1

CREATE USER nu_liquidity WITH PASSWORD '[password]';

grant privileges (SELECT, INSERT) and usages

GRANT SELECT, INSERT ON data to nu_liquidity;
GRANT USAGE, SELECT ON SEQUENCE data_id_seq TO nu_liquidity;

create update user 2

CREATE USER nu_block WITH PASSWORD '[password]';

grant privileges (SELECT, INSERT) and usages

GRANT SELECT, INSERT ON data to nu_block;
GRANT USAGE, SELECT ON SEQUENCE data_id_seq TO nu_block;

Put the correct authentication and server details for rpc and database in the various python files.

Set the block check environment variable

add NU_BLOCK_PID="0" to /etc/environment then run

    source /etc/environment

run nu-liquidity.py

    nohup python nu-liquidity.py &

Hit up your server for data