pilerimport | error parsing the configfile: /usr/local/etc/piler.conf

[Former user]

Setup

piler 0.1.25-master-branch, build 845, Janos SUTO [e-mail]

Build Date: Sat Sep 28 14:41:12 CEST 2013 ldd version: ldd (Debian EGLIBC 2.13-38) 2.13 gcc version: gcc version 4.7.2 (Debian 4.7.2-5) Configure command: ./configure --localstatedir=/var --enable-starttls --enable-tcpwrappers --with-database=mysql

Debian Wheezy, fresh install

Problem

The web interface is running fine. When I wanted to import email via pilerimport, I always get the same error

error parsing the configfile: /usr/local/etc/piler.conf
ERR: reading cipher key

I did recreate the cipher key via

dd if=/dev/urandom bs=56 count=1 of=/usr/local/etc/piler.key
chmod 640 /usr/local/etc/piler.key

It did not help.

Under "health status" I can see that there is a

SMTP status: piler: ERROR

which I cannot figure out.

Any more info needed to troubleshoot this?

[Janos SUTO]

Try setting PILER_HOST in config-site.php to solve the smtp status error. And try running pilerimport in a directory where the user piler has read and write permissions, eg. /tmp

[Former user]

Thanks for your reply.

I couldn't check the PILER_HOST setting yet but what solved the parsing error turned out to be correct permissions. All in all I needed to run

chown piler:piler /usr/local/etc/piler.conf
dd if=/dev/urandom bs=56 count=1 of=/usr/local/etc/piler.key
chmod 640 /usr/local/etc/piler.key
chown piler:piler /usr/local/etc/piler.key

to make pilerimport work.

However, when running in a Docker container (Ubuntu Server 14.04), importing does not appear to work at all. I just get an endless array of this:

failed to import: 12425-imap-1.txt (id: 4000000053ab4233175e1fd400c2448c041d)
error importing '12425-imap-1.txt'
failed to import: 12425-imap-2.txt (id: 4000000053ab42331955060400d55692b410)
error importing '12425-imap-2.txt'
failed to import: 12425-imap-3.txt (id: 4000000053ab42331b64851400f994f1d7af)
error importing '12425-imap-3.txt'

This happens in the /tmp directory as well.

When I run the same setup within a full VirtualBox virtual machine (Ubuntu Server 14.04 minimal), it works.

Any ideas?

[Janos SUTO]

I'm not sure what a docker container is, but make sure that user piler has read/write rights on the current directory.

[Former user]

To make sure, I created a new directory with mkdir import && chmod 777 import and ran pilerimport in there. Same result.

Info about docker can be found here: http://www.docker.com

It's the talk of the town and especially handy for development environments and application distribution and cloud scaling.

[Janos SUTO]

Yes, in the meantime I've checked their site. Regarding the import, is there any clues in the mail log?

[Former user]

Sorry for the late reply, I've been swamped.

Today, I have been able to completely get piler to work inside an Ubuntu 14.04 Docker container when I started from scratch, following a guide I prepared documenting my first attempt. Having it not working before was probably due to my own fault. Whatever that was.

Now I have problems with viewing the actual imported mail. I ran pilerimport, then created a regular user. When logging in as that user, I cannot see any imported mail, although there are several thousand mails already imported. The built-in auditor user cannot see anything at all as well. What could now be broken – any ideas?

Here's the documentation/guide I created (which also reflects every step I did):

Piler Install on Ubuntu Server 14.04 LTS

Update the system

apt-get update && apt-get upgrade

Prepare MariaDB repository

apt-get install software-properties-common
apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db
add-apt-repository 'deb http://mirror.netcologne.de/mariadb/repo/5.5/ubuntu trusty main'
apt-get update

Install needed packages

apt-get install openssl libssl-dev apache2 php5 php5-gd php5-mysql mariadb-server libmariadbclient-dev php5-ldap memcached php5-memcache sphinxsearch libtre-dev libzip2 libzip-dev sysstat gcc libwrap0 libwrap0-dev catdoc libpoppler44 poppler-utils unrtf make tnef

Start services

service mysql start
service apache2 start

Enable Apache mod_rewrite

a2enmod rewrite

Add piler group and user

groupadd piler
useradd -g piler -s /bin/sh -d /var/piler piler
usermod -L piler

Get piler source code and compile

cd /tmp
wget https://bitbucket.org/jsuto/piler/downloads/piler-0.1.25-rc2.tar.gz
tar xfvz piler-0.1.25-rc2.tar.gz
cd piler-0.1.25-rc2
./configure --localstatedir=/var --with-database=mysql --enable-starttls --enable-tcpwrappers --enable-memcache
make
make install

Install piler Apache site template

cp contrib/webserver/piler-apache-2.x.conf /etc/apache2/sites-available/000-default.conf
sed -i -e 's/piler.yourdomain.com/<DOMAIN_NAME>/' /etc/apache2/sites-available/000-default.conf
sed -i -e 's/apach2/apache2/' /etc/apache2/sites-available/000-default.conf

Set up piler

make postinstall

Example output

This is the postinstall utility for piler
It should be run only at the first install. DO NOT run on an existing piler installation!


Continue? [Y/N] [N] y


Please enter the path of your documentroot [/var/www/ba6c3596bbd8] /var/www/piler.example.com

Please enter the webserver groupname [apache] www-data

Please enter mysql hostname [localhost]

Please enter mysql socket path [/var/run/mysqld/mysqld.sock]

Please enter mysql database [piler]

Please enter mysql user name [piler]

Please enter mysql password for piler []

Please enter mysql root password []
mysql connection successful


Please enter the path of sphinx.conf [/etc/sphinxsearch/sphinx.conf]

Please enter smtp relay []

Please enter smtp relay port [25]
no crontab for piler



INSTALLATION SUMMARY:

piler user: piler
keyfile: /usr/local/etc/piler.key

mysql host: localhost
mysql socket: /var/run/mysqld/mysqld.sock
mysql database: piler
mysql username: piler
mysql password: *******

sphinx indexer: /usr/bin/indexer
sphinx config file: /etc/sphinxsearch/sphinx.conf

vhost docroot: /var/www/piler.example.com
www group: www-data

smtp relay host:
smtp relay port: 25

piler crontab:
### PILERSTART
5,35 * * * * /usr/local/libexec/piler/indexer.delta.sh
30   2 * * * /usr/local/libexec/piler/indexer.main.sh
*/15 * * * * /usr/bin/indexer --quiet tag1 --rotate
*/15 * * * * /usr/bin/indexer --quiet note1 --rotate
30   6 * * * /usr/bin/php /usr/local/libexec/piler/generate_stats.php --webui /var/www/piler.example.com
*/5 * * * * /usr/bin/find /var/www/piler.example.com/tmp -type f -name i.\* -exec rm -f {} \;
### PILEREND



Correct? [Y/N] [N] y

Continue and modify system? [Y/N] [N] y

Creating mysql database... Done.
Writing sphinx configuration... Done.
Initializing sphinx indices... Sphinx 2.0.4-id64-release (r3135)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/etc/sphinxsearch/sphinx.conf'...
indexing index 'main1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.017 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'main2'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.005 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'main3'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.002 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'main4'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.002 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'dailydelta1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.003 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'delta1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.009 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'tag1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.002 sec, 0 bytes/sec, 0.00 docs/sec
indexing index 'note1'...
collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.002 sec, 0 bytes/sec, 0.00 docs/sec
total 8 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 40 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
Done.
installing cron entries for piler... Done.
installing keyfile (piler.key) to /usr/local/etc/piler.key... Done.
Making an ssl certificate ... Generating a 4096 bit RSA private key
................................................................................................................................................................................................++
................................................++
writing new private key to '/usr/local/etc/piler.pem'
-----
Copying www files to /var/www/piler.example.com... Done.

Done post installation tasks.

Restart Apache

service apache2 restart

Edit piler web settings

/var/www/<DOMAIN_NAME>/config-site.php

Run piler and sphinx

ldconfig
/etc/init.d/rc.piler start
/etc/init.d/rc.searchd start

Fix permissions

chown piler:piler /usr/local/etc/piler.conf

Create cypher key

dd if=/dev/urandom bs=56 count=1 of=/usr/local/etc/piler.key
chmod 640 /usr/local/etc/piler.key
chown piler:piler /usr/local/etc/piler.key

[Janos SUTO]

Nice docs :-) Please check if you have actual data under /var/piler/sphinx and /var/piler/store/00. Then try to retrieve a single email with pilerget. It expects a piler id as an argument. You can see it in the maillog (search for 400000.... strings) or in the mysql metadata table. Also try "mysql -h 127.0.0.1 -P 9306", and "select * from main1,dailydelta1,delta1;". It should return at least 20 hits. Finally just hit the search button, and check the sphinx query in maillog whether it's correct. If nothing helps, let's arrange a teamviewer session.

[Former user]

I just researched Sphinx more and found that reindexing the Sphinx index by running indexer --rotate --all does the trick.

However, when I manually run the cronjob /usr/bin/indexer --quiet --rotate postdelta threaddelta, all messages disappear again and running indexer --rotate --all shows that there is no data at all.

Here's the output of the manual cronjob run:

WARNING: no such index 'postdelta', skipping.
WARNING: no such index 'threaddelta', skipping.

I'm stumped.

[Janos SUTO]

Running "indexer --all" is a mistake. It resets the sphinx data, so don't do it. See the FAQ section Q5. The mysql errors are also concerns, they should be fixed.

[Janos SUTO]

Any update on the matter?

[Janos SUTO]

No news is good news.