pilerimport entire directory

[Former user]

Hi, I have just downloaded the VM image. I compiled libpst-0.6.54, used readpst to get mail into a readable format for pilerimport.

I output all the folders within the PST to a directory, then issued: pilerimport -d /dir/name

It issued output like this:

This is UnRTF, version 0.19.2 By Dave Davey and Marcos Serrou do Amaral Original Author: Zach T. Smith Processing 4000000051005bb62ce766449f2f0046e9ab.a1.bin... Done. 240 words were hashed. This is UnRTF, version 0.19.2 By Dave Davey and Marcos Serrou do Amaral Original Author: Zach T. Smith Processing 4000000051005bb62ce766449f2f0046e9ab.a3.bin... Done.

Then finishes with "processed: 13". On the web interface when I check the health/health page it shows 13 mails have been imported - but I cannot search for any of them.

Any idea what I'm doing wrong?

I ran readpst from /var/mail/pst which I created earlier and chown'ed permissions to piler:piler and the pilerimport from the same directory.

[Janos SUTO]

OK, as the 1st step, please connect to sphinx, and verify it's indexed all the 13 messages.

$ mysql --host 127.0.0.1 -P 9036 mysql> select * from main1;

you should see 13 rows returned.

[ally_wilson]

Hi, So, after posting I thought might have done something incorrect when performing the readpst part (so I reran it [to a new directory] with "readpst -D -o /var/mail/pst/MyName/ /var/mail/pst/MyName.pst").

I then performed the "pilerimport -d /var/mail/pst/MyName/" command again, and i got a lot more information returned. It import >2,000 of the emails (instead of just the 13 I mentioned earlier).

Ever hopeful, I issued: "reindex -f 1 -t 2000 -p" ...but still nothing in the search.

I've just issued the command you suggested above, here is the output (this is the very last command I have issued prior to commenting, it shows 20):

mysql> select * from main1
    -> ;
+------+------------+------------+-----------+-----------+--------+-------------+
| id   | arrived    | sent       | size      | direction | folder | attachments |
+------+------------+------------+-----------+-----------+--------+-------------+
|    1 | 1358968001 | 1347863852 |   3547803 |         1 |      0 |           9 |
|    2 | 1358968003 |          0 |    311943 |         0 |      0 |           0 |
|    3 | 1358968003 | 1347890319 |   5284095 |         0 |      0 |          12 |
|    4 | 1358968005 | 1348840928 |  61180367 |         1 |      0 |          15 |
|    5 | 1358968086 | 1347894560 |  16711692 |         1 |      0 |          11 |
|    6 | 1358968092 | 1346765146 |    127218 |         0 |      0 |           3 |
|    7 | 1358968092 | 1332783271 | 123083682 |         1 |      0 |          15 |
|    8 | 1358968712 | 1347793067 |    461046 |         3 |      0 |           3 |
|    9 | 1358968713 | 1347873929 |   8539381 |         3 |      0 |          15 |
|   10 | 1358968715 | 1347866568 |   2521520 |         1 |      0 |           8 |
|   11 | 1358968717 | 1347631998 |   2059253 |         0 |      0 |           6 |
|   12 | 1358968717 | 1347973107 |   2853196 |         1 |      0 |           4 |
|   13 | 1358968718 | 1347793496 |   5734299 |         1 |      0 |          15 |
|   14 | 1358978722 | 1347905997 |   1309360 |         1 |      0 |           1 |
|   15 | 1358978722 | 1347963417 |      9750 |         1 |      0 |           0 |
|   16 | 1358978722 | 1347963347 |      8971 |         1 |      0 |           0 |
|   17 | 1358978723 | 1347963298 |    299602 |         1 |      0 |           2 |
|   18 | 1358978723 | 1347954703 |    832088 |         0 |      0 |           1 |
|   19 | 1358978723 | 1347956237 |    833348 |         0 |      0 |           1 |
|   20 | 1358978724 | 1347955640 |    835893 |         0 |      0 |           1 |
+------+------------+------------+-----------+-----------+--------+-------------+
20 rows in set (0.00 sec)

[ally_wilson]

After leaving it over night, I performed the following this morning.

Disabled the reindex cronjobs then:

root@myhost:/var/mail/pst# reindex -f 1 -t 2191

put 2191 messages to sph_index table for reindexing

root@myhost:/var/mail/pst# /usr/local/bin/indexer delta1 --rotate

Sphinx 2.0.6-release (r3473) Copyright (c) 2001-2012, Andrew Aksyonoff Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/usr/local/etc/sphinx.conf'... indexing index 'delta1'... collected 2191 docs, 5.8 MB sorted 2.7 Mhits, 100.0% done total 2191 docs, 5801551 bytes total 2.292 sec, 2530873 bytes/sec, 955.80 docs/sec total 3 reads, 0.005 sec, 3272.9 kb/call avg, 1.7 msec/call avg total 27 writes, 0.013 sec, 740.3 kb/call avg, 0.4 msec/call avg rotating indices: successfully sent SIGHUP to searchd (pid=1279).

root@myhost:/var/mail/pst# /usr/local/bin/indexer --merge main1 delta1 --merge-dst-range deleted 0 0 --rotate

Sphinx 2.0.6-release (r3473) Copyright (c) 2001-2012, Andrew Aksyonoff Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/usr/local/etc/sphinx.conf'... merging index 'delta1' into index 'main1'... read 0.7 of 0.7 MB, 100.0% done merged 112.4 Kwords merged in 0.572 sec total 6215 reads, 0.011 sec, 3.7 kb/call avg, 0.0 msec/call avg total 44 writes, 0.006 sec, 231.1 kb/call avg, 0.1 msec/call avg rotating indices: successfully sent SIGHUP to searchd (pid=1279).

But there's still nothing showing up in the search function on the website, and the command you had me run previously to show the sphinx data still has just those 20 rows.

[ally_wilson]

I've just found the commandline tool to search (/usr/local/bin/search) and that seems to be working, seems to find all the mail and displays matches...

Instead of using the master admin account I created myself, I logged in with the default auditor account - and it works fine!

Sorry for wasting your time about this.

[Janos SUTO]

OK, no problem. I'm glad that it works :-)