pilerexport | ssh | pilerimport
Issue #1132
resolved
I would like to move our archive of 40 million emails from our old piler server to our new piler server. Does pilerexport support writing to STDOUT?
Comments (26)
-
repo owner
-
reporter
Thanks for this. The old server is running 1.1.1 and the new server is running 1.3.9. Do I have to upgrade both to latest or can I compile the latest piler and copy over pilerexport and pilerimport and use those on the older versions?
-
repo owner
No need to upgrade from 1.1.1 to 1.3.9. I suggest to apply this patch to 1.1.1 source code, and recompile. Let me know if you need a hand doing so.
-
reporter
Instructions on how to apply the patch would help me out. Also, does pilerimport need a patch?
-
repo owner
I see it now. Yes, pilerimport also needs a patch to make this work. I’ll let you know when I’m ready with the complete feature.
-
repo owner
I’ve updated the master branch to let pilerimport read exported data by pilerexport from stdin.
You may try something like pilerexport -o -A | ssh otherhost pilerimport -y
Notes:
- You may need the -t option for ssh
- The ssh user’s home dir on the remote host where pilerimport will write some temp files must be read-writable by user piler.
I’ll attach the modified pilerexport.c for 1.1.1 release soon. So try it, and let me know how it goes. It might be a nice section in the FAQ.
-
repo owner
- attached pilerexport.c
Try overwriting piler-1.1.1/src/pilerexport.c with this file and recompile.
-
reporter
Export worked fine.
-
reporter
As for pilerimport, can I just download the latest, run ./configure; make and then copy over the binary manually into /usr/local/bin ?
-
reporter
[root@wdc-piler1 jsuto-piler-3729ed1ecfe6]# make
Making all in src
make[1]: Entering directory `/home/karlr/piler-mater/jsuto-piler-3729ed1ecfe6/src'
gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2 -Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -c dirs.c -o dirs.o
gcc: error: unrecognized command line option ‘-Wimplicit-fallthrough=2’
make[1]: *** [dirs.o] Error 1
make[1]: Leaving directory `/home/karlr/piler-mater/jsuto-piler-3729ed1ecfe6/src'
make: *** [all-recursive] Error 1
-
repo owner
Yes, you can just download the latest, however don’t just overwrite the pilerimport binary, run a complete make install. Assuming this is an empty host at the moment. Also your gcc (nad perhaps the OS itself) is a bit old and doesn’t know the -Wimplicit-fallthrough compiler option. Anyway, you can fix it by finding all generated Makefiles, and removing the ‘-Wimplicit-fallthrough=2’ string at their beginnings.
-
reporter
Installed gcc7 and got the following on yesterday’s master branch.
pilerexport.c:17:10: fatal error: zip.h: No such file or directory
#include <zip.h>
^
~~compilation terminated.
make[1]: *** [Makefile:63: pilerexport] Error 1
make[1]: Leaving directory '/home/karlr/piler-mater/jsuto-piler-3729ed1ecfe6/src'
make: *** [Makefile:41: all-recursive] Error 1
-
repo owner
You need the libzip-dev or similar package.
-
reporter
pilerexport.c: In function ‘write_to_zip_file’:
pilerexport.c:331:4: error: unknown type name ‘zip_source_t’; did you mean ‘in_port_t’?
zip_source_t *zs = zip_source_file(z, filename, 0, 0);
^
~~~in_port_t
pilerexport.c:331:23: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
zip_source_t *zs = zip_source_file(z, filename, 0, 0);
^
~~~~~~pilerexport.c:332:13: warning: implicit declaration of function ‘zip_file_add’; did you mean ‘zip_fread’? [-Wimplicit-function-declaration]
if(zs && zip_file_add(z, filename, zs, ZIP_FL_ENC_UTF_8) >= 0){
^
~~~zip_fread
pilerexport.c:332:43: error: ‘ZIP_FL_ENC_UTF_8’ undeclared (first use in this function); did you mean ‘ZIP_FL_ENCRYPTED’?
if(zs && zip_file_add(z, filename, zs, ZIP_FL_ENC_UTF_8) >= 0){
^
~~~ZIP_FL_ENCRYPTED
pilerexport.c:332:43: note: each undeclared identifier is reported only once for each function it appears in
make[1]: *** [Makefile:63: pilerexport] Error 1
make[1]: Leaving directory '/home/karlr/piler-mater/jsuto-piler-3729ed1ecfe6/src'
make: *** [Makefile:41: all-recursive] Error 1
-
reporter
libzip-devel.x86_64 0.10.1-8.el7
-
repo owner
What Linux distro is this?
-
reporter
CentOS Linux release 7.9.2009 (Core)
-
repo owner
I’ve updated the master branch to support gcc 4 and 0.10.x libzip in centos7. Please get the master branch and recompile.
-
reporter
Works. Thanks!
-
repo owner
Great, let me know if you can verify that the export-import over ssh feature works as expected.
-
reporter
The export/import seems to be complaining some.
I haven’t kept track of the errors that closely. Console is showing:
Syntax Error (490804): Dictionary key must be a name object
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Catalog object is wrong type (null)
Syntax Error: Couldn't read page catalog
Syntax Error: Couldn't read xref table
Syntax Warning: Invalid Font Weight
500000005fea57201a4f198c004dd9890876.a1.bin is not OLE file or Error
-
repo owner
These are warnings only from the used attachment extractor utilities. We can’t do much about it. Did you need to specify any extra command line options for ssh?
-
repo owner
- changed status to resolved
I conclude that the issue is resolved by now.
-
reporter
I upgraded to piler-1.3.10 and i’m now getting the following error
-sh-4.1$ pilerexport --start-date 2019.01.01 --stop-date 2020.01.01 -o |ssh -t newpiler pilerimport -y
Pseudo-terminal will not be allocated because stdin is not a terminal.
error: didnt write 1 bytesshould pilerimport -y work with piler-1.3.10?
-
repo owner
Try without the -t option.
-
reporter
removing -t worked.
- Log in to comment
Just added a commit to enable it. Check out https://bitbucket.org/jsuto/piler/commits/2b4f27372d550a3ebeafe691f55bab3417fc8556 and let me know if it works as expected. In a nutshell add -o option, and pilerexport will dump data to stdout.