1. Thorn Technologies Avatar Thorn Technologies
  2. Products
  3. sftpgateway-public
  4. Issues

Files Detected, Not transferring to S3

Issue #19 new
Samuel Dixon created an issue

Interesting issue here. I have a user who's files are not uploading to s3. It appears that the directories are detected properly, however they are not transferring to s3. I have used the users ssh key and was able to initiate a transfer myself which makes this issue a bit odd.

Please see the below logs and let me know if you will be able to assist or if you have seen this issue before. I have sanitized some of the user information.

incrontab -l 

/home/<user>/home/<user>/uploads/835 IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/835 $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/835/ $%
/home/<user>/home/<user>/uploads/835/HMHP IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/835/HMHP $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/835/HMHP/ $%
/home/<user>/home/<user>/uploads/835/MHPN IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/835/MHPN $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/835/MHPN/ $%
/home/<user>/home/<user>/uploads/835/WCOH IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/835/WCOH $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/835/WCOH/ $%
/home/<user>/home/<user>/uploads/835/CMHP_WCOH IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/835/CMHP_WCOH $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/835/CMHP_WCOH/ $%
/home/<user>/home/<user>/uploads/837 IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/837 $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/837/ $%
/home/<user>/home/<user>/uploads/837/HMHP IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/837/HMHP $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/837/HMHP/ $%
/home/<user>/home/<user>/uploads/837/WCOH IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/837/WCOH $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/837/WCOH/ $%
/home/<user>/home/<user>/uploads/837/MHPN IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/837/MHPN $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/837/MHPN/ $%
/home/<user>/home/<user>/uploads/837/CMHP_SWOH IN_CREATE,IN_DELETE,IN_CLOSE_WRITE /usr/local/bin/movetos3.sh /home/<user>/home/<user>/uploads/837/CMHP_SWOH $# sftpgateway-i-05eff45d7572f93ce/<user>/uploads/837/CMHP_SWOH/ $%

/var/log/cron

Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/835/CMHP_WCOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/HMHP - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/WCOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/MHPN - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/CMHP_SWOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: table for user root changed, reloading
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/835/CMHP_WCOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/HMHP - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/WCOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/MHPN - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/CMHP_SWOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: table for user root changed, reloading
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/835/CMHP_WCOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/HMHP - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/WCOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/MHPN - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: access denied on /home/<user>/home/<user>/uploads/837/CMHP_SWOH - events will be discarded silently
Jan 26 14:23:21 ip-10-1-12-226 incrond[2559]: cannot create watch for user root: (2) No such file or directory
Jan 26 15:01:01 ip-10-1-12-226 CROND[1815]: (root) CMD (run-parts /etc/cron.hourly)
Jan 26 15:01:01 ip-10-1-12-226 run-parts(/etc/cron.hourly)[1815]: starting 0anacron
Jan 26 15:01:01 ip-10-1-12-226 run-parts(/etc/cron.hourly)[1824]: finished 0anacron

Directory file permissions:

drwxr-xr-x 2 <user> <user> 4096 Jan 26 15:52 EPIC
drwxr-xr-x 2 <user> <user> 4096 Jan 24 12:32 SERIES
drwxr-xr-x 2 <user> <user> 4096 Jan 23 18:14 STAR_CMHP_SWOH
drwxr-xr-x 2 <user> <user> 4096 Jan 23 18:10 STAR_HMHP

File permissions in directories:

-rw-r--r-- 1 <user> <user> 175249238 Jan 25 21:58 file.zip

I am a bit confused on how there are permission issues. Any assistance would be helpful.

Comments (4)

  2. Robert Chen

    Can you provide the /var/log/movetos3/movetos3.log file? (feel free to redact the contents, and/or email the file to support@thorntech.com)

    Thanks! Robert

  4. Robert Chen

    So, what strikes me is that you were able to initiate a transfer from your end. I would like to confirm whether the issue is with the user's SFTP client. Here's a few tests you can try:

    Let's say the directory in question is:

    /home/<user>/home/<user>/uploads/835/HMHP

    Try creating a file at that location:

    sudo su
cd /home/<user>/home/<user>/uploads/835/HMHP/
touch test1.txt

    The file should create, get moved to S3, and then get deleted (this should all happen instantaneously). If not, let me know (e.g. if the test1.txt file remains on the Linux file system)

    If that's working, try looking for files within the subdirectories of that 835/HMHP folder. For example, you might see:

    -rw-r--r-- 1 <user> <user> 175249238 Jan 25 21:58 file.zip

    Then search for file.zip within the /var/log/movetos3/movetos3.log file:

    cd /var/log/movetos3/
grep -iR file.zip .

    What I'm looking for in particular are file extensions, e.g. file.zip.filepart, or file.zip.tmp. Let me know if you come across something like this.

    Depending on what you find out, we can troubleshoot from there.

    Thanks! Robert

  5. Log in to comment
Assignee
–
Type
bug
Priority
major
Status
new
Votes
0
Watchers
1