Issue #55 closed
Jan Galkowski
created an issue
Filesystem                            1K-blocks       Used  Available Use% Mounted on
/dev/mapper/vg_sys-lv_sys_root          8125880    1962036    5728032  26% /
devtmpfs                                7962000          0    7962000   0% /dev
tmpfs                                   7984924          0    7984924   0% /dev/shm
tmpfs                                   7984924      25200    7959724   1% /run
tmpfs                                   7984924          0    7984924   0% /sys/fs/cgroup
/dev/md0                                1031064     192404     786284  20% /boot
/dev/mapper/vg_sys-lv_sys_var          10190136     887020    8762444  10% /var
/dev/mapper/vg_sys-lv_sys_home         20511356     187036   19259360   1% /home
/dev/mapper/vg_sys-lv_sys_vartmp        5029504     224108    4526868   5% /var/tmp
/dev/mapper/vg_sys-lv_sys_tmp           5029504      10264    4740712   1% /tmp
//u141408.your-storagebox.de/backup 10485760000 5737588713 4748171287  55% /media/jan-one
//u141128.your-storagebox.de/backup 10485760000 4600785123 5884974877  44% /media/datarefuge
tmpfs                                   1596988          0    1596988   0% /run/user/1003
tmpfs                                   1596988          0    1596988   0% /run/user/1006
/dev/mapper/vg_pool-lv_data_jan      4227420072 2049108740 1985023552  51% /home/jan/local_data
/dev/mapper/vg_pool-lv_data_maxwell  2113646512      81944 2006174004   1% /home/maxwell/local_data
[jan@azi01 doe-eia]$ sudo lsof | grep 'maxwell-one'
lsof: WARNING: can't stat() cifs file system /media/maxwell-one
      Output information may be incomplete.
bash      28509      maxwell  cwd   unknown                                         /media/maxwell-one/INCOMING (stat: Permission denied)
htop      28528      maxwell  cwd   unknown                                         /media/maxwell-one/INCOMING (stat: Permission denied)
bash      28529      maxwell  cwd   unknown                                         /media/maxwell-one/INCOMING (stat: Permission denied)
[jan@azi01 doe-eia]$ sudo cd /media/maxwell-one
/bin/cd: line 2: cd: /media/maxwell-one: Not a directory
[jan@azi01 doe-eia]$

Comments (38)

  1. Sakari Maaranen

    Yeah, it's definitely the same problem. We need to migrate away from the Storage Box CIFS mounts. I am starting a third root server, so we should have enough capacity.

    Please use local storage on azi02 until this issue is resolved. Do not use the azi01:/media/* CIFS mounts for anything else but moving data away from there.

  2. Jan Galkowski reporter

    There now enough room on azi03 /home/maxwell/local_data to accommodate /media/maxwell-one:

    [jan@azi03 ~]$ sudo df -k
    Filesystem                            1K-blocks       Used  Available Use% Mounted on
    /dev/mapper/vg_sys-lv_sys_root          8125880    1814728    5875340  24% /
    devtmpfs                                7963432          0    7963432   0% /dev
    tmpfs                                   7985140          0    7985140   0% /dev/shm
    tmpfs                                   7985140      25132    7960008   1% /run
    tmpfs                                   7985140          0    7985140   0% /sys/fs/cgroup
    /dev/md0                                 498980     180499     292296  39% /boot
    /dev/mapper/vg_sys-lv_sys_var          10190136     214956    9434508   3% /var
    /dev/mapper/vg_sys-lv_sys_home         20511356      48116   19398280   1% /home
    /dev/mapper/vg_sys-lv_sys_tmp           5029504      10292    4740684   1% /tmp
    /dev/mapper/vg_sys-lv_sys_vartmp        5029504      10232    4740744   1% /var/tmp
    /dev/mapper/vg_pool-lv_data_maxwell  2992041504    2844064 2867559744   1% /home/maxwell/local_data
    /dev/mapper/vg_pool-lv_data_jan      8460161116 5981201840 2135081404  74% /home/jan/local_data
    //u141477.your-storagebox.de/backup  5242880000 1660155312 3582724688  32% /media/maxwell-one
    tmpfs                                   1597032          0    1597032   0% /run/user/1003
    //u141408.your-storagebox.de/backup 10485760000 5737588713 4748171287  55% /media/jan-one
    tmpfs                                   1597032          0    1597032   0% /run/user/1006
    
  3. Sakari Maaranen

    Scott is only using 1.6T of maxwell-one. He has 2.7T free in his local_data. It fits very well.

    Last login: Sun Jan  8 20:38:30 2017 from REDACTED
    [sam@azi03 ~]$ sudo df -hT
    Filesystem                          Type      Size  Used Avail Use% Mounted on
    ...
    /dev/mapper/vg_pool-lv_data_maxwell ext4      2.8T  2.8G  2.7T   1% /home/maxwell/local_data
    
    /dev/mapper/vg_pool-lv_data_jan     ext4      7.9T  5.6T  2.0T  74% /home/jan/local_data
    
    //u141477.your-storagebox.de/backup cifs      4.9T  1.6T  3.4T  32% /media/maxwell-one
    
    //u141408.your-storagebox.de/backup cifs      9.8T  5.4T  4.5T  55% /media/jan-one
    
  4. marsroverdriver

    I'm not sure what would account for such different results. I also get 1.6TB when I df -h /media/maxwell-one, but (a) the rsync to my azi03:local_data that supposedly has more space than that failed with an out of space error, and (b) rsyncing to azi02:local_data has already copied >1.9TB and is still going.

    I do not immediately understand this. I wonder if this might be connected to the odd behavior we've seen from the storage boxes already.

  5. Sakari Maaranen

    @marsroverdriver Please confirm you have deleted possible duplicate / unfinished copies. For example, if you were successful on azi02, please make sure the same partial data has been erased from azi03. So we get a good picture of how much we actually have and don't have unwanted redundancy.

  6. Jan Galkowski reporter

    I'm going to finish off all pending tasks and then take a look at possible redundanices, including sizes. I might use rsync -c to copy over any potentially missing files. But this will be a day or more into the future.

    Meanwhile, focus today and tomorrow will be on monitoring, and cleaning up /media/datarefuge.

  7. marsroverdriver

    I have

    • Finished the rsync from /media/maxwell-one to azi02:local_data.
    • Deleted /media/maxwell-one.
    • Deleted the partial copy of /media/maxwell-one in azi03:local_data.
    • Resumed PODAAC download on azi02.
    • Tied off climate.nasa.gov backup. (It was completed, just waiting for SHA-256 checksums and the like. Will update issue later today, after work.)
    • Deleted my partial oceandata backup, since Jan was also doing that one.

    ... Um, I think that's it for this morning.

  8. Sakari Maaranen

    Awesome! We have now pretty much sorted out the mess those storage boxes gave us. We could do even more with our team especially now that everything is back in order.

    I am now killing off maxwell-one to terminate the cost.

    If I recall right, Jan still wanted to double-check something with jan-one(?).

  9. Jan Galkowski reporter

    Yes, I've a comparison working against /media/jan-one. Sry, thought it would have been finished by now. Just confirmed it is still crankin'. Progress is being recorded in /home/jan/local_data/jan-one/rsync.listing.

  10. marsroverdriver

    @Sakari Maaranen I don't see the process you refer to:

    [maxwell@azi03 ~]$ ps auxw | fgrep maxwell
    root     24968  0.0  0.0 145044  5076 ?        Ss   03:58   0:00 sshd: maxwell [priv]
    maxwell  24970  0.0  0.0 145044  2192 ?        S    03:58   0:00 sshd: maxwell@pts/0
    maxwell  24971  0.0  0.0 115384  2024 pts/0    Ss   03:58   0:00 -bash
    maxwell  24992  0.0  0.0 151056  1836 pts/0    R+   03:58   0:00 ps auxw
    maxwell  24993  0.0  0.0 112648   892 pts/0    S+   03:58   0:00 grep -F --color=auto maxwell
    

    AFAIK, you can delete maxwell-one forever.

  11. Jan Galkowski reporter

    The rsync of /media/jan-one/ has been completed.

    It is permissible to expunge /media/jan-one/ at this time.

    I'll leave this ticket open until that gets done, documented in a comment, and then the ticket can be closed.

  12. Log in to comment