ibdata1 filled up root

I got caught out on our piler server when ibdata1 used all the free space on root. It’s my own fault for not monitoring root. I’m not sure if I lost data. I suspect we have since the last write date on ibdata1 was March 10th.

We are running piler 1.3.11 build 1001. I think I built piler from source and not an appliance.

I ended up bandaid fixing the problem by mounting a new volume, copying /var/lib/mysql to the new volume and remounting it to /var/lib/mysql

This is what it looks like now:

[root@piler mysql]# ls -lathr /var/lib/mysql
total 47G
drwx------. 2 mysql mysql 6 Dec 16 2020 test
drwx------. 2 mysql mysql 4.0K Dec 16 2020 mysql
drwx------. 2 mysql mysql 4.0K Dec 16 2020 performance_schema
drwx------ 2 mysql mysql 4.0K Dec 16 2020 piler
drwxr-xr-x. 37 root root 4.0K Dec 17 2020 ..
-rw-rw----. 1 mysql mysql 52 Mar 22 14:12 aria_log_control
-rw-rw----. 1 mysql mysql 16K Mar 22 14:12 aria_log.00000001
srwxrwxrwx 1 mysql mysql 0 Mar 22 14:28 mysql.sock
drwxr-xr-x 6 mysql mysql 190 Mar 22 14:28 .
-rw-rw----. 1 mysql mysql 5.0M Mar 22 14:44 ib_logfile1
-rw-rw----. 1 mysql mysql 47G Mar 22 14:45 ibdata1
-rw-rw----. 1 mysql mysql 5.0M Mar 22 14:45 ib_logfile0

Next step will be to get caught up with the MSFT Journal and try to fixup ibdata1 with some downtime.

I hope no one else get caught by this in the future.

‌

Comments (2)