raw_logs mysql table corruption - how to recover?

[mrdini]

Hello all,

Recently, I noticed I was having issues with the Zimbra logger. As a result, I found this wiki page.

So I issued a REPAIR TABLE statement, and after 5+ hours wait, I decided to investigate, and found I'd run into the exact same issue as outlined in this MySQL bug report. My du/df returns exactly the same results as outlined in that bug (an empty and yet full /tmp/ directory)

Doing a tail servername.com.err returns several lines similar to the below..

071219 15:56:04 [ERROR] /opt/zimbra/logger/mysql/libexec/mysqld: Disk is full writing '/tmp/STYAnvMQ' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs

Any thoughts? Right now, I'm EXTREMELY loath to abort the "REPAIR TABLE" process, yet I am pretty sure it's stuck...

TIA!

[jholder]

Are you out of space in /tmp?

[mrdini]

Originally Posted by jholder

Are you out of space in /tmp?

No.

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             51625340   2993668  46010232   7% /
none                   1046016         0   1046016   0% /dev/shm
/dev/sda3             51614140  25925372  23067328  53% /opt
/dev/sda4              2015824   2004140         0 100% /tmp
# du -s /tmp
52      /tmp

[jholder]

Look closer.

Yes you are.

[mrdini]

Originally Posted by jholder

Look closer.

Yes you are.

So how come du -s (at the bottom of my paste in my previous post) reports only 52K used up?

In any case, if I AM out of space, how should I progress? Kill "REPAIR TABLE"?

[jholder]

Not sure on the -s switch. Man indicates:
"-s Display an entry for each specified file. (Equivalent to -d 0)"

/tmp isn't a file, it's a directory. It's possible that i'm wrong on that (actually it's likely), but your df clearly indicates that /tmp is 100% full.

Can you free up space in /tmp? If you do, it should redo the operation, fixing it.

[mrdini]

Originally Posted by jholder

Not sure on the -s switch. Man indicates:
"-s Display an entry for each specified file. (Equivalent to -d 0)"

/tmp isn't a file, it's a directory. It's possible that i'm wrong on that (actually it's likely), but your df clearly indicates that /tmp is 100% full.

Can you free up space in /tmp? If you do, it should redo the operation, fixing it.

du -s = summarise du.

Let me run it without du -s & you'll see it's not full

# du /tmp/
4       /tmp/.ICE-unix
4       /tmp/hsperfdata_zimbra
20      /tmp/zimbra/afb99765-2950-47e2-9ae7-a95cc575e374-deferred
24      /tmp/zimbra
4       /tmp/hsperfdata_root
52      /tmp/

Convinced yet?

[mmorse]

Deleted files not freeing their handle in /tmp

[jholder]

Nope.

Let's think through this logically:
1) You try to repair a table and get an error: "Disk is full writing '/tmp/STYAnvMQ' (Errcode: 28). Waiting for someone to free space..."
2) I ask if you're out of space on /tmp
3) You say no, but display a df which says that /tmp is 100% full
4) df also says that there is zero k left on that mount

Which is more likely:
You're out of space on /tmp which is indicated by mysql and df?
or
You're misinterpreting the results of du -s ?

[mmorse]

For instance:

Originally Posted by Centurion

According to the standard system tools:

[root@mail tmp]# df -h
Filesystem            Size  Used Avail Use% Mounted on
...snipped...
/dev/md3              3.0G  1.8G  1.1G  63% /tmp

But looking at the /tmp directory with "du" reveals....

[root@mail tmp]# du -hs /tmp
820K    /tmp

Digging deeper with "lsof" we see the following:

[root@mail tmp]# lsof | grep "/tmp"
bash       1970    root  cwd       DIR        9,3       4096          2 /tmp
screen    13507   jgray    3r     FIFO        9,3                 15747 /tmp/uscreens/S-jgray/13507.pts-0.mail
perl      19932  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
perl      19932  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
zmlogger  19937  zimbra    4w      REG        9,3 1884374605         26 /tmp/zmlogger.out (deleted)
mysqld_sa 21321  zimbra    1w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
mysqld_sa 21321  zimbra    2w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
logswatch 21391  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
logswatch 21391  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
mysqld    21402  zimbra    5u      REG        9,3          0         20 /tmp/ibD4jO0l (deleted)
mysqld    21402  zimbra    6u      REG        9,3          0         21 /tmp/ibzy1fOB (deleted)
mysqld    21402  zimbra    7u      REG        9,3          0         22 /tmp/ibXukIBR (deleted)
mysqld    21402  zimbra    8u      REG        9,3          0         23 /tmp/ibJj1dq7 (deleted)
mysqld    21402  zimbra   12u      REG        9,3          0         25 /tmp/iblBE0Vn (deleted)
perl      21423  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
perl      21423  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
zmlogger  21446  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
zmlogger  21446  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
zmlogger  21446  zimbra    4w      REG        9,3     373318         27 /tmp/zmlogger.out
mysqld_sa 21877  zimbra    1w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
mysqld_sa 21877  zimbra    2w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
mysqld    21972  zimbra    5u      REG        9,3       3075         28 /tmp/ibagshl0 (deleted)
mysqld    21972  zimbra    6u      REG        9,3          0         29 /tmp/ibY1LZ2w (deleted)
mysqld    21972  zimbra    7u      REG        9,3          0         30 /tmp/ibakKIK3 (deleted)
mysqld    21972  zimbra    8u      REG        9,3          0         31 /tmp/ib4yAKFB (deleted)
mysqld    21972  zimbra   12u      REG        9,3          0         32 /tmp/ibk4EXla (deleted)
java      22007  zimbra  mem       REG        9,3      32768     299138 /tmp/hsperfdata_zimbra/22007
zmtomcatm 22790    root    1w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
zmtomcatm 22790    root    2w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
java      22791  zimbra  mem       REG        9,3      32768     330626 /tmp/hsperfdata_root/22791
swatch    23099  zimbra    1w      REG        9,3         79         34 /tmp/swatch.out
swatch    23099  zimbra    2w      REG        9,3         79         34 /tmp/swatch.out
perl      23117  zimbra    1w      REG        9,3         79         34 /tmp/swatch.out
perl      23117  zimbra    2w      REG        9,3         79         34 /tmp/swatch.out
lsof      27703    root  cwd       DIR        9,3       4096          2 /tmp
bash      27704    root  cwd       DIR        9,3       4096          2 /tmp
lsof      27705    root  cwd       DIR        9,3       4096          2 /tmp
logswatch 28631  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
logswatch 28631  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
sshd      32234   jgray    7u     unix 0xc67d0380              98791141 /tmp/ssh-JGggs32234/agent.32234

BTW, don't know if your profile is current (4.5.6_GA_1044.RHEL4_64_20070706142644 CentOS4_64) but 4.5.10 is out