Trouble with NE (Standard) Backup

[Jesster]

Hello,

Last night something odd happened during our backup (which was very close to finishing).

I happened to get a few commands in before/during/after the problem occurred. Essentially what happened -- a NE standard backup is running, and my script polls the zmbackupquery roughly every 5 minutes. At some point, the command returned a Connection Refused, then an error message. I logged in to verify everything was up and running - including the ldapmaster side of things. In the end, Zimbra does not see any backups on the system, but looking closer at the /opt/zimbra/backup filesystem, I see a backup. Any chance of getting Zimbra to recognize this backup?

Command History Follows:

#
# Mail Store Output
# (uptime, zmbackupquery, sleep 300, followed by ^C and other commands)
00:51:39 up 6 days, 23 min, 1 user, load average: 9.37, 8.06, 8.52

Label: full-20080915.015948.711
Type: full
Status: in progress
Started: Sun, 2008/09/14 18:59:48.711 PDT
Redo log sequence range: 1322 .. 1568
Currently backing up: 50165@example.org
Number of accounts: 11531
11174 out of 11531 accounts have been backed up


00:56:42 up 6 days, 28 min, 1 user, load average: 10.21, 16.31, 12.61

Label: full-20080915.015948.711
Type: full
Status: in progress
Started: Sun, 2008/09/14 18:59:48.711 PDT
Redo log sequence range: 1322 .. 1569
Currently backing up: 50109@example.org
Number of accounts: 11531
11177 out of 11531 accounts have been backed up


01:01:44 up 6 days, 33 min, 1 user, load average: 16.05, 13.89, 12.29

Error occurred: Connection refused

01:06:54 up 6 days, 38 min, 1 user, load average: 1.73, 7.11, 9.89

Error occurred: backup full-20080915.015948.711 not found: /opt/zimbra/backup/sessions/full-20080915.015948.711 not found


[zimbra@mailstore ~]$ zmcontrol status
Host webmail.example.org
mailbox Running
mta Running
snmp Running
spell Running
stats Running

[zimbra@mailstore ~]$ while true; do uptime; echo; zmbackupquery -lb full-20080915.015948.711; echo; sleep 300; done
01:12:05 up 6 days, 43 min, 1 user, load average: 4.31, 4.42, 7.84

Error occurred: backup full-20080915.015948.711 not found: /opt/zimbra/backup/sessions/full-20080915.015948.711 not found


[zimbra@mailstore ~]$ zmbackupquery
No backups found

[zimbra@mailstore ~]$ df -h /opt/zimbra/backup
Filesystem Size Used Avail Use% Mounted on
/dev/sde1 914G 454G 414G 53% /opt/zimbra/backup
[zimbra@mailstore ~]$



[zimbra@mailstore ~]$ ls -al /opt/zimbra/backup/
total 32
drwxr-xr-x 5 zimbra zimbra 4096 Sep 14 18:59 .
drwxr-xr-x 62 root root 4096 Sep 14 18:57 ..
drwx------ 2 root root 16384 Sep 14 18:52 lost+found
drwxr-x--- 2 zimbra zimbra 4096 Sep 14 18:59 sessions
drwxr-x--- 3 zimbra zimbra 4096 Sep 14 18:59 tmp
[zimbra@mailstore ~]$ ls -al /opt/zimbra/backup/tmp
total 12
drwxr-x--- 3 zimbra zimbra 4096 Sep 14 18:59 .
drwxr-xr-x 5 zimbra zimbra 4096 Sep 14 18:59 ..
drwxr-x--- 5 zimbra zimbra 4096 Sep 14 18:59 full-20080915.015948.711
[zimbra@mailstore ~]$ ls -al /opt/zimbra/backup/sessions/
total 8
drwxr-x--- 2 zimbra zimbra 4096 Sep 14 18:59 .
drwxr-xr-x 5 zimbra zimbra 4096 Sep 14 18:59 ..
[zimbra@mailstore ~]$ mailstore


#
# LDAP Master Commands During this error
#
================================================== ==============

[root@ldapmaster ~]# su - zimbra
[zimbra@ldapmaster ~]$ ldap status
slapd running pid: 21072
[zimbra@ldapmaster ~]$ ps wuax |grep slapd
zimbra 21072 69.4 17.2 519692 361024 ? Ssl Aug28 19415:20 /opt/zimbra/openldap/libexec/slapd -l LOCAL0 -4 -u zimbra -h ldap://zldap.example.org:389 -f /opt/zimbra/conf/slapd.conf

[zimbra@ldapmaster ~]$ top

top - 01:10:45 up 81 days, 8:20, 1 user, load average: 1.69, 1.52, 1.47
Tasks: 72 total, 3 running, 69 sleeping, 0 stopped, 0 zombie
Cpu(s): 60.2%us, 38.7%sy, 0.0%ni, 1.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2097344k total, 2049360k used, 47984k free, 147864k buffers
Swap: 1001448k total, 24k used, 1001424k free, 1602976k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21072 zimbra 21 0 507m 352m 195m S 100 17.2 19415:45 slapd