[SOLVED] Error running mailboxd after script backup

[ttortosa]

Hello,

I'm using this script for do a daily backup in our Zimbra Installation (Debian 4.0, 5.0.8. Network Edition).

# Stop Zimbra Services
before2="$(date +%s)"
echo "Parant Zimbra"
sudo -u zimbra /opt/zimbra/bin/zmcontrol stop

sleep 20
 
# Sync to backup directory
echo "Fent backup"
rsync -avHK --delete /opt/zimbra/ /backups/zimbra > /backups/rsync.log

# Restart Zimbra Services

echo "Iniciant Zimbra"
sudo -u zimbra /opt/zimbra/bin/zmcontrol start
# Calculates and outputs amount of time the server was down for
 after="$(date +%s)"
 elapsed="$(expr $after - $before2)"
 hours=$(($elapsed / 3600))
 elapsed=$(($elapsed - $hours * 3600))
 minutes=$(($elapsed / 60))
 seconds=$(($elapsed - $minutes * 60))
 echo El servidor ha estat fora de servei durant: "$hours hores $minutes minuts $seconds segons"


sudo -u zimbra zmcontrol -v > /backups/zimbra/zimbra_version.txt

echo Mostrant estat de Zimbra...

sudo -u zimbra /opt/zimbra/bin/zmcontrol status

tar -zcvf /tmp/mail.backup.tgz -C /backups/zimbra .
ncftpput -u our_ftp_server -p our_password IP /backup_zimbra /tmp/mail.backup.tgz
rm -R /backups/zimbra/*
rm /tmp/mail.backup.tgz
cp /var/log/zimbra.log /backups/zimbra.log
echo Hora de backup finalitzada = $(date +%T)

This is a customized script very similar to open source scripts for backup in zimbra wiki.

I can see in the zimbra log this, just after zimbra start in the script:

Oct 20 00:34:07 obertic zimbramon[7949]: 7949:err: Service status change: obertic.dedi.acens.net mailboxd changed from running to stopped 
Oct 20 00:34:07 obertic zmmailboxdmgr[31287]: stale pid 11337 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Oct 20 00:34:07 obertic zmmailboxdmgr[31287]: assuming no other instance is running
Oct 20 00:34:07 obertic zmmailboxdmgr[31287]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Oct 20 00:34:07 obertic zmmailboxdmgr[31287]: assuming no other instance is running
Oct 20 00:34:07 obertic zmmailboxdmgr[31287]: no manager process is running
Oct 20 00:34:07 obertic zmmailboxdmgr[31285]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Oct 20 00:34:07 obertic zmmailboxdmgr[31285]: assuming no other instance is running
Oct 20 00:34:07 obertic zmmailboxdmgr[31285]: no manager process is running
Oct 20 00:34:07 obertic zimbramon[12931]: 12931:err: Service status change: obertic.dedi.acens.net mailboxd changed from running to stopped 
Oct 20 00:34:07 obertic zimbramon[10383]: 10383:err: Service status change: obertic.dedi.acens.net mailboxd changed from running to stopped 
Oct 20 00:34:08 obertic postfix/postdrop[31441]: warning: uid=1008: No space left on device
Oct 20 00:34:08 obertic postfix/postdrop[31443]: warning: uid=1008: No space left on device
Oct 20 00:34:08 obertic postfix/postdrop[31440]: warning: uid=1008: No space left on device
Oct 20 00:34:08 obertic postfix/postdrop[31422]: warning: uid=1008: No space left on device
Oct 20 00:34:08 obertic postfix/postdrop[31424]: warning: uid=1008: No space left on device
Oct 20 00:34:08 obertic postfix/postdrop[31425]: warning: uid=1008: No space left on device
Oct 20 00:34:08 obertic postfix/sendmail[31417]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:08 obertic postfix/sendmail[31418]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31323]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31371]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/postdrop[31423]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/sendmail[31320]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31324]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/postdrop[31429]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/sendmail[31322]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31416]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/postdrop[31426]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/sendmail[31319]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/postdrop[31430]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31431]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31433]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31432]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31428]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31436]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31434]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31438]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31427]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31439]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/postdrop[31437]: warning: uid=1008: No space left on device
Oct 20 00:34:09 obertic postfix/sendmail[31373]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31407]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31410]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31409]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31412]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31372]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31411]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic postfix/sendmail[31413]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:09 obertic zmmailboxdmgr[31550]: status requested
Oct 20 00:34:09 obertic zmmailboxdmgr[31550]: stale pid 11337 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Oct 20 00:34:09 obertic zmmailboxdmgr[31550]: assuming no other instance is running
Oct 20 00:34:09 obertic postfix/sendmail[31321]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31558]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31557]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/sendmail[31414]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/sendmail[31415]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic zmmailboxdmgr[31550]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Oct 20 00:34:10 obertic zmmailboxdmgr[31550]: assuming no other instance is running
Oct 20 00:34:10 obertic zmmailboxdmgr[31550]: no manager process is running
Oct 20 00:34:10 obertic postfix/postdrop[31556]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31553]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31555]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31554]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic zimbramon[10383]: 10383:err: Service status change: obertic.dedi.acens.net mailbox changed from running to stopped 
Oct 20 00:34:10 obertic zmmailboxdmgr[31538]: status requested
Oct 20 00:34:10 obertic zmmailboxdmgr[31538]: stale pid 11337 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Oct 20 00:34:10 obertic zmmailboxdmgr[31538]: assuming no other instance is running
Oct 20 00:34:10 obertic zmmailboxdmgr[31538]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Oct 20 00:34:10 obertic zmmailboxdmgr[31538]: assuming no other instance is running
Oct 20 00:34:10 obertic zmmailboxdmgr[31538]: no manager process is running
Oct 20 00:34:10 obertic postfix/postdrop[31576]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/postdrop[31552]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/postdrop[31595]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31593]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31591]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31590]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31580]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31575]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31569]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31568]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31578]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31567]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31581]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31570]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31583]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31577]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31613]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/postdrop[31589]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31587]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31604]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31601]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/sendmail[31551]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic zmmailboxdmgr[31530]: status requested
Oct 20 00:34:10 obertic postfix/sendmail[31572]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/sendmail[31607]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31600]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31573]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic zmmailboxdmgr[31530]: stale pid 11337 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Oct 20 00:34:10 obertic zmmailboxdmgr[31530]: assuming no other instance is running
Oct 20 00:34:10 obertic zmmailboxdmgr[31530]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Oct 20 00:34:10 obertic zmmailboxdmgr[31530]: assuming no other instance is running
Oct 20 00:34:10 obertic postfix/postdrop[31605]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic zmmailboxdmgr[31530]: no manager process is running
Oct 20 00:34:10 obertic postfix/sendmail[31586]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31596]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31594]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31609]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31606]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic postfix/postdrop[31592]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31579]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic zmmailboxdmgr[31549]: status requested
Oct 20 00:34:10 obertic zmmailboxdmgr[31549]: stale pid 11337 found in /opt/zimbra/log/zmmailboxd_manager.pid: No such process
Oct 20 00:34:10 obertic zmmailboxdmgr[31549]: assuming no other instance is running
Oct 20 00:34:10 obertic zmmailboxdmgr[31549]: file /opt/zimbra/log/zmmailboxd.pid does not exist
Oct 20 00:34:10 obertic zmmailboxdmgr[31549]: assuming no other instance is running
Oct 20 00:34:10 obertic zmmailboxdmgr[31549]: no manager process is running
Oct 20 00:34:10 obertic zimbramon[7949]: 7949:err: Service status change: obertic.dedi.acens.net mailbox changed from running to stopped 
Oct 20 00:34:10 obertic postfix/postdrop[31599]: warning: uid=1008: No space left on device
Oct 20 00:34:10 obertic postfix/sendmail[31598]: fatal: zimbra(1008): queue file write error
Oct 20 00:34:10 obertic zimbramon[12931]: 12931:err: Service status change: obertic.dedi.acens.net mailbox changed from running to stopped

....

Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: antispam: Running 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: antivirus: Running 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: archiving: Running 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: imapproxy: Stopped 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: ldap: Running 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: logger: Running 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: mailbox: Stopped 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: mta: Running 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: snmp: Running 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: spell: Running 
Oct 20 00:34:13 obertic zimbramon[30904]: 30904:info: 2008-10-20 00:34:02, STATUS: obertic.dedi.acens.net: stats: Stopped

That only happens some days, and we discover it on the morning (after 8 hours from backup script has been executed). I wonder if the problem is related to disk space...

This shows my df command:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             10325780   5723280   4077980  59% /
tmpfs                   517736         0    517736   0% /lib/init/rw
udev                     10240        52     10188   1% /dev
tmpfs                   517736         4    517732   1% /dev/shm
/dev/sda2               101105     12356     83528  13% /boot
/dev/sda7             10325748    731560   9069668   8% /home
/dev/sda6             15488716   2245184  12456752  16% /usr
/dev/sda5            116555916   3803200 106831964   4% /var

[uxbod]

Yep I would say so as you ran out of space. Now some has been cleaned up what happens if you completely shutdown ZCS and then start it ?

[ttortosa]

Thank you uxbod

When I stop and start Zimbra it works fine, except imapproxy that says

bind(): Address already in use
failed to listen
nginx is running
memcached is not running

Mailboxd is started and working.

There's no space error messages in the zimbra log.

Maybe I must change the steps order in the backup and wait to start Zimbra for the erase of the temporary backup files?

[uxbod]

You are tar'ing to the root mountpoint so if that runs out of space it will effect ZCS Why not move /backups too /var/backups as you appear to have plenty of free space there

[ttortosa]

OK! I will try with /var/backups directory

Thanks very much, uxbod

[ttortosa]

Originally Posted by uxbod

You are tar'ing to the root mountpoint so if that runs out of space it will effect ZCS Why not move /backups too /var/backups as you appear to have plenty of free space there

These two days the backup has worked fine, and there isn't any problem. Thank you very much, uxbod.