OK I think I figured it out.
I edited some sysctl values and it looks like it messes up the network stability.
My advice: do not touch net.ipv4.tcp_tw_reuse or net.ipv4.tcp_tw_recycle as mentionned here:
Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki (unless you know what you're doing).