OutOfMemoryError with new java?

[simonpg]

I have /opt/zimbra/java -> /usr/lib/j2sdk1.5-sun. I run zmprov normally and get: zimbra@xanadu:~/bin$ zmprov
Exception in thread "main" java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:203)
at java.util.jar.JarFile.(JarFile.java:132)
at java.util.jar.JarFile.(JarFile.java:70)
at sun.misc.URLClassPath$JarLoader.getJarFile(URLClas sPath.java:579)
at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:546)
at sun.misc.URLClassPath$3.run(URLClassPath.java:324)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java: 313)
at sun.misc.URLClassPath.getLoader(URLClassPath.java: 290)
at sun.misc.URLClassPath.getResource(URLClassPath.jav a:160)
at java.net.URLClassLoader$1.run(URLClassLoader.java: 192)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.j ava:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:3 06)
at java.lang.ClassLoader.loadClass(ClassLoader.java:2 99)
at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:2 51)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:319)

If I run zmlocalconfig, all is fine:

zimbra@xanadu:~/bin$ zmlocalconfig
ldap_host = ldap.thesimonet.org
ldap_is_master = false
ldap_master_url =
ldap_port =
ldap_root_password = *
ldap_url =
logger_mysql_bind_address = localhost
logger_mysql_data_directory = ${zimbra_home}/logger/db/data
...

If I run com.zimbra.cs.localconfig.Main using the opened up defines that zmprov uses, I get the same error:

zimbra@xanadu:~/bin$ java -client -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib com.zimbra.cs.localconfig.Main
Exception in thread "main" java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:203)
at java.util.jar.JarFile.(JarFile.java:132)
at java.util.jar.JarFile.(JarFile.java:70)
at sun.misc.URLClassPath$JarLoader.getJarFile(URLClas sPath.java:579)
at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:546)
at sun.misc.URLClassPath$3.run(URLClassPath.java:324)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java: 313)
at sun.misc.URLClassPath.getLoader(URLClassPath.java: 290)
at sun.misc.URLClassPath.getResource(URLClassPath.jav a:160)
at java.net.URLClassLoader$1.run(URLClassLoader.java: 192)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.j ava:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:3 06)
at java.lang.ClassLoader.loadClass(ClassLoader.java:2 99)
at sun.misc.Launcher$AppClassLoader.loadClass(Launche r.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:2 51)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:319)

Any pointers would be appreciated.

[anand]

Add -verbose:load to the Java command line. Or run it under strace. It would be interesting to see which jar file is causing the VM to trip over itself.

[anand]

btw what is your OS, hardware, zimbra version etc?

[simonpg]

If you would like the full dumps, let me know where to send them. The following line worked: /opt/zimbra/java/bin/java -client -Dzimbra.home=/opt/zimbra -cp /opt/zimbra/lib/zimbrastore.jar:/opt/zimbra/lib/commons-logging.jar:/opt/zimbra/lib/commons-cli-2.0.jar:/opt/zimbra/lib/dom4j-1.5.jar:/opt/zimbra/lib/log4j-1.2.8.jar:/opt/zimbra/lib/commons-httpclient-2.0.1.jar -verbose com.zimbra.cs.localconfig.Main (Note: 'man java' showed no -verbose:load, so I choose just -verbose). It loaded a few more classes:

Good load:
[Loaded sun.misc.ExtensionDependency from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.security.PrivilegedActionException from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.util.zip.ZipEntry from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
Bad load:
[Loaded sun.misc.ExtensionDependency from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.util.zip.ZipEntry from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
Good load:
[Loaded java.lang.Math from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded sun.misc.Resource from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded sun.misc.URLClassPath$JarLoader$1 from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.lang.Package from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.util.jar.Manifest from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
Bad load:
[Loaded java.lang.Math from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.util.jar.Manifest from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]


The following line did not work: /opt/zimbra/java/bin/java -client -Dzimbra.home=/opt/zimbra -Djava.library.path=/opt/zimbra/lib -Djava.ext.dirs=/opt/zimbra/java/jre/lib/ext:/opt/zimbra/lib -verbose com.zimbra.cs.localconfig.Main The class loading really started to differ here:

Good load:
[Loaded sun.misc.ASCIICaseInsensitiveComparator from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.util.jar.JarVerifier from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.io.ByteArrayOutputStream from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded sun.security.util.ManifestEntryVerifier from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.security.Provider from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded sun.security.provider.Sun from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.security.Provider$ServiceKey from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.security.Provider$EngineDescription from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]


Bad load:
[Loaded sun.misc.ASCIICaseInsensitiveComparator from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.util.jar.JarVerifier from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.io.ByteArrayOutputStream from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.io.IOException from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.io.FileNotFoundException from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded java.security.PrivilegedActionException from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded sun.reflect.ClassFileConstants from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]
[Loaded sun.reflect.AccessorGenerator from /usr/lib/j2sdk1.5-sun/jre/lib/rt.jar]

As to particulars, Debian testing on AMD64 using ZCS 3.0 M2 source. Java provided using jdk-1_5_0_06-linux-amd64.bin. The following mod to get_plat_tag.sh allows ZimbraBuild to work very well.
grep "testing" /etc/debian_version > /dev/null 2>&1
if [ $? = 0 ]; then
echo "DEBIAN3.1"
exit 0
fi

Kudos to the Zimbra team for an already amazing build system!

[anand]

-verbose:load output looks ok - atleast the part you posted.

Don't have 64 bit JVM here to try.

How how strace to see what few files the JVM touches before the error?

Is it the case that 32-bit JVM from zimbra works ok but 64-bit JVM does not? In that case I would look into ZimbraNative - a little bit of our native code and see if that needs recompiling.

[simonpg]

Found that "attach files" button so I have attached straces and a build output showing that ZimbraNative gets built along with everything else. Then did ./install.sh and it still blows up at the same spot. My environment is "pure64", so no 32-bit. I do have other machines; are you suggesting that I install jdk-1_5_0_06-linux-i586.bin and see if that works?

[anand]

yes, use the 32 bit JDK.

[simonpg]

Installed in a 32-bit Debian testing environment and it works great; I get a prov> prompt. I used the CVS version because it was easier to download. Should I try the CVS version on the 64-bit system? Or is it Something Else? Since I have seen another post with the same problem, it would appear to be a 64-bit issue.

[anand]

don't have any 64 bit installs handy. Please let us know if you narrow this down.

[simonpg]

So I added -Xrunhprof to zmjava to start debugging and it works great! So, other than slow startup and the creation of a large debugging file, 64-bit works just fine. This kind of majic typically points to an internal problem with the vendor supplied code. I haven't seen anything after googling about. Any ideas?