Java-monitor user Pate interpreted the symptoms of his system very well.
He set out to answer the correct question: "what is causing the large number of full GC's when my system is does not use much memory?". I have really no idea where he got the idea to check for a call to System.gc().
It did occur to me (and to another user on Java-monitor; Kees de Kooter) that there is a tool that could have helped him. It is aptly named findbugs. Actually, there are several (PMD and CheckStyle being other, similar tools), but findbugs is nice because you don't actually need the application's source code to check it for bugs. All you need are the JAR files and/or class files. Another great feature is that each report is documented very well. You don't just get vague warnings that you sort-of give up on after a few minutes, but you can find an explanation for each warning on-line
Here is a sample output of findbugs, when you would run it against Java-monitor's probe.
$ findbugs -textui -effort:max java-monitor-probes/build/WEB-INF/classes
The following classes needed for analysis were missing:
Missing classes: 7
From this report you can see that findbugs cannot find any issues with the code itself. There are some classes that the Java-monitor code depends on. These were not included in the analysis. Since these are all Sun-provided classes I presume them to be bugfree. *cough*
Anyway. Let's make the analysis a bit more juicy. Here is the output when running findbugs against the well-known MySQL JDBC driver. I'm sure that many of you use it in production today. This code produces significantly more results and findbugs takes a while to analyse the whole package. I have shown only a few lines of the 154 warnings.
$ findbugs -textui -effort:max mysql-connector-java-5.1.5-bin.jar
H C NP: Method call in com.mysql.jdbc.profiler.ProfilerEvent.pack() passes null for unconditionally dereferenced parameter of writeBytes(byte, byte, int) Method invoked at ProfilerEvent.java:[line 375]
M D NP: Possible null pointer dereference of s1 on path that might be infeasible in com.mysql.jdbc.ConnectionImpl.nullSafeCompare(String, String) Dereferenced at ConnectionImpl.java:[line 341]
M C NP: Method call in com.mysql.jdbc.DatabaseMetaData.getInstance(ConnectionImpl, String) passes null for unconditionally dereferenced parameter of new DatabaseMetaData(ConnectionImpl, String) Method invoked at DatabaseMetaData.java:[line 632]
H S SQL: Method com.mysql.jdbc.DatabaseMetaData.getColumnPrivileges(String, String, String, String) passes a nonconstant String to an execute method on an SQL statement At DatabaseMetaData.java:[line 2156]
H S SQL: Method com.mysql.jdbc.DatabaseMetaData.getTablePrivileges(String, String, String) passes a nonconstant String to an execute method on an SQL statement At DatabaseMetaData.java:[line 4638]
M S SQL: Method com.mysql.jdbc.ConnectionImpl.setSessionVariables() passes a nonconstant String to an execute method on an SQL statement At ConnectionImpl.java:[line 5074]
M M IS: Inconsistent synchronization of com.mysql.jdbc.CallableStatement.outputParameterResults; locked 50% of time Unsynchronized access at CallableStatement.java:[line 1948]
M M IS: Inconsistent synchronization of com.mysql.jdbc.StatementImpl.wasCancelledByTimeout; locked 83% of time Unsynchronized access at PreparedStatement.java:[line 1756]
Warnings generated: 154
Learning to read the output of findbugs takes a little time. What I do is just work through a certain error or warning when I'm bored or frustrated with the code that I should be writing. It's my procrastination work. :-)
I have only picked out the things that as a developer strike me as problematic: "passes null for unconditionally dereferenced parameter". Eek. NullPointerException anyone? Of course, this could well be test code, or even unused code that is still under development. How about this: "passes a nonconstant String to an execute method on an SQL statement". Hmm. If left unchecked, this could be a vector for an SQL injection vulnerability.
I said earlier that findbugs does not require you to have access to the source code of an application to find bugs in it. Just for laughs, let's have a look at one of the cornerstones of our JEE application servers: The Oracle JDBC driver.
$ findbugs -textui -effort:max ojdbc6.jar
M B Dm: oracle.sql.ConverterArchive.openArchiveforRead() invokes System.exit(...), which shuts down the entire virtual machine At ConverterArchive.java:[line 375]
M B Dm: oracle.sql.ConverterArchive.closeArchiveforRead() invokes System.exit(...), which shuts down the entire virtual machine At ConverterArchive.java:[line 390]
M B ES: Comparison of String objects using == or != in oracle.jdbc.connector.OracleConnectionRequestInfo.equals(Object) At OracleConnectionRequestInfo.java:[line 104]
H C IL: There is an apparent infinite recursive loop in oracle.jdbc.rowset.OracleCachedRowSet.updateBlob(int, InputStream, long) At OracleCachedRowSet.java:[line 6365]
H C IL: There is an apparent infinite recursive loop in oracle.jdbc.rowset.OracleCachedRowSet.updateClob(int, Reader, long) At OracleCachedRowSet.java:[line 6445]
H C IL: There is an apparent infinite recursive loop in oracle.jdbc.rowset.OracleCachedRowSet.updateNClob(int, Reader, long) At OracleCachedRowSet.java:[line 6535]
Warnings generated: 1028
That driver produces no less than 1028 warnings. Wow. I'm not suggesting that the Oracle JDBC driver is actually a bad piece of code. I just find that it smells a little
. :-) Oracle's developers might want to have a crack at resolving the findbugs-reported warnings. There are many little performance and stability suggestions in there.
And yes: findbugs checks for uses of System.gc()
PS. Please be aware that I used findbugs 1.3.7. This version of findbugs has a bug that causes it to generate false positives for cleaning up database resources
PPS. Who am I kidding: I do think that the Oracle driver is bad. Infinite loops? System.exit()? Please.