It appears that even with a _dev Zimlet the JSP is compiled and cached; see post
Debugging Zimlet JSP.
That being said, it would be helpful for _dev Zimlets that any JSP files be re-compiled at each browser F5 hard refresh. Or at the very least, make the recompilation each refresh a configuration option.