Tuple23

Scala's upcoming dynamic capabilities

2011-02-03T18:01:00.000-08:00

I was surprised to see the Scala team check in a new "Dynamic" trait into Scala trunk recently. Once this trait is mixed in to your class, any unresolved method calls are passed to an invokeDynamic() method, much like Ruby's method_missing.

I've been thinking about this a lot, and I see three decent reasons for the inclusion of the Dynamic trait:
1. Integration with dynamic languages on the JVM.
2. DSLs for accessing data where there's already no type safety, ie JSON and XML (written by @jorgeortiz85, I think)
3. DSL for reflection. Reflection libraries are ridiculously verbose, and like JSON/XML/etc libraries, offer little static typing benefits anyway.

I think #1 is very convincing. LinkedIn and their LinkedIn Signal is one of the high-profile usages of Scala in industry, and it is a mixed Scala/JRuby project. And why the heck not? I'm very much warming to the idea that there is no Next Big Language, and programmers of different mindsets will be able to work together with the best tools possible.

Using the Dynamic trait is very easy, and really fun. I thought I'd try hooking up Scala to Ruby using Dynamic, JRuby, and the Java Scripting Engine (JSR 223). I was really pleased with how easy it was. Here's some Ruby:

With a little setup (installing JRuby to get the "jruby" scripting engine and getting Scala trunk running), I had this simple Scala class:

The Ruby above can be annotated with type information, allowing jrubyc to compile down to normal old class files with complete type information. In that case, we'd only need Dynamic for sugaring calls to "dynamic finders" which aren't declared in the class at all. I imagine the situation is similar for other dynamic languages on the JVM. I have to wonder why dynamic invocation would be preferable to calling static bindings of these dynamic classes. I would love to learn more about how this will all work out in practice.

I have no information on this whatsoever, but from what I can tell this would be in Scala 2.9, which is slated for release in "weeks not months".

Update 3/28/2011: For Scala 2.9, This method is now called "applyDynamic", but is only available with the -Xexperimental flag.

"Intro to Scala for Java Programmers": slides, code, and links

2010-11-20T18:41:00.000-08:00

Last week, I presented a talk titled "An Introduction to Scala for Java Programmers". I had a lot of fun making it, and I learned a ton. I thought it was interesting that the more I learned, the more I liked Scala. This is certainly not always true of technologies that look good at first!

Here's the slides. They're probably pretty reliant on the speaking that typically goes with them, but maybe you will see something interesting.

Introduction to Scala for Java Programmers

Here's the code I used to develop the slides. It's in the form of an Eclipse workspace. To use it, you'll need the Scala plugin installed on Eclipse. Unfortunately, you'll need to use Eclipse 3.5 (Galileo) instead of the latest. Once it's installed, make sure you activate the Scala perspective, and import the sample code using File...Import...Existing Project Into Workspace, and use "Select Archive File" to import talk.zip. The code is a downright odd collection of snippets, but I suspect it would be interesting to play with if you were curious about Scala.

Finally, I wanted to give better credit for some of the ideas and samples in the slides.

3 interesting uses of traits: lifted from this awesome article
Actors: inspired by this great intro on Actors
As useful as named parameters and default values are, I didn't have a good example to illustrate them until I found this intro on Artima
I think the format of the slides was guided by this article on combining OO and functional principles.
Many times, I ended up referring back to Programming in Scala for clarity. It's a great general-purpose programming book.

Update: Some people have asked for a downloadable pdf - here it is

3 easy wins using Google Guava

2010-10-26T09:06:00.000-07:00

Google Guava used to be known as Google Collections. It is a large set of utility methods used within Google. It aims to help you create concise and easy-to-read code.

1. Concise collection initialization - No example of Java being overly verbose is complete without showing a repetitive and noisy constructor like this:

Java 7 intends to introduce a "diamond operator" which allows the code to be converted to this:

This kind of change probably won't prevent any bugs, but in my opinion it greatly reduces code "noise". Noise does not matter in a single-line example, but makes a big difference when multiplied across an entire system.

Google Guava provides similar functionality, today, and it works all the way back to Java 5. It takes advantage of one of the few places Java does infer types: method return values.

Here, newHashMap is a static method on com.google.common.collect.Maps, with a static import. It gets better, Guava can also be used to initialize collections with values:

java.util.Arrays.asList does something similar, but is actually immutable. In combination with other techniques (many from Guava), type inference for collections and easy collection initialization can really clean up your code.

2. Multimap - How often are the values in your map actually a collection? Imagine taking a group of people and grouping these people by last name using a Map.

Current way:

Using Multimap:

3. One-liners
Here's a small sampling of clever utility methods provided:

I've got to stop here, or I'll never finish. I have only covered a few of the classes I really love, honorable mention goes to:

BiMap - Easily create the inverse of your map (keys become values and values become keys)
Ordering - Lots of nice extensions to Comparator
MultiSet - A set that keeps count of duplicates
Predicates and Functions - reusable objects for filtering and transforming your collections
AbstractIterator - Writing your own iterator is annoyingly tricky - this can really help
Forwarding Collections - Makes it easy to write your collection classes by delegating to existing collections classes (w/o boilerplate)

There's also a huge portion of the library I don't know yet, and really want to investigate:

MapMaker - Factory for producing maps for caches - makes a very tricky job easier.
Immutables - I'm a big fan of immutability, but haven't investigated Guava's extensive support for it yet

Scala's Option will save you from the most common cause of NullPointerException

2010-10-22T12:05:00.000-07:00

A while back, Cedric Beust claimed "Scala’s “Option” and Haskell’s “Maybe” types won’t save you from null". He's rightly convinced that Option is not a silver bullet for unexpectedly referencing null values at runtime. However, I believe there are "Good" NullPointerExceptions(NPE) and "Bad". Option goes very, very far in saving you from "Bad" NPEs by enlisting the compilers help in tracking what is intended to be optional.

I believe there are two very different cases of NullPointerException:
"Bad" - The code incorrectly assumes the value is guaranteed to be set, but the value is null.
person.getPreferredTheme().getBackgoundColor();
In this case "getPreferredTheme" is optional, but the programmer accesses it as if it will never be null in the current context. This is programmer error, due to the fact that the developer was unaware/not caring the value can sometimes be null. This is class of NPE I feel is the most common, and I believe the Option "pattern" can greatly, if not completely, reduce.

"Good" - The code correctly assumes the value is guaranteed to be set, but the value is null.
manager.getCompany().getName()
In this case, getCompany is designed to be set on a manager object. In my opinion, the developer should NOT try to accommodate getCompany returning null if it is specified that it never will. In the spirit of failing fast, the user should boldly use the object as it's specified.

In both Good and Bad scenarios, how was it specified that getPreferredTheme was optional and getCompany was required? In Java, I've seen a few approaches:

1. Sheer mental horsepower and subjective judgement. "Preferred Theme sounds pretty wishy washy to me, I better check for null". If you're not sure, check the javadoc, source, database schema, and any number of artifacts to inform your guess. Good luck!

2. "Sentinel values" - Sentinel values are when you return a non-null value of the correct type, but with a wacky value to signify, well, null! For example, a binary search that returns -1 if the element isn't found. This is not the same as returning a default value, which is awesome when appropriate. Not realizing you're working with a sentinel value is identical to not realizing you are working with null. When you accidentally use it as a real value, it's gonna be ugly.

3. Annotations - JSR 305 defines some annotations which allow you to mark up your fields, variables and parameters as @Nonnull, @Nullable, @CheckForNull, @ParametersAreNonNullByDefault, and @ParametersAreNullableByDefault. A tool like windbags can process these annotations and give you warnings when you break the specification. I tried this approach for several months, and definitely had some success with it. However, it has a fair share of false negatives and false positives, and can be noisy. I gave up on it when I encountered code like this (paraphrased):
@Nonnull Person manager = ...
....
checkNonNull(manager);
manager.getCompany(); //findbugs warns here - it doesn't know the semantics of checkNonNull

4. Naming conventions - I've tried preceding method names that can return null with a "try", ie tryGetPreferredTheme, and naming fields and vars using "optional", ie
private Theme optionalTheme.
This visually sets of the null access, and worked better than I'd thought. However, it seems a little noisy, and is very easy to overlook.

5. The "Null Object" Pattern - implement your interface using a "null" implementation. Exactly like returning null, but more confusing.

Scala/Haskell and many other languages offer a 6th option: the Option type. In Java, our "getPreferredTheme" would change to this if it used Option:

Option is an interface with two implementations: "Some" to represent a real value and "None" to represent a null value. Here's the code, adapted from this great article:

With the change above this would no longer compile:

person.getPreferredTheme().getBackgoundColor();

Instead, the two choices are:
1. You just know it won't be null in this one special case:

person.getPreferredTheme().get().getBackgoundColor().getName();

get() will get you the actual Theme value if set, or throw a NullPointerException if not. What, a NPE? I thought we were avoiding those? We can't. I'm striving only to get rid of "bad" NPEs, ones which the programmer access the null reference out of ignorance or neglect. We only used get() here because we have assured ourselves it should be an error condition if it wasn't set - a "Good" NPE. Think of it as a shortcut for "check for null and error out if it is null".

2. You're glad you were reminded it was potentially null, and can move on without it.

if (person.getPreferredTheme().isDefined()) {
...process the data

Yep, just a standard null check with a fancy name.

The important thing to note here is that accessing optional data without considering that fact is disallowed by the compiler. This cannot be said of the four approaches described at the beginning of the post. You are moving what was previously developer busywork (which we're notoriously bad at) into the domain of the compiler, which excels at busywork. And not some whiz-bang language feature like Nullable Types or annotations either - it takes advantage of very simple classes.

It's also important to note Option should be used only on types that are specified to be optional. I think a lot of people think Option is intended to protect you from all NPEs by forcing that extra check each time. Not so - just use it on your truly optional variables, and use your normal variables with confidence.

Option is absolutely worth trying in your Java project - I intend to try soon. I fear it may require too much boilerplate in Java. Scala (and Haskell and others) make Option more appealing due to much stronger type inference (less noise) and a host of other tools for processing them beyond get() and isDefined(). The other tools available in Scala require much more of a leap of faith/new way of thinking. They're also the source of most of Cedric's skepticism, I believe. I hope to cover them in the next post.

"If it's worth telling another developer, it's worth telling the compiler"
-Guy Steele

A new syntax for Java closures/lambdas

2010-07-09T18:56:00.000-07:00

I'm surprised this isn't making more news. Brian Goetz of Oracle has posted a dramatic revision of Java 7's Project Lambda. The syntax is much less noisy than the original version, thanks to a lot more type inference for parameters and return types. I'm also really happy with how accessible the document is - the document is very readable, even for a "blue-collar" Java developer like me.

Oracle pushes first version of Lambda/Closures implementation

2010-05-27T12:16:00.000-07:00

Test cases may not be the best source of coding gems, as they're probably testing the weirder stuff. But you can now see some test cases, and therefore the evolving syntax here.

Oracle has a tough task in adding closures to a language that has checked exceptions, minimal type inference, and without breaking compatibility of zillions of lines of existing code. Go Snoracle!

Some JUnit tricks for easier and better test cases

2010-05-17T10:40:00.000-07:00

Whenever I start a new project, the first thing I'll write is a stack trace filterer/clarifier. The second would be several enhancements to JUnit. For test driven development to be effective, you need good coverage and lots of tests. For that to happen, tests need to be dead-simple to write, easy to maintain, and fast. The following enhancements have really helped me towards these goals.

1. Easy Fixtures - Many bugs/regressions will involve a set of complicated data relationships. Create an XML/JSON/whatever format to easily import large sets of data. For example, let's say bug #535 comes in: "Exception is thrown when trying to delete users when they are in a group". You can quickly go out, and create 535.xml:

Then, your test case could be:

Without an extremely easy way to create test fixtures and import them, 99% of us will write far less tests.

2. Automatic transaction handling and application bootstrapping. Ideally, the example test method I gave above should be all it takes to run the test. No explicit setUp, tearDown for bootstrapping your environment, setting up transactions, etc. It is worth investing your time in some means (maybe a AbstractTest superclass?) to set these things up automatically for the 95% of test cases that require no special environmental setup. Don't Repeat Yourself.

3. More powerful Set and List comparison - When I do assertEquals(expected, actual) where "expected" and "actual" are Collections with dozens of elements, the last thing I want to hear that they are not equal, maybe with a toString of both. So I always implement a assertSetsEqual and assertListsEqual. assertSetsEqual compare the two collections, where order doesn't matter. Here's a sample of assertSetsEquals:

The List comparison, where order matters, requires more work to produce a nice message, but it's worth it:


Expected: [a, b, c, d, e, f, g, h, i, j, k]
Actual:   [a, b, c, e, f, g, h, i, j, k, l]
Missing 'd' @ 3
Unexpected l @ 10

4. Type-checking assertEquals - Due to limitations of Java's type system, assertEquals takes two parameters of type Object. There can be no compile-time checking that the two arguments are actually of types that can actually have equal objects. For example:

I discovered org.junit.Assert as I wrote this, much better than the junit.framework. Why is there two? Just printing the types goes a long way. I'm surprised how often an assertEquals LOOKS right, but it happily returns false because you are passing incompatible types in.

5. An Eclipse template for creating a new test case.

Just laziness mostly, but I do think it's important to have that fail("No Asserts") in there from the start. Too often I'll write a test case as I'm exploring a problem, but then I get distracted and abandon it before I really nail down the asserts. Without asserts, your test case is mostly just slowing things down.

These little changes seem like overkill in the context of writing a silly little test case. But if your goal is to write hundreds, I found these tricks to be really worth the investment.

Update #1: In the comments, Crias points out that junit.framework is for backwards compatibity with JUnit 3 and earlier and should be avoided if possible. I think a custom assertEquals which uses reflection determine if the two arguments even can be equal is worthwhile, but the new org.junit goes a long way just by printing class names.

Update #2: In the comments, David Karr correctly points out the the test runner will print any exception that bubbles out a test method created with my template. The reason I took this weird approach is the the Eclipse JUnit runner does not output the trace to the console but to the JUnit View, which I find is a goofy place to inspect the exception. More importantly, the Eclipse runner won't filter my stack traces, which I'm addicted to.

Update #3: Tym The Enchanter points out that Hamcrest helps with my #3 and #4 points about better asserts. I know very little about Hamcrest, but here's my naive reaction:

is/equalTo for safer equal to comparisons. These basically force the type of the "actual" to be a subclass (or the same class) as the "expected". This goes a long way to reducing dumb-dumb comparisons like new Integer(2) and "2". In my experience, these are exactly the kinds of comparison errors that slow you down. It's not perfect, however - it's quite possible that the compile time type of your expected is not a subclass of the actual, but they ARE assignable - see the "Garfield" example below. Of course you can fall back to more traditional assertEquals if it bites you, and I'm unsure how often I'd see this in real life.
Set/List comparisons - For sure, Hamcrest beats JUnit asserts here. However, these comparisons basically fail fast - they only tell you the first problem they find - which could lead you to running the test several times before you get it right. Tell me all of the problems right off the bat!

Thanks for the comments!

Another reason Virtual Machines are amazing

2010-04-20T10:29:00.000-07:00

http://github.com/codahale/yoink

The details aren't important, but basically:
1. Clojure is a Lisp-like language which runs on the JVM, Scala a more C/Java style language which also runs on the JVM.
2. Clojure writes high performance immutable data structures which can handle small updates w/o copying the entire structure.
3. Clever Scala programmer just wraps the Clojure collection in a thin Scala wrapper.

I know language interop on the JVM has been evolving for some time, but it's especially neat to see when neither of the languages are Java! We're entering an era where you will not know or care what language your libraries are written in.

Filter your stack traces

2010-03-25T09:29:00.000-07:00

Probably the first utility I write when starting up a new project is ExceptionUtils.getFilteredStackTrace(Throwable e)
The idea here is to:
1. "Flatten" the trace. e.printStackTrace attempts to display the stack trace of the creation of each Exception in the chain. In my experience, only the stack of the "bottom" exception is interesting.
2. Filter out third party lines. 99% of the time, the offending code is mine, not my library's. So why show those endless lines from Tomcat, JUnit, Hibernate, etc?

Filtering isn't always appropriate. Sometimes the trace provides good insight into what your libraries are trying to do when the messages aren't clear. I also recommend turning filtering off in production, where you really need all of the info you can get. However, most of the time, you mainly just want to see what your own code was up to when the exception was thrown, and this approach removes massive amounts of noise.

Here's an example. I took the standard Hibernate Person model and hooked it up to the H2 database. I altered the model to make username unique, so that this test fails:

Here's the output from the filtered run:


HibExample.test: Exception: org.hibernate.exception.ConstraintViolationException: could not insert: [hib.Person]
Caused by: org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "CONSTRAINT_INDEX_8 ON PUBLIC.PERSON(FIRSTNAME)"; SQL statement:
insert into PERSON (PERSON_ID, age, firstname, lastname) values (null, ?, ?, ?) [23001-132]
 37 lines skipped for [org.h2, org.hibernate, sun., java.lang.reflect.Method, $Proxy]
 at hib.HibExample.test(HibExample.java:18)
 24 lines skipped for [sun., java.lang.reflect.Method, org.junit, org.eclipse]

And here's e.printStackTrace


org.hibernate.exception.ConstraintViolationException: could not insert: [hib.Person]
 at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:94)
 at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
 at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:64)
 at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2176)
 at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2656)
 at org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentityInsertAction.java:71)
 at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
 at org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:321)
 at org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:204)
 at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:130)
 at org.hibernate.event.def.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:154)
 at org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:110)
 at org.hibernate.event.def.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:61)
 at org.hibernate.impl.SessionImpl.firePersist(SessionImpl.java:646)
 at org.hibernate.impl.SessionImpl.persist(SessionImpl.java:620)
 at org.hibernate.impl.SessionImpl.persist(SessionImpl.java:624)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:342)
 at $Proxy4.persist(Unknown Source)
 at hib.HibExample.test(HibExample.java:16)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
 at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
 at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
 at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
 at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
 at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
 at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
 at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
 at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "CONSTRAINT_INDEX_8 ON PUBLIC.PERSON(FIRSTNAME)"; SQL statement:
insert into PERSON (PERSON_ID, age, firstname, lastname) values (null, ?, ?, ?) [23001-132]
 at org.h2.message.DbException.getJdbcSQLException(DbException.java:316)
 at org.h2.message.DbException.get(DbException.java:167)
 at org.h2.message.DbException.get(DbException.java:144)
 at org.h2.index.BaseIndex.getDuplicateKeyException(BaseIndex.java:157)
 at org.h2.index.PageBtree.find(PageBtree.java:115)
 at org.h2.index.PageBtreeLeaf.addRow(PageBtreeLeaf.java:137)
 at org.h2.index.PageBtreeLeaf.addRowTry(PageBtreeLeaf.java:92)
 at org.h2.index.PageBtreeIndex.addRow(PageBtreeIndex.java:88)
 at org.h2.index.PageBtreeIndex.add(PageBtreeIndex.java:79)
 at org.h2.table.RegularTable.addRow(RegularTable.java:112)
 at org.h2.command.dml.Insert.insertRows(Insert.java:120)
 at org.h2.command.dml.Insert.update(Insert.java:82)
 at org.h2.command.CommandContainer.update(CommandContainer.java:70)
 at org.h2.command.Command.executeUpdate(Command.java:198)
 at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:141)
 at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:127)
 at org.hibernate.id.IdentityGenerator$GetGeneratedKeysDelegate.executeAndExtract(IdentityGenerator.java:94)
 at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(AbstractReturningDelegate.java:57)
 ... 44 more

Update: Here's a good starter version of the filtering code. It may have bugs: I quickly ported it. It depends on Google Collections, because I can no longer imagine coding without them!

File upload using Jersey Client

2010-03-05T09:44:00.000-08:00

This post is nothing special - I just wanted to do an http file upload using Jersey's client API. There's a lot of "noise"/outdated code out there on this topic, just wanted to show an example that worked for me:

This requires Jersey 1.1.5 and the optional jersey-multipart module (the two jars at the bottom of this section)

If you're just curious about Jersey, look at how simple the server side is for file upload using Jersey:

Beautiful!

Embed code in Blogger/Blogspot posts

2010-03-05T04:58:00.000-08:00

Many source code snippet sites allow you to easily embed source code in your posts by providing a simple <script> tag for each snippet. Blogger seems to disable these tags, and I was surprised how little info I could find on the topic. Luckily, Pastebin provides <iframe>, which works like a charm on Blogger!

Scala features for Java Programmers: Case Classes

2010-03-03T20:38:00.000-08:00

I'm a Java programmer who's been casually evaluating Scala for about 6 months. I find it very hard to objectively compare the two languages. Would concise collection literals save me significant time every day? Do boilerplate getter methods really ever cause bugs, or do they just offend my sense of style? In what ways can Java be improved to allow me to write better code? What Java-related factors cause me to waste the most time, day-to-day?

I think that's a very difficult question to answer. So far, I've come up with three principles to help me evaluate various differences between languages:

1. Would it make my code easier to read?

2. Would I be able to more easily write correct code?
3. Does it addresses common problems I've encountered that are related to Java?

For my first comparison, can you quickly tell me what the bug is here:

Did you catch the bug? The problem is I had Eclipse generate hashCode/equals before I added the "_firstName" field. This mistake is very easy to make, and can be brutal to track down in a bigger system. Here's an example of how things break when you screw up hashCode:

I feel this is an example of where another language may make it easier to write this code correctly. At 96 lines to essentially represent a struct of 5 fields, I'd say another language has ample opportunity to help me make this code more readable. These type of "Bean" objects - heavy on data, light on code, are so common in Java, language level support seems justifiable.

One of my favorite Scala features addresses the readability and correctness problems mentioned above: Case Classes. Case Classes build on Scala's already succinct class definition format, and adds some extra goodies such as compiler-generated equals and hashCode methods. Of course, compiler-generated means that as you add/subtract fields, hashCode and equals automatically adjust accordingly.

I know it seems small, having the compiler maintain hashCode and equals methods on "Bean"-style classes. However, it contributes much to both the readability and correctness of these common classes. Here's the full definition of Person and the test main method in Scala:

That one little line at the top gives you: enforced immutable access to all 5 fields, correct hashCode and equals, and even a decent toString! This is 1/96th of the code, and with less bugs. And I'm holding back some additional tricks that Case Classes have.

Of course, Case Classes are not always appropriate. I've often written hashCodes that did not consider certain fields, etc. However, many, many classes I've written in Java would have greatly benefitted from a Case Class-type functionality.

I certainly don't spend my days debugging hashCode methods, but Case Classes absolutely address all 3 of my principles for evaluating language differences.

Nice online JSON formatter

2010-01-22T08:44:00.000-08:00

http://www.bodurov.com/JsonFormatter/

Formats to nice collapsible/expandable format, processed completely client side.

4 Great Resources For Getting Started With Scala

2010-01-12T05:32:00.000-08:00

I've been aware of Scala for years, and it never really caught my interest. Every time I'd seen a Scala example, it was an insanely dense block of code solving some problem I'd never had before. However, once the creator of Groovy essentially endorsed Scala, I decided it was worth a closer look. I've since looked at a LOT of Scala content and these 4 resources really stand out for me. If you're interested in Scala, I highly recommend you start here:

Programming In Scala - This is simply a great general-purpose programming book. In addition to being a gentle, clear introduction to the language, it's also a fantastic introduction to functional programming concepts and language design. Even if you hate Scala, this book will make you a better programmer.
Pragmatic Real-World Scala - This video shows off all kinds of things that would make a Java developer drool.
Scala For Java Refugees - Very well-written mostly gentle introduction to major Scala concepts.
Daily Scala - Once you've done 1-3, and have written a few Scala apps, subscribe to Daily Scala. Thoughtful, bite-sized examples nearly every day.
Another tour of Scala - A Java-centric breakdown of fundamental Scala features.

Keep terminated launch consoles in Eclipse

2008-07-17T11:58:00.001-07:00

This one bugs me every time I upgrade to a new release of Eclipse. I often like to look back at the consoles from main methods/junits/tomcat launches I've terminated. I guess I expect to find a preference right in the context menu preferences for the Console view, or for it to match "terminated" in the prefs search - no luck.

So that I never have to search for this again, this preference can be found under
Run/Debug => Launching => "Remove terminated launches when a new launch is created"

Wednesday Afternoon Reflections

2008-07-16T12:59:00.000-07:00

What does this print?

import java.lang.reflect.Method;

interface IMessage {
public CharSequence getMessage();
}

public class SayWhat implements IMessage {
public String getMessage() {
return "Hi!";
}

public static void main(String[] _args) {
try {
SayWhat instance = new SayWhat();
for (Method m : SayWhat.class.getMethods()) {
if (m.getName().equals("getMessage")) {
System.out.println("SayWhat.main: " + m.invoke(instance));
}
}
}
catch (Throwable e) {
System.out.println("SayWhat.main: " + e);
}
}
}

The answer is
Hi!
Hi!

That's right, the getMessage method appears twice in SayWhat.class.getMethods(). The key here is that the SayWhat class is overriding IMessage with a covariant return type (CharSequence vs. String). Looks like the compiler makes covariant return types possible by creating a behind-the-scenes "synthetic" implementation of getMessage which just calls through to my implementation.

If you find yourself looping through a list of Method objects looking for a match, be sure to check isSynthetic on each Method.

Download sources for javax libraries

2008-03-24T14:11:00.001-07:00

If you are lucky like me, you often find yourself wanting to see the source code for various javax libraries (JavaMail, JSTL, EL, etc). My success in tracking down the complete zips of these sources varied from project to project. Today, I stumbled across this:
http://download.java.net/maven/1/

This looks like a good starting place if you are after javax sources or jars.

ical4j, JavaMail, Exchange and Outlook secret sauce

2008-03-11T07:40:00.000-07:00

Last summer, I was tasked with sending out Outlook "meeting invites" for events in our product. This meant integrating with iCal4j. iCal4j itself has a truly bizarre API, but it wasn't bad to create a new invite, as there are so many examples out there. What really stumped me for a while was getting the headers right so that Outlook would pop up an "Accept/Decline" dialog for automatically adding the event to their calendar. I should have blogged about this last summer, as I've now forgotten what I finally did to make it work - I believe the real coup was getting the headers right. I remember finding lots of people with similar problems, so I wanted to get this code out there.

This code requires iCal4j and JavaMail 1.4. I believe the JavaMail 1.4-only ByteArrayDataSource is crucial for getting this to work. This code might not even work for you, I just wanted to get it out there as a starting point.

public MimeMessage createMimeMessage(Session _javaMailsession) throws Exception {
MimeMessage mimeMessage = new MimeMessage(_javaMailsession);
mimeMessage.setSubject(getSubject());
mimeMessage.addFrom(getFromInternetAddresses());
mimeMessage.addRecipients(Message.RecipientType.TO, getFromInternetAddresses());
Multipart multipart = new MimeMultipart();
MimeBodyPart iCalAttachment = new MimeBodyPart();
byte[] invite = createICalInvitation(getMeetingID(), getSubject(), getContent(), getMeetingStart(), getMeetingEnd(), getMeetingTimeZone());
iCalAttachment.setDataHandler(new DataHandler(new ByteArrayDataSource(new ByteArrayInputStream(invite), "text/calendar;method=REQUEST;charset=\"UTF-8\"")));
multipart.addBodyPart(iCalAttachment);
mimeMessage.setContent(multipart);
return mimeMessage;
}

private byte[] createICalInvitation(String _meetingID, String _subject, String _content, Date _start, Date _end, TimeZone _tz) throws Exception {
CompatibilityHints.setHintEnabled(CompatibilityHints.KEY_OUTLOOK_COMPATIBILITY, true);

VEvent vEvent = new VEvent();
vEvent.getProperties().add(new Uid(_meetingID));
vEvent.getProperties().add(new Summary(_subject));
vEvent.getProperties().add(new Description(_content));
vEvent.getProperties().add(new DtStart(new DateTime(_start)));
vEvent.getProperties().add(new DtEnd(new DateTime(_end)));

net.fortuna.ical4j.model.Calendar cal = new net.fortuna.ical4j.model.Calendar();
cal.getProperties().add(new ProdId("-//iloveoutlook//iCal4j 1.0//EN"));
cal.getProperties().add(net.fortuna.ical4j.model.property.Version.VERSION_2_0);
cal.getProperties().add(CalScale.GREGORIAN);
cal.getProperties().add(net.fortuna.ical4j.model.property.Method.REQUEST);
TimeZoneRegistry registry = TimeZoneRegistryFactory.getInstance().createRegistry();
VTimeZone tz = registry.getTimeZone(_tz.getID()).getVTimeZone();
cal.getComponents().add(tz);
cal.getComponents().add(vEvent);

ByteArrayOutputStream bout = new ByteArrayOutputStream();
CalendarOutputter outputter = new CalendarOutputter();
outputter.output(cal, bout);
return bout.toByteArray();
}

Firefox 3

2008-02-22T08:59:00.000-08:00

I've been playing around with beta 3 of Firefox. It's definitely stable enough to use all day, I can't say I've had any real problems yet. The speed is dramatically improved. I'm also enjoying much smarter auto-complete in the address bar.

One must-have config tweak if you plan on using Google Reader with ff3, via crazybob:

To load all new Firefox tabs in the background, set "browser.tabs.loadDivertedInBackground" to "true" in "about:config".
...
My last tweet makes the "v" shortcut in Google Reader imminently more useful.

Tuesday auto-boxing nugget

2008-02-06T14:02:00.000-08:00

What does the following print?

public static void main(String[] _args) {
System.out.println(boolean.class.isAssignableFrom(Boolean.class));
System.out.println(Boolean.class.isAssignableFrom(boolean.class));
System.out.println(Boolean.class.isInstance(true));
System.out.println(boolean.class.isInstance(true));
}

The answer:
false
false
true
false

This makes sense when you think about it, but it can definitely trip you up when you've become accustomed to auto-boxing. Incidentally, this is my first true "gotcha" w/ auto-boxing in the 2.5 years I've been using it.

Analyzing huge HeapDumpOnOutOfMemoryError heap files

2008-02-05T18:25:00.000-08:00

I previously mentioned a JVM flag that has the potential to really clarify things when you encounter a OutOfMemoryError in production.

When I first utilized this flag, my heap dump file was probably about 500M. It's a year later, and we just had another OutOfMemoryError (OOM). This time, the heap dump size was more like 1.1G. If you've ever struggled with Java OOMs on 32-bit Windows in the past, you know 1.1 is effectively the largest heap size you can get away with. I don't necessarily agree with -Xmx1.1G in the least, but that's a different post.

It appears that to analyze a 1.1G heap dump file, jhat requires greater than 1.1G of heap allocated for the analysis. In other words, if you max out heap on your application, you won't be able to analyze the heap dump on a similar machine. In other words, if you max out heap on a 32-bit OS, you'll need a 64-bit OS to analyze it.

I also tried SAP Memory Analyzer and YourKit on my 32-bit machine, just to see. Same experience - the OOM analysis tools crash with an OOM. Finally, I tried SAP Memory Analyzer on a 64 bit machine, and it worked well. After seeing the contents of the heap, it was very easy to diagnose the problem. I'm not sure I ever would have solved the problem w/o it.

I've since found out the YourKit 7.1 EAP can handle massive heap dumps on 32 bit machines: I was able to analyze the file by passing in -Xmx700M to YourKit. Overall, I was really impressed w/ SAP and YourKit in terms of polish and responsiveness from support.

Timezone problems, a few weeks early

2007-02-22T06:58:00.000-08:00

Due to some serious ambiguities (Australia has an EST, China has a CST), Sun deprecated three-digit timezone IDs as of Java 1.4. Well, they really meant it: as of Java 1.5_08, "EST" shifts from traditional Eastern Standard Time to a 1980s System V timezone name for "Indiana, with DST not observed." The bug and related thread are an interesting insight into how TimeZones are managed in the software world.

For reasons long forgotten, we had some ancient code which allowed us to configure the VM-wide TimeZone, and we defaulted to "EST" - ouch. Changing that to America/New_York fixed the problem, but I didn't understand how default timezone was affecting our date formatting. I conceptualized databases storing Timestamps as what Joda would call an "Instant", or others would call "epoch time": what you get when you call System.currentTimeMillis. Instants don't have timezones, but represent a moment precisely:

1149138000000 == 6/1/2006 2:00 PM EST == 6/1/2006 11:00 AM PST

I was assuming the database essentially stores "1149138000000". When you read that value out, you get a Timestamp back, with this Instant as its backing time. After all, Timestamp/Date essentially represent Instants: look at their getTime() method. Once you have a Timestamp, you can easily format it to any timezone using DateFormat and TimeZone objects. With these horribly flawed ideas, I didn't see where default timezone fit in.

Boy, was I wrong. Consider the following SQL:
INSERT INTO DateTest (DateField) VALUES {ts '2006-06-01 00:14:00.00'}
How do you convert that to an Instant? Of course, you can't - that could be 2 PM in any one of dozens of timezones - all different Instants. Really, the database is just holding the various month/day/hour/etc values - it's up to the code reading the value to make it into an Instant. Considering that java.sql.Timestamp really is an Instant, it's pretty weird to reconsider this code:
Timestamp ts = rs.getTimestamp("DateField");
Which 2PM instant will "ts" represent?? JDBC will apply the VM default timezone in this case. getTimestamp provides a method which will probably make things more clear:
public Timestamp getTimestamp(String columnName, Calendar cal);
I'm still unclear as to why this takes Calendar, and not TimeZone, but I'm done with investigating for a while :)

Lessons:
1. Don't use three-digit timezone codes for looking up timezones!
2. DATE and TIMESTAMP columns do not store TimeZone info, or even Instant info.
3. Web programmers should be very careful with TimeZone.setDefaultTimeZone, its scope is either VM-wide or ThreadLocal, depending on your permissions.
4. Instead of setDefaultTimeZone, consider the Calendar-based permutations of getTimeXXX(....) in ResultSet. This is much more clear, and will avoid the scoping problems described in #3
5. TimeZone.getTimeZone("someIDThatDoesntExist") .equals(TimeZone.getTimeZone("GMT") )- BOO!

Excellent VM flag

2007-02-21T07:43:00.000-08:00

This is a bit dated, but if you're using Java 1.5 or 1.4.2, there's an excellent flag in more recent VM updates: HeapDumpOnOutOfMemoryError. OOMs have to be one of the hardest problems to debug: traditionally, you'll have next to zero information on the cause of the crash. This flag apparently has no runtime cost, and will provide a thorough snapshot of the heap at the time of the crash. I hadn't had a "out of heap space"-style OOM in years, but recently had one. Luckily, this flag was enabled, and I was able to use the "jhat" tool to pretty quickly find the culprit. I had some problems with the java.net HAT tool, but the jhat which is distributed with Mustang worked perfectly. This is one of those simple changes that can really save you later.

All the info

When generic type inference fails

2007-02-08T06:22:00.000-08:00

99% of the time, using generics is perfectly natural and pleasurable. The other 1%, it can be downright humbling.

Look at the signature of Arrays.asList:

public static <T> List<T> asList(T... a)

Going to take in an array of type T, and return a List containing the same type. But what is T? The parameter "a" is just a set of parameters the caller may have just defined in-place - with no explicit declaration of T. This is only a problem when the varargs passed in are of diverse types:

Arrays.asList(new Thing<Integer>(), new Thing<String>(), new Thing<String>());

In this case, the compiler must find the lowest common denominator type. Unfortunately, with the eclipse compiler at least, that's a tad insane - it would decide the common type is this:

Thing<? extends Object&Comparable<?>&Serializable>>

Yuck! It's true, that in a vacuum, there's not much else the compiler can do. But what about when I give it more info?

public List<Thing>> getThings() {
return Arrays.asList(new Thing<Integer>(), new Thing<String>(), new Thing<String>());
}

The compiler complains that it can't convert its insane type to what would appear to be a proper "super" type: Thing. It doesn't seem to me the compiler is taking advantage of all the info I'm giving it. To be fair, I'm sure there are tons of situations where the compiler has no additional context to take advantage of.

However, there's hope. In his post Java 7 Wish List , the author describes a way (which is in Java 5, btw) to give the compiler more of a nudge:

Arrays.<Thing<?>>asList(new Thing<Integer>(), new Thing<String>());

Hey, it aint pretty, but my only guess to the alternative is a bit too verbose:
List<Thing> things = new ArrayList<Thing>();
things.add(new Thing<Integer>());
things.add(new Thing<String>());

Actually, you CAN cast:

(List<Thing<?>>)Arrays.asList(new Thing<Integer>(), new Thing<String>(), new Thing<String>());

But this works only on eclipse, not javac. Plus, doesn't it feel a little to ironic to add a cast to accomodate generics :)

PS. Dear Google programmers, please use your 15% time to make it a lot easier to add code snippets to blog entries.

Portable SQL?

2007-01-11T11:56:00.000-08:00

The app I work on is going on 7 years old. We're slowly migrating to Hibernate, but there's plenty of "old fashioned" JDBC assets left. So, I maintain DDL for about 80 tables and 40 views. We've always tried to keep it portable. So far, the same DDL has run on Oracle, MySQL, Postgres, Derby, and Mckoi. You can imagine my shock when this didn't hold up when I tried SQL Server 2000. SQL Server has no DATE type, and has a bizarre (and deprecated) TIMESTAMP type which really has nothing do w/ storing traditional timestamps. They want you to use the "DATETIME" type for both.

I desperately want to a single set of portable, "pure" DDL for all of these databases, but I may be out of luck. I was able to fix the lack of DATE type by adding an "alias type":

CallableStatement cs = getConnection().prepareCall("{call sp_addtype(?, ?, ?)}");
cs.setString(1, "DATE");
cs.setString(2, "DATETIME");
cs.setString(3, "NOT NULL");
cs.executeUpdate();

But I can't do the same for TIMESTAMP, since it's a reserved word. There must be something clever that can be done...

Acrobat replacement

2007-01-11T11:47:00.000-08:00

Over the years, my hate affair with Acrobat Reader has grown.
1. The UI seemed to block until the entire document was downloaded
2. The slow startup time, complimented by their oh-so-impressive patent list
3. The ever present, marketing driven, "Update Reminder"
4. The downright goofy interface.

It got so bad, I'd skip PDF hits in search results - that's insane. I was SO happy to stumble across
Foxit Reader. For my purposes at least, it's a fully functional drop-in replacement for Acrobat. I haven't had a single problem with it, and it's downright snappy. And it's free.

Resource bundle syntax gotchas

2006-11-29T07:37:00.000-08:00

What's wrong with this property resource bundle entry?

importantMessage={0}'s account status is now {1}

For people who are experienced with resource bundles, this is probably obvious: that single ' escapes the rest of the string from MessageFormat processing. Also, since it's an escape character, it will not be displayed, regardless of MessageFormat. It's too bad MessageFormat doesn't fail when the ' goes unclosed. A double ' acts as a replacement for '. Our team knows this rule, but inevitably some messages get the single tick anyway. I was pleased to find out that I could mass-fix this using Eclipse's search and replace feature. I know the search feature supported regexes, but was unaware the replace feature was also regex aware. The trick is I wanted to replace only single ticks, not ticks part of a double tick. The regex for finding these:
[^']{1}'[^']{1}
aka "one non tick followed by a tick, followed by one non-tick"
But to do a replace, I needed to preserve the non-tick values on either side of the tick. Luckily, Eclipse replace understands regex groups. I changed the match to capture the non-ticks:
([^']{1})'([^']{1})
And replaced matches with
$1''$2

I thought that was handy.

On a side note, don't use colons in your key name in a PropertyResourceBundle.
important:Message=Watch out!
Properties parses that as
key = important
value = Message=Watch out!
Properties.load equates ":" with "=". Who knew? Not me.

mailing lists as rss

2006-06-01T09:40:00.000-07:00

I can't stand mailing lists. They're just so goofy to subscribe/unsuscribe/track. I was pleasantly suprised to find most major Java related mailing lists (and lots more) are published by gmane as rss feeds:

http://blog.gmane.org/index.php?prefix=gmane.comp.java

Inflammatory Ruby Questions

2005-12-16T13:39:00.000-08:00

I have some honest questions for Ruby experts to help me understand where I am on this Ruby debate. Not that any Ruby experts, or anyone reads this blog, but it makes me feel good, ok?
I honestly want to understand why and how Ruby is going to be a revolution. I absolutely love Java, but I am deathly afraid I will my love for it will blind me from seeing the next big thing.

I'd like to seperate Ruby and Ruby on Rails for clarification.

Ruby:
What is it about this language that seperates it from the dynamically-typed crowd? Does it hit the sweet spot between loose typing and "true" OO features? Is it fair to Ruby a great dynamically typed language, and short circuit this half of the discussion as the ancient "staticly versus dynamically typed languages" debate? My point is I dont see much of a difference between "Ruby v. Java" and "Static v. Dynamic typing" debates. Is there something novel Ruby adds to that debate? Perhaps Ruby does for dynamically typed languages what Java did for statically typed languages?

Ruby On Rails:
What does Ruby on Rails provide that other language web frameworks cannot? I'm sure "convention over configuration" is a fantastic innovation for small, simple webapps, but what precludes another framework from mimicking this, such as Trails? I understand Ruby ensusiasts' frustration w/ "XML hell", but that's not a function of Java in the least, but in the frameworks they choose.

If these were both true statements:
"Ruby is the best dynamically typed language available"
"Ruby on Rails is an innovative web framework for writing simple webapps with very little work"
I see plenty of room left for Java.

Simple logging improvement

2005-12-13T05:56:00.000-08:00

No matter how clever you are about logging, you will run in to problems in production where you need more information. Often times, I'll resort to access logs to try to understand more. I recently made two changes to the default access log format in Tomcat which have drastically increased the utility of these logs. I added session ID (%S) and "Time taken to process the request" (%D). If you take care to log session ID in your normal logs, you'll have a nice "foreign key" into the access logs.

Tiger Beat!

2005-12-12T17:28:00.000-08:00

1. Enhanced for loop is pure coding joy, especially in nested loops, but you will be suprised at how often you can't use it: you need the index, special exit condition (I'm not a "break" man), or you have an iterator.

2. Generics will improve your API dramatically. The rap on them is that creating your own generic classes can get messy, and that's true. But it's not _that_ bad, and 99% of the time you are simply using generic classes, which is completely simple.

Before generics, we often deliberated about what type to return when a method produced a group of objects (Not a group of "expensive" objects, like objects read from a database - they pretty much necessitate an Iterator). Arrays were succinct and type-safe. Collections were what the method often used internally and were more flexible, but lacked type safety. For us, in a big system, readability won out, and we chose arrays. Overall, I was happy with the choice, but it didn't sit well how often we'd convert a List to an array at the end of the method. Generics resolved this completely. Now all methods return "generified" collections objects, and we get the best of both worlds.

Map.get/containsKey/remove and Set.contains are NOT generic. Good discussion:
http://forum.java.sun.com/thread.jspa?threadID=465357&messageID=2139377

It takes a while to really get used to the fact that generics info is not available at runtime. Once you understand that, you'll find yourself more a user of generics classes than an author of generics classes.

3. java.util.concurrent - Great libraries change how you think about problems. java.util.concurrent is no exception. Simple concurrency problems are now actually simple to solve. I'm able to express amazingly powerful concurrent problems in simple and safe code. I don't know how I lived before ConcurrentHashMap, Executor, Callable, and Future. I now look forward, rather than dread, the next concurrency programming problem.

4. @Override - Love it. Even though I haven't programmed Perl in 8 years, I still get a thrill when the compiler points out a problem I never would have noticed. Errors that @Override solve are pretty rare, but they've saved me once, and that's enough for the "Love it" designation. While I understand it would be trickier, an @Implements would be cool to. @SuppressWarning allows to turn certain compiler warnings on which I avoided in the past, because I didn't want warnings for the handful of "special cases" of a warning.

5. String.contains - Can you believe it? I feel like a Programmer of the Future every time I use this baby.

6. OutOfMemory actually TELLS you what kind of memory we ran out of! Before 1.5, you had to figure out what kind: heap? perm heap? thread stack? They were clearly emboldened by the rejoicing of this fix, and tackled an even MORE amazing problem in Mustang: ClassCastException will tell you what was casted!

7. Thread.getStackTrace/getAllStackTraces. If you run into race condition/locking problems, and don't have the luxury of reproducing in your IDE, a JSP that displays Thread.getAllStackTraces is pretty useful.

8. enums - Typesafe enums are essential for clarity in big APIs, and I hope they get more attention now that they're first class in Java. If you are leary of Generics introducing clutter to API, you won't like this:
public abstract class Enum>
I tried writing a utility method for enums, and it actually set my brain on fire.

Eclipse has a neat compiler option - to warn/error if a switch statement doesn't cover every member of the enum. This is a wonderful tool, as you often want to take an action based on the value of an enum, and it's imperative to cover every possible value of the enum. However, I haven't used switches or breaks since I was a C programmer. I was torn between the joy of a new compiler warning, and the mess that is "switch". I tried and tried, and eventually just had to drop the switch. Its freaky scoping and flow was just too much to swallow.

9. Monitoring - Anybody who's had problems in production will love JConsole and java.lang.management.ManagementFactory. You'll also love System.nanoTime if you've ever tried timing operations with System.currentTimeMills. currentTimeMillis could not measure down to the millisecond, so any "fast" operation would result in times of 0, 16 or 32 milliseconds. Not a problem with nanos!

I'm sure I'm forgetting quite a bit, but in short, I'm in love. Tiger represents a good balance between Sun's traditional conservatism and the pressure they face from their emerging competition. I hope they maintain their discipline as well as their fear.

i18n zip file woes

2005-10-23T06:51:00.000-07:00

Update: This is being addressed in JDK7. It will still be up to your to figure out what encoding your zip file is in - no small feat if you've got zips coming in from various sources/locales.

Our app allows customers to upload zip files of content. Recently, customers in Brazil were having problems with a file they were uploading. We'd get this exception when trying to unzip it:


java.lang.IllegalArgumentException
at java.util.zip.ZipInputStream.getUTF8String (ZipInputStream.java:291)
at java.util.zip.ZipInputStream.readLOC (ZipInputStream.java:230)
at java.util.zip.ZipInputStream.getNextEntry (ZipInputStream.java:75)

Inspecting the file, I found they were including Portuguese characters in the file name. I was pretty suprised to find this issue was related to not one, but two of Java's Top 25 Bugs. The Sun engineers point out the unfortunate fact that the Zip spec doesn't say anything about encoding of file names. The only thing Java could do better is allow people to pass in their own encoding when instantiating a ZipFile. A commentor even provides a patch for ZipInputStream to allow this. I found this solution didn't even cut it for me. The customers were using some German zip program (a truly international problem!), and I couldn't get a clean unzip with any encoding. When I discoved both WinZip and Windows XP "compressed folders" feature were baffled as well, I threw in the towel. People waiting for Sun to fix this are probably in for a nasty suprise.

For the Brazillian customers, I had them re-create the zip using a nice little graphical jar utility - ZipOutputStream uses UTF-8, which will have no trouble with their i18n file names. Another good alternative would be 7-zip, which considered i18n issues from the start. But, suprisingly, their Java support seems pretty lacking.

For the grim details of this problem, I can't say it better than the WinZip tech support person:


>There is unfortunately some ambiguity within Zip files as to the
>character set that the filename is stored in. Whenever a file can be
>stored in the OEM character set (which is the character set originally
>used by MS-DOS), WinZip does so for compatibility with other Zip
>utilities. In this case, it marks the Zip file as made by MS-DOS, since
>the original Zip utilities were DOS utilities that used the OEM
>character set for filenames.
>
>For files whose names can't be stored in OEM, because they contain
>characters not present in the OEM character set, WinZip stores the
>filename in ANSI and marks the file as having been made on an
>NTFS-based system. WinZip is here following the lead of InfoZip, a Zip
>utility that since 1993 used this method of marking filenames as being
>stored in the ANSI character set. (ANSI is the character set used by
>Windows 95, 98, and Me, and by many Windows applications. Windows 2000
>and Windows XP use yet another character set, Unicode, but they also
>provide support for the ANSI character set.)
>
>However, some other Zip utilities handle this differently. In
>particular, Microsoft's Compressed Folders program always marks the
>files it creates as having been made on an NTFS-based system, but
>stores the filename in the OEM character set.
>
>So when WinZip sees a file marked as NTFS-based, it has no way to be
>absolutely sure whether the filename is in the OEM character set (and
>needs to be translated to ANSI), or if it is already in the ANSI
>character set. WinZip uses some complicated heuristics to try to decide
>which character set is involved, and it almost always comes up with the
>"right" answer, but there is no way to be absolutely right in every
>case.
>
>We are looking into how to handle this better in a future version of
>WinZip.

Crystal ate my log4j!

2005-10-22T11:29:00.000-07:00

When solving a problem on a live site, logs are indespesnable. So imagine my frustration last spring when I went to solve a problem, and no logs were there. None. Everything had just stopped logging. Solving a problem with access logs alone is not an enviable task. I searched around a bit, and found this

I was indeed using the log4j WatchDog to periodically watch log4j.xml for changes. And we do hot restarts of the context from time to time. I wasn't convinced this was the problem, as I could not reproduce the problem no matter what I did with the WatchDog and hot restarts. So I set the system property "log4j.debug" to "true", in the hopes that next time it happened, I'd get some kind of info. I also wrote code that ensured any time a Logger object is acquired, a System.out Console appender was attached to it, at the very least.

The logs continued to disappear, but thankfully output went to stdout, and was captured. Then, log4j.debug paid off. In the middle of a log, I saw this:

log4j: Reading configuration from URL jar:file:/E:/webapps/app/WEB-INF/lib/celib.jar!/META-INF/CrystalEnterprise.Trace/basic.properties
log4j: Parsing for [root] with value=[ERROR, A1].
log4j: Level token is [ERROR].
log4j: Category root set to ERROR

Whoah! Sure enough, all of my categories and appenders were blown away at this point. It didn't happen at startup, but whenever the first Crystal report was launched. After much consternation and trying to decipher the log4j comment in the release notes, my coworker came up with setting the system property "crystal.enterprise.trace" (yes, not even mentioned in the release notes) to "false". Now we can launch Crystal reports AND use log4j - oh, the decadence!

I'm no log4j expert, so I wonder how Crystal should have played more nicely here. Surely, there's a away to use PropertyConfigurator to amend logging configuration, rather then obliterate it.

Lessons learned:
1. Abstract your code such that you can intercept acquisition of Logger objects. Use this code to ensure that a Logger is at least guaranteed to have a Console appender, and escalate the problem if it indeed has zero appenders.
2. Set log4j.debug to true at all times
3. When using crystal, be sure to set crystal.enterprise.trace to true (unless, of course, you care about their logs more than your own).

Microsoft to buy Redhat?

2005-10-22T11:20:00.000-07:00

I've finally gotten over my fear that I'd create a totally useless, misguided, and confused blog. I've gotten over it - I now know this blog will be be useless, misguided, and confused. My only three goals are:

Document weird problems I've encountered in the hopes that I can help someone avoid my frustration
Have inflamatory titles to attract people to read my blog, and respond to it.
Learn from responses to my posts
Vent.

I'm a Java programmer who has been working on roughly the same product since about 1998. That being said, I won't have many smart things to say about the hot topics of the day: AJAX, RoR, or Web 2.0. I will hopefully be able to share interesting experience on more mundane topics, like JDBC, i18n, webapp performance, maintainable code, and other such things.