Archive for the 'java' Category

Project Euler Test Harness

Project Euler is a collection of mathematics/computer science problems, as you probably already know.  I've solved almost 50 of them so far, and I've developd a collection of utilities to make my job easier.  Some of them (factorization routines, prime generators, etc.) I'm not going to share as they are fundamental to solving certain problems.  However, I will share my test harness, as it's probably generally useful, and it contains no "secrets".

Why build a custom harness?  Well, I originally considered using JUnit (or whatever) as a simple runner, but since there isn't anything to test, it didn't feel right.  It also doesn't provide a good way to get timing information for the runs.  And since I was originally using strictly Groovy, I didn't want to impose any more "ceremony" than I needed to.  I've since started doing some problems in Java simply for the performance (as well as converting some of those "secret" routines), and have generalized my harness.

First, here are the file templates for Groovy:

import util.Runner

new Runner() {
  // put some code here, eh?
}

and for Java:

import util.Runner;
import util.java.Solver;

public class BruteForce {
  public static void main(String[] args) {
    new Runner(new Solver() {
      public Object solve() {
        // put some code here, eh?
        return null;
      }
    });
  }
}

As expected the Java version requires quite a bit more ceremony, even though the semantics of the two templates are identical.  The Solver interface (used for Java) is as follows:

package util.java;

public interface Solver {
   public Object solve();
}

Finally, the Runner class, which is implemented in Groovy.  As you can see from the templates, it accepts either a Solver instance or a Groovy closure, and treats the two in exactly the same way, excepting that it picks between Solver.solve() and Closure.call() based on the type.

package util

import util.java.Solver

class Runner {

  private static ThreadLocal timer = new InheritableThreadLocal()
  private static int timerCount = 0

  def Runner(solver) {
    this(true, solver)
  }

  def Runner(doIt, solver) {
    if (! doIt) {
      return
    }
    startTimer()
    // this will send an update every five or so seconds
    volatile def updateThread
    updateThread = Thread.startDaemon {
      while (true) {
        Thread.sleep(5000)
        if (updateThread == null) {
          break // after the sleep, before the log
        }
        log("still executing...")
      }
    }
    try {
      log("starting...")
      def result;
      if (solver instanceof Solver) {    // the closure/Solver switch
        result = solver.solve()          //  |
      } else { // better be a Closure    //  |
        result = solver.call()           //  |
      }                                  //  V
      stopTimer()
      if (result instanceof String) {
        try {
          result = new BigInteger(result)
        } catch (e) {
          // oh well
        }
      }
      if (result instanceof Integer || result instanceof Long || result instanceof BigInteger) {
        log("result: $result")
      } else {
        def s = "" + result
        if (s.length() > 100) {
          s = s[0..<97] + "..."
        }
        log("result was not numeric: $s", System.err)
      }
    } finally {
      updateThread = null
    }
  }

  def elapsed() {
    get().elapsed
  }

  def formatElapsed(e) {
    def seconds = e.intdiv(1000)
    def millis = e % 1000
    def s = ""
    if (seconds >= 60) {
      s += seconds.intdiv(60) + ":"
      seconds = seconds % 60
    }
    if (s.length() > 0 && seconds < 10) {
      s += "0$seconds"
    } else {
      s += seconds
    }
    if (millis < 10) {
      s += ".00$millis"
    } else if (millis < 100) {
      s += ".0$millis"
    } else {
      s += ".$millis"
    }
    s
  }

  def log(Object msg, out=System.out) {
    def t = get()
    out.println "[${t.index}: ${formatElapsed(t.elapsed)}] $msg"
  }

  private startTimer() {
    Runner.timer.set(new TimerData(++timerCount))
  }

  private stopTimer() {
    get().stop()
  }

  private get() {
    def t = Runner.timer.get()
    if (! t) {
      throw new IllegalStateException("No timer has been started...")
    }
    t
  }
}

class TimerData {
  def index
  def startDate
  def stopDate

  def TimerData(threadIndex) {
    index = threadIndex
    startDate = new Date()
  }

  def stop() {
    stopDate = new Date()
  }

  def getElapsed() {
    if (running) {
      return new Date().time - startDate.time
    } else {
      return stopDate.time - startDate.time
    }
  }

  boolean isRunning() {
    stopDate == null
  }
}

There's a lot of stuff going on there, but primarily it's setting up a timing framework, a background thread to tell you it's still running every 5 seconds, and doing some magic with the returned solution.  As you can see, the constructor takes an optional first parameter for whether to execute.  Useful if you have multiple Runners in a single script/class, but only want to run some of them.

It wasn't my intent, but building this out was quite educational in the way the Java/Groovy interplay works.  Write Groovy for whatever you can, and write Java for the bits that need it.  As you can see from the templates, Java requires a lot more work, but it has benefits.  I was quite pleased to be able to use my Groovy harness (which leverages all kinds of Groovy goodness) for my Java solutions.  I originally figured I'd have to port it to Java to use it with both languages, but not the case.

Eclipse and JUnit

I started building a little Java app this evenings.  First time I've done Java SE work in a long time, and it's kind of nice.  Not sure why I picked Java instead of Groovy, but whatever.  Probably the same reason people go home for the holidays.

In any case, being the good developer I am, after designing the core interfaces, I wrote some unit tests using JUnit 4.  Right click on the file in Package Explorer, hit "Run as JUnit Test" and it fires up in a fast view to show you how you did.  All very handy.  After a few tests, i figured I should build some test suites, but I really didn't want to.  I don't know about anyone else, but I always end up forgetting to register some test or another, and then I create a regression that would get caught by that test, but since it's not registered it doesn't run, and it's just bad.  So I didn't, and continued to run the tests individually.

After half an hour or so so, I had the idea to try right clicking on a package and see if I could run all the test in that package.  I was quite pleased to see that you can.  Same thing works at the source folder and project levels.  Little efficiency improvements like that make Barney a happy boy.  Of course, still have to write the suites if I want to run them from CI or the like, but this project at least will very likely never get there.

Java SE 1.6.0_10

Just did the 1.6.0_10 upgrade on my server.  Oh how I love Linux.  Just unarchved the new JDK, repointed a single symlink and restarted my Java servers.  There didn't seem to be a significant difference in spinup time for my ColdSpring AOP-heavy apps - perhaps a touch faster.  No problems with Magnolia either as near as I've been able to tell.  WordPress, not surprisingly, was unaffected.

Even More Groovy

I have a confession to make.  My neat little code counter utility is partially a lie.  I wrote the entire thing as a single Groovy script, and only split it up into individual files for each class to release it.  271 lines of code (419 if you count the blank lines) simply does not justify seven distinct files.  That said, it does justify a runner script and six distinct classes.  So if you want the "real" source, just take the six classes, copy them directly into Runner.groovy and then delete the files.  Everything works exactly the same, except it's all in one concise package.

The project started with a pile of procedural code (i.e. a Groovy script), all inline with no classes at all.  My basic concepts proved sound, and I started refactoring.  Does anyone NOT like refactoring?  If so, raise your hand so I'll know who to slap.  ;)  Classes quickly started falling out of the proof-of-concept code, and soon I had my four core classes (CodeCounter, ProjectInfo, PathInfo, and FileInfo), plus some no-longer-quite-functioning output code.

Once the "backend" was refactored into a stable model, the front-end went just the same.  XML was the format of the original script, and so I reimplemented the same output based on the model, rather than emitting it inline.  To ensure I wasn't being stupid and coupling where I shouldn't, I threw together the text renderer as well.

I'm a huge fan of this style of development, and do much the same thing with CF.  Start writing one big-ass file with no real concern for parameterization, abstraction, or encapsulation.  Once it works "enough", start refactoring for those three characteristics, as well as extending the functionality into something useful to the real world.

Doing Java is a bit harder, since you have the object nature foisted on you from the start.  You can still begin with one class and a pile of static methods, but the refactoring is a bit harder, I think, because of the static/dynamic transition.  With CF and Groovy, you pretty much bypass all of that (thank you compilers!), which is very nice.

What's the downside to this style of development?  Unit testing is hard.  With a rapidly evolving project (structurally), having to evolve unit tests can be a significant burden.  Not to mention that you can't really unit test a single large script; you're stuck waiting until you have an class which is after most of the work is done.  So if you write unit tests at all (I didn't), they have to come late in the cycle.

The main benefit is that you don't have to think about architecture before implementing functionality.  If you know exactly what your desired functionality is, and know how you're going to approach implementing it then architecture is the "hard" part, and unit tests can help ensure your implementation matches what you expect/require.  However, that's not the case a lot of time.

CF Groovy Preso

Last night I talked at the PDX RIA group about using Groovy to extend CFML applications and better leverage some of the benefits that Java has to offer (in addition to the benefits Groovy itself brings).  Unfortunately we had some technical difficulties, but the presentation was recorded on Connect, and can be accessed at http://experts.na3.acrobat.com/p71343823/.  I apologize for the delay in posting, I usually try to get things up immediately afterward, but just didn't happen this time.

In addition to the Connect recording, I've posted my slides (in PDF format).  The sample app I demoed is just the CF Groovy demo app available at https://ssl.barneyb.com/svn/barneyb/cfgroovy/trunk/demo.  Within the demo app is the framework itself, including Hibernate (whose JARs are the source of the large download).  The demo app is also available online should you wish to run it without having to download anything.  Of course, there's not much to see without the code.

Thread.setContextClassLoader Doesn't Work on CF 8.0.0

I know the number of CF 8.0.0 installs is probably pretty minimal compared to CF 8.0.1, but thought this was worth pointing out.  If you use Thread.setContextClassLoader on CF 8.0.0, it raises no exception, but it doesn't actually set the ClassLoader for the thread.

I ran into this today using my CF Groovy/Hibernate integration on one of my existing CF installs that is still 8.0.0.  It totally bombs, because the classloading trickery fails if it can't set up a classloader.  After spending a bunch of time on it, I just gave up, as nothing I tried had any effect.

Both CF 8.0.1 and Railo 3.0b2 correctly allow overriding the context ClassLoader, and can therefore use the Hibernate via CF Groovy.  I haven't tried in a while, but OBD exhibited similar symptoms as CF 8.0.0 when trying to run Hibernate, so it may suffer from the same problem.  This only affects the Hibernate stuff I'm doing, the base Groovy integration requires no ClassLoader trickery, so it works on all four without issue (and probably CF 7 too).

I should also note that this is NOT the underlying JVM.  I use the same JVM (a 1.6 series from Sun) for all my stuff, both development and production.

Fun with Classloading in Java

With my CF Groovy project, I've been doing a lot of Java coding of late. It's all been written in CFML, but it's mostly Java code in CFML syntax. As an aside, doing that sucks. CFML is horrible for writing anything more than the simplest Java code. But on to the topic at hand….

To this point in my career, basically all of the Java code I've written has been application code. Most of the CFML, JavaScript, and ActionScript I write is also application code, but I've contributed significantly to Fusebox and built a number of micro-frameworks in all three languages. However, none of that touches on the stuff I've done in the past couple weeks. The primary source of my issues getting Groovy and Hibernate to play together was classloading.

Unlike Grails (GORM, really), any of the various examples on the web, and I'm guessing Joe Rinehart's Hiberailooving project, three of the inviolate framework objectives were zero dependency on an IDE beyond text editing, no manual compilation, and no container restarts. Obviously, Hibernate provides a set of IDE tooling that can be brought to bear, but I'm talking about just the CF integration portion. I want to be productive with emacs and a web browser.

This presented a number of issues with classloading, because Groovy can only be loaded by the GroovyClassLoader (and subclasses). Basically this forced my hand to using compiled Groovy code, and ensuring it and Hibernate are loaded by the same ClassLoader. No GroovyClassLoader at all.

Now all that's well and good, but you have to be a bit careful, because two of the goals where no manual compilation and no container restarts. Doing compilation from CF is fairly simple (instantiate FileSystemCompiler and call 'main' with the right args), though again, getting all the right stuff on the classpath in the right order proved a nightmare. This is further complicated by the while JEE classloader situation, which isn't straightforward at all.

The details of this intricate dance completely escaped me for a week and a half. Then one night, while lying in bed, it all clicked together (thank you, Heather, for going to England). The solutions seem fairly obvious now, but they took a lot of fussing and fighting to deduce. I'd never even instantiated a ClassLoader before, aside from a couple one-off URLClassLoaders in CFML to lazy-load some JARs. Now I've got stacks of ClassLoaders, in some places two or three deep.

After my initial post-epiphany implementation was complete, I realized I can do some streamlining, but that's for after the release. I think I can even remove the need to drop the Groovy JAR in your /WEB-INF/lib folder, without adding any code complexity or runtime overhead, which would a nice bonus for ease of deployment. I.e., unzip the engine, write an entity, save it to the DB. No JARs, no restart the container, nothing.

It's been a very interesting (and pleasurable) experience, since it's so different from my day job of banging out boring little presentation apps all day. I've learned a lot, and even just the past few days of thinking about how other Java apps/frameworks work, I've realized how applicable that knowledge really is.

CF Groovy Takes a Nap

After close to two weeks of struggling, I finally managed to deploy pure source to a CFML runtime (Railo, in this case), and get Groovy entities in and out of the database with Hibernate.  No compliation, no IDE, no development-mode server, just my Groovy source along with a hacked up CF Groovy.  This is very exciting for me, because while Groovy is cool and all, CF's dynamic nature has become a must have.

I don't have source to download yet, just a horribly hacky proof of concept, but here's what the user of the "framework" does.  First, create your entity:

package com.barneyb
import javax.persistence.*

@Entity
class Person {

  @Id
  @GeneratedValue
  Long id
  Long version
  String name
  Date dob

  def getAgeInSeconds() {
    (new Date().getTime() - dob.getTime()) / 1000
  }

  def getAgeInDays() {
    ageInSeconds / 86400
  }

  def getAgeInYears() {
    ageInDays / 365.249
  }

  String toString() {
    "$name is $ageInYears years ($ageInDays days) old"
  }

}

That's the same class as my initial CF Groovy demo used, except with the of JPA annotations added.  Because I'm binding to Hibernate, you can use any Hibernate-supported annotations, but I opted to stick with vanilla JPA.  Now we need our hibernate.cfg.xml, which is totally stock:

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
  "-//Hibernate/Hibernate Configuration DTD//EN"
  "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>

  <session-factory>

    <property name="current_session_context_class">thread</property>
    <property name="cache.provider_class">org.hibernate.cache.NoCacheProvider</property>
    <property name="show_sql">true</property>
    <property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
    <property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
    <property name="hibernate.connection.url">jdbc:mysql://localhost:3306/dbname</property>
    <property name="hibernate.connection.username">username</property>
    <property name="hibernate.connection.password">password</property>
    <property name="hibernate.hbm2ddl.auto">update</property>

    <mapping class="com.barneyb.Person" />

  </session-factory>

</hibernate-configuration>

You could also map your classes manually with XML, if you wanted, but for the sake of simplicitly, I've done it all with annotations.  You'll also notice the database connection information embedded directly in the file. This is far from ideal.  I'm hoping to extract it from a CFML DSN, but that might only be possible on Adobe ColdFusion (via the Admin API).  The last thing is the test script:

<g:script>
  import java.text.SimpleDateFormat

  import org.hibernate.*
  import org.hibernate.cfg.*

  import com.barneyb.Person

  sdf = new java.text.SimpleDateFormat("yyy/MM/dd")
  sdf.lenient = true

  def sf = new AnnotationConfiguration()
      .configure().buildSessionFactory()
  def sess = sf.openSession()
  def tx  = sess.beginTransaction()
  sess.createQuery("delete Person").executeUpdate()
  sess.save(new Person(
    name: attributes.name ?: "Barney",
    dob: sdf.parse(attributes.dob ?: "1980/06/10")
  ))
  tx.commit()
  sess.close()
  sf.close()
</g:script>

No need to compile anything, no need to drop JARs all over your server, nothing.  Write, refresh, persist.  That includes adding new entities, new properties to existing entities, etc.

It's not ready for public consumption at the moment, but that's my top "free time" priority.  For the incredibly impatient, there's a branch in my SVN repository with the current source.  Don't expect anything pretty.  ;)

Closures for Java

Wanting some light reading for this evening, I decided to dig into the Closures for Java draft spec (homepage).

Oh. My. God.

I'm a huge fan of closures. They're elegant, simple, and easy to use. Their semantics are subtle, but not confusing. They allow for very concise implementation of otherwise complicated algorithms, trimming a vast number of "normal" control structures, particularly various looping constructs.

Java already has (anonymous) inner classes which provide similar semantics, though with a marginally clunkier interface. This mechanism is basically how closures in Groovy are implemented, with some compiler magic to do the heavy lifting.

If you go look through the Closures for Java spec, however, what you see is appalling. Closures are about simplicity and elegance, and the spec outlines exactly the opposite. Closures themselves are exactly what you'd expect, using the Groovy (or Ruby) syntax, though types are required for the arguments. But what you don't expect is the function type syntax, the closure conversion rules, generics support, etc. It's a mess. Check this example:

String name;
{int => long} transform;

The first line declares a variable of type "java.lang.String". The second declares a variable of type "function that takes one int argument and returns a long". Note that that is different from a variable declared as "{int => int}", of course. Even better, any closure that can be assigned to the 'transform' variable implicitly implements every single-method interface whose method accepts an int and returns a long. It's unclear to me whether the converse is true - I couldn't take any more dry spec-speak.

The spec is pretty short (less than nine pages), and if you do any Java, it's definitely worth perusing. After reading, you'll realize that anonymous inner classes are just fine, thank you very much, even with the requirement of having to declare the interfaces explicitly. With the amount of baggage they seem to be lugging into Java (mostly because of it's typing), all I can say is "yuck." Closures require a bit more dynamic an environment to be effective, I think, and Java is static. Totally static.

ActionScript 3 provides an interesting middle ground that provides the benefits but culls a lot of the garbage. In AS3 you declare functions with typed arguments and an explicit return type. However, every function is of type function, which means if I have an algorithm that requires an {T, T => int} closure (that's the Comparator interface, by the way), I can happily pass in any old function I want and the compiler won't care. I'll get an error at runtime, of course.

Much the dancing the spec details is to protect developers from this potential situation, and the rest is to deal with backwards compatibility an interoperability with older libraries. I understand that Java is a strongly typed language, and that the compiler is omnipotent, but c'mon. Backwards compatibility is also very useful, but again, to what extent. You can already thrash a generics-based library with reflection, because it's absolved from compiler checks. It seems like there should be a point where you recognize the fact that the language has very deliberately poked holes in it's own security structures, and realize that that means trying to create airtight compiler checking is pointless.

I guess that's why Groovy got created, though. Backwards compatibility is important for Groovy as well, but the compiler (javac, mind you, not groovyc) and all it's type checking wonder has basically been given the finger since everything is just an Object and duck-typed as needed. Quack! I'm a developer that likes transparency in my compiler. Compiler writers are supposed to be smart, they can figure out what I'm doing and tell me if it'll work or not. I shouldn't have to give them explicit instructions, at least not that explicit.

NB: Function pointers (a la CFML) are NOT closures. Closures bind to their definition scope, function pointers just point at a function with no bound scope. I've heard the argument that function pointers are equivalent/sufficient. They're not.

Scriptlets in CF Anyone?

My last post about Comparators via CF Groovy was simplistic in nature, but the underlying concept is incredibly powerful.  Here's a similar snippet (from the demo app), this time using a Comparator class:

<g:script>
Collections.sort(variables.a, new ReverseDateKeyComparator())
</g:script>

So what do we have here?  Why it's a snippet of Java embedded directly in your CFML page, and it gets dynamically compiled and executed just like the CFML.  Let me say that again: it's a snippet of Java embedded directly in your CFML page.  It's even better than that, though, because it's Groovy, which is Java plus a whole lot more.

In addition to the scriptlet functionality, CF Groovy provides a classloading framework for Groovy scripts and classes, in addition to the ones provided by the CFML runtime and the JVM.  You can define an extra Groovy classpath for your scriptlets like this:

<g:setPath path="#expandPath('/groovy')#,#expandPath('/packages')#" />

There's no way to tell how the ReverseDateKeyComparator class in the first example is implemented, but it happens to be a Groovy class that I've added to the classpath with the <g:setPath> tag.  That class, just like the scriptlet, is dynamically compiled and executed just like the CFML.  This is also very powerful, because it eliminates the compile/build/deploy cycle that is usually required for Java code in a JEE environment.

Combining these two, you can use <g:script> to execute arbitrary scripts from the Groovy classpath, instead of inlining them:

<g:script name="myScript.groovy" />

This lets you reuse scripts across CFML pages, as well as reusing classes between scripts.

I've added CF Groovy to my Projects page, so that's the place for the latest info, as well as some documentation and the all-important Subversion information.