Macports voodoo distortion field

Today I was building the opensprints-core gem. The Rakefile depends on Jeweler which depends on the Git ruby gem. No problem. I got all that flotsam and jetsam installed and wouldn't you know my version of git was too old. Shaving yaks is dangerous work. I broke out into a cold sweat because I knew git was installed via macports and macports works by rube-goldberg-inter-dimensional-voodoo-distortion-field, 800 rolls of duct tape and a buttplug. Let's just say I've had some bad experiences with it. On top of that I haven't tried updating anything ports related for a year or two. So let's upgrade.

sudo port selfupdate
sudo port upgrade outdated

Sure enough, I start upgrading and BOOM. Some obscure library fails to build.

--->  Building cyrus-sasl2
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_security_cyrus-sasl2/work/cyrus-sasl-2.1.23" && /usr/bin/make -j1 all " returned error 2
Command output: /usr/bin/make  all-recursive
Making all in include
make[2]: Nothing to be done for `all'.
Making all in sasldb
ar cru .libs/libsasldb.a db_ndbm.o allockey.o
Making all in plugins
make[2]: Nothing to be done for `all'.
Making all in lib
make[2]: Nothing to be done for `all'.
Making all in utils
/bin/sh ../libtool --mode=link /usr/bin/gcc-4.0  -Wall -W -O2 -arch i386  -L/opt/local/lib -Wl,-rpath,/opt/local/lib -L/opt/local/lib -o pluginviewer  pluginviewer.o ../lib/libsasl2.la   -lresolv -lresolv  -lpam
/usr/bin/gcc-4.0 -Wall -W -O2 -arch i386 -L/opt/local/lib -Wl,-rpath,/opt/local/lib -L/opt/local/lib -o .libs/pluginviewer pluginviewer.o -L../lib/.libs -lsasl2 -ldl -lresolv -lresolv -lpam -lresolv -lresolv -lpam
Undefined symbols:
"_auxprop_plugin_info", referenced from:
   _main in pluginviewer.o
   _main in pluginviewer.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make[2]: *** [pluginviewer] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Luckily the Google knew how to fix it.

{{{
sudo port deactivate cyrus-sasl2
sudo port clean cyrus-sasl2
sudo port upgrade cyrus-sasl2
}}}

Then I hit the wall

--->  Building MPlayer
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_multimedia_MPlayer/work/MPlayer-1.0rc2" && /usr/bin/make -j1 all " returned error 2
Command output: /usr/bin/gcc-4.0 -I../libavcodec -I../libavformat -Wdisabled-optimization -Wno-pointer-sign -Wdeclaration-after-statement -I. -I.. -I../libavutil -Wall -Wno-switch -Wpointer-arith -Wredundant-decls -O4 -march=i386 -mtune=i386 -pipe -ffast-math -fomit-frame-pointer -mdynamic-no-pic -falign-loops=16 -DSYS_DARWIN -shared-libgcc -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAVE_CONFIG_H -I/opt/local/include/lzo -I/opt/local/include -I/usr/local/include   -c -o ao_mpegpes.o ao_mpegpes.c
/usr/bin/gcc-4.0 -I../libavcodec -I../libavformat -Wdisabled-optimization -Wno-pointer-sign -Wdeclaration-after-statement -I. -I.. -I../libavutil -Wall -Wno-switch -Wpointer-arith -Wredundant-decls -O4 -march=i386 -mtune=i386 -pipe -ffast-math -fomit-frame-pointer -mdynamic-no-pic -falign-loops=16 -DSYS_DARWIN -shared-libgcc -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAVE_CONFIG_H -I/opt/local/include/lzo -I/opt/local/include -I/usr/local/include   -c -o ao_null.o ao_null.c
/usr/bin/gcc-4.0 -I../libavcodec -I../libavformat -Wdisabled-optimization -Wno-pointer-sign -Wdeclaration-after-statement -I. -I.. -I../libavutil -Wall -Wno-switch -Wpointer-arith -Wredundant-decls -O4 -march=i386 -mtune=i386 -pipe -ffast-math -fomit-frame-pointer -mdynamic-no-pic -falign-loops=16 -DSYS_DARWIN -shared-libgcc -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAVE_CONFIG_H -I/opt/local/include/lzo -I/opt/local/include -I/usr/local/include   -c -o ao_pcm.o ao_pcm.c
/usr/bin/gcc-4.0 -I../libavcodec -I../libavformat -Wdisabled-optimization -Wno-pointer-sign -Wdeclaration-after-statement -I. -I.. -I../libavutil -Wall -Wno-switch -Wpointer-arith -Wredundant-decls -O4 -march=i386 -mtune=i386 -pipe -ffast-math -fomit-frame-pointer -mdynamic-no-pic -falign-loops=16 -DSYS_DARWIN -shared-libgcc -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAVE_CONFIG_H -I/opt/local/include/lzo -I/opt/local/include -I/usr/local/include   -c -o ao_openal.o ao_openal.c
ar r libao2.a audio_out.o ao_mpegpes.o ao_null.o ao_pcm.o ao_openal.o
ar: creating archive libao2.a
ranlib libao2.a
/usr/bin/make -C input
/usr/bin/gcc-4.0 -I../libavcodec -I../libavformat -Wdisabled-optimization -Wno-pointer-sign -Wdeclaration-after-statement -I. -I.. -I../libavutil -Wall -Wno-switch -Wpointer-arith -Wredundant-decls -O4 -march=i386 -mtune=i386 -pipe -ffast-math -fomit-frame-pointer -mdynamic-no-pic -falign-loops=16 -DSYS_DARWIN -shared-libgcc -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAVE_CONFIG_H -I/opt/local/include/lzo -I/opt/local/include -I/usr/local/include   -c -o input.o input.c
/usr/bin/gcc-4.0 -I../libavcodec -I../libavformat -Wdisabled-optimization -Wno-pointer-sign -Wdeclaration-after-statement -I. -I.. -I../libavutil -Wall -Wno-switch -Wpointer-arith -Wredundant-decls -O4 -march=i386 -mtune=i386 -pipe -ffast-math -fomit-frame-pointer -mdynamic-no-pic -falign-loops=16 -DSYS_DARWIN -shared-libgcc -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAVE_CONFIG_H -I/opt/local/include/lzo -I/opt/local/include -I/usr/local/include   -c -o ar.o ar.c
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:29,
                from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/DriverServices.h:32,
                from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:125,
                from /System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20,
                from /System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:21,
                from /System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:20,
                from ar.c:27:
/usr/lib/gcc/i686-apple-darwin9/4.0.1/include/xmmintrin.h:35:3: error: #error "SSE instruction set not enabled"
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/DriverServices.h:32,
                from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:125,
                from /System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20,
                from /System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:21,
                from /System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:20,
                from ar.c:27:
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:216: error: syntax error before '__m128'
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:218: error: syntax error before 'sd'
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:220: error: syntax error before '}' token
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:264: error: array type has incomplete element type
make[1]: *** [ar.o] Error 1
make: *** [input/libinput.a] Error 2

Error: Unable to upgrade port: 1

The good news is git got upgraded. Mplayer and who knows what else is not. Oh well, while I am being a hater I might as well mention that the real suckiness is the fact that Leopard doesn't have a package manager. And that IS the dumbest thing about OSX.

ShellSink adds inline tagging

If you don't know about shellsink you can find more info here. Release 0.2.2 adds a new feature. You can add tags to a command at the same time as you execute it. This is achieved by using inline comments. Here is an example:

cat /proc/cpuinfo #sysadmin:diagnostic

That command will show up in your shellsink history with the tags sysadmin and diagnostic. This nifty idea was given to me by Jesse Hallett at my shellsink presentation at the open source bridge conference in portland this summer. Thanks Jesse.

A JS Bach's Crab Canon on a Mobius Strip

If you don't know what a crab canon is see here. video link

Ruby is a Bloodsport

8th Light's 2009 ruby sparring tournament was posted at the end of August, but I just heard about it last week. I followed the simple instructions for getting started and was quickly rewarded with a very cool limelight app that drives the code you write. The coolest part of the app is that it analyzes your player and assigns a score. The score is based on efficiency, lines of code, test coverage, various metrics and of course how well your player does at hangman. I was immediately sucked in to the fun and challenge of the problem and I have created and refined my own player and submitted it for entry in the tournament. I thought I would share a few of the things I learned that might help others improve the scores of their own hangman players.

1. "Simplicity Score" depends on lines of code, including comments, so delete all comments from the code to maximize your simplicity score.
2. "Flog Score" is an easy one to maximize. Run flog separately on your code. Anything with a flog score above 9 will cost you points. Refactor bigger methods to smaller ones to keep their flog score below the limit.
   
3. "Time Score" can be tricky to maximize. One approach is to create a performance test to help drive out the bottlenecks in your code. The hangman uses a list of 10,000 words so any time you iterate over that complete list you kick holes in performance.

Those were the 3 biggest things that helped me get my player score up there. It was definitely lots of fun and a good way to sharpen skills. I'm looking forward to seeing how the tournament plays out. Drop me a line if you wanna talk shop on this. Hope you decide to join in. And a big thank you to the folks at 8th light for making this happen.

LambdaJ: the wild man's closure

A couple weeks back I was talking about using anonymous inner classes to lend a more functional style to your Java. A common response to this was "hey champ, just use Scala" or "hey tough guy, have you tried Clojure?" Thanks. In the immortal words of the Dude, "Obviously you're not a golfer." That aside, I wouldn't have anything useful to say about closures in Java but for a posting on tss that linked to me. As luck would have it that post also had a link to something REALLY interesting: Mario Fusco's lambdaJ. What is lambdaJ? Only one of the most interesting and creative uses of Java syntax I've seen since Szczepan Faber's Mockito came out a couple years ago.

LambdaJ is a library that very artfully plasters over some of the uglier bits of Java, like the collections framework and the lack of closures. I'm not going to waste prose trying to describe it. I'll just cut to the chase with some code:

//example of lambdaJ closure
Closure println = closure(); { of(System.out).println(var(String.class)); }
println.each("one", "two", "three");

//example of lambdaJ forEach
List personInFamily = asList(new Person("Domenico"), new Person("Mario"), new Person("Irma"));
forEach(personInFamily).setLastName("Fusco");

//example of lambdaJ sort
List sortedByAgePersons = sort(persons, on(Person.class).getAge());

The Catch
If you haven't seen that before, I bet you are excited. I was too the first time I saw it and I nearly ran out and refactored all the code I was working on to use lambdaJ. Then I decided to slow down and do some benchmarks. That is where I hit a snag. It turns out that all this magic comes with a price. Most iterative operations are an order of magnitude slower than java's for loops. Here is some benchmarking code.

import static ch.lambdaj.Lambda.forEach;
import static ch.lambdaj.Lambda.forEach;
import static ch.lambdaj.Lambda.on;
import static ch.lambdaj.Lambda.sum;
import static org.junit.Assert.assertTrue;

import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;

import org.junit.Test;

public class LambdaJBenchmark {
public class Sequence{

private List<BigDecimal> sequence;

public Sequence(int size){
sequence = new ArrayList<BigDecimal>();
for(int i = 0; i<size; i++){
sequence.add(new BigDecimal(i));
}
}

public int iterativeSum(){
BigDecimal sum = new BigDecimal(0);
for(BigDecimal value : sequence){
sum = sum.add(value);
}
return sum.intValue();
}

public void iterateEntireSequence(){
for(BigDecimal value : sequence){
value.intValue();
}
}

public int lambdaJSum(){
return sum(sequence, on(BigDecimal.class).intValue());
}

public void forEachEntireSequence(){
forEach(sequence).intValue();
}

}

@Test
public void lambdaJIsSlowerThanForForTenThousandItems() {
long startTime = System.currentTimeMillis();
new Sequence(10000).lambdaJSum();
long endTime = System.currentTimeMillis();
long lambdaJRunningTime = endTime - startTime;

startTime = System.currentTimeMillis();
new Sequence(10000).iterativeSum();
endTime = System.currentTimeMillis();
long javaRunningTime = endTime - startTime;

System.out.println("Summing 10000 numbers.  LambdaJ run time: " + lambdaJRunningTime + " and Java: " + javaRunningTime);
}

@Test
public void lambdaJIsSlowerThanForForOneItems() {
long startTime = System.currentTimeMillis();
new Sequence(1).lambdaJSum();
long endTime = System.currentTimeMillis();
long lambdaJRunningTime = endTime - startTime;

startTime = System.currentTimeMillis();
new Sequence(1).iterativeSum();
endTime = System.currentTimeMillis();
long javaRunningTime = endTime - startTime;

System.out.println("Summing 1 number.  LambdaJ run time: " + lambdaJRunningTime + " and Java: " + javaRunningTime);
}

@Test
public void lambdaJNestedLoopsAreMoreOrLessAsFastAsFlatLoopOfSameSize() {
List<Sequence> sequences = new ArrayList<Sequence>();
for(int i = 0; i<100; i++){
sequences.add(new Sequence(100));
}

//Seems that "warming up" lambdaJ makes a big difference.
//Without this line the second test is an order of mag. faster.
new Sequence(10).lambdaJSum();

long startTime = System.currentTimeMillis();
new Sequence(10000).lambdaJSum();
long endTime = System.currentTimeMillis();
long flatRunTime = endTime - startTime;

startTime = System.currentTimeMillis();
forEach(sequences).lambdaJSum();
endTime = System.currentTimeMillis();
long nestedRunTime = endTime - startTime;

assertTrue("Summing 100 numbers 100 times.  Run time: " + nestedRunTime + " but summing 10000 numbers took: " + flatRunTime, (nestedRunTime < (flatRunTime + 15)) && (nestedRunTime > (flatRunTime - 15)));
}
}

Here is sample output:

Summing 10000 numbers.  LambdaJ run time: 259 and Java: 19

All those tests show you how much slower things are with lambdaJ. I haven't really looked into the lambdaJ code, but I imagine reflection is what is slowing this down... still, for some applications perhaps the trade-off between speed and readability is worth it. Another thing to bear in mind is that lambdaJ is a pretty new project. They might still be optimizing this stuff. Looking at the LambdaDemoTestMain class (in the lambdaJ source) it is clear that Mario and company are thinking about this performance stuff. They have a more exhaustive set of speed comparisons like the ones I created.

More Catches
Bear in mind that I'm new to lambdaJ and might be missing a few tricks here. But here are some gotchas that have tripped me up. That closure trick is really clever, but when you try to use it I think you will find the behavior of lambdaJ closures to surprising. If you try something like this:

Sequence sequenceOfOne = new Sequence(1);
Closure foo = closure(); { sequenceOfOne = new Sequence(2); }

You realize you can't really do assignment because you need to use of() to bind sequenceOfOne to the active closure. Another surprise to me was that you can't have a closure with a return value. See this example:

private Integer plusTen(int lambdaJSum) {
return lambdaJSum + 10;
}

Closure sequenceSumPlusTen = closure();{ of(this).plusTen(var(Sequence.class).lambdaJSum()); }
Integer sequenceOfOnePlusTen = (Integer)sequenceSumPlusTen.apply(new Sequence(1));

I expected sequenceOfOnePlusTen to be 11, but sequenceOfOnePlusTen gets set to null. And no, you can't just inline a return statement in there. I hope I don't seem to be picking on lambdaJ. I think it is great. I would love to be able to use the sort methods and forEach methods in my codebase. But while we are talking about gotchas, here are some more: Using forEach on a null or zero sized collection requires using a different forEach method that has a slightly different syntax. And using lambdaJ on classes that are final will cause problems because lambdaJ can't proxy those classes. I suppose Strings must play hell with lambdaJ for that reason. And trying this code out, sure enough, I get a class cast exception.

List<String> strings = new ArrayList<String>();
strings.add("foo");
forEach(strings).toUpperCase();

Other interesting behavior I noticed was that the first lambdaJ operation you do is much slower than subsequent ones. See my benchmarking tests from before. The last test case will fail horribly if you comment out the lambdaJ "warm up code". I haven't dug through the lambdaJ code enough to understand why.

Conclusion
In the end you see how lambdaJ is a fantastic project full of great ideas. It offers you a taste of how things could be different if Java implemented closures. However it is just papering over this ugliness in Java. And because of the amount of proxy, thread local and reflection magic they had to use to do it, you should be prepared for some surprises when things don't work as you would expect. Regardless, my hat is off to the lambdaJ team. This is a great project that deserves to get more attention. I hope the project matures, inspires more DSL style code and whets the Java communities appetite for real closures and a better collections framework.

Anonymous Inner Classes: A Poor Man's Closure?

Yesterday I had a discussion with a colleague about whether Java has closures or not. We both could immediately agree that, formally speaking, Java does not have them, but the part of the discussion that went on and on was about passing anonymous inner classes vs. passing around an anonymous function. I argued that the anonymous inner classes were a very poor substitute because a class has so much extra baggage, both conceptually and behaviorally. Classes understand and do all sorts of things that you don't care about when you are simply trying to pass along a little ol' function. Stuff like: inheritance, encapsulation, polymorphism. Worst of all, classes require lots of messy syntax. My colleague made an important point though: What else is a Java developer gonna do? When you are working in Java and you need something like a closure you've got no choice but to go for the anonymous inner class, the poor man's closure.

Google's Collection Framework is the best example I know of Java programmers making do with their poor man's closures. Here is a quick bit of sample code I created that demonstrates applying a function to every element of a list (Sorry you can't see my generics. I can't seem to escape properly for html):

import com.google.common.collect.Lists;
import com.google.common.base.Function;
import java.util.ArrayList;
import java.util.List;
public class PoorMansClosure{
  public static void main(String[] args){
    //Demonstrate something similar to the map function on a collection in ruby
    ArrayList strings = Lists.newArrayList("feeling", "bad");
    Function upcase = new Function(){
      public String apply(String from){
        return from.toUpperCase();
      }
    };
    List upperCaseStrings = Lists.transform(strings, upcase);
    for(String val : upperCaseStrings){
      System.out.println(val);
    }
  }
}

Martin Fowler has a nice article about how closures naturally compliment collections. Here is a quick bit of ruby to demonstrate that.

class Closure
  def demonstrate
    strings = ["feeling", "good"]
    upper_case_strings = strings.map{|elem| elem.upcase }
    p upper_case_strings
  end
end
Closure.new.demonstrate

In this example the closure frees us from the burden of specifying the names and signatures for both class and method and we simply give the function that maps to upper case without the noise. There have been rumblings in the Java world for some time that Java might someday support closures. Here's hoping that Java 7 will. If that does happen it will be interesting to see what they do to the collection framework. The existance of Google's collection framework is proof that the standard Java framework has major drawbacks. I think a major rewrite will be in order.

Hooray, Wes Anderson! Hooray, Roald Dahl!

Roald Dahl + Wes Anderson = The next movie I go to at the cinema. Dig the preview here.