Archive | November, 2011

Clojure Conj 2011 Recap

17 Nov

Having attended first Conj last year I was looking forward to one more Clojure deep dive. And I was not disappointed once again. Similar to the last year the conference took place in Raleigh, NC but this time hotel was right in the middle of downtown with walking distance access to restaurants, shops, etc. All sessions were recorded, look forward to upcoming videos. Here is my attempt at quick recap with some personal comments.

Day 1, Thursday, Nov 10th

  • Session 1, Ideas Rant, Stuart Sierra
    Stuart, with occasion inspired blue hair, started the conference with rant-like collection of ideas/suggestions. Among them were: using clojure read/print for serialization, how to implement your own clojure structures for drop-in replacement for closure ones. He also spent some time talking about java.util.concurrent goodies as an option for cases where Clojure provided state management primitives don’t fit the bill. Somewhat mixed message IMO, given the clojure’s goal to hide much of the Java. Stuart also mentioned clojure.test.generative, input test generation library inspired by Haskell’s QuickCheck.
  • Session 2, Concurrent Stream Processing, David McNeil, Revelytix, Inc.
    One of two very good session on “Clojure in the real World”. Problem is to process SPARQL federated queries against multiple SQL datasources, aggregate and combine results. Solution was inspired by unix pipe processing.  s-expressions via custom DSL were compiled into tree of pipe connected nodes, where node could represent query or sub-query. Aggregation and post processing of the query results was done via fork/join pool. Few interesting notes:
    Fork/join is well suited only for cpu-bound work so any blocking io was done prior when fetching results.
    To allow for large streams system monitored virtual and transparently switched to memory buffered pipes.
  • Session 3, Clojail, Anthony Grimes
    Anthony quite possibly is the youngest member of Clojure community, he  is 17. Avid cat lower and Stuart Sierra’s groupie  with occasion inspired green hair. Main message : code is dangerous, but dangerous is good except there few cases when it is not: IRC channel, learning, etc. where sandboxing provides an answer. Reuse JVM sandboxing where possible. Sandboxing dynamic Clojure features is quite hard, Clojail is an attempt to achieve this. It has good customization and default security profile.
  • Session 4, ClojureScript, Chris Houser
    Chris presented some low-level details of ClojureScript (CLJS) from data types -> protocols -> macros -> compiler. CLS vector is copy-on-write wrapper over js array. 2 types of map implementations: ObjMap (when keys are string-like), HashMap when keys are objects. Protocols can extend foreign class into CLJS. Chris warned about potential implementation clash if you don’t own protocol and data structure.
  • Session 5, Striving To Make Things Simple and Fast, Phil Bagwell, Typesafe
    Phil made low level data structure talk bearable. He started on showing how easy it is to make Scala data structure code parallel via recently added support for parallel collections. Then he delved into details on solving the problem of developing fast immutable vector. It was fascinating to see his description of how work progressed step by step. In the end it took team 18 months to come up with performing solution. For more details see RRB-Trees: Efficient Immutable Vectors
  • Session 6, Intro to Logic Programming with Clojure, Ambrose Bonnaire-Sergeant
    Logic programming was one of the main themes of the Conj. Rich Hickey and Clojure/Core expressed a strong interest in this area. But in terms of the presentation, Ambrose came up short. He is young, sharp student from Australia but his delivery lacked conviction, coherence and deep understanding of the subject matter. After a while many attendees faded away into typing. IMO, this area would have been better covered by David Nolan, the author of core.logic but he was presenting next day.
  • Session 7, Hacking The Human Genome, Arnoldo J. Muller-Molina
    A definite hit, an excellent example of power of Clojure in the real world. Arnoldo used Clojure and similarity search to discover the rest of “landing sites” sequences of DNA of which only 10% have been known.
  • Session 8, Ousterhout’s Dichotomy Isn’t (Part 2 of the Simplicity/Power/Focus Series), Stuart Halloway
    Another rant like session from “Stue”. No many notes on this one, I must have been mentally fatigued plus there was very little to take away.
  • After Session, Scheme’s miniKarnen, William Byrd and Daniel Friedman
    These guys stole the day 1 show, they rock. Although I came up few minutes late, I was able to see them in the act, going through Scheme code, changing and trying things on the fly, making statements like “We can make our program to run backwards, you do want to run your programs backwards, don’t you?” Checkout their “Schemer Series” book The Reasoned Schemer

Day 2, Friday, Nov 11th

  • Session 1, Master Plan for Clojure Enterprise Domination, Neal Ford, ThoughtWorks
    Neal discussed how to sneak in Clojure into enterprise. He shared his experience on well-known CTO prospective of bringing new programming language into a mix. According to him Clojure can help solve enterprise problems in the areas of: RESTful integration, ORM, and rule engines. Neal proposed “cat burglary” approach which worked fine for Rails/Cucumber and Groovy/Gradle. Interesting thought on perceived rivalry with Scala – at this point Scala is Clojure’s friend and the enemy is Java’s status quo. When selling Clojure to management one should start in the following order: another language for JVM -> functional -> dynamic. At the end he appealed for wrapping everything in order to build bridges to other technologies.
  • Session 2, Predicate Dispatch, David Nolen
    David is a great example of young hacker generation drinking from a Clojure’s fire hose. His talk centered around a finding a better way of function dispatch. Although Clojure’s multimethods dispatch on more than first argument, they are essentially closed, the dispatch function is hardwired – potential extenders cannot add new dispatch conditions not already defined by the original dispatch function. Similarly, pattern matching favored in some functional languages exhibits the same problem. Predicate dispatch generalizes traditional method dispatch mechanisms by permitting arbitrary predicates to control method applicability and by using logical implication between predicates as the overriding relationship. As of now predicate dispatch is not in Clojure but David showed that it is plausible and could be added in the near future.
  • Session 3, Extending Javascripts from Clojure, Kevin Lynagh
    One of the least informative sessions – a lot of fluff not enough stuff. Kevin reiterated Javascript’s shortcomings: global vars, lack of modules, etc. and showed how ClojureScript overcomes them, nothing new here.
  • Session 4, From Linear to Incremental, Christophe Grand
    Christophe took on an interesting problem of incremental parsing/compilation in order to prevent full parsing/recompilation for small code changes. I wish I could give you more details but Christophe’s heavy accent diminished my note talking capability. The only other interesting detail I noted is that Christophe used technique of stack transplantation via diff memoization.
  • Session 5, Logs as Data, Mark McGranaghan, Heroku
    Mark presentation centered around Clojure implementation of central distributed logging at Heroku. Don’t treat logging as lines in the text file but as logging events with multiple uses: log aggregation, system status, system performance, etc. Parse log lines and convert them to Clojure map to be forwarded to logger processors.
  • Session 6, Modeling the world Bayesian network in Clojure, Chas Emerick
    This presentation covered Bayesian network (BN) from AI realm and it’s applicability to problem domains of process control, decision systems, and prediction. Chas mentioned that he used BN in his own business in document processing system to learn about different document formats. Chas announced that he will release probability library “raposo” soon.
  • Session 7, Clojure and Adroid, Daniel Gomez
    Daniel summed up the state of Clojure development for Android platform. He started with describing high level architecture of Dalvik VM and how it is different from JVM. Through rigorous testing and profiling Daniel showed that Clojure lags in the areas of code size and startup time. His presentation sparked a discussion on how to minimize Clojure’s runtime footprint and startup time.
  • Session 8, Keynote, Rich Hickey
    No rant-like talk from Rich this time. His presentation centered around future work stream for Clojure/Core team. Future areas of interest are: making Clojure leaner with different build targets, unification with ClojureScript, elusive goal of ClojureInClojure, getting rid of dependency on ASM byte code manipulation library, InvokeDynamic support in Java 7, leveraging core.logic for type checking and predicate dispatch, parallelism with algoritms not collections, transients and pods, accumulators and extensible reader.

Day 3, Saturday, Nov 12th

  • Session 1, Cascalog, Nathan Martz, Twitter
    Nathan’s presentation centered around anaylyzing Twitter’s Big Data with Clojure custom DSL on top of Hadoop’s MapReduce. I immediately draw parallel with earlier David’s McNeil’s presentation. Both share similar architecture but differ in query DSL and in task running backend component – one uses fork/join while other Hadoop. Cascalog supports different type of predicates: filters, aggregators, generators, and user defined functions. Joins in Cascalog are implicit and subqueries are supported. Nathan ran trough a lot of examples from trivial ones to more advanced.
  • Session 2, Functional Data Structures, Daniel Spiwek
    Daniel’s presentation was most enthusiastic by far. My first thought was – how many cups of coffee he had in the morning? But other people whom seen him present before pointed out that this is his style. Two groups of data structures: sequential and associative. Provide immutability and use structural sharing for performance. Implementation goal is to minimize copying and maximize sharing. Amortization as a way of distribute cost of infrequent slow operation. The latest state of the art is hybrid approach: combine sequential and associative characteristics. Pioneered by Rich Hickey and then ported to other languages. CPU locality of reference and JVM heap locality are the underlying low level reasons why they are fast enough.
  • Session 3, Macronomicon, Michael Fogus, Clojure/Core
    Another rant-like talk, somewhat more coherent and useful, centered primarily on the subject of macro use and abuse. Fogus echoed Christophe Grand’s presentation from last year on suggested macro use cases: creating binding forms, control flow and syntactic sugar when warranted. Drew attention to his work in progress on pre/post condition library Trammel. At the end Fogus showed how macros could be used to shift runtime calculation to a compile time.
  • Session 4, Lightning Talks, Ten minutes/Various Presenters
    CyclicTime – jodatime like library for sane date computations.
    Immutant – application server for clojure on top of Jboss using JRuby.
    UberLisp – lisp1 interpreter for the Arduino. The best by far, whittingly presented, auidence bursted into applauses several times.
    Korma– Sugar coated DSL on top of SQL.
  • Session 5, Performance in the Wild, Craig Andera, Clojure/Core
    Good presentation on how to optimize Clojure in a real-life project. Many common sense, but useful nevertheless techniques:
    Establishing performance baseline early in development cycle helps to catch deceptively small changes that could effect performance.
    Performance is not a scalar metric.
    Be aware of assumptions it a data model.
    Measure distribution of latency under most likely use case/s.
    Optimization loop: bechmark -> analyze -> recommend -> optimize; done when performance targets are met.
    Don’t make mistake of equating number of threads with concurrent clients.
    Tools used: jmeter plugin to chart the results, YourKit for profiling.
  • Session 6, Programming Music With Overtone, Sam Aaron
    Skipped this one to catch the flight.

Overall, it was a great experience. I talked to few dozen of people about Clojure. Similarly to the last year there were quite a few at playing/evaluating/trying to introduce stage. But I did see signs of wider adoption. Met few guys who took a stance “No mas Java” and found 100% Clojure jobs after a while. Met one guy from London who works at a bank where IT director had been bitten by Clojure bug and was pushing it from the top. Clojure developer community is quite diverse with three main backgrounds: young (mostly Ruby) hackers, old time Lispers, and Java converts. Relevance (Clojure/Core) does an admirable job of evolving language and ecosystem.

Advertisements