Slamhound 1.3.0 - Cleaning up all your namespaces

Cleaning up namespaces in Clojure is a chore. So noone does it, and the namespaces get worse.

Enter Slamhound. Slamhound's a library by Phil Hagelberg (the Leiningen guy) that attempts to reconstruct a namespace for a file.

For a while now Slamhound has been pretty unusable, do to it not having a good heuristic for figuring out which namespace to use for a require/as clause. For me, it would fail on most real-world namespaces I tried it on.

Over Christmas break I decided to get it working again, because I was pretty sick of fixing up legacy namespaces by hand. It now works, and also, as an added bonus, converts all your uses to requires, as per the Clojure 1.4 good practice.

The original Slamhound `reconstruct` function returned a string representing the new namespace, but it was still tedious to run it on 50+ namespaces and copy-n-paste the new namespaces in.

For this reason, I decided to automate the process. Long story short, the slam.hound namespace in version 1.3.0 now also has a -main function that takes a file or directory and replaces the namespace and writes it back to the file, or to all the files in the directory. Like magic.

Also, if you're on Leiningen 2 you can define a partial alias:

and then call it from within your project's root directory:

Then take a look at the diff for your repo. You'll see a lot of files have been altered.

Caveats: Slamhound is still not perfect. It can lose metadata, or comments in the namespace form, and sometimes it picks the wrong dependencies to use in a namespace.

Reducing Complexity

I had written some complicated looking code at work recently (shhh, let's keep that between us ok?). Something about it didn't seem right to me, but I was busy at the time so I wrote a TODO to come back and find a way to make it simpler.

Since the code is so confusing it is virtually impossible to read at a glance and understand its intent, I will explain what it does. It takes in a map where the keys are numbers of days over a given SLA, and the values are the number of shipments that were over the SLA by that amount of days. The map actually can and does have more keys on it than that, but for the purposes of this article we can ignore them. The function takes that map and converts it to a version where the values are not counts of the individual SLA, but instead the current running percentage totals.

An example:

What makes this code end up so complicated is that it is complecting three things: generating the series of sums, dividing them, and creating a new map to return.

The solution is to break those 3 parts up.

First we generate the series of sums using reductions.

Second, we divide each by the total to convert them to percentages.

Thirdly, we create a new map with percents as the values instead of counts.

Another thing you might notice in this code is that I purposely used the more verbose `#(get count-based-past-SLA-map %)` when I could have just used `count-based-past-SLA-map`, as it is already a function! I prefer to err on the side of making my code as obvious as possible, so I try to avoid using data structures as functions except under a few special cases. But I guess that would be a different topic for a different blog.

The Many Faces of Midje Fakes

The Midje Clojure testing library enables top-down development, and the primary tool it has for enabling it is the fake.

In Midje, you don't have mock objects like you do in object-oriented languages, instead you fake out functions. The simplest way you can fake out functions is by using the provided form within your fact.

Fakes

(Things that look like ..x.. are called Midje metaconstants. For now just pretend they are constants, or skip to the end of this article for a link to the wiki.)

In the above example we've managed to define six-times in terms of a function, doubler, that is not yet defined (it is unfinished).

Presumably the next thing we'd do when designing a piece of software like this is to write the fact for doubler.
Or perhaps write a more thorough test:
(Tabular is a way to write one fact that runs against multiple input sets.)

The Myriad Uses of Fakes

Fakes have other uses besides simple stubbing.

They can also be used to verify call counts. Imagine that you want to check that when some data is not valid there are no calls to send-email, but otherwise there will be 1 call sent to each recipient?

We can write this kind of test using the :times n prerequisite option. (where n is some non-negative integer)

Checkers

Using checkers we can even expand on the idea of what can be checked by a provided. In the above examples first we used the anything checker to check that the email function was never called with anything. Then we simply checked that the email function was called with :alex and :brian once each.

In this example we want to check that the query function is called with 10 things that look like query maps. To do that we first need to create our own checker that checks whether the parameter looks like a query.
Midje will say that any arg matches this checker if it has a non-false/non-nil :target-db and :query-plan.

Let's confirm query is called 10 times with params that look like queries.

The Fake Chain Gang

If you've ever violated The Law of Demeter in an object-oriented language and then tried to use mock objects with the offending class, you may have experienced a chain of mocks.

A chain of mocks is when you mock a method on a mock object that itself returns another mock object, that has a method on it mocked to *finally* return the value you want.

when(mockA.call()).thenReturn(mockB);
when(mockB.call()).thenReturn(mockC);
when(mockC.call()).thenReturn(42);

In Midje the naive approach might be to write something like:
Midje supplies a feature called Folded Prerequisites that enable a nice condensed syntax for this, though it is probably a code smell if you are doing this too often.

Fakes That Cover Multiple Facts

There are other ways to make fakes in Midje as well, by using the background macro and the against-background macros. These give you the ability to write fakes that cover a group of fakes. I think I'll talk about them in another blog though time permitting.

Wiki Links

Metaconstants
Using Checkers in Fakes
Folded Prerequisites

TDD in Midje in a Nutshell

Midje is a Clojure testing library supporting a top-down mock function style of development, but also equally well supports a bottom-up style.

This is an introductory article, and depending on the response, there will be more articles about some of the intermediate and advanced features.

Elegant Examples

Midje starts with a simple premise: that tests should be elegant and readable.

When was the last time you saw code samples like this in somebody's blog, gist, or book?

EXAMPLE:
As you can see the foo function always returns :a-ham-sandwich

(foo :bar :baz)
;=> :a-ham-sandwich

As developers, I think we write examples like this, because it reads well to us. We're used to this sort of pseudo-code example style, and so we can all tell at a glance what's happening.

This is where Midje enters. The below is executable Clojure Midje code:
You can take that code, and run it, now, and if you've defined foo, then you can run `lein midje` or use midje-mode in Emacs to confirm that foo does in fact always return :a-ham-sandwich. (Forget for now that this particular fact doesn't very well cover every possible input. Using a generated test input style of testing could be very useful here, but that is for another blog.)

Describing Code That Doesn't Exist - Yet

Let's try describing some code that doesn't yet exist with these kinds of examples. I think if we jot down some example cases, it will help us think more clearly about what we're trying to get our code doing.
We can run these examples now. First thing you'll notice is that we're getting errors; that's of course because we have not yet written any code. Let's add the my-func function.

(defn my-func [n] )

Let's be cautious and just try to get one of these facts working first.

(defn my-func [n]
  (* 2 n))

This seems to work... but for only the first fact. (There's a lot to be said for how effectively TDD can help you to stay focused on the task at hand.)

So let's get my-func to pass the second fact as well.

(defn my-func [n]
  (if (odd? n)
    (* 2 n)
    (* 3 n))

Running those examples again with lein midje or midje mode, we see that they do indeed pass.

But there's one more thing. There is a little bit of duplication left in my-func. Let's take a second, and refactor this a little.

(defn my-func [n]
  (let [multiplier (if (odd? n) 2 3)]
    (* multiplier n))

Now that we've revealed a new concept of a multiplier in our code, it looks like we are done. Our code works as advertised.

TDD Recap

You just learned:

• how to use Midje facts as a form of executable examples
• how to do TDD

Nice.

I was sneaky just then. Without letting you know what I was doing, I've walked you through the 4 steps of TDD. Decide what you are testing, write a failing test, make that test pass, refactor the code to have a good design.

TDD Steps:

1. We chose a new piece of functionality we wanted the code to have.
2. We wrote a failing test. We ran it. We saw it fail. In Midje, we like to think of this step as taking some time to describe the facts of the future version of the code.
3. We wrote as little code as possible to make the test pass. In Midje, we think of this step as bringing the code up to date with the new facts.
4. We took some time to reorganize the code - giving it better structure, removing duplication, adjusting the naming, etc.

Repeat.

That's TDD in Midje in a nutshell.

What did you think? Interested in hearing some more intermediate or advanced TDD and Midje concepts and exercises?

Technicalities

B.A.Z.N.E.X. - Boston Autonomous Zone Nerd Nexus

B.A.Z.N.E.X.: Boston Autonomous Zone Nerd Nexus

The short version is that we’re a self-directed organization focused on the joys of software development, community, and learning.

BAZNEX

Before we get into more details about what we are let’s start with what it is NOT

• BAZNEX is not a meetup, but we do meet together.
• BAZNEX is not a forum, but we do discuss.
• BAZNEX is not a lesson, because there is no one teacher.
• BAZNEX is not corporate sponsored, because we don’t have ulterior motives of hiring you.  We’re here to better ourselves and, ultimately, the world.  But who knows who you might meet, and what connections might be born.
• BAZNEX is not an open source programming club, but we might work on some open source projects.
• BAZNEX is not a place to teach or be taught, but it is a place to grow together with others, and to teach when called upon to do so.
• BAZNEX has NO agenda. There are no lectures and no prearranged topics. You come, you meet others, and you join the party. That good vibe you feel here is what we call the Nerd Nexus.
• BAZNEX has no leader, but for the force of willpower generated through a phenomenon called a Nerd Nexus - that moment with the vibes so delicious, and the love of computer science and software development, as well as the inspired action for the community, and for the history of computing.

 

Okay, BAZNEX isn’t any of those things. So what IS it??

• BAZNEX is appreciating and revering the history of our profession.
• BAZNEX is sharing cool software stuff.
• BAZNEX is experiencing the greater nerd community of Boston.
• BAZNEX is pairing together with others to share expertise, and to contribute back to the Open Source community, and the community at large.
• BAZNEX is about the community of software developers.
• BAZNEX is about appreciating the venerable greats of our industry and mooching off of their awesomeness by osmosis.
• BAZNEX is about bringing up novices.
• Yes. BAZNEX is a software osmosis, free-form, group software block-party.
• On any given BAZNEX meeting (we call them nexuses) pretty much anything could happen.
• But you can generally count on working on some code, and learning cool stuff from those you pair up with at the nexus.  You are encouraged to write on the wiki what your interests are; that way others who are interested in learning with you can say hello and you may pair together, and vice versa! 
• BAZNEX grows organically; your presence makes you a part of it all, and so your desires matter. 

What you can do if you're interested in joining the Nexus

We're very much in a primordial state. We have had one small (successful!) gathering of three at the Andala coffee shop in Central Square Cambridge where we discussed why I love index cards, a bit about teaching others programming, took a look at the SeeSaw Clojure Swing-wrapping library, and took time to work on adding a feature to Midje, a Clojure testing/mocking framework.

 

The current process we've done (a total of one time) is to have everyone write down on index cards  the different activities they might want to do at that gathering.  Then put them all together and we split up into pairs or triples. Last gathering index cards were written such as " want to look at SeeSaw?", "talk more about teaching programming", "fix the code Alex broke in Midje", "do some remote pairing, or at least learn more about how we could", "create a Content Management system in Clojure", "can anyone help me with my 5 minute lightning talk on Clojure I'm doing tomorrow?"

 

We're looking into finding larger venues preferably closer to downtown Boston for easier access by all of us, and places that can fit more of us than just a coffee shop can handle.  Sprout in Davis Square is going to be our next location to try out on Wednesday Nov 1.  

 

Also, if this all is jiving with you, we've got a BAZNEX Google Group we've just started, where all future discussion of BAZNEX will transpire. 

 

Please tell anyone you think may be interested. The more the merrier.  Any suggestions, offerings, additions, or displays of exuberance, also invited.

A Few Nice Features Of Clojure, Part #1: Using Macros To Freeze Time

I thought it might be fun to show some of Clojure's features off by taking a look at how I use some of them in my own project EmailClojMatic.  I've got 4 features in mind and will show one feature each post.

Problem #1:

We want to unit test a function that depends on the current date.

Solution #1:

Fortunately we are using JodaTime, a great Java date/time library.  JodaTime has built in features for setting the 'now' time.  Let's use those to create a function to wrap our test code in, that will set the time at a given DateTime.

(defn do-at* [date-time f]
  (DateTimeUtils/setCurrentMillisFixed (.getMillis date-time))
  (try
    (f)
    (finally (DateTimeUtils/setCurrentMillisSystem))))

;; you might use this code like this:

(do-at* (DateMidnight. 2000 1 1)
  (fn [] (DateMidnight.))) ;;=> (DateMidnight. 2000 1 1)

If you are like me and you find yourself writing test code using this construct frequently you can use Clojure's macro functionality to essentially extend the Clojure language to have a new feature for evaluating an expression at a given DateTime.  This way we won't have to always wrap the expression in a function like we are doing in the do-at* function.

Let's see the macro version that I decided to call `do-at`, because it is just like the built in Clojure function `do` except it evaluates at the given time. [It's probably worth noting that xyz* function with xyz companion macro is a common Clojure idiom. -@darevay]

(defmacro do-at [date-time & body]
  "like clojure.core.do except evalautes the
  expression at the given time"
  `(do-at* ~date-time
    (fn [] ~@body)))

;; nice. Now we can see the actual code that was hiding
;; behind all of those parens and fn creation:

(do-at (DateMidnight. 2000 1 1)
  (DateMidnight.)) ;;=> (DateMidnight. 2000 1 1)

It's pretty powerful the way that macros give you the ability to shape your library so that it is free of boiler plate.

Simplicity is a Prerequisuite for Reliability

(This blog post is based on Rich Hickey's keynote speech at Strange Loop 2011)

Word origins

Could we go back to what these words REALLY mean?

Simple: sim-plex - one fold/braid

Complex: many folds/braids

Easy: ... lie near

Simple

One fold/braid

One role

One task!

One concept

One dimension

But not

    one instance

    one operation

    it's about lack of interleaving, it's not cardinality

Interleavage is an Objective notion

Easy

Near, physically

    on our own hard drive, in our toolset, IDE, apt get, gem install.

Near, to our current understand/skill set

    familiar - mentally near.

We're generally overly fixated on these two meanings of easy.

If you want everything to be familiar, you learn NOTHING.

There's a third meaning of easy:

Near, to our capabilities.

Easy is RELATIVE - unlike simple

Limits

We can only hope to make reliable those things we understand.

We can only consider a few things at a time.

Intertwined things must be considered together. This is a mental burden and the intertwining tends to grow combinatorially.

Complexity undermines understanding.

Simplicity Driven Development

If you ignore complexity - you will invariably sloooowwww down over time.

If you focus on ease - you will have a fast at start, and slow down every iteration to redo everything.

If you focus on simplicity - you will go a bit slower at the start, but then ramp up to a higher steady state of productivity.

What is In Your Toolkit?

Complex                                Simple

state, objects                             values

methods                                     functions, namespaces

vars                                             managed refs

inheritence, switch, matching  polymorphism

syntax                                         data

imperative loops, fold               set functions

actors                                          queues

ORM                                           declarative data manipulation

conditionals                               rules

inconsistency                            consistency

Complect

archaic word meaning "to braid together"

don't do it

complecting things is a source complexity

best to avoid in the first place

Compose

meaning "to place together"

composing simple components is the way we write robust software!

We can make simple systems by making them modular.

What do we want to make these two parts of code allowed to think about??

In other words, program towards abstractions!

Don't be fooled by code organization. Two things can be physically separate, but to change one means you need to change the other - they are complected

State is Never Simple

It complects value and time

But it is so EASY!

It interleaves everything that touches it, directly or indirectly.

Programming Features That Complect

Feature                    Complects

State                            everything they touch

Objects                        state, identity, value

Methods                      function and state, namespaces

Syntax                         meaning, order

Inheritance                 types

switching/matching   multiple who what pairs

variables                    value, time

loop, fold                    what/how

actors                         what to do, who does it

ORM                           OMG

Simplicity is NOT the same as Easy - Remember that - It is KEY

The Simplicity Recipe

You do not need C#/Java/C++.  You can make big systems with dramatically simpler tools.

We make hundreds of variations that make it tough to manipulate the essence of this stuff - data.  We should use simple data so we can write code to work on the essence of this stuff.

Develop sensibilities around entanglement - learn to see the complecting.

Tools for reliability like testing, refactoring, etc are all safety nets that are orthogonal to this problem.

Choose simple constructs over constructs that encourage braided together code.

Simplify the problem space before you start.

Simplicity often mean you have more things... it's not about counting.

Reap the benefits!