Jef Claes

GES scavenging and the hidden cost of link events

Sun, 12 Aug 2018 14:58:00 +0200

Somewhere around a year ago, we started using GES in production as the primary data store of our new loyalty system. The system stores two types of data.

External services push batches of dumb downed events to the loyalty system. For example: a user logged on, played a game or participated in a competition. These events are transient by nature. Once the loyalty system has processed them, they only need to be kept around for a few days.
When these ingress events are processed, they go through a mini pipeline in which each event is assigned a specific weight, for then to be aggregated and translated to a command. This command awards a user with virtual currency to spend in the loyalty shop and a number of points contributing to a higher rank - unlocking more priviliges. The state machine that stores a user’s balance and points is backed by a stream which is stored indefinitely. Unless the user asks to be forgotten that is.

As a rough estimate, for every 1000 ingress transient events, only 1 needs to be stored indefinitely as part of a state machine.

When implementing this more than a year ago, I thought I had done my homework and knew how to make sure the ingress events would get cleaned up. First you make sure the $maxAge metadata is set on the streams you want to clean up, for then to schedule the scavenging process (other databases use the term vacuum). This worked without any surprises. Once scavenging had been run, I could see disk space being released. However, after a few months I started to become a bit suspicious of my understanding of the scavening process. GES was releasing less disk space than I expected.

Even though, we had been quite generous while provisioning the nodes with disk space, we would run out very soon. Much to my frustration, the “Storage is cheap” mantra gets thrown around too lightly. While the statement in essence is not wrong, for a database like GES that’s built on top of a log, more data also means slower node restarts (index verification), slower scavenges and slower $all subscriptions.

GES has no built-in system catalog that allows you to discover which streams are taking up all this space. However, you can implement an $all subscription and count events per stream or even count the bytes in the event payload.

static void Main(string[] args) {
    using (var conn = EventStoreConnection.Create(
        ConnectionSettings.Create().Build(),
        new IPEndPoint(IPAddress.Parse("127.0.0.1"), 1113))) {

        conn.ConnectAsync().Wait(5000);

        conn.SubscribeToAllFrom(
            Position.Start,
            CatchUpSubscriptionSettings.Default,
            eventAppeared: Count, liveProcessingStarted: Print, subscriptionDropped: Warn,
            userCredentials: new UserCredentials("admin", "changeit"));

        Console.ReadLine();
    }
}

private static void Print(EventStoreCatchUpSubscription sub) {
    foreach (var count in  counts.ToList().OrderByDescending(x => x.Value)) {
        Console.WriteLine($"{count.Key}: {count.Value}");
    }
}

private static Task Count(EventStoreCatchUpSubscription sub, ResolvedEvent e) {
    var ev = e.Link ?? e.Event;
    if (!counts.ContainsKey(ev.EventStreamId)) {
        counts.Add(ev.EventStreamId, 1);
    } else {
        counts[ev.EventStreamId] += 1;
    }

    return Task.CompletedTask;
}

// Output
// $et-LoggedIn : ...
// $et-GamePlayed : ...
// ...

Inspecting the results, we found that the streams emitted by the built-in system projections contained a disproportionate amount of events. Most of them were ingress events and should have been long scavenged! As it turns out, this was a wrong assumption on my part. Built-in projections build new streams by linking to the original event (instead of a emitting a new one). But when an event is scavenged, events linking to the original event still linger around on disk. Although link events are much smaller than the original event usually - it’s just a pointer, the bytes used to store the pointer and the event envelope still take up quite some space when there’s billions of them!

Luckily, I was only using a small portion of the built-in projections. I created a custom projection that only created streams I was actually interested in, pointed my code in the right direction, stopped the built-in projections and deleted the now irrelevant streams.

Running the scavenging process after being able to delete all these streams was very satisfying. The scavenging process loops through all the transaction file chunks one by one. It reads a chunk and writes a temporary new one, only containing the events that haven’t been deleted. Once it reaches the end of the file, it swaps out the newly written file with the old one. Since writes are slower than reads, this makes that scavenging is actually way faster when there’s more to scavenge - or less data to be written to a new file. After all the chunks have been scavenged, the process merges the now smaller files into new chunks when possible. This process is quite transparant by design; all you have to do is list the files in the data directory when scavenging.

When this whole process was complete, used disk space went down from 410GB to 47GB! Having trimmed all this excessive data, scavenging is faster (hours not days), node restarts are faster and resetting an $all subscription makes me less anxious.

Amazon Redshift - Fundamentals

Tue, 01 May 2018 14:17:00 +0200

Late 2017, we set out to replace and upgrade our existing reporting and analytics infrastructure with something that would be a better fit for our workloads. Keeping costs and required maintenance at a minimum would be a nice plus, making for an easy sell. After a bit of research, it was obvious Amazon Redshift had the potential to tick all the right boxes. While steadily porting the most problematic workloads away from our existing infrastructure, I started writing an investigative article on the fundamental concepts of Amazon Redshift. I learned a lot studying each individual building block, allowing me to make some small, but impactful changes to our own setup along the way.

The outcome is a 10.000 word document (1 hour reading time), covering 7 topics:

Storage
Distribution
Importing data
Table maintenance
Exporting data
Query processing
Workload management

The text is available in three formats:

HTML
EPUB
MOBI (Kindle)

The project is open source and available on Github.

Thanks to everyone who proof-read earlier iterations and provided me with indispensable feedback.

I hope this work can teach you as much as it thought me. I’m looking forward to your feedback.

Consumed in 2017

Wed, 03 Jan 2018 21:28:00 +0100

Another year, another 17 books, 6 shows and 3 movies consumed. Here’s this year’s highlights.

Books

The Zen and Art of Motor Cycle Maintenance

The author is a tormented soul on a quest to define quality. You’re his passenger, driving shot gun on a CB77 Super Hawk, in for an exhausting intellectual journey through the high mountains of reasoning. You will often fear getting lost and feel slightly anxious that the driver might drive of a cliff any moment, but he won’t. Once you see the top of the mountain for the first time, you’ll be happy he doesn’t make it too easy on you, and you’ll be more appreciative of the road that took you there.

Throughout the process of fixing the machine things always come up, low-quality things, from a dusted knuckle to an accidentally ruined “irreplaceable” assembly. These drain off gumption, destroy enthusiasm and leave you so discouraged you want to forget the whole business. I call these things “gumption traps.”

Peace of mind isn’t at all superficial to technical work. It’s the whole thing. That which produces it is good work and that which destroys it is bad work.

It’s the style that gets you; technological ugliness syruped over with romantic phoniness in an effort to produce beauty and profit by people who, though stylish, don’t know where to start because no one has ever told them there’s such a thing as Quality in this world and it’s real, not style.

Amazon

The Soul of a New Machine

The type of writing I wish there was more of. It’s the closest I’ll ever get to experience building a mini-computer. It makes one appreciate how much we’re standing on the shoulders of giants. So much has changed in 40 years and even more hasn’t changed at all. People will be people.

Adopting a remote, managerial point of view, you could say that the Eagle project was a case where a local system of management worked as it should: competition for resources creating within a team inside a company an entrepreneurial spirit, which was channeled in the right direction by constraints sent down from the top. But it seems more accurate to say that a group of engineers got excited about building a computer.

In the sixties there was proposed a “National Data Bank,” which would, theoretically, improve the government’s efficiency by allowing agencies to share information. The fact that such a system could be abused did not mean it would be, proponents said; it could be constructed in such a way as to guarantee benign use. Nonsense, said opponents, who managed to block the proposal; no matter what the intent or the safeguards, the existence of such a system would inevitably lead toward the creation of a police state.

Amazon

Winter is Coming: Why Vladimir Putin and the Enemies of the Free World Must Be Stopped

I’ve been trying to get better educated on the history of world politics and long-lasting international conflicts. Growing up in Western Europe with very little direct conflict, you’re never really taught why other parts of the world seem to be so messed up and why they’re so angry at us.

Although you might find Kasparov to be a bit too convincing, he has good reasons to hold strong opinions and is in a unique position to shed light on what’s been happening under the Putin regime. It’s like listening to someone who just escaped an abusive relationship. I was remembered of Hintjens’ The Psychopath Code more than once.

Like a weed, evil can be cut back but never entirely uprooted. It waits for its chance to spread through the cracks in our vigilance. It can take root in the fertile soil of our complacency, or even the rocky rubble of the fallen Berlin Wall.

If the road to hell is paved with good intentions, compromises on principles are the streetlights.

He and his junta have turned the country into a petro-state, and exporting natural resources to an insatiable global market doesn’t require entrepreneurs or programmers, let alone writers and professors.

Putin and his defenders abroad bragged about Russia’s rising GDP, but it was like taking the average temperature of all the patients in a hospital.

The hypocrisy of condemning weak dictatorships while embracing strong ones destroys American and European credibility and undermines any attempt at global leadership; in fact, it seems to encourage smaller autocracies to aspire to greater ambitions.

Amazon

Shows

Westworld

In a not so distant future, the rich will be spending their free time visiting amusement parks inhabited by lifelike robots. Once you pay your entry ticket, you can be the protagonist of any story you want. Maybe you want good value for money and go on an epic quest chasing a bad guy to the edge of the park, or maybe you just want to drink, gamble and maybe kill a few randoms for fun in the saloon just feet away from the drop-off point? Like playing Red Dead Redemption post-virtual reality.

The setting, cast and especially the story line are out of this world. Reading up on fan theories after the show is half the fun. It’s amazing how many plausible theories were put out and how a small community is able to dissect every little scene looking for hints to figure out the park’s mysterious past.

This show made me question if I understood what it is that makes us human. Aren’t we all just the result of our pre-programmed genetics and the events we experience throughout our lives?

IMDB

Stranger Things 2

I just finished watching this one last night. It’s like reading a really good Stephen King novel, but in color. I noticed yesterday’s season finale was 61 minutes long; good things do happen when you’re not constrained to TV time.

Netflix

Podcasts

Commutes have gotten earlier and longer this year. I’ve cut back on the technology podcasts in favor of a more broad range of topics.

The Joe Rogan Experience

It’s a bit like sitting in on a conversation between two people at a bar. One person is a talkative and enjoyable guy, not afraid to ask questions and the other is happy to drop knowledge on a specific (and often fringe) topic. Topics range from diet and fitness, to economy and politics. Some episodes that stood out for me recently are the ones with Colin Moriarty and Nina Teicholz.

Conversations with Tyler

The same concept, being a one-on-one conversation covering a wide range of topics. However, this one is more formal and often a bit (too?) academic. This podcast on Marcroeconomics, Mentorship and Avoiding Complacency might give you a good idea on what to expect.

Dan Carlin’s History X

Extremely thoroughly researched lectures on important periods of our history. The Destroyer of Worlds is a 6 hour long, but captivating piece on the history of nuclear warfare and filled in a lot of gaps for me.

Passing the AWS Certified Solutions Architect exam

Fri, 15 Dec 2017 20:34:00 +0100

Before last week, the only certification exam I ever passed was the Microsoft .NET Framework Application Development Foundation certification. This was almost eight years ago. My manager back then thought getting certified was the best way for me to get a raise. It would be a win-win. I for one would learn something along the way, and the company would have less trouble keeping its Microsoft Gold partnership. As far as I remember, I spent a good six months reading, studying and memorizing this 794 pages thick book. Although the book did teach me a fair amount of solid .NET framework internals, most time was spent force feeding myself the ins and outs of framework API’s you only need once in a blue moon and should just Google for when needed.

This time around though, it was my own decision to get certified. Mid 2016, our components were getting more and more structured in a way that allowed us to deploy them away from our on-premise data center. Components that didn’t own data bound to a specific territory by regulatory requirements and that would allow for some down-time were the obvious candidates.

Moving some of our infrastructure to the cloud, we had a few goals in mind:

Take advantage of managed cloud services to reduce operation cost significantly.
More freedom to scale up or down. The structure of the contracts with our data center (and regulations that require us to own our own racks) generally forces us to over provision our infrastructure. Making changes halfway the contract takes time and is costly.
Ease into learning how to run software on the cloud for when we move to other markets or when we build services that have less strict territoriality constraints.

Getting started with AWS is easy enough. Starting an EC2 instance, attaching a disk, using a managed database, configuring a load balancer is child’s play. But when it came to networking, security, fault-tolerance and properties guaranteed by AWS, I had a lot of questions. I hoped to find answers going over the AWS Certified Solutions Architect material. Why not set myself an artificial goal and get certified while I was at it?

After three months of studying, I passed the exam with a score of 95%. Here’s a list of the resources I used, including how much money and time were spent.

Exam Blueprint

You should go over the exam blueprint to understand what they expect you to know to pass the exam. It’s a good idea to go over the document while studying and to tick off domains you feel comfortable with.

Money spent: €0, time spent: 30 minutes

A Cloud Guru

A good collection of short videos and mini-exams covering all the topics needed to pass the exam. Details that require extra attention to pass the exam are highlighted throughout the course.

You can get the videos on acloud.guru for €99. I got it through Udemy for only €10. Worth every cent.

Money spent: €10, time spent: 26 hours

FAQs and Whitepapers

AWS advises you to read a specific set of whitepapers and FAQs. The material can be a bit dry, but it’s extremely useful. Not just to pass the exam, but to avoid nasty surprises in production.

Money spent: €0, time spent: 6 hours

AWS Open Guide

An open-source effort to document real world experiences running environments on AWS.

Money spent: €0, time spent: 1 hour

Whizzlabs

Somewhere around 500 practice questions that tease out the topics you don’t completely master. When I got a question wrong, I would read up on the topic and play around in the AWS console until I felt like I got it.

Although there were some questions that were very similar, you can’t pass the exam by just studying these questions.

Money spent: €20, time spent: 12 hours

Exam Guru

Mobile app affiliated with A Cloud Guru containing more practice questions. These are less scenario based and less in-depth. The Whizzlabs questions are much more in the direction of what to expect on the actual exam. Disappointing to be fair.

Money spent: €20, time spent: 2 hours

Test exam

A small set of questions provided by AWS in the style of the actual exam. This was very much a waste of time and money. Whizzlabs had copied all of these questions word for word.

Money spent: €20, time spent: 20 minutes

Test day

The least enjoyable part of the experience. 55 multiple choice questions need to be answered in 80 minutes. Half of the questions are quite straight forward. The other half are more involved. For the longer questions, I first read the answers and wrote down the options. For then to read the question and strike through the options that definitely were not a part of the answer. This helped me to focus on the important bits of the question and to gain momentum plowing through the questions at a steady pace. I finished with 25 minutes left.

Money spent: €135, time spent: 90 minutes. Extra money spent on parking in the city center of Brussels: €10, searching for a spot: 60 minutes

In short… Go over the material, practice, take notes, practice some more, review your notes until you get sick of them.

Fast projections

Sun, 30 Jul 2017 14:34:00 +0200

Most EventStore client libraries allow you to subscribe to a stream by passing in a callback which is invoked when an event occurs (either a live or historic event).

type Envelope = {
  Id : Int64
  Event : Event
  Historic : bool
}

type EventOccurred = Envelope -> unit

Let’s say we subscribe to a stream of a popular video service, and we want to project a read model that shows how many videos a viewer has watched. We don’t care about the bookmarked videos for now.

type Event =
  | WatchedVideo of WatchedVideo
  | BookmarkedVideo of BookmarkedVideo
and WatchedVideo = { ViewerId : string; VideoId : string; At : DateTime }
and BookmarkedVideo = { ViewerId : string; VideoId : string; At : DateTime }

We’re sitting on top of storage that can execute a single statement and a batch of statements.

The statements supported are limited:

Set a checkpoint
Increment the view count for a viewer

type Statement =
  | Checkpoint of Int64
  | IncrementViewCount of string * int

type Batch = Statement seq

type Storage = {
  ExecuteStatement : Statement -> unit
  ExecuteBatch : Batch -> unit
  CalculateCost : unit -> decimal
}

The storage engine exposes a method which calculates the cost of executed statements:

Executing a single statement costs 1 execution unit
Executing a batch also costs 1 execution unit plus 0.1 execution unit per statement in the batch

The stream

For this exercise the stream contains 3500 historic views, 50 historic bookmarks and 100 live views.

let stream = seq {
  for i in 1L .. 3500L do
    yield { Id = i; Event = WatchedVideo { ViewerId = "1"; VideoId = "1"; At = DateTime.UtcNow }; Historic = true }
  for i in 3551L .. 3650L do
    yield { Id = i; Event = WatchedVideo { ViewerId = "2"; VideoId = "1"; At = DateTime.UtcNow }; Historic = false }
  for i in 3501L .. 3550L do 
    yield { Id = i; Event = BookmarkedVideo { ViewerId = "1"; VideoId = "1"; At = DateTime.UtcNow }; Historic = true }
}

First attempt

The first attempt at projecting the stream to state, executes a statement for each event we’re interested in and checkpoints after each event (even the ones we’re not interested in).

let ``First try`` (storage : Storage) : EventOccurred =
  fun envelope ->

      match envelope.Event with
      | WatchedVideo x -> IncrementViewCount (x.ViewerId, 1) |> storage.ExecuteStatement
      | _ -> ()

      Checkpoint envelope.Id |> storage.ExecuteStatement

// ## First try ##
// -----------------------------------------
// - Viewcount: seq [[1, 3500]; [2, 100]]
// - Checkpoint: 3650L
// -----------------------------------------
// = Cost: 7250.0M

The cost of this projection is high: 7250 execution units - even though there are only 3600 events we’re interested in. We execute a statement for each event we handled and checkpoint immediately after, even for the events we didn’t handle.

Less checkpointing

It’s not hard to get rid of some of the checkpointing though.

let ``Less checkpointing`` (storage : Storage) : EventOccurred =
  fun envelope ->

    match envelope.Event with
    | WatchedVideo x ->
      IncrementViewCount (x.ViewerId, 1) |> storage.ExecuteStatement
      Checkpoint envelope.Id |> storage.ExecuteStatement
    | _ -> ()

// ## Less checkpointing ##
// -----------------------------------------
// - Viewcount: seq [[1, 3500]; [2, 100]]
// - Checkpoint: 3650L
// -----------------------------------------
// = Cost: 7200.0M

The cost has improved, but only marginally. We saved 50 execution units by avoiding checkpointing after events we do not handle. Time for a bigger improvement..

Batching

Instead of handling each event individually, we will buffer them as soon as they come in. When we’re catching up and seeing historic events, we only flush the buffer every 100 events. When we’re caught up, we flush on each event. We want to always make a best attempt at showing fresh data.

When the buffer gets flushed, events are mapped into a sequence of statements, which are sent in batch to the storage engine. The checkpoint is appended to the tail of the batch.

let ``Batching`` (storage : Storage) : EventOccurred =
  fun envelope ->

    let flushOn = if envelope.Historic then 100 else 1

    let flush () =
      let batch =
        buffer
        |> Seq.map (fun env ->
          match env.Event with
          | WatchedVideo x -> IncrementViewCount (x.ViewerId, 1) |> Some
          | _ -> None
        )
        |> Seq.choose id

      if not (batch |> Seq.isEmpty) then
        [ Checkpoint envelope.Id ]
        |> Seq.append batch
        |> storage.ExecuteBatch

      buffer.Clear()

    match envelope.Event with
    | WatchedVideo _ -> buffer.Add envelope
    | _ -> ()

    if buffer.Count >= flushOn then
      flush()
      
// ## Batching ##
// -----------------------------------------
// - Viewcount: seq [[1, 3500]; [2, 100]]
// - Checkpoint: 3650L
// -----------------------------------------
// = Cost: 508.5M

This approach makes a significant difference. Execution cost has reduced by 93%! Batching of historic events makes replays much faster, but with some extra effort we can take this optimization even further.

Batching with transformation

It always pays off to understand the guarantees and intricacies of the storage you’re using. Looking closely at the storage interface, we find that we can increment the view count by any number. If we use a local data structure to aggregate the view count up front, we can reduce the number of statements even further.

In practice, we filter the for events we’re interested in, group by the viewer id, count the values and map that into a single statement per viewer.

let ``Batching with transformation`` (storage : Storage) : EventOccurred =
  fun envelope ->

    let flushOn = if envelope.Historic then 100 else 1

    let flush () =
      let batch =
        buffer
        |> Seq.map (fun env -> 
          match env.Event with 
          | WatchedVideo x -> Some x 
          | _ -> None
        )
        |> Seq.choose id
        |> Seq.groupBy (fun e -> e.ViewerId)
        |> Seq.map (fun (viewerId, e) -> (viewerId, e |> Seq.length))
        |> Seq.map (fun (viewerId, length) -> IncrementViewCount (viewerId, length))

      if not (batch |> Seq.isEmpty) then
          [ Checkpoint envelope.Id ]
          |> Seq.append batch
          |> storage.ExecuteBatch

      buffer.Clear()

    match envelope.Event with
    | WatchedVideo _ -> buffer.Add envelope
    | _ -> ()

    if buffer.Count >= flushOn then
      flush()
 
// ## Batching with transformation ##
// -----------------------------------------
// - Viewcount: seq [[1, 3500]; [2, 100]]
// - Checkpoint: 3650L
// -----------------------------------------
// = Cost: 162.0M

This further reduces costs by more than 2/3th. The optimization makes the code a bit more elaborate, but not necessarily that more complex - it’s still a local optimization.

Conclusion

In three steps, we brought cost down from 7250 execution units to only 162 units. That makes me a 44x engineer, right?

In general, storage is one of the slowest components of your system. Making your system faster often involves making it do less work. Avoiding waste by batching and some more work up front, can make a big impact when you want to make your projection faster.

You can find the complete F# script here.

From human decisions, to suggestions to automated decisions

Sun, 23 Jul 2017 08:48:00 +0200

I’ve been wanting to share this experience for a while, but it took me a while to come up with a story and example I could use in a blog post.

I help out during the weekends in a small family run magic shop. I’m the third generation working in the shop. My great-grandfather always hoped that his only son would follow in his footsteps as a carpenter. But at only eighteen years old, my grandfather said goodbye to the chisels and sawdust, and set out for the big city to chase his dream of becoming a world class magician. The first few years were tough, he was no Houdini. He would (hardly) get by performing at kid birthday parties, weddings and store openings. That’s how he met my late grandmother. She worked as a shop girl in one of the first malls that were built in the city, and happened to show up each time my grandfather performed in one of the stores. After getting married, having a baby (my dad) and saving every dime they earned, my grandfather was able to rent a hole in the wall and open up his own tiny magic shop - in that same mall. Once my dad finished school, he worked as a middle school teacher for a few years, giving up on that job to join his father in the family business. He loves to tell you how he can now still teach children, without the chore of grading their homework. I’ve been running around and helping out in the store since I could barely walk. I guess you can say that magic runs in our blood.

Since the beginning of time, our trade has relied on secrecy. However, due to the rise of the internet, magic is dying a slow death. Even the greatest of tricks and illusions are challenged and destroyed in the open by non-believers. Our craft is now reduced by many to a cheap fairground attraction.

My grandfather, even after suddenly losing grandma last year, isn’t willing to give up on the business though. “There will be a time when the people need magic once more, and we will be waiting right here.” He decided not to see modern technology as the nemesis of magic, but rather as a potential assistant.

Instead of fiddling in his study all night with a book of cards, a hat, a scarf and the lonely rabbit, I’ve been working with him trying to show him what happened in technology over the last 30 years. Being a programmer, I started by showing off some of my unfinished hobby projects experimenting with micro controllers. I hadn’t gotten much further than making some LEDs blink controlled by my voice, but that was enough to spark my grandfather’s creativity. “Can that chip make the lights go out? Can it blow smoke? Can it sense if I flip it around real fast?” One evening, tired after brainstorming and testing ideas all night, he told me “being able to tell these little computers what to do might not be magic, but a miracle”.

I would like to tell you the details of what we came up with, but I’d have to make you disappear after I did. To make a long story short, it was an overwhelming success. Neighbourhood magicians picked it up, and even mere mortals thought it was a great gimmick they could show off with. A friend of mine even told me he used our invention to pick up girls. It wasn’t for long before I got daily emails and tweets from all over the world, begging me to ship our gadget their way.

Thanks to some lovely open source software, I was able to set up a full blown web shop in a matter of days. While orders started rolling in, we got a grip on how to actually produce our new product at a sufficient pace. Shipping overseas turned out to be surprisingly easy. What we didn’t anticipate for was how to handle returns and refunds.

This is where the actual story starts…

When our usual customers visit the store, we take our sweet time to show them how to perform the trick. This results in us knowing our customers pretty well and hardly ever having anyone return an item or ask for a refund. Admittedly, growing the same connection with our customers online hasn’t been a success. A lot of them lack the magician mindset. You can’t just buy magic, you have to put the practice in to make the magic happen. These maladjusted expectations make for quite a few phoney complaints.

Me, my dad and my grandfather in turn have been performing the chore of handling returns and refunds. This domain of the open source shop is very much underdeveloped. Even something as simple as looking up a customer’s details and order history, requires scanning multiple pages of information or even querying the database by hand. Making a well-informed decision takes up way more time than it should.

A use case specific view

The first thing I did in an attempt to speed up this process was building a use case specific view. I asked my dad and grandfather which heuristics they use and which data is needed to feed those heuristics. To get the full picture as soon as possible, I imposed the rule that only this specific view can be used to make a decision. If a piece of data was lacking, I would add it the same day.

This process was more useful than I expected. What we learned is that we all used different heuristics, but were also victim to different biases. For example, I learned that my grandfather used to have a Dutch neighbour who would leave for work very early, and slam the door so loud, it woke my grandfather up each morning. He has grown a disliking for the Dutch ever since. When customers were Belgian, he would much more lean towards issuing a refund, since he believes Belgians are less likely to lie about the cause of a broken item. We also discovered that we used different words for specific numbers. I would use “Items purchased sum”, but my dad would use “Items purchased lifetime” to define the total amount of money spent purchasing items. We decided on being more explicit and making a composition of all those words.

I ended up with a simple screen that rendered a read model that looked something like this.

type ReadModel = {
    DaysSinceItemPurchased : int
    ItemPrice : decimal
    ItemPurchasedSumLifetime : decimal
    RefundedToPurchasedItemsRatio : decimal
    CustomerId : string
    CustomerName : string
    CustomerCity : string
    CustomerCountryCode : string
}

Based on heuristics in the head and a snapshot of information available in the world, we would make a decision and click a button to execute a specific command.

type Command =
    | IssueRefund
    | IssuePartialRefund
    | AwardCoupon
    | ShipReplacement
    | DenyRefund
    | DeferDecision

type Snapshot = {
    SnapshotId : string
    Value : ReadModel
}

type Decision = {
    BasedOnSnapshotId : string
    Command : Command
    MadeBy : string
}

type MakeDecision = Snapshot -> Decision

let makeHumanDecision by : MakeDecision =
    // Heuristics in the head, fed by knowledge in the world
    fun snapshot ->
        {
            BasedOnSnapshotId = snapshot.SnapshotId
            Command = IssueRefund
            MadeBy = by
        }

Making suggestions

By now, I had gotten quite interested in this domain. Each day I would go over all of the decisions and see whether I understood why a decision was made. I would call the shop each time I didn’t understand and scribble down notes whenever I discovered a new implicit rule. In the meanwhile, I started experimenting with codifying these rules to make automated decisions, but when I ran the older snapshots through my routine the results were not 100% there yet. Instead of making automated decisions, I switched to making suggestions instead.

type Suggestion = {
    Command : Command
    Confidence : int
}

type Suggestions = {
    BasedOnSnapshotId : string
    Options : Suggestion seq
}

type MakeSuggestion = Snapshot -> Suggestions

let machineMakesSuggestions snapshot =
    let returnWindowInDays = 14
    let itemInReturnWindow = snapshot.Value.DaysSinceItemPurchased <= returnWindowInDays

    let options =
        if itemInReturnWindow then
            [
                { Command = ShipReplacement; Confidence = 51 }
                { Command = IssueRefund; Confidence = 49 }
		{ Command = DenyRefund; Confidence = 0 }
                { Command = AwardCoupon; Confidence = 0 }
                { Command = IssuePartialRefund; Confidence = 0 }
            ]
        else
            [
                { Command = DenyRefund; Confidence = 50 }
                { Command = AwardCoupon; Confidence = 30 }
                { Command = IssuePartialRefund; Confidence = 10 }
                { Command = ShipReplacement; Confidence = 10 }
                { Command = IssueRefund; Confidence = 0 }
            ]

    { BasedOnSnapshotId = snapshot.SnapshotId; Options = options }

I rendered these suggestions on top of the existing view and observed the decisions that were made.

type PickSuggestion = Suggestions -> Decision

let humanPicksSuggestion by : PickSuggestion =	
    fun suggestions ->
        {
            BasedOnSnapshotId = suggestions.BasedOnSnapshotId
            Command = IssueRefund
            MadeBy = by
        }

My partners in crime were quite ecstatic with this new feature. They had a bit more room to breathe and could spend more time doing things they liked.

After observing and comparing the suggestions with the decisions made, I kept tweaking the routine a bit more. I got close but I felt as if I wasn’t quite there yet.

Automating decisions

It was my grandfather who eventually pushed me to fully automate making these decisions. He said “I almost always find myself picking the first option. It’s fine if the machine is a bit off now and then. Just defer decisions and call in a human when the machine is not confident enough.” How can I question my grandfather’s wisdom? And so this happened…

let machinePicksSuggestion : PickSuggestion =
    fun suggestions ->
        let confidenceTreshold = 50

        let command =
            suggestions.Options
            |> Seq.filter (fun x -> x.Confidence >= confidenceTreshold)
            |> Seq.sortByDescending (fun x -> x.Confidence)
            |> Seq.map (fun x -> x.Command)
            |> withFallbackTo DeferDecision
            |> Seq.head

        {
            BasedOnSnapshotId =  suggestions.BasedOnSnapshotId
            Command = command
            MadeBy = "Machine"
        }

From human decisions, to suggestions, to automated decisions

With that, we’ve come full circle. I’m happy to report that my dad, grandfather and I are back to spending more time in my grandfather’s study coming up with new tricks.

I regularly have a look at the data to check for anomalies and to tweak the routine a bit further. But even that is taking up less and less time. To be fair, I didn’t invest in a full blown test suite even though this small routine has grown into 200 lines of code. I find much relief in the fact that when I replay past decisions, I hardly ever find a regression.

Maybe if we invent another popular trick like this one, we will acquire enough data to let the machine do the learning for me. For now, it’s automagical enough.

How to organize a meetup

Mon, 13 Feb 2017 21:21:00 +0100

I’ve organized a few DDDBE meetups in the past, and always succeed in forgetting something. Either someone points it out well in advance, or I end up stressing last minute. This post partly serves as a checklist for myself, but it would be a welcome side effect to also see it encourage others to help out organizing future meetups. Organizing a meetup is not rocket science, having a list of what to take care of is a good start.

Finding a speaker

One of the crucial ingredients of an interesting evening is the speaker and the content they bring. My strategy is to use Twitter to keep a tab on people that produce content that’s of interest to me. A meetup is a great excuse to meet in person and to hear them out. When it’s someone who isn’t local, I keep an eye on whether they will be attending any conferences nearby in the future.

Contacting a speaker

The medium you use doesn’t matter that much, whether it’s through Twitter, email, Slack or in person, as long as you give them enough context to work with.

Things to mention when contacting someone:

Introduce yourself and the user group.
Tell them why you think he or she would make a great guest. Which was the talk, tweet or blog that piqued your interest? What is it that makes the content so relevant to your audience?
Tell them about the expected composition of the audience. Will there be 20 people or 100? Should most of them be quite familiar with the topic, or do you think an introductory talk would work better?
Propose a date or a small set of dates from the get-go. Even if none of those dates work, you might settle on an alternative date instead which allows you to move forward and start planning.
To avoid awkward situations, mention up front whether expenses can be covered or not. Not having a budget normally isn’t an issue when speakers are local or when they’re around for a conference or customer. When they have to get on a plane, it’s another matter.

Gathering speaker requirements

Ask the speaker what he needs, to be able to do his session:

A projector
A whiteboard
Modelling space
Markers and sticky notes
…

With those requirements in mind, you can start looking for a location sponsor to host the meetup.

It’s a luxury to have a pool of location sponsors. Depending on the speaker and format of the session, you can make a selection on what would be the best fit. When the speaker is a big name in the community, try to find a location that fits a lot of people. A large auditorium perhaps. When it’s a workshop, make sure there’s plenty of modelling space. An open space with lots of walls or windows you can use to your advantage. If you think the interest in a certain topic will be rather limited, aim for something a bit smaller and more cozy, which could benefit the quality of interactions between the speaker and the audience.

Once you’ve come up with a short list of ideal locations, you should be aware of the type of relationship your community has with the sponsor. Even if they don’t get much out of it, they might not mind doing you a favor once in a while, but hosting a weekly meetup might be a bit much. I like to order them in a way that makes a best attempt at a round-robin distribution. This also benefits attendees and avoids over concentrated geographical communities. Not everyone’s up to drive half way across the country on a weekday for a one hour talk.

Keep a list of location sponsors and the people you need to contact, and who inside your community knows them best. You have a higher chance of getting a positive response if you have someone that knows them well contact them. You can send an email, but there’s a high risk that it ends up in a low priority queue somewhere - nobody likes sending out those reminder emails. It’s more efficient to just call them, to ping them on Slack or to send them a direct message on Twitter - in that particular order.

Prepare a list of things the location sponsor needs to know:

The date
The name of the speaker and the subject
The speaker requirements
The amount of people you’re expecting
Expectations towards food: sandwiches, pizzas, soda, beers?

I try to contact location sponsors one after the other. This avoids broadcasting the request, to have to turn it down later on. This has never been a problem, as long as response times are low. If you have to wait weeks to get a reply, you’re going to end up stressing out once the deadline approaches.

Meetup.com

You can schedule a meetup as soon as the speaker and the date are confirmed.

Opening the RSVP’s is best done after the location is also confirmed and not more than one month in advance. You want to avoid people reserving their seat to eventually not show up.

Details you want to include when making the announcement:

The date
The location and the maximum number of attendees
The speaker and the abstract of the session
Where to park your car
Whether food is provided or not. Include what to expect if possible, so that people with a specific diet can make the necessary arrangements.
The agenda of the evening: doors open, start of the session, debriefing

Not all these details need to be present when you schedule the meetup, you can still enrich later on. My own experience tells me I mostly care about the speaker, date and location. I only look at the specifics one week up front.

When you have people on the waiting list, it’s a good idea to send a personal reminder one week in advance, asking people to update their RSPV’s in case they can’t make it. These seem to be more effective than the auto generated ones.

Even with those reminders, you will often find a percentage of people not showing up. Many factors influence the no-show rate. Count the empty chairs, and allow for some overbooking once you get a feel for it. The worst thing that can happen is that the organizers have to watch the session standing up.

Speaker gift

Like Christmas gifts, you also want to think of a speaker gift up front. This doesn’t have to be something super expensive, a small token of appreciation will do. I’m going to guess this is a culture thing. In Belgium people often end up gifting each other beers or wine. This isn’t always appropriate or practical. Not everyone likes to drink and when you’re travelling it’s quite literally a burden on your shoulders. I’m actually thinking that something small, like a book with a personal note might be a better idea.

Recording

Before recording a session, there are a few things you need to consider.

Does the speaker mind being recorded? Some bring experimental content that’s rough around the edges which isn’t necessarily something they’re ready to show to a large audience. Others see it as a perfect opportunity to be able to pitch a conference talk for later on, or to spread their content.

When you announce that the talk will be recorded up front, more people tend to stay at home. They can catch the video later on, but eventually often end up never doing so. Even when they do, they’ve missed out on the best bits: the interactions before and after the talk.

Day of the meetup

Make sure you set aside enough time in your agenda. You don’t want to be stressing out last minute, or God forbid, be late.

Your job is to make the speaker as comfortable as can be, and to think fast when something falls apart last minute.

If the speaker has been travelling, go get him at the station or at the airport. Travel is tiring. Reassure him he can just relax from now on, you will take care of anything he needs. The basics first: is he thirsty, hungry, does he need to use the toilet, an internet connection? You will also help him set up his laptop and put a bottle of water nearby before the session starts.

Arrive at the venue in time, at least 20 minutes up front. Say hi to the host and quick check whether all requirements are met.

Once the first people start pouring in, say hi, point them in the right direction and make some small talk. Once a small crowd has found its way to sandwiches and the social area, you can focus on ensuring the room and speaker are ready to go. The next batch of people to come in will be able to find their way on their own.

Once people are well fed and settled, take the stage and do a short introduction:

Community announcements: scheduled meetups, oncoming befriended conferences…
Sponsor raffles: software licenses, conference tickets…
Mention the types of sponsoring the community is still looking for: locations, markers, budget…
Thank the sponsor(s) and allow them to say a few words if they like to
The agenda for the night
The speaker(s)

Now the session is well on its way, you can mostly just sit back and relax. Pay attention though. In case nobody wants to go first during QA, you should lead by example and have a question ready.

Once questions dry up, or people are thirsty, thank the speaker and don’t forget to hand off your speaker gift.

After the session, the location sponsor usually has a fridge with beverages you’re free to plunder. Out of courtesy, you don’t want to make it too late and see yourself out in time. Ask the host what time that should be. Usually 30% of the people leave right after the talk, the rest sticks around for one or two drinks. When the time comes to leave, not everyone wants to head home right away. As part of your prep, find a bar that’s nearby and easily accessible, where you can gather afterwards in case not everyone has had enough.

Give thanks

Whew, congratulations, you made it! The only thing left to do is to thank the speaker and the location sponsor once more. Give them a shout out in public, and send them a personal thank you note through a more personal medium.

Thanks to the whole DDDBE community for the inspiration and the platform. Thanks to Mathias, Yves, Stijn, Gien, Antonios in particular for reviewing.

Printing money - hacking loyalty points

Sun, 12 Feb 2017 00:20:00 +0100

At work, we’ve all grown quite vigilant when it comes to customers trying to cheat the system. A necessary trait when working in a domain where money flows back and forth and customers are always trying to find an edge that will turn the tables.

While most tend to leave this mindset at work, one colleague in particular, is always probing, trying to find cracks in the surface of everyday models. Years of practice have made him exceptionally good at it too. When you want more than a shallow review of your design, you go see him.

I happened to be a part of his last quest, in which we reverse engineered a case of security by obscurity involving a large supermarket chain.

A bit of backstory first. This supermarket - as many others - offers a loyalty program. In return for your data, you earn one point per two euros spent. As soon as you’ve saved up 500 points, you get a 5 euro coupon which you can use to get a discount or to buy an item in their loyalty shop. To identify yourself, you receive a card which you show to the cashier on checkout or which you scan yourself when using the self-checkout. When you forgot your card in the car or at home, they print a code on the receipt. This allows you to claim your points afterwards by handing it to the cashier, or again by scanning it yourself at the self-checkout.

In this example, we purchased something small without using our loyalty card. The item cost less than four euros, earning us one meager loyalty point. If you take a closer look at the ticket, you can see the amount of points being repeated at the tail of the barcode.

If you’re familiar with the anatomy of some of the established barcode standards, you might notice they’re using the UPC symbology to encode loyalty points.

After proving the manufacturer code was connected to the store the ticket was printed at - by purchasing multiple products from different branches, we figured we could just go ahead and forge our own loyalty points. A quick Google search and a printer were all the tools we needed.

So we first purchased two products on two different days. This gave us a ticket worth one point and another one worth two points. We printed our own barcode worth three points, and claimed it at the self-checkout. We weren’t stopped by security on our way out, and the points were credited to our points wallet, so I guess it was a successful experiment.

To be fair, if it wasn’t for the self-checkout option, it would be much harder to get away with this. If we had to hand over our ticket to a human, we would have needed to counterfeit a lot more than just the barcode. There are reasons it’s so incredibly hard to print fake money.

The system used is quite simple; both from a customer perspective as a technical perspective. It’s a lightweight stateless model which should hardly require any maintenance.

If you wanted to make this system more secure - not necessarily bullet proof, I can think of two obvious options:

Leave the validation of points to humans. I’m not sure how hard it would be to make a high quality copy of a ticket with lots of points on it.
Spawn a new state machine identified by a random token each time you hand out points. This state machine allows points to be claimed and to expire.

Please chime in if you know of battle-tested models for this type of functionality.

Closing off, we didn’t do any harm during our research: we didn’t take more than we paid for.

Everyone has to serve somebody, but software has to serve more than one

Fri, 13 Jan 2017 15:08:00 +0100

When people get paid to write software, we very often find some form of friction between the people that build the software and those that pay to have it built.

The company I joined three years ago was no exception. When I joined, they had just launched three months ago, weren’t seeing any return on investment and prospects weren’t too bright either.

I won’t lie, that first year was quite stressful. The business model sucked, the code was even worse. That first year, I wouldn’t have been too surprised if the whole team had been let go. In that situation, I could have played it safe, but to be fair, there was something quite exciting about actually having skin in the game and trying to make something out of nothing.

The best thing to come out of this situation, is that each and every engineer acquired a high sense of ownership. When you’re a team of only a few people, you have to carry your weight and make an impact. And the only impact that mattered was coming up with changes that would make customers return to us instead of to our competitors. A product without customers, isn’t a product at all.

Being extremely customer-focussed, we did set aside many of our own concerns. Both from a personal and technical perspective. We spent too much time at the office, putting in more hours than ideal. We were lucky to have a young team, that didn’t have a large day-to-day personal workload though. Since we were also the ones running the operational side of things in the early days, the tooling that was built covered the bare minimum, in a very crude way. On the technical side of things, we played the hand we were dealt. We tried not to make things worse - like treating a slowly healing wound. Improving things when constraints allowed it, still taking the rare short cut when our hand was forced. I still believe today that had we invested in the large technical overhaul the code base deserved - putting ourselves in a position where we would have shipped less features and changes - we would have found ourselves completely pushed out of the market and without a job six months later.

Once things started to turn around for the better, we acquired more and more return customers, but we also acquired a set of customers we didn’t anticipate.

When we were still small, we would handle customer tickets ourselves. The owners didn’t think the workload warranted another hire. After we threw away the homegrown, not very inviting to use, ticket system, we switched to an off-the-shelf live chat system. This gave us a lot more customer feedback - more than we could manage. Where tickets can be processed in batch, chats are expected to be replied to in real time. Nothing you can juggle in-between programming work, giving us a good reason to hire extra people. For these people to be productive, and actually able to help customers, they also needed access to those crude tools we built earlier. Turns out, those tools didn’t really cut it. We needed to listen to their feedback and work towards building tooling that’s obvious and easy to use. And there you have it, an extra type of “customer” we needed to satisfy.

But it didn’t stop right there. Like a scurry of squirrels after a bird feeder, there was more and more pressure from within the organization to provide more sophisticated tools for the expanding operational needs, like invoicing, content management, customer segmentation, localization and so much more.

So far, we had always put the end customer above all. We had always been very conservative with our technical budgets. We did refactor aggressively, but only changing the inside doesn’t always cut it. Larger structural, organizational changes require you to move mountains. Well, more like small hills. We knew the domain and its characteristics well enough to have a pretty good idea where we wanted to go, but the clock had always kept us from making big technical leaps forward.

One could ask: if we have been able to postpone these changes for so long, are they necessary at all? If you would ask non-technical stakeholders, they would probably prioritize the next “one more feature”. Hell, they would probably only start caring when the system comes down crashing every few days. Who we should really be asking however is those maintaining the system. If they feel as if they’re operating a system that’s headed toward needing life support, we should probably pay attention. Eventually something has to give. Eventually you might see your software become a serious liability, or you might see people jump ship even before the tipping point. Engineers are just as much of a customer as your end customer - code is a product too. Engineers also have certain needs you want to satisfy to turn them into happy return customers, who like spending their time, energy and creativity on your product.

It might sound backwards, but even the people you’re paying at the end of the month, are very much customers of your product. A monthly pay check might buy you a higher pain barrier, but it’s not unbounded either. You might even see some of the most passionate people succumb early on. People on the payroll are as much a vital part of the ecosystem as the end customer. If they don’t like being a customer, chances are, your end customer won’t either. Highly contagious. While the end customer is still King in this story, to build something truly sustainable for the future, you need to reckon with each and every entity that interacts with the software.

Consumed in 2016

Tue, 27 Dec 2016 22:10:00 +0100

I’m keeping the tradition alive, sharing how much I’ve consumed over the last year highlighting the things that stood out. 18 books, 8 movies and 9 shows. Looks like I consumed more than other years, which probably also explains why I produced less after-hours.

Books

I finished the Dark Tower series after 3 years. Following Roland Deschain and his ka-tet throughout the 8 books has been an epic adventure. Finishing Harry Potter by the time I was 18, I had little hopes to be ever dragged into such a long and captivating tale ever again. The Stand, another epic by Stephen King is also high up on my list. I’ve seem to have taken a liking to stories that are set in a post-apocalyptic world, in which the antagonist is not necessarily a horde of zombies.

Control the things you can control, maggot. Let everything else take a flying fuck at you, and if you must go down, go down with your guns blazing. - Stephen King, The Dark Tower

Show me a man or a woman alone and I’ll show you a saint. Give me two and they’ll fall in love. Give me three and they’ll invent the charming thing we call ‘society’. Give me four and they’ll build a pyramid. Give me five and they’ll make one an outcast. Give me six and they’ll reinvent prejudice. Give me seven and in seven years they’ll reinvent warfare. Man may have been made in the image of God, but human society was made in the image of His opposite number, and is always trying to get back home. - Stephen King, The Stand

Culture and Empire and The Psychopath Code by the late Pieter Hintjens were thought provoking, yet easy reads. I wish more authors could break apart complex problems in such an understandable fashion. Pieter’s ability to research any given topic and to build a convincing case is a skill I’d like to acquire one day.

For the sake of argument, let’s divide society into four roughly equal chunks. We have the bandits, who specialize in taking from others. Then, we have the beggars, who specialize in getting something for nothing. Middle management, perhaps. Then, we have the bureaucrats, who specialize in making rules and keeping things organized. Finally, we have the bakers, who specialize in making things that other people need. - Pieter Hintjens, Culture and Empire

When the cost of secrets held by one person or group outweighs the benefits to society, then it’s right that those secrets be leaked. Security does not just trump Liberty, he takes her into a dark back alley, violates her repeatedly, and then beats her senseless with a heavy stick. - Pieter Hintjens, Culture and Empire

As far as technical books go, I’ve been mostly reading about things that are relevant to my current job. I learned most from Site Reliability Engineering and SQL Server 2012 Internals. They are long, very slow reads, but the information has a depth to it which is hard to get by in other formats. In retrospect, I often wonder if I might have been better of reading them more selectively though.
Covering the same domain, I finally got around to reading the classic Release it! This book was published in 2007, and it shows. However, that doesn’t mean it’s not worth your time. It’s fascinating to see how much the infrastructure landscape has changed over the years. Knowing where we’re coming from deploying and running large systems helps me understand why we’re doing things in a certain way today.

Movies

The Revenant. If you’re looking for a rich storyline, you might end up disappointed. However, it’s the high quality cinematography and breathtaking scenery that make this film. I’m a sucker for the romance of the American frontier though.

Snowden. Not a documentary, but a Hollywood picture telling the story of the most disputed whistleblower of the century. To be fair, having worked with government tech over the years, I used to be ignorant of the fact that a government agency would be able to acquire the talent needed to build and run a mass surveillance system successfully. Being Belgian, I totally ignored the fact that in other countries there are some really smart people patriotic enough to serve their country no matter what. Although I read up on the subject when it was early days, the massive scale of it all hit me like a ton of bricks halfway through the movie. Generally I’m pretty excited to work in a field that has changed how we communicate, distribute and do work in such a small period of time. However, it has become more and more apparent to me that the centralized nature of the things we build and use, is also a huge threat to our freedom. It’s crucial that we learn from this incident, and grow towards a world where pardoning Snowden is the obvious thing to do, and in which the people in power toying with our privacy are the outcasts.

I think the greatest freedom that I have gained, the fact that I don’t have to worry about what happens tomorrow, because I’m happy with what I’ve done today. - Edward Snowden

Shows

I got around to watching the first episode of Westworld yesterday, and my expectations are sky-high. Friends have been hyping me up over this show since the first episode.

Earlier in the year, I watched Narcos and Stranger Things. Netflix focussing more on content creation instead of rapidly expanding their portfolio with third party content might pay off big time. Both shows bring high quality, bingeable content.

Podcasts

This American Life has been a favorite of mine for a long time. The Monday morning commute almost becomes enjoyable. Almost.

Software Engineering Daily is the podcast that I use to break out of my technological echo chamber. The pace at which these have been published is unheard-of. Five days a week there’s a fresh one hour interview with a high-profile guest involved in building incredibly interesting systems at scale.

Being the accidental DBA at work, the SQL Server Pain Relief podcast has been compensating for the lack of people I know that have first hand experience running large, high available and performance intensive SQL Server production systems.

If there’s anything I really missed out on that I should watch or read, let me know!

Commands and events with JustSaying and AWS

Sun, 18 Sep 2016 20:52:00 +0200

I’ve been looking into handing a bit of our messaging infrastructure over to a managed alternative. Managing your own messaging infrastructure that should be highly available is not always an investment you want to make in this day and age. Going through the documentation and relying on experiences from some people I trust, I ended up looking at AWS and SNS/SQS.

Making the Github repository rounds, looking for inspiration, I stumbled on JustSaying: a library by the people from JustEat implementing a message bus on top of AWS.

I wanted to find two messaging patterns in this library:

Command queuing. A common pattern in our components is to react to an event by making an HTTP request to an external partner. To improve reliability and throughput, we generally don’t make that HTTP request in the projection itself, but rather drop a command onto a queue which will then be processed in parallel using a bounded amount of retries. When things do go wrong, we either retry the messages by moving them from the error queue back to the input queue or we change the reaction and reset the projection checkpoint, sending the commands again.
Pub-sub. Another pattern used when there is a certain level of familiarity between components, is to have a component publish events. Other components can subscribe to these messages and have them delivered to their own queues.`

Both these styles are supported by JustSaying.

In this example, I have two commands: BookFlight and CancelBooking, with two related events: FlightWasBooked and BookingWasCancelled.

Since JustSaying requires messages to inherit from a base class, these message definitions live on the outside, far from the domain. This allows to decouple the domain from the outside contracts and to make sure the events go out to the world in the format I want them to be.

type BookFlight() = inherit Message()
type CancelBooking() = inherit Message()
type FlightWasBooked() = inherit Message()
type BookingWasCancelled() = inherit Message()

To handle these messages, JustSaying requires you to implement the IHandler interface.

type HandleBookFlight() =
  interface IHandler<BookFlight> with
      member this.Handle msg = printfn "Book flight"; true

type HandleCancelBooking() =
  interface IHandler<CancelBooking> with
      member this.Handle msg = printfn "Cancel booking"; true

type HandleFlightWasBooked() =
  interface IHandler<FlightWasBooked> with
      member this.Handle msg = printfn "Flight was booked"; true

type HandleBookingWasCancelled() =
  interface IHandler<BookingWasCancelled> with
      member this.Handle msg = printfn "Booking was cancelled"; true

Having this out of the way, we need to configure the bus (publishers and subscribers).

First of all, Amazon needs to know who we are and what we’re allowed to do.

let bus =
  CreateMeABus.DefaultClientFactory <- new Func<IAwsClientFactory>(
    fun () -> new DefaultAwsClientFactory(credentials) :> IAwsClientFactory)

We should define which region our infrastructure lives in.

CreateMeABus.InRegion("eu-west-1")

Now we can configure our command queue. Commands should be published using an SQS publisher, directly dropping messages into the “Commands” queue. A point-to-point subscriber will directly pull messages from the “Commands” queue and hand them over to the command handlers.

.WithSqsMessagePublisher<BookFlight>(new Action<SqsWriteConfiguration>(fun x -> x.QueueName <- "Commands"))
.WithSqsMessagePublisher<CancelBooking>(new Action<SqsWriteConfiguration>(fun x -> x.QueueName <- "Commands"))
.WithSqsPointToPointSubscriber()
.IntoQueue("Commands")
.WithMessageHandler(new HandleBookFlight())
.WithMessageHandler(new HandleCancelBooking())

Events are not directly dropped to an SQS queue, but will be created as an SNS topic. We can use SQS to subscribe to these topics and have them delivered to an “Events” queue.

.WithSnsMessagePublisher<FlightWasBooked>()
.WithSnsMessagePublisher<BookingWasCancelled>()
.WithSqsTopicSubscriber()
.IntoQueue("Events")
.WithMessageHandler(new HandleFlightWasBooked())
.WithMessageHandler(new HandleBookingWasCancelled())

Once the bus has been created, we can start listening and publishing messages.

bus.StartListening()

bus.Publish(new BookFlight())
bus.Publish(new FlightWasBooked())

JustSaying will create two SNS topics and four SQS queues: two input queues and two error queues.

Those topic and queue names are not that descriptive once you introduce multiple components and might cause names to collide. JustSaying allows you to define a custom naming strategy. I’ve settled on a strategy that is based on the message type and prefixed with the component name. This has the added advantage that each message type now goes into its own queue.

type ComponentNamingStrategy () =
  interface INamingStrategy with
    member this.GetQueueName (sqsConfig, messageType) =
      "component_" + messageType.ToLower()
    member this.GetTopicName (topicName, messageType) =
      "component_" + messageType.ToLower()

.WithNamingStrategy(new Func<INamingStrategy>(fun _ -> new ComponentNamingStrategy() :> INamingStrategy))

This whole experiment has had a scary low learning curve (maybe a bit too low). While I’m still in the assess-phase, I’m fairly optimistic that running on top of SNS/SQS might take away some of our operational burden. Going over the JustSaying API and code base, it’s quite opinionated and there are things I might have approached differently. Some features I’d like to see, like the library providing a message envelope as a first-class citizen (a base message class is something I’ve regretted in the past) is being worked on, so I’m keeping my eye on those. Since I’m only using command queuing at the moment, I should be pretty safe from future breaking changes to the message format and such.

My InfoQ interview on DDD, events and legacy

Sun, 21 Aug 2016 15:07:00 +0200

Seems that it’s impossible to beat the Gaussian curve of blogging frequency. On the other hand, I spent quite some of my mental blogging budget on an interview with InfoQ.

I’m a bit bummed out that it’s such a large wall of text. When submitting the answers, I highlighted some snippets which should make for easier scanning. Too bad the formatting was lost when publishing it. I included some highlights below.

The interview itself can be found here.

Extracting components: Starting out, this can be as trivial as trying to model boundaries as namespaces or modules.

Invariants: Having core properties enforced deep within the model, allows for a better night’s sleep.

Sizing aggregates: Make your aggregates as small as they can be, but not any smaller. There’s a big difference between an invariant that needs to be strongly held and data that helps the aggregate to make a decision, but which doesn’t require strong consistency.

ORM pitfalls: Being able to navigate through a graph, which basically walks through your whole database, is a great way to lose any sense of transactional boundaries.

The value of bounded contexts: Now when I switch between bounded contexts, it feels like walking through a door, entering a separate room where you can tackle a problem with an unbiased mindset, allowing you to use the right tool for the job.

Introducing domain events: When you don’t want to or can’t afford to invest in the full paradigm shift there’s a middle ground. You can try a hybrid approach in which you, next to persisting state, also persist the events that led up to a specific state. This does entail the risk of introducing a bug which causes split-brain in which your events do not add up to your state.

Designing contracts: If you get the semantics wrong, you will end up with a system that’s held together by brittle contracts that break constantly.

Pieter Hintjens

Wed, 27 Apr 2016 22:47:00 +0200

Writing doesn’t necessarily always come naturally to me. It often takes me days, weeks or even months of toying with an idea, before I think it’s mature enough to put it down into writing. I can’t afford that luxury this time though, I wouldn’t think of myself as much of a friend if Pieter didn’t get to read this in time.

I met Pieter the first time in a bar in Vilnius, December 2013, I accidentally ended up sitting next to him during the traditional pre-conf drinks. The first thing that stood out, was what a comfortable warm sweater he was wearing - I still cherish the memory of that grey woolen sweater on cold winter nights. I’m still unsure whether it was the sweater or his undeniable radiant charisma that made its way into my memories. When Pieter talks, people tend to listen, or at least pay attention. That’s what I ended up doing that night - listening, sipping in the knowledge, afraid to make a fool out of myself joining the conversation.

That next day Pieter opened the conference with probably the most motivational keynote I ever attended, aptly titled “Building stuff changes everything”. Him being a fellow countryman and me having a few Lithuanian beers in me, helped me gather enough courage to properly introduce myself and talk for a bit.

From that moment on, we would only meet a few times a year, traveling to Vilnius with the Belgian delegation or as a guest at the Domain Driven Design Belgium meetup. During that time, we had a few - far from enough - lengthy conversations. Me mostly asking questions, him sharing his point of view, and me trying hard to keep up, taking it all in. Although he was always more than happy to entertain each question you would throw at him, I would always feel a bit selfish keeping him to myself for too long.

The most memorable talk I had with Pieter was during a tête-à-tête in the Vilnius sky bar. We would mingle Dutch and English, whichever language made the next sentence sound best. We shared some personal experiences, he laid out most of the groundwork for what a good year later materialized into “The Psychopath Code”, but most importantly he allowed me a peek through his eyes, looking at his playground we like to call life.

You don’t need his Mensa membership card, to realize he is a highly gifted individual. He could have pursued anything he wanted and been good at it, but he chose all-out for people, freedom and love - making it his mission to evangelize his core beliefs.

His words - both spoken and written - have inspired me more than he can imagine. And they will likely continue to inspire others for many more years to come. His work has given me a framework to build on for the rest of my life. There’s so much to learn from how he is capable of dissecting the world, to document things that are broken and his tireless effort to make it whole again - one protocol, one word, one hug at a time. Where I would often feel overwhelmed by dark hopeless sentiments, he has given me enough tools to overcome those. From his mouth to my heart: “We’re not much more than a pile of ants, but together we can achieve tremendous things”.

Pieter, I’m not half the writer you are, but I hope these words can serve as a testimony to your children what a great dad they had. If your time comes, know that I’m grateful that I’ve been able to call you my friend.

Using a batch layer for fast(er) aggregations

Sun, 24 Apr 2016 22:43:00 +0200

In the oldest system I’m maintaining right now, we have an account aggregate that, next to mutating various balances, produces immutable financial transactions. These transactions are persisted together with the aggregate itself to a relational database. The transactions can be queried by the owner of the account in an immediate consistent fashion.

The table with these transactions looks similar to this:

CREATE TABLE [dbo].[Transaction] (
	[Id] [int] IDENTITY(1,1) NOT NULL,
	[Timestamp] [datetime] NULL,
	[AccountId] [int] NOT NULL,
	[TransactionType] [varchar](25) NOT NULL,
	[CashAmount] [decimal](19, 2) NOT NULL,
	[BonusAmount] [decimal](19, 2) NOT NULL,
	[...] [...] () NULL /* Too much metadata I'm not very happy about */
CONSTRAINT [Tx_PK] PRIMARY KEY CLUSTERED ( [Id] ASC) 
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 100) ON [PRIMARY]
) ON [PRIMARY]

There’s an index on the timestamp, the account identifier and the transaction type, which allows for fast enough reads for the most common access patterns which only return a small subset.

In a use case we recently worked on, we wanted real-time statistics of an account’s transactions over its life time.

SELECT
  TransactionType,
  SUM(CashAmount) AS CashAmount,
  SUM(BonusAmount) AS BonusAmount,
  COUNT(*) AS Count,
  /* ... */
FROM Transaction
WHERE AccountId = @AccountId
GROUP BY TransactionType, /* ... */

Running this query would seek the account id index, to look up all rows that match given predicate. In case one account has tens of thousands of transactions, this results in a high amount of reads. In case your database fits into memory, SQL Server can probably satisfy your query looking in its buffer cache. Although this still has overhead, it’s supposed to be a lot faster than when SQL Server is forced to do physical reads - reading pages straight from disk. In this case, where transactions are often years old, and the database does not fit into memory, odds are high that SQL Server will be reading from disk - which is dog-slow.

One option would be to create a new covering index (including columns like CashAmount etc) for this specific workload. The problem is that indexes don’t come for free. You pay for them on every write, and depending on your performance goals, that might be a cost you want to avoid. It might even be impossible, or too expensive to create such an index on environments that have no maintenance window and no license that allows for online index builds. Assuming that when you don’t own said license, you don’t have read replicas available either.

Considering the workload, the never-changing nature of financial transactions and constraints in place, we applied Lambda Architecture theory on a small scale, starting by building daily aggregations of transactions per account.

This translates into scheduling a job which catches up all days, by performing a query per day and appending the results to a specific table.

/** Aggregate transactions by day **/
INSERT INTO [dbo].[DailyTransactionAggregation]
   SELECT
     CONVERT(DATE, Timestamp) AS Datestamp,
     AccountId
     TransactionType
     COUNT(*) AS Count,
     SUM(CashAmount) AS CashAmount,
     SUM(BonusAmount) AS BonusAmount,
     /* ... */
   FROM [dbo].[Transaction]
   WHERE Timestamp >= :start AND Timestamp < :end
   GROUP BY CONVERT(DATE, Timestamp), TransactionType, AccountId,   /* ... */

On our dataset, this compresses the transaction table by a factor of more than 300. Not just that, by separating reads from writes, we give ourselves so much more breathing room and options, which makes me sleep so much better at night.

As you probably noticed, for real-time statistics on this data, we’re still missing today’s transactions in this table. Since today’s transactions are a much smaller subset and likely to live in SQL Server’s cache, we can query both the batch table and the transaction table, to eventually merge the results of both queries. For our use case, resource usage and query response times have dropped significantly, especially for the largest accounts.

I don’t see it happening in the near future, but in case the usage of these queries grows, we can still borrow more Lambda Architecture practices and push further.

Notifications from an event log

Sun, 17 Apr 2016 17:37:00 +0200

User notifications are a feature that came as an afterthought, but turned out to be rather easy to implement - without touching (read: breaking) existing functionality - thanks to having an immutable event log.

In the domain I’m working in at the moment, we will often give users incentives to return to the website, or to extend their stay on the website. These incentives were only communicated by email at first, and this is a decent medium when you want users to return to the website. However, when you want to extend their stay on the website, you want to avoid users switching contexts between your website and their mail client. But also, as soon as they return to your website, you want to show them a crisp overview of all relevant calls to action. Having most calls to action map to a specific page, the list of notifications can serve as a one-click starting point, lowering the hurdle to browse to a relevant page.

Notifying a user is one thing. Another use case we wanted to solve, is to dismiss notifications as soon as they are no longer relevant.

Two examples of when a notification might no longer be considered relevant:
1.When a bonus is awarded to a user, he might ignore the notification and activate the bonus by directly browsing to the specific page. 2.When a bonus is awarded to a user, he might not visit the website before the bonus expires.

In these cases, to avoid confusion and unsatisfied customers, we want to dismiss the notification automatically.

Let’s say that we’re going to implement notifications for bonuses. We have these type of events to work with.

type Events =
    | BonusAwarded of payload : BonusAwarded
    | BonusActivated of payload : BonusActivated
    | BonusExpired of payload : BonusExpired
and BonusAwarded = { BonusId : Guid; UserId : Guid; ExpiryDate : DateTime }
and BonusActivated = { BonusId : Guid; UserId : Guid; }
and BonusExpired = { BonusId : Guid; UserId : Guid; }

On the other hand, we have a set of commands that interact with notifications.

type Commands =
    | NotifyUser of payload : NotifyUser
    | ReadNotification of payload : ReadNotification
    | DismissNotification of payload : DismissNotification
and NotifyUser = { NotificationId : Guid; UserId : Guid; TemplateId : Guid; Data : Map<string, string> ; LinkId : string }
and ReadNotification = { NotificationId : Guid; UserId : Guid }
and DismissNotification = { UserId : Guid; LinkId : string }

A notification has an identifier, references a user, contains some data, and most importantly can be linked to something.

Working from an immutable event log, we can project the events to commands (to dispatch them eventually).

let react log =
  log
  |> Seq.map (fun e ->
     match e with
     | BonusAwarded payload -> NotifyUser {
         NotificationId = Guid.NewGuid()
         UserId = userId
         TemplateId = "BonusAwarded"
         Data = [ ("ExpiryDate", payload.ExpiryDate.ToString(CultureInfo.InvariantCulture) ) ] |> Map.ofList
         LinkId = "bonus/" + payload.BonusId.ToString("N") }
     | BonusActivated payload -> DismissNotification {
         UserId = payload.UserId
         LinkId = "bonus/" + payload.BonusId.ToString("N") }
     | BonusExpired payload -> DismissNotification {
         UserId = payload.UserId
         LinkId = "bonus/" + payload.BonusId.ToString("N") } )

log |> react |> dispatch |> ignore

When a bonus is awarded to a user, we will notify the user, providing the template id and data that can be used inside of the template. In this example, the notification can be linked to a specific bonus, leveraging the bonus identifier.

The user might now see something like this.

Being aware of the events which a bonus produces over its lifetime, and their significance, we choose to dismiss the notification as soon as the bonus is activated or expired (leveraging the bonus identifier as the link again).

Now it’s up to the UX team (if you’re lucky enough to have one) to decide on how to visualize the difference between a read and a dismissed notification (if at all).

Functional one-liner for running totals in C#

Mon, 28 Mar 2016 16:22:00 +0200

Visualizing some data earlier this week I had to compute the running total of a sequence of numbers.

For example, if the input sequence was [ 100; 50; 25 ] the result of the computation would be a new sequence of [ 100; 150; 175 ].

Muscle memory made me take a procedural approach, which works, but made me wonder if I could get away with less lines of code and without mutable state.

static IEnumerable<int> ToRunningTotalProcedural(IEnumerable<int> sequence)
{
    var runningTotal = 0;

    foreach (var item in sequence)
    {
        runningTotal += item;

        yield return runningTotal;
    }
}

Although C# doesn’t try very hard to push a functional approach, the BCL does give you some useful tools.

The first thing that comes to mind is using IEnumerable’s Aggregate function, which will apply a function over each item in the sequence and will pass the aggregated partial result the next time the function is applied. Each time the function is applied, we can take the last item (if it exists) of the aggregated partial result and add the current item’s value to it, and append that sum to the aggregated partial result.

static IEnumerable<int> ToRunningTotalFunctionalAggregate(IList<int> sequence)
{
    return sequence.Aggregate(
        ImmutableList<int>.Empty, 
        (acc, item) => rt.Add(acc.Any() ? acc.Last() + item : item));
}

Another more compact - but less efficient approach - I could think of, is using the index of each element in the sequence, to take subsets and to sum their values.

static IEnumerable<int> ToRunningTotalLinqTake(IList<int> sequence)
{
   return sequence.Select((item, i) => sequence.Take(i + 1).Sum());
}

Running out of ideas, I ported F#’s Scan function which allows more compact code, without giving up efficiency. This function, similar to the Aggregate function, applies a function over each item in the sequence. However, instead of passing the aggregated partial result each time the function is applied, the value of the last computation is passed in, to finally return the list of all computations.

With a bit of good will, C# allows you to be more functional too.

Consumed in 2015

Fri, 01 Jan 2016 14:49:00 +0100

I started in 2014 to keep lists of everything I consume. I’ve continued this effort throughout 2015 and can now share the items I particularly enjoyed.

In 2015, I read 16 books and 3 papers, watched 3 movies and 4 shows, listened to 1 audio book and no podcasts. A lot less TV compared to 2014, but most of that time went to playing video games with some of my friends. Also a lot less time spent in the car listening to audiobooks or podcasts, since I’m now dropping off my girlfriend every day.

Going over the books I’ve read this year here are my recommendations:

Most of the TV shows I’ve watched this year were more than great:

Visualizing event streams

Sun, 20 Dec 2015 17:59:00 +0100

In my recent talk on Evil by Design, I showed how I’ve been visualizing event streams as a means to get a better grip on how aggregates behave in production. The talk’s scope kept me from showing the code that goes together with the examples shown. Consider this post as an addendum to that talk.

First off, we need a few types: a string that identifies a stream, an event containing a timestamp and its name. A stream which is a composition of an identifier and a sequence of events. We also need a function that’s able to read a stream based on its identifier.

type StreamId = string
type Event = { Timestamp : DateTime; Name : string; Payload : string; }
type Stream = { StreamId : StreamId; Events : seq<Event> }
type ReadStream = StreamId -> Stream

let readStream ( streamId : StreamId ) =
  {
    StreamId = streamId
    Events = seq {
      // implementation goes here
    }
  }

Once we’ve implemented that, we want to go ahead and visualize a single stream. Having some experience with Google Charts, I used the XPlot.GoogleCharts package.

I want to visualize my event stream as a timeline. For that, it makes only sense to use the Timeline graph. This means that I’ll have to make sure I transform my data into a format the Timeline chart can work with, which is a sequence of tuples.

// Timeline data
data : seq<string * string * #value * #value>
data : seq<string * #value * #value>

let asTimelineEventStream stream =
    stream.Events
    |> Seq.map (fun e ->
    (
        stream.StreamId,
        e.Name,
        e.Timestamp,
        e.Timestamp.AddMilliseconds(float 1)
    ))

So we write a function which accepts a stream, and returns a sequence of tuples containing the stream identifier, the event name and the timestamp of the event.

With just a few lines of code, we can already compose our way to a timeline.

"withdrawrequest/..."
|> readStream
|> asTimelineEventStream
|> Chart.Timeline
|> Chart.Show

The result tells a small story: a withdrawal to a casino was requested at 9:44PM, approved at 12:15PM the next day, and eventually completed 7 hours later.

From an operational perspective, this visualization can be used as visual assistance for your support team when users have a question or a complaint. From a more technical perspective, it can be used to get a feel of the domain language and business processes without having to look at code or tests. I could even see this being used in the front-end, where you enable users to monitor a process; think package tracking, document verification and so on.

Once you start exploring aggregates, you will notice that some aggregates look healthier than others; lean and short-lived. While other aggregates are fat and long-lived which can introduce a set of problems:

rebuilding state from a large event stream might kill performance
there’s often more contention on larger aggregates making optimistic (or pessimistic) concurrency very annoying

Spotting one of these instances is an invitation to review your model, to revise true invariants and to break things apart.

We’ve now looked at an aggregate’s event stream in isolation, but often something happening in one place leads to a reaction somewhere else. A simple example: when a new user registers, a promotion is awarded. We can visualize this by rendering multiple streams on one timeline.

Technically, we need to transform a sequence of streams to a single sequence of tuples which we can feed the chart. It’s as simple as mapping each stream for then to flatten the result into a single sequence.

let asTimelineEventStreams streams =
    streams
    |> Seq.map asTimelineEventStream
    |> Seq.concat

correlatedStreams
|> asTimelineEventStreams
|> Chart.Timeline
|> Chart.Show

This one extra step makes the result even more useful.

There’s more potential though; consider showing the payload when hovering over an event, adding commands in the mix, zooming out, zooming in, filtering…

If this is something you could see being useful to you or your organization, let me know! Maybe I can port some bits and polish the concept in the open.

Slides from my talk "Evil by Design" at Build Stuff

Wed, 18 Nov 2015 15:47:00 +0100

Third time attending Build Stuff, first time doing a talk. I’m happy that it’s out of the way and can now just enjoy the conference, but I’m even more excited that it was well-received! The talk should have been recorded, but you can already find the abstract and slides below.

Last year I ventured into the domain of (online) gambling. Given that the industry has been around since forever, I expected most problems to be of the technical kind. As it turned out, the struggle with technology was only part of a bigger problem; to move forward we needed to fully grasp the industry and its consumers. Events started out as a way to dismantle a legacy system, but quickly proved to be an effective tool to gain a deeper understanding of our domain. Visualising event streams, we discovered patterns that helped us identify what drives different types of users. Having a better understanding of what customers are looking for, we dove into existing literature to learn which techniques and models casinos use to cater for each type of user. We learned how to program chance while staying true to the Random Number God. Even when variance is brutal, casinos have enough data and tools to steer clear from the pain barrier. All of this entails interesting problems and software, but isn’t my code damaging society? Or is gambling just another human trait?

Slides

Defining big wins

Mon, 16 Nov 2015 21:31:00 +0100

Casinos invest a lot of energy selling the dream. One way to do this is by showing off people winning big in your casino. Everyone has seen those corny pictures of people holding human-sized cheques right? It’s a solid tactic, since empirical evidence shows that after a store has sold a large-prize winning lottery ticket, the ticket sales increase from 12 to 38% over the following weeks.

If we look at slot machine play, what exactly defines a big win? The first stab we took at this was quite sloppy. We took an arbitrary number and said wins bigger than 500 euro are impressive. This was quick and easy to implement, but when we observed the results we noticed that when you have players playing at high stakes, a win of 500 euro really isn’t that impressive, and we would see the exceptional high roller often dominate the results.

What defines a big win, is not the amount, but how many times the win multiplies your stake. Betting 1 euro to win 200 euro sounds like quite the return right? Coming to this conclusion, we had to define a multiplier threshold that indicates a big win.

Having each win correlate to a bet, we could project the multipliers, and look at the distribution.

In this example I’m using matlab, but we could do the same using Excel or code.

So first we load the multipliers data set.

multipliers = csvread('C:\data\multipliers.csv')

For then to look at its histogram, visualizing how the multipliers are distributed.

histfit(multipliers)

Here we notice that there is a skewness towards large values; a few points are much larger than the bulk of data. Logarithmic scales can help us here.

histfit(log(multipliers), 8)

This shows us a pretty fitting bell curve, meaning the multipliers are somewhat log normally distributed. We could now use the log standard deviation to pick the outliers.

But we can also tabulate the data set and hand pick the cut-off of normal wins.

tabulate(log(round(multipliers)))

    Value    Count   Percent
        0    54905     21.54%
 0.693147    68548     26.89%
  1.09861    29680     11.64%
  1.38629    16421      6.44%
  1.60944     8900      3.49%
  1.79176     8102      3.18%
  1.94591     2238      0.88%
  2.07944     5953      2.34%
  2.19722     1044      0.41%
  2.30259     3297      1.29%
   2.3979      625      0.25%
  2.48491     1128      0.44%
  2.56495      687      0.27%
  2.63906      544      0.21%
  2.70805      820      0.32%
  2.77259     1402      0.55%
  2.83321      364      0.14%
  2.89037      344      0.13%
  2.94444      185      0.07%
  2.99573     2406      0.94%
  3.04452      162      0.06%
  3.09104      139      0.05%
  ...

We could now write a rule in our projection of big wins which states that a log(multiplier) larger than 3 is considered to be a big win.

Matlab, Excel and the like are great domain specific tools for data exploration which can help you reach a better feel and understanding.

Bulk SQL projections with F# and type providers

Sun, 18 Oct 2015 15:30:00 +0200

Early Summer, I had to set up an integration with an external partner. They required of us to daily provide them with a relational dataset stored in SQL Server. Most, if not all of the data was temporal, append-only by nature; think logins, financial transactions..

Since the data required largely lived in an eventstore on our end, I needed fast bulk projections. Having experimented with a few approaches, I eventually settled on projections in F# taking advantage of type providers.

Let’s say we have an event for when users watched a video and one for when users shared a video.

type Events =
   | WatchedVideo of payload : WatchedVideo
   | SharedVideo of payload : SharedVideo
and WatchedVideo = { UserId : string; Title : string; Timestamp : DateTime }
and SharedVideo = { UserId : string; Title : string; Timestamp : DateTime }

We want to take streams from our eventstore and project them to a specific state; a stream goes in and state comes out.

type EventStream = seq<Events>
type Projection<'state> = EventStream -> 'state

Then we want to take that state, and store it in our SQL Server database.

type StoreState<'state> = SqlConnection -> SqlTransaction -> 'state -> unit

Some infrastructure that reads a specific stream, runs the projection, stores the state and checkpoints the projection, could look like this.

let runSqlProjection input =
   let checkpoint = readCheckpoint input.ProjectionName
   let events = readEvents input.SourceStreamName checkpoint
   let newCheckpoint = events |> Seq.map(fun x -> x.EventId) |> Seq.max
   let state = events |> unwrap |> input.Projection

   use conn = new SqlConnection(destinationConnectionString)

   conn.Open()

   use tx = conn.BeginTransaction(IsolationLevel.ReadCommitted)

   input.StoreState conn tx state
   writeCheckpoint input.ProjectionName newCheckpoint conn tx

   tx.Commit()

To avoid data corruption, storing the state and writing the checkpoint happens in the same transaction.

With this piece of infrastructure in place, we are close to implementing an example. But before we do that, we first need to install the FSharp.Data.SqlClient package. Using this package, we can use the SqlProgrammabilityProvider type provider to provide us with types for each table in our destination database. In the snippet below, I’ll create a typed dataset for the WatchedVideos table and add a row.

type Destination = SqlProgrammabilityProvider<"Data Source=...">
// or type Destination = SqlProgrammabilityProvider<"Name=Destination">

// **magic**
let dt = new Destination.dbo.Tables.WatchedVideos()
dt.AddRow("Jef", "Tesla's New Autopilot", DateTime.UtcNow)

I haven’t defined this type, nor was it generated by me. The SqlProgrammabilityProvider type provider gives you these for free, based on the meta data it can extract from the destination database. This also means that when you change your table, without changing your code, the compiler will have no mercy and immediately feed back where you broke your code. In this usecase, where you rather rebuild your data than migrate it, the feedback loop of changing your database model becomes so short, that it allows you to break stuff with much confidence. The only caveat here is that the compiler must always be able to access that specific database, compiling without fails. In practice, this means you need to ship your source with a build script that sets up your database locally before you do any work.

Going from a stream to a dataset is quite declarative and straightforward with the help of pattern matching.

let toWatchedVideosDataset stream =
  let dt = new Destination.dbo.Tables.WatchedVideos()

  stream 
  |> Seq.iter(fun e ->
    match e with
    | Events.SharedVideo x -> dt.AddRow(x.UserId, x.Title, x.Timestamp)
    | _ -> ())

  dt

Storing the result in an efficient fashion is also simple, since the dataset directly exposes a BulkCopy method.

let storeState conn tx ( state : Destination.dbo.Tables.WatchedVideos ) =
    state.BulkCopy(conn, SqlBulkCopyOptions.Default, tx)

When we put this all together, we end up with this composition.

runSqlProjection {
   ProjectionName = "WatchedVideos"
   SourceStreamName = "$all"
   Projection = toWatchedVideosDataset
   StoreState = storeState
}

Executing this program, we can see the data was persisted like expected.

In the real world, you also want to take care of batching and logging, but that isn’t too hard to implement.

Having this approach in production for some time now, I’m still quite happy with how it turned out. The implementation is fast, and the code is compact and easy to maintain.

Aspect ratio calculation

Fri, 11 Sep 2015 23:40:00 +0200

Earlier today I was writing a migration script in F# where I had to calculate the aspect ratio based on the given screen dimensions. This is one of those problems where I don’t even mind breaking my head over, but directly head over to Stackoverflow to find an accepted answer which I can just copy paste. Since I didn’t find an F# snippet I could use, I ported some JavaScript, and embedded the result below for future snippet hunters.

The aspectRatio function does two things:

Recursively find the greatest common divisor between the width and height
Divide the width and height by the greatest common divisor

let aspectRatio width height =
  let rec greatestCommonDivisor x y =
    match y with 
    | 0 -> x
    | _ -> greatestCommonDivisor y (x % y)

  let gcd = greatestCommonDivisor width height

  width / gcd, height / gcd
  
// > aspectRatio 1200 900;;
// val it : int * int = (4, 3)
// > aspectRatio 100 100;;
// val it : int * int = (1, 1)
// > aspectRatio 900 1200;;
// val it : int * int = (3, 4)

Basic casino math

Mon, 22 Jun 2015 22:52:00 +0200

In a previous series of posts, I went over the models used by casinos to spin a wheel (spinning, manipulating the odds, clustering and near misses). I did not yet expand on the basic mathematical models that ensure a casino makes money.

Let’s pretend we are spinning the wheel again. The wheel has 5 pockets, and just one of those is the winning one. Given we will be using an unmodified wheel, you win 1 out of 5 spins. Each bet costs you 1 euro. Looking at the true odds (1/5), the casino should pay out 4 euro for you to break even.

 - 1M (Loss)
 - 1M (Loss)
 - 1M (Loss)
 - 1M (Loss)
 + 4M (Win)
 = 0M (Total)

Respecting the true odds would not make the casino any money, they pay out less to ensure that the house has an edge on you. So instead of paying out 4 euro, it will be a tad less.

-  1M (Loss)
-  1M (Loss)
-  1M (Loss)
-  1M (Loss)
+  3M (Win)
= -1M (Total)

The house edge can be cast into a fairly simple formula.

let houseEdge oddsOfWinning winnings oddsOfLosing stake =
  oddsOfWinning * winnings - oddsOfLosing * stake

houseEdge ( 1M / 5M ) 3M ( 4M / 5M ) 1M |> printfn "%A"

// 0.20M

In this example, the house edge is a whopping 20%, meaning statistically 20% of each bet will go to the casino. So the higher the house edge, the better?

Not really, if players constantly go through their bankroll in a matter of minutes, it’s not very likely they will keep returning to your casino. The inverse to the house edge, and maybe even a more important number, is the payout percentage. When the house edge is 20%, the player’s payout percentage will be 80%. For each bet you make, you will statistically see a return of 80%. As a player to get maximum value for money - to play as long as possible - you should aim to play in a casino that has the highest payout percentages.

Often misunderstood is that this does not mean you will get to keep 80% of your bankroll by the end of the night. The payout percentage relates to a single bet. The casino’s hold, or money eventually left on the table, is several times the house edge, since players tend to circulate through the same money more than once. So the longer you play, the more the house edge will nibble at your bankroll.

Knowing the house edge, it’s pretty simple for a casino to predict a customer’s worth; multiply the house edge, the average stake and the number of games per hour.

let customerValue houseEdge stake handsPerHour =
       houseEdge * stake * handsPerHour

customerValue 0.2M 1M 60M |> printfn "%A"

// 12M

Given we spin the wheel 60 times an hour for a stake of 1 euro, we will make the casino 12 euro an hour on average. The higher this number, the bigger your potential, the harder a casino will try to make you a regular.

Understanding how casinos make a living, it’s safe to say casinos aren’t the place to play for money, but to play with money.

Consumed: Queries and projections (F#)

Sun, 24 May 2015 18:00:00 +0200

This is the third post in my series on porting a node.js application to an F# application.

So far, I’ve looked at parsing command line arguments, handling commands and storing events. Today, I want to project those events into something useful that can be formatted and printed to the console.

In the original application, I only had a single query. The result of this query lists all items consumed grouped by category, sorted chronologically.

type Query = | List

type ListResult = { Categories : seq<Category> }
       and Category = { Name : string; Items : seq<Item>; }
       and Item = { Id : string; Timestamp : DateTime; Category : string; Description: string; Url: string }

Handling the query is done in a similar fashion to handling commands. The handle function matches each query and has a dependency on the event store.

Where C# requires a bit of plumbing to get declarative projections going, F#’s pattern matching and set of built-in functions give you this for free.

We can fold over the event stream, starting with an empty list, to append each item that was consumed, excluding the ones that were removed later. Those projected items can then be grouped by category, to be mapped into a category type that contains a sorted list of items.

let handle read query =
        match query with
        | Query.List ->
            (
                let folder state e =
                    match e with
                    | Event.Consumed data ->
                        {
                            Id = data.Id;
                            Timestamp = data.Timestamp;
                            Category = data.Category;
                            Description = data.Description;
                            Url = data.Url
                        } :: state
                    | Event.Removed data ->
                        List.filter (fun x -> x.Id <> data.Id) state

                match read "$all"  with
                | EventStream.NotExists _ -> { Categories = Seq.empty }
                | EventStream.Exists ( _ , events ) ->
                    (
                        let items = Seq.fold folder [] events
                        let categories =
                            items
                            |> Seq.groupBy (fun x -> x.Category)
                            |> Seq.map (fun ( x , y ) -> { Name = x; Items = y |> Seq.sortBy (fun x -> x.Timestamp) })
                        { Categories = categories }
                    )
            )

The result can be printed to the console using a more imperative style.

match parse argv with
| Success(Query(query)) ->
   (
       let result = QueryHandling.handle (read path) query

       for c in result.Categories do
           printfn "%s" c.Name
           printfn "%s" (String.replicate c.Name.Length "-")
           for i in c.Items do
               let ts = i.Timestamp.ToString("dd/MM/yyyy")
               printfn "%s - %s | %s (%s)" ts i.Id i.Description i.Url
   )
| _ -> ...

And that’s it, we’ve come full circle. We can now consume items, remove items and query for a list of consumed items.

λ Consumed.exe -help

Following commands are available:
-n consume -c category -d description -u url
-n remove -id id
-n list

λ Consumed.exe -n consume -c book -d "The Drawing of the Three" -u "..."

Yay! Something happened = Success ...

λ consumed.exe -n list

book
----
24/05/2015 - 24052015125831 | The Drawing of the Three (...)

Compared to the node.js implementation, the F# version required substantially less code (two to three times less). More importantly, although I wrote tests for both, I felt way more confident completing the F# version. A strong type system, discriminated unions, pattern matching, purity, composability and a smart compiler makes way for sensible and predictable code.

Source code is up on Github.

Consumed: Handling commands (F#)

Sun, 17 May 2015 17:51:00 +0200

As I wrote earlier, I’ve been working on porting a node.js web application to an F# console application. It’s an application I wrote to learn node.js but still use today to keep track of all the things I consume.

The application is able to consume an item, to remove a consumed item and to query all consumed items.

In the previous post, I parsed command line arguments into typed commands and queries. Today, I’ll look at handling the two commands.

type Command =
	| Consume of data : ConsumeData
	| Remove of data : RemoveData
and ConsumeData =  { Id : string; Category : string; Description : string; Url : string }
and RemoveData = { Id : string; }

I’ve refactored the command discriminated union to contain records with data that go along with the command - I found that this makes for more discoverable and refactor-friendly deconstruction later on.

Validation

Before we do anything with the command, we need to make sure it passes basic validation. The validate function takes a command, and returns a success or failure result. Validation can fail because an argument is empty, its structure is invalid or it’s out of range. Inside the function we match the command discriminated union with each case, validate the data and return a result.

type Result<'TSuccess,'TFailure> =
	| Success of 'TSuccess
	| Failure of 'TFailure

 type ValidationFailure =
        | ArgumentEmpty of string
        | ArgumentStructure of string  
        | ArgumentOutOfRange of string

type HandlingFailure =
	| Validation of ValidationFailure
	
let succeeds s = Success s
let validationFails f = Failure(Validation(f))
	
let validate cmd =
	match cmd with
	| Consume data -> 
	    (
	        if data.Id = "" then validationFails(ArgumentEmpty("id"))
	        else if data.Category = "" then validationFails(ArgumentEmpty("category"))
	        else if data.Description = "" then validationFails(ArgumentEmpty("description"))
	        else if data.Url = "" then validationFails(ArgumentEmpty("url"))
	        else if not ( [| "book"; "movie" |] |> Seq.exists (fun x -> x.Equals(data.Category, StringComparison.OrdinalIgnoreCase) ) ) then validationFails(ArgumentOutOfRange("category"))         
	        else if not ( data.Url.Contains("http://") || data.Url.Contains("https://") ) then validationFails(ArgumentStructure("url"))
	        else succeeds cmd
	    )
	| Remove data -> 
	    (
	        if data.Id = "" then validationFails(ArgumentEmpty("id"))
	        else succeeds cmd
	    )

Producing events

Having validated the command, we can start thinking about doing something useful. I want the command handlers to be pure, to be able to focus on computation, without having to worry about side effects.

Since the node.js web application stores its data in the form of events, this one will too. I can now migrate the existing event store to a simple text file living in my Dropbox, for then to drop the existing Postgres database.

This means that command handlers will need to produce events.

type Event =
	| Consumed of data : ConsumedData
	| Removed of data : RemovedData
and ConsumedData = { Timestamp : DateTime; Id : string; Category : string; Description : string; Url : string }
and RemovedData = { Timestamp : DateTime; Id : string }

type CmdResult = | Event of stream : string * event : Event

Dependencies

Looking at the tests the command handlers need to satisfy, we know that a command handler depends on the system time and the eventstore.

The dependency on time is just a function that takes no arguments and returns a datetime.

unit -> DateTime

An implementation could look like this.

let thetime() = DateTime.UtcNow

Reading an event stream is a function that takes a stream name and returns an event stream.

type EventStream =
	| NotExists of name : string
	| Exists of name : string * events : seq<Event>

string -> EventStream

Implementing a naïve event store takes more than one line on code.

An eventstore

This implementation stores events in a text file. When an event is stored, it gets serialized to JSON for then to be appended to a text file. When reading a stream it will read all events from disk, deserialize them to then filter by stream name before returning it as an event stream - it’s not exactly web scale.

type StoredEvent =  { Stream : string; Body : Event  }

let store path stream e =
        let serialize e = JsonConvert.SerializeObject e

        let writeToDisk ( line : string ) = 
            use wr = new StreamWriter(path, true)
            wr.WriteLine(line)
        
        let eventOnDisk = { Stream = stream; Body = e } 

        serialize eventOnDisk |> writeToDisk 
        Success eventOnDisk

let read path stream =
	let deserialize x = JsonConvert.DeserializeObject<StoredEvent>(x)

	let readFromDisk = File.ReadAllLines path |> Seq.map deserialize

	let eventsFromDisk =
		match stream with
		| "$all" -> readFromDisk
		| _ -> readFromDisk |> Seq.filter (fun e -> e.Stream = stream)

	match eventsFromDisk |> Seq.isEmpty with
	| true -> EventStream.NotExists stream
	| false -> EventStream.Exists (stream, eventsFromDisk |> Seq.map (fun e -> e.Body))

The signature for reading a stream doesn’t satisfy the signature we defined earlier though. We can satisfy it by creating a partially applied function.

// path:string -> stream:string -> EventStream to string -> EventStream

let read stream = read "eventstore.txt" stream

Handlers

Handlers focus on pure computation, they just need to return an event or a failure.

We can only consume an item once, and we can only remove items that exist. It shouldn’t be possible to consume items that have been removed. There isn’t much needed on the inside to cover these use cases.

We inject the event store and time dependencies by passing in the relevant functions - since I’m already using this function further on in program.fs, the compiler can infer the signatures, no need to explicitly state the signatures I defined earlier.

type CommandFailure =
	| ItemAlreadyConsumed
	| ItemDoesNotExist

type HandlingFailure =
	| Validation of ValidationFailure
	| Command of CommandFailure
	
let commandFails f = Failure(Command(f))

let handle read thetime cmd =
	match cmd with
	| Command.Consume data ->
	    let stream = sprintf "consumeditem/%s" data.Id
	
	    match read stream with
	    | EventStream.Exists _ ->
	        commandFails ItemAlreadyConsumed
	    | EventStream.NotExists stream -> 
	        succeeds ( Event ( stream, Consumed { Timestamp = thetime(); Id = data.Id; Category = data.Category; Description = data.Description; Url = data.Url } ) )
	| Command.Remove data ->
	    let stream = sprintf "consumeditem/%s" data.Id
	
	    match read stream with
	    | EventStream.NotExists ( _ ) ->
	        commandFails ItemDoesNotExist
	    | EventStream.Exists ( stream , _ ) ->
	        succeeds ( Event ( stream, Removed { Timestamp = thetime(); Id = data.Id } ) )

Side effects

So far we have been able to avoid intentional side effects - we did introduce functions that might have accidental side effects (reading from disk and reading the system time ). It would be nice to be able to restart the application without losing all state, so we need to take the result the command produced and persist it. A small function takes care of matching each result to invoke the relevant side effect. So far, we only want to store events. With this, we successfully isolated side effects to one small function.

let sideEffects store input =
	match input with
	| Event ( stream, event ) -> store stream event

Putting it all together

By now, we can validate commands, handle them and take care of their side effects. We can now compose those pieces together using Railway Oriented Programming and invoke the pipeline. The output gets matched, so we can print something relevant for the user to see.

let read stream = read path stream
let handleCommand cmd = 
	cmd 
	|> validate
	>>= handle read thetime 
	>>= switch ( sideEffects (store path) )

match handleCommand cmd with
| Success e 
	-> printfn "Yay! Something happened = %A" e
| Failure(Command(ItemAlreadyConsumed)) 
	-> printfn "Item was already consumed"
| Failure(Command(ItemDoesNotExist)) 
	-> printfn "Item does not exist"
| Failure(Validation(ArgumentEmpty x)) 
	-> printfn "Argument empty = %A" x
| Failure(Validation(ArgumentStructure x)) 
	-> printfn "Argument structure invalid = %A" x
| Failure(Validation(ArgumentOutOfRange x)) 
	-> printfn "Argument out of range = %A" x

Next time, we’ll look at implementing queries.

Finding unused code (F#)

Sun, 26 Apr 2015 18:06:00 +0200

Coming from C#, I’m used to the compiler warning me about unused variables. Relying on the compiler to help me with checked exceptions in F#, I noticed that unused values (and functions) would go unnoticed. Having accidentally read earlier that Haskell has a compiler flag to check for unused bindings, I looked for the F# equivalent but failed to find it, until Scott Wlaschin pointed me in the right direction.

By using the –warnon:1182 flag, the compiler will warn you about unused bindings.

For example, compiling Paket.Core with this flag enabled, outputs the following warnings.

1>C:\Paket\src\Paket.Core\Utils.fs(88,9): warning FS1182: The value 'fi' is unused
1>C:\Paket\src\Paket.Core\Utils.fs(361,46): warning FS1182: The value 'econt' is unused
1>C:\Paket\src\Paket.Core\Utils.fs(361,52): warning FS1182: The value 'ccont' is unused
1>C:\Paket\src\Paket.Core\PackageSources.fs(97,25): warning FS1182: The value 'uri' is unused
1>C:\Paket\src\Paket.Core\RemoteDownload.fs(147,18): warning FS1182: The value 'downloaded' is unused
1>C:\Paket\src\Paket.Core\RemoteUpload.fs(71,16): warning FS1182: The value 'whyIsThisNeeded' is unused
1>C:\Paket\src\Paket.Core\RemoteUpload.fs(85,17): warning FS1182: The value 'progressSubscription' is unused
...

Looking into these warnings revealed values and functions that can be deleted, but no apparent bugs. There are also cases where unused bindings make sense, for example when you pass in a function that does not use all of its arguments or when pattern matching. In these cases you can suppress the warning by prefixing the bindings with an underscore.

fun (cont,_econt,_ccont) -> ...

A useful compiler feature which strangely enough is opt-in. I plan on using it from now on.

Consumed: Parsing command line arguments (F#)

Sun, 19 Apr 2015 16:54:00 +0200

Last year, I set out to write my first node.js application; a small web application for keeping lists of everything I consume. I had something working pretty quickly, deployed it to Heroku and still find myself using it today. Since there’s very little use for having it running on a server, and because I wanted something to toy with getting better at F#, I decided to port it to an F# console application.

With the UI gone, I need to resort to passing in arguments from the command line to have my program transform those into valid commands and queries that can be executed.

The set of commands and queries is limited; consume an item, remove an item and query a list of everything consumed.

module Contracts =

    type Command =
        | Consume of id : string * description : string * url : string
        | Remove of id : string

    type Query =
        | List

Ideally I go from a sequence of strings to a typed command or query. However, when the list of arguments can’t be parsed, I expect a result telling me what failed just the same.

type Result<'TSuccess,'TFailure> = 
	| Success of 'TSuccess
	| Failure of 'TFailure
	
type ParserFailure =
	| ArgumentsMissing 
	| KeyMissing of string
	| KeyLooksLikeValue of string
	| NotFound
	
[<Test>]
let ``Parsing consume command``() =  
    let expected = Consume("2", "The Dark Tower", "http://thedarktower.com")
    let actual = parse [| "--n"; "consume"; "--id"; "2"; "--d"; "The Dark Tower"; "--u"; "http://thedarktower.com"; |]
    match actual with
    | Success(Command(x)) -> x |> should equal expected
    | _ -> Assert.Fail()

Since we need the name to identify the command or query, I expect the input to have at least two arguments.

let ensureEnoughElements input =
    match ( input |> Seq.length > 1 ) with
    | true -> Success input 
    | false -> Failure ArgumentsMissing

Arguments come in pairs; a key and a value. My first thought was to build a map here, but that made key validation, key transformations and pattern matching harder. I can actually get away with transforming the input to a sequence of tuples.

let pair input =    
	input 
	|> Seq.pairwise   
	|> Seq.mapi (fun i x -> if i % 2 = 0 then Some(x) else None)
	|> Seq.choose id

Hoping to avoid some mistakes in the input, basic validation makes sure the keys actually look like keys, instead of a value. Keys start with a single or double dash.

let ensureKeysDontLookLikeValue ( arguments : seq<string * string> ) =
	let looksLikeValue = 
		arguments 
		|> Seq.tryFind ( fun ( k, v ) -> not (k.StartsWith("-") || k.StartsWith("--")) )
	match looksLikeValue with
	| Some ( key, value ) -> Failure(KeyLooksLikeValue key)
	| None -> Success arguments

Once that validation is out of the way, I strip away those dashes. That should make the two last steps easier.

let stripKeys ( arguments : seq<string * string> ) =        
    arguments |> Seq.map (fun ( k, v ) -> ( k.Replace("-", "").ToLower(), v ))

The name is required, so I wrote a small function that makes sure a specific key exists.

let ensureKeyExists key arguments =      
	match arguments |> Seq.exists (fun ( k, v ) -> k = key ) with
	| true -> Success arguments
	| false -> Failure(KeyMissing key)

Now that I have a list of arguments, I can map them into a typed command or query using pattern matching.

let toCommandOrQuery arguments =
	match arguments |> Seq.toList with
	| [ ( "n", "consume" ); ("id", id ); ( "d", description ); ( "u", url ) ] ->
	Success(Command(Consume(id, description, url)))
	| [ ( "n", "remove" ); ( "id" , id ) ] ->
	Success(Command(Remove(id)))
	| [ ( "n", "list" )] ->
	Success(Query(List))
	| _ -> Failure NotFound

Having written all these small functions, I can simply compose them using Scott Wlaschin’s Railway oriented programmming.

let result = 
	input 
	|> ensureEnoughElements
	>>= switch pair
	>>= ensureKeysDontLookLikeValue
	>>= switch stripKeys
	>>= ensureKeyExists "n"
	>>= toCommandOrQuery

This is far from a generic command line parser, but it’s simple and covers my needs.

Next up, executing those commands and queries, and printing feedback.

Checked errors in F#

Sun, 29 Mar 2015 17:19:00 +0200

In the land of C#, exceptions are king. By definition exceptions help us deal with “unexpected or exceptional situations that arise while a program is running”. In that regard, we’re often optimistic, overoptimistic. Most code bases treat errors as exceptional while they’re often commonplace. We are so confident about the likelyhood of things going wrong, we don’t even feel the need to communicate to consumers what might go wrong. If a consumer of a method wants to know what exceptions might be thrown, he needs to resort to reading the documentation (or source) and hope it’s up-to-date.

Java on the other hand has a concept of unchecked and checked exceptions. Unchecked exceptions are exceptions that are caused by a programming mistake and should be left unhandled (null reference, division by zero, argument out of range etc); while checked exceptions are exceptions that your program might be able to recover from. They become part of the method signature and the Java compiler forces consumers to handle them explicitly.

While checked exceptions might bloat the method’s contract and enlarge the API surface area, they might have every right to. Dealing with errors is an important part of programming. Having discoverable errors which require thoughtful care, should improve overall quality. Having said that, it also requires careful consideration from the designer to decide what’s truly exceptional.

Coming up with something that can compete with the mechanics of checked exceptions in C# seems to be impossible. We could return a result with an error from a method, but the compiler doesn’t force you to do anything with that result.

F# on the other hand doesn’t allow for the result of an expression to be thrown away. That is, unless you explicitly ignore it, or bind it and leave it unused.

Let’s look at an example. We start by defining two discriminated unions. The first type defines a generic result; it can either be success or failure. The second type defines all the errors that can be returned after deleting a file.

type Result<'TSuccess,'TError> = 
    | Success of 'TSuccess
    | Error of 'TError

type DeleteFileError = 
    | FileInUse 
    | OpenHandle 
    | UnauthorizedAccess

Then we write a function that deletes a file, but instead of throwing exceptions when an error occurs, it returns a specific error. When no errors occur, success is returned.

let deleteFile name =
	match name with
	| "inuse.txt" -> Error(FileInUse)
	| "openhandle.txt" -> Error(OpenHandle)
	| "unauthorizedaccess.txt" -> Error(UnauthorizedAccess)
	| _ -> Success()

When I now use this function, the compiler will tell me that it has a return value which needs to ignored or binded.

deleteFile "inuse.txt"

// This expression should have type 'unit', but has type 'Result<unit,DeleteFileError>'. 
// Use 'ignore' to discard the result of the expression, or 'let' to bind the result to a name.	

deleteFile "inuse.txt" |> ignore
let res = deleteFile "inuse.txt"

While ignoring a result stands out, an unused binding is easier to go unnoticed. I wish the F# compiler had a flag to detect unused bindings.

Assuming I don’t ingore the result, I can use pattern matching to address each error specifically.

let deleteFileAggressively name retries =
	let rec deleteFileAggressivelyIn name retries retry =
		match deleteFile name with	
		| Success _ -> ()
		| Error(FileInUse) | Error(OpenHandle) ->
			match retry <= retries with
			| true ->
				Thread.Sleep 1000
				printfn "Trying to delete file: retry %A of %A" retry retries
				deleteFileAggressivelyIn name retries (retry + 1)
			| false -> printfn "Failed to delete file agressively"
		| Error(UnauthorizedAccess) -> printfn "Unauthorized access"
			
	deleteFileAggressivelyIn name retries 1

deleteFileAggressively "inuse.txt" 5

By not including a wildcard pattern, extending the contract by adding errors will introduce a breaking change. We’ll have to consider what to do with newly added errors.

For example, if I add the error PathTooLong, the compiler shows me this warning.

// Incomplete pattern matches on this expression. 
// For example, the value 'Error (PathTooLong (_))' may indicate a case not covered by the pattern(s).

In summary, it might be more safe to be a bit less optimistic when it comes to errors. Instead of throwing exceptions, making errors part of the public interface, communicating errors explicitly, and handing responsibility on what to do with the error to the caller, might lead to more robust systems. While this can be achieved with C#, the mechanics are error-prone. Expressions and pattern matching make that F# allows for stronger, yet still not ideal, mechanics.

Scaling promotion codes

Sun, 15 Mar 2015 18:01:00 +0100

In our system a backoffice user can issue a promotion code for users to redeem. Redeeming a promotion code, a user receives a discount on his next purchase or a free gift. A promotion code is only active for a limited amount of time, can only be redeemed a limited amount of times and can only be redeemed once per user.

public bool TryRedeem(UserId userId)
{
	if (HasAlreadyBeenRedeemedByUser(userId)) return false;
	if (NoLongerActive()) return false;
	if (Depleted()) return false;

	Apply(new PromotionCodeRedeemed(userId.Value));

	return true;
}

In code these requirements translated into a promotion code aggregate which would guard three invariants.

var promotionCode = _promotionCodeRepository.GetById(promotionCodeId);

if (promotionCode.TryRedeem(userId)) 
{
	_promotionCodeRepository.Update(promotionCode);
		
	return RedeemPromotionCodeResponse.Success();
}

return RedeemPromotionCodeResponse.Unavailable();

The command handler looked something like this.

Depending on the promotion code, we would often have a bunch of users doing this simultaneously, leading to a hot aggregate, leading to concurrency exceptions.

Studying the system, we discovered that the limit on the amount of times a promotion code could be redeemed was not being used in practice. Issued promotion codes all had the limit set to 999999. Just by looking at production usage, we were able to remove an invariant, saving us some trouble.

The next invariant we looked at, is the one that avoids users redeeming a promotion code multiple times. Instead of this being part of the big promotion code aggregate, a promotion code redemption now is a separate aggregate. The promotion code aggregate now picks up a new role; the role of a factory, it decides on the creation of new life.

public Maybe<PromotionCodeRedemption> Redeem(UserId userId)
{
	if (NoLongerActive()) return Maybe<PromotionCodeRedemption>.Empty();

	return new Maybe<PromotionCodeRedemption>(new PromotionCodeRedemption(Id, userId));
}

The promotion code redemption’s identifier is a composition of the promotion code identifier and the user identifier. Thus even when the aggregate is stored as a stream, we can check in a consistent fashion whether the aggregate (or stream) already exists, avoiding users redeeming a promotion code multiple times. On creation of the stream, the repository can pass to the event store that it expects no stream to be there yet, making absolutely sure we don’t redeem twice. The event store would throw an exception when it would find a stream to already exist (think unique key constraint).

var promotionCodeRedemptionId = new PromotionCodeRedemptionId(promotionCodeId, userId);
if (_promotionCodeRedemptionRepository.Exists(promotionCodeRedemptionId))
	return RedeemPromotionCodeResponse.Unavailable();

var promotionCodeRedemption = promotionCode.Redeem(userId);

if (promotionCodeRedemption.HasValue())
{
	_promotionCodeRedemptionRepository.Add(promotionCodeRedemption.Value);
	
	return RedeemPromotionCodeStatus.Success();
}

return RedeemPromotionCodeResponse.Unavailable();

In this example, we were able to remove an annoying and expensive invariant by looking at the data. Even if we had to keep supporting promotion code depletion, we might have removed this invariant and replaced it with data fed into the aggregate/factory from the read model. Ask yourself, how big is the cost of having a few more people redeem a promotion code? Teasing apart the aggregate even further, we discovered that the promotion code had a second role; a creational role. It now helps us spawning promotion code redemptions while still making sure this only happens when the promotion code is active. Each promotion code redemption is now a new short-lived aggregate, while the promotion code itself stays untouched. By checking the existence of the aggregate up front and by using the stream name to enforce uniqueness, we avoid users redeeming a promotion code more than once. This has allowed us to completely avoid contention on the promotion code, making it perform without hiccups.

Domain Language: The Playthrough Bonus

Mon, 23 Feb 2015 19:05:00 +0100

Since online gambling has been regulated in Belgium, basically each eligible license holder has complemented their land based operations with an online counterpart. Being such a small country, everyone wants to secure their market share as soon as possible. The big players have been pouring tons of money in to marketing and advertising, it’s everywhere: radio, television, (online) newspapers, bus stops, billboards, sport events, airplane vouchers - you name it. While regulations for land based casinos are very strict and almost overprotective, regulations for online play are much more permissive. This makes that online casinos can be rather aggressive acquiring new customers.

You will often see online casinos hand out free registration bonuses: “You get 10 euro for free when you sign up, no strings attached!”. This makes it look like casinos are just handing out free cash right? We should all know better than that though.

There are always conditions to cash out a bonus. Bonuses come in different forms and flavors and preconditions to clear them vary wildly. The Playthrough Bonus is the favorite among players by far; it’s straightforward and requires zero investment.

When you receive a Playthrough Bonus, you receive an amount of bonus money, which can be converted to cash by wagering it a specific amount of times. For example; you receive a Playthrough Bonus of 10 euro, which needs to be wagered 30 times before the bonus is cleared. This means that you need to bet 300 euro (10 euro multiplied by 30) in total to clear the bonus and to receive what’s left off your balance.

So is betting the bonus amount 30 times realistic, or will you always close your browser empty handed with nothing to show for on your balance? The answer depends heavily on the payout rate. This percentage represents how much of the money that goes into the casino, is returned to players. In a formula, this is the wins divided by the bets. This percentage is generally a lot higher than most people expect. Casinos aim for a payout rate between 95 and 99 percent. They want to cultivate a long term relationship with happy and social customers, not clear your bank account as soon as you open the door. Note that the payout percentage is an average, not all games are a smooth ride. Some players like big wins, and big losses, while others feel more comfortable losing small, but don’t mind winning small either. Casinos also prefer a smooth ride, especially when it comes to bonuses. They might even tweak games to have less aggressive, more equally distributed wins when bonus money is in play.

Now let’s look at how much money would be left on our balance when we try to clear a Playthrough Bonus of 10 euro with a playthrough of 30 and a payout percentage of 98.

I defined two records (excuse my primitive obsession). First a Bonus record that contains an amount, a balance, the total amount of bets and a playthrough. A few functions are associated with the Bonus record; they allow the bonus to be created, to bet, to win, to check if it still accepts bets and to check if the bonus has been cleared. The second record, the GameSettings, define the payout percentage and the stake of a bet.

type Bonus = { Amount : decimal; Balance : decimal; Bets : decimal; Playthrough: decimal; }
    with    
        static member Create amount playthrough =
            { Amount = amount; Balance = amount; Bets = 0M; Playthrough = playthrough }       
        member x.Win amount =            
            { x with Balance = x.Balance + amount }
        member x.AcceptsBet amount =
            x.Balance - amount >= 0M
        member x.Cleared =
            ( x.Playthrough * x.Amount ) - x.Bets <= 0M
        member x.Bet amount =
            match x.AcceptsBet amount with
            | true -> { x with Bets = x.Bets + amount; Balance = x.Balance - amount }
            | false -> invalidOp "Bonus can't accept bet" 
type GameSettings = { Payout : decimal; Stake : decimal; }

After defining these structures, I defined a function that recursively keeps playing (bet and win) until either the bonus is cleared, or the bonus no longer accepts bets (out of money).

let rec playUntilCleared ( bonus : Bonus ) settings =          
    match bonus.Cleared with        
    | true -> bonus
    | false -> 
    (         
        let bet = settings.Stake
        let win = settings.Stake / 100M * settings.Payout

        match bonus.AcceptsBet bet with
        | false -> bonus
        | true -> playUntilCleared ( bonus.Bet(bet).Win(win) ) settings            
    )

When we run this function we know the answer to our question. On average, we will clear the bonus with four euro left on our balance.

printfn "%A" ( playUntilCleared ( Bonus.Create 10M 30M ) { Payout = 98M; Stake = 0.2M } )

// {Amount = 10M;
// Balance = 4.000M;
// Bets = 300.0M;
// Playthrough = 30M;}

When we turn down the payout to be one percent lower, we only have 1 euro left on our balance. When we turn it down even more, there won’t be anything left.

Given a few parameters which should be available to you (bonus amount, playthrough and even payout percentage), you can calculate how feasible it is to clear a Playthrough Bonus. Unless variance is on your side, I guess it will rarely turn out to be a lucrative grind.

Side by side

Sun, 15 Feb 2015 17:34:00 +0100

This week marked my first year at my current employer. While that event went by rather silently, one year in, a few of my observations are finally shaping up to be cast into writing.

Where I used to work in the typical battery cage, I’m now part of a team of just four people, having the luxury of a big dedicated room to ourselves - a whole floor actually. The room is set up almost symmetrically; two desks on one side of the room and two more on the other side, with quite some space in between. Having only four people in the room makes it easy to casually throw something at the group - be it a question, a critique or a random idea.

I made good use of this perk early on, but noticed that I would too often find myself amid a Mexican Standoff. We would often get ourselves into discussions that quickly turned into a my-opinion-versus-your-opinion and would lead nowhere.

It didn’t make sense how I got myself into this situation time after time, until I read somewhere how to approach petting an unfamiliar dog.
Our species have more in common than you would think. Our shared history of pack hunting has made both our species highly social and interdependent.
For example, when you approach an unfamiliar dog, you shouldn’t pet him on the head since this can be very threatening. It’s better to approach him from the side to rub his ears, neck or back. This behaviour is an evolutionary remnant of pack hunting; members of the pack would rub each other’s shoulder constantly chasing their next meal.

It occurred to me that the cause of our unproductive discussions might be as simple as our desks being in an aggressive position, desk-to-desk or face-to-face.
Looking back at my previous jobs, I found no precedents of having discussions in this position. The horrors of the open plan had always forced me to either walk over or to find a meeting room.

But when I look at my personal relationships and discussions, I find more situations that confirm this theory. When going for drinks, most of my friends prefer a noisy and crowded bar, which force you side-by-side just to make yourself understood. Even when communicating with my girlfriend, it’s not the cliché tete-a-tete dinner dates that yield the best conversations, it’s taking a walk, long road trips or even cooking together.

When it came to avoiding these unproductive situations at work, I now carefully consider which things I can just throw out there, or which things require a side-by-side approach. Even when you got yourself in trouble face-to-face, you can still guide things into a more constructive direction by changing the situation.

Getting side-by-side shouldn’t be too hard in a professional environment. Both pair programming and whiteboard sessions (or one of its more exotic evolutions - model storming, mob programming etc) are ingrained in most places now.

Evolution has wired our brain in a way that makes being side-by-side, preferably chasing the same goal, extremely amicable. Although we’re no longer hunting for food, we still find ourselves chasing different means of prosperity and success. Just like before, it’s still the case that we’re at our best as a team, side-by-side, in pack.

Averages are not good enough (F#)

Sun, 18 Jan 2015 18:40:00 +0100

Let’s (no pun intended) look at a set of response times of a web service.

let responseTimes = [ 18.0; 21.0; 41.0; 42.0; 48.0; 50.0; 55.0; 90.0; ]

People like having a single number to summarize a piece of data. The average is the most popular candidate. The average is calculated by dividing the sum of the input elements by the number of input elements.

let average input =
	let length = input |> Seq.length
	match length with
		| 0 -> raise <| new ArgumentException("Input sequence is empty")
		| _ -> (input |> Seq.sum) / float length 	

// Average = 45.625
// (18 + 21 + 41 + 42 + 48 + 50 + 55 + 90) / 8

The average is a measure of centre which is fragile to outliers; one or two odd irregular values might skew the outcome. The median on the other hand is always representative of the centre, not just when the data distribution is symmetric. The median is determined by sorting the input elements and picking the one in the middle.

let (|Even|Odd|) x =
	if x % 2 = 0 then Even
	else Odd

let median input = 
	let sortedInput = input |> Seq.sort
	let length = input |> Seq.length
	match length with
		| 0 -> raise <| new ArgumentException("Input sequence is empty")
		| 1 -> input |> Seq.nth 0
		| _ -> match length with
			| Even -> ( let first = sortedInput |> Seq.nth (length / 2 - 1)
						let second = sortedInput |> Seq.nth (length / 2)
						(first + second) / float 2)                             
			| Odd -> sortedInput |> Seq.nth ((length - 1) / 2)
				
// Median = 45.0
// 18 21 41 42 48 50 55 90
//          __ __

Both measures are terrible at showing how the data is distributed though. The average and median response time might look fair, but maybe there are a few outliers which are giving a few customers a bad time.

Instead of reducing our input down to a single number, it might be better to extract a table that displays the frequency of various outcomes in the input.

let frequencyDistribution input =
	input |> 
	Seq.groupBy (fun x -> 
		match x with 
		| x when x < 30.0 -> [ 0, 30 ]
		| x when x < 70.0 -> [ 30, 70 ]
		| x when x < 90.0 -> [ 70, 90 ] 
		| _ -> [ 90, System.Int32.MaxValue ] ) |>
	Seq.map (fun (x, y) -> x, y |> Seq.length) 

// Frequency Distribution = seq [
// ([(0, 30)], 2); 
// ([(30, 70)], 5); 
// ([(90, 2147483647)], 1)]

Now this is more useful; we can clearly see that the data is not distributed equally and there are a few outliers in our response times we need to investigate further.

This table takes up quite a bit of ink though. What if we want to get rid of the table, but maintain a feel for the distribution of the data?

The standard deviation measures the amount of variation from the average. A low standard deviation means that the data points are very close to the mean. A high standard deviation indicates that the data points are spread out over a large range of values.
It is calculated by taking the square root of the average of the squared differences of the values from their average value.

let standardDeviation input  =
	let avg = input |> Seq.average
	let x = input |> Seq.map(fun x -> System.Math.Pow(float x - avg, float 2)) |> Seq.sum
	let y = input |> Seq.length |> float
	let variance = x / y
	System.Math.Sqrt variance
	
// Average = 45.625; Standard Deviation = 20.87425148

The standard deviation is even more useful when you put the average at the centre of a graph, lay out the input elements according their deviation of the average and see a bell graph drawn. This means that we can use the empirical 68-95-99.7 rule to get a feel of how the data is distributed.

In statistics, the so-called 68–95–99.7 rule is a shorthand used to remember the percentage of values that lie within in a band around the mean in a normal distribution with a width of one, two and three standard deviations, respectively; more accurately, 68.27%, 95.45% and 99.73% of the values lie within one, two and three standard deviations of the mean, respectively.

For our set of response times, this means that 68.27% of the response times lies within the 24.8 and 66.4 range, 95.45% lies within the 4 and 87.2 range, while 99.73% lies within the -16.8 and 108 range.

When we calculate the standard deviation, we can put one extra number next to the average and derive from just two numbers how the data is distributed.

In conclusion, the mean and the median hide outliers. Looking at the frequency distribution gives you a more complete picture. If we insist on having less data to look at, the standard deviation and the 68–95–99.7 rule can compress the same complete picture into just two numbers.

Consumed in 2014

Sun, 11 Jan 2015 17:45:00 +0100

Starting 2014, I wanted to look more closely at everything I consume. So I started keeping a list of everything I read, watch and listen to.

I started off with a markdown file on Github that quikcly evolved into a good excuse to dabble with an alternative stack. I ended up writing an event sourced node.js application on top of postgres, hosted on Heroku. I could write a post on that particular experience, but it’s safe to say this blog post captures it better than I ever could.

In 2014, I listened to 10 audiobooks and 18 podcasts, read 20 books and watched 12 movies and 13 shows.

I just went over the list and cherry picked three items per category that stuck with me the most.

Books

Addiction by Design: An incredibly complete overview of the gambling industry, with insights into the human psyche which apply far outside the domain of gambling.
11/22/63: I try to read a fiction book for every non-fiction book I read. This was my first “real” King book, and I was hooked instantly. I devoured the 880 pages thick book in less than two weeks, and was left with a big gaping hole in my heart - the journey was more than worth it.
SQL Performance Explained: The author succeeds in capturing the essentials in only 204 pages. This is one of those books you want to have going around at the office.

Movies

Be Here To Love Me: Not-your-average documentary about Townes van Zandt; my musical and philosophical discovery of 2014. A troubled soul, struggling with the futility of existence. Simple poetic songs beautifully crafted, capturing in a few words what would take others books. While darkness inspired his best work, it was also the search for relief of that same darkness that led him further down the path of self-destructive behaviour. Painful to watch. Rake, Waiting around to die, Snake Mountain Blues, Dollar Bill Blues.
3-10 to Yuma: A true modern Western, capturing the American West beautifully, making me nostalgic.
Dallas Buyers Club: Is there one role Matthew McConaughey can’t hack?

Audio books

On Writing - A Memoir of the Craft: “What follows is an attempt to put down, briefly and simply, how I came to the craft (of telling stories on paper), what I know about it now, and how it’s done. It’s about the day job; it’s about the language.” If you’re not a King fan, you will be after finishing this one.
Getting to Yes: Separate people and issues, focus on interest, generate options and use objective criteria. But also, what to do when the other party is more powerful or when they use dirty tricks.
Blink: Making decisions in the blink of an eye; why you should go with instinct and why you shouldn’t.

Shows

True Detective: “Life’s barely long enough to get good at one thing. So be careful what you get good at.” “If the only thing keeping a person decent is the expectation of divine reward then, brother, that person is a piece of shit.” “People incapable of guilt usually do have a good time.” Heavy stuff.
[Rick and Morty](http://www.imdb.com/title/tt2861424/?ref_=fn_al_tt_1: “Listen, Morty, I hate to break it to you but what people call love is just a chemical reaction that compels animals to breed.” “Nobody exists on purpose. Nobody belongs anywhere. Everybody’s gonna die. Come watch TV? “Snuffles was my slave name, you can call me snowball because my fur is white and pretty.” Must be the best animated series of the moment hands down, looking forward to next season. The first season is available for free.
The Walking Dead: Addictive soap opera that features zombies.

Papers

Mostly classics on Functional Programming.

TDD as the crack cocaine of software

Sun, 28 Dec 2014 17:29:00 +0100

The psychologist Mihaly Csikszentmihalyi popularized the term “flow” to describe states of absorption in which attention is so narrowly focused on an activity that a sense of time fades, along with the troubles and concerns of day-to-day life. “Flow provides an escape from the chaos of the quotidian,” he wrote.

This is a snippet from the highly recommended book Addiction by Design, which not only gives you an incredibly complete overview of the gambling industry, but also insights into the human psyche which apply far outside the domain of gambling.

For me, this book was an eye-opener, with the biggest realization being that most gamblers don’t play to win. They play to lose. To lose themselves. Slot machines and video poker are for many people the quickest and surest way to reach flow. It’s this phenomenon that has earned machine gambling the title of “the crack cocaine of gambling.”

It’s not just gamblers that crave for flow though, we all do.

Some of us get up early on the weekends, to drive halfway across the country for a few hours of intensive mountain biking. Others come home after work, throw their laptop in the corner, to engage in an online shooter, zoning out for a good hour. Others will accidentally waste their entire Sunday morning solving a crossword puzzle they bumped on reading the newspaper.

All these activities meet a specific set of preconditions.

Csikszentmihalyi identified four preconditions of flow: first, each moment of the activity must have a little goal; second, the rules of attaining that goal must be clear; third, the activity must give immediate feedback so that one has certainty, from moment to moment, on where one stands; fourth, the tasks of the activity must be matched with operational skills, bestowing a sense of simultaneous control and challenge.

Machine play has all these properties. Let’s look at video poker. The goal is to make a winning combination. The set of winning combinations should be easy enough to remember; they’re similar to live poker. After pushing “deal” you get five cards. The player decides which cards to “hold”. Pushing the “deal” button the second time will draw new cards from the same virtual deck. After the draw, you immediately know whether you’ve won or lost. Feedback is instantaneous. A game is over in a few seconds. Although the outcome is determined by chance, there is some degree of skill involved; it’s up to you to hold the right cards.

As programmers we’re lucky enough to inadvertently end up in the zone frequently. Without a doubt, it’s in this state most of us do our best work. In the zone, it’s constant feedback and a sense of moving forward that keep me going. One could argue that the zone is inherent to the activity of programming. I’d say that the length of the feedback loop and the size of the goals are critical and hard to maintain without working at it.

In this regard, there are a few techniques that help me reach a state of flow. At first I could get by just trying to get the code to compile or to just launch whatever it was I was working on. But once you’re comfortable with a code base, getting it to compile isn’t much of a challenge, and having to start your application to get feedback gets old real quick. Most often it’s TDD that helps me get there these days. You start of with a failing test, your mission: to make it pass. The rules are simple; when your test goes from red to green, you’re allowed to move on. It’s important that tests are fast to be able to give you that immediate feedback. How fast? Fast enough for you not to lose focus. It stands for itself that the fourth precondition is met too; you’re writing the code, doing your best to bend it your way.

When TDD is sold as a productivity booster, it are often strengths such as automated continuous validation of correctness, partitioning of work in smaller units and cleaner and better designed code that are used as arguments. While these are valid arguments, it’s a shame that the power of TDD as a consistent gateway to flow gets neglected or undersold.

Getting in the zone by yourself is one thing, getting there surrounded by a group of people is often out of the question. Here Event Storming has helped me out. Small goals; what happens before this event? Rules; write the previous event down on a yellow post-it. Feedback; once the post-it is up, we see that we’re reaching a better understanding of the big picture. Control and challenge; you’re the one searching for deeper insight, writing and putting up the post-its.

The activities that get me in a state of flow are the ones that I enjoy the most and which enable me to output my best results. If you reread the four preconditions, and assess the things that get you going, you might learn that the same goes for you.

Spinning the wheel: clustering and near misses

Sun, 14 Dec 2014 17:13:00 +0100

The previous post showed a simple model casinos use to manipulate the odds. Instead of relying on the physical wheel for randomness, they rely on a virtual list of indexes that maps to the physical wheel.

Using that same model, it’s easy to fiddle with the virtual indexes so that they map to misses right next to the winning pocket, creating “near misses”. “Near misses” make players feel less like losing, since you “almost won”. Casinos use this technique to get the next spin out of you.

Let’s create more specific labels - a label for each individual pocket.

type WheelSpaceLabel =
    | Miss1
    | Miss2
    | Miss3
    | Win

The winning pocket is in the physical wheel at index two. We need the virtual indexes to make clusters next to the winning label. Four indexes map to Miss2, one maps to Win and three map to Miss3. We intentionally ignore Miss1.

let physicalWheel = [Miss1; Miss2; Win; Miss3] |> Seq.ofList
let virtualWheel = [1; 1; 1; 1; 2; 3; 3; 3] |> Seq.ofList

Spinning the wheel one million times reveals the pattern; Miss1 gets ignored, while we hardly ever win but very often “just” miss.

Results seq [(Miss3, 37501); (Miss2, 49901); (Win, 12598)]

Since the law states that randomness and visualization are two separate concepts, casinos are free to operate in this gray zone, as long as randomness stays untouched.

Spinning the wheel: manipulating the odds

Thu, 11 Dec 2014 19:24:00 +0100

The previous post defined a basic set of data structures and functions to spin a wheel of fortune in F#.

There was very little mystery to that implementation though. The physical wheel had four pockets and spinning the wheel would land you a win one out of four spins. As a casino, it’s impossible to come up with an interesting payout using this model.

To juice up the pot, casinos started adding more pockets to the wheel of fortune. This meant that the odds were lower, but the possible gain was higher. More pockets also allowed casinos to play with alternative payouts, such as multiple smaller pots instead of one big one.

Adding pockets to the wheel didn’t turn out the way casinos hoped for though. Although players were drawn to a bigger price pot, they were more intimidated by the size of the wheel - it was obvious that the chances of winning were very slim now.

Today, instead of having the physical wheel determine randomness, randomness is determined virtually.

Casinos now define a second set of virtual indexes that map to the indexes of the physical wheel.

let virtualIndexes = [0; 0; 1; 1; 2; 3; 3] |> Seq.ofList

There are seven virtual indexes; six map to a miss pocket and only one maps to a win pocket - one out of seven is a win.

Instead of picking a random index in the physical wheel, we now pick a random index in the virtual indexes and map that back to an index in the physical wheel.

let rng = new Random()       
let random (rng : Random) virtualIndexes = 
	virtualIndexes |> Seq.nth (rng.Next(0, virtualIndexes |> Seq.length))
let spin physicalWheel virtualIndexes rng = 
	physicalWheel |> Seq.nth (random rng virtualIndexes)

When we now spin the wheel a million times, the outcome is different. Although the physical wheel has four pockets, we now only win one out of seven times or 14% of the time.

seq [(Miss, 856903); (Win, 143097)]

Using this technique, the physical wheel only serves for interaction and visualization. Randomness is determined virtually, not physically.

In my next post, I’ll describe how casinos have tweaked this model to create “near misses”, making players feel as if they just missed the big pot.

Spinning the wheel

Tue, 09 Dec 2014 19:56:00 +0100

In this post, I’ll define a basic set of data structures and functions to spin a wheel of fortune. In the next post, I’ll show you the simple model casinos use to build a bigger, more attractive pot, without touching the physical wheel and without losing money. Finally, I’ll show you how casinos tweak that model to bend the odds and create near misses.

Let’s say we have a physical wheel with four pockets, which are labeled either miss or win.

type PocketLabel = Miss | Win
type Pocket = { Label : PocketLabel }
type PhysicalWheel = Pocket seq

Three out of four pockets are labeled as a miss, one is labeled as a win. This makes the odds to win one out of four, or 25%.

let (physicalWheel : PhysicalWheel) = 
	[ 
		{ Label = Miss }; 
		{ Label = Miss }; 
		{ Label = Win }; 
		{ Label = Miss }
	] |> Seq.ofList

Spinning the wheel, we should end up in one of four pockets. We can do this by picking a random index of the physical wheel.

let rng = new Random()       
let randomIndex (rng : Random) physicalWheel = 
  rng.Next(0, physicalWheel |> Seq.length)
let spin physicalWheel rng = 
  physicalWheel |> Seq.nth (randomIndex rng physicalWheel)

To avoid a shoulder injury spinning the wheel multiple times, we’ll define a function that does this for us instead.

let spin physicalWheel rng times = 
  [1 .. times] |> List.map (fun x -> spin physicalWheel rng)

Now I can spin the wheel one million times.

let results = spin physicalWheel rng 1000000

If the math adds up, we should win 25% of the time. To verify this, we’ll group the results by label and print them.

let groupedResults = 
	results
	|> Seq.groupBy (fun x -> x.Label) 
	|> Seq.map (fun (x, y) -> (x, (y |> Seq.length)))
 
printfn "%A" groupedResults

Give or take a few hundred spins, we’re pretty close to winning 25% of the time.

seq [(Miss, 750505); (Win, 249495)]

When the odds are this fair, it’s impossible to come up with an attractive enough payout without the casino going broke. What if we wanted to advertise a bigger pot, while keeping the same physical wheel, without losing money? Tomorrow, I’ll write about the simple model casinos have been using to achieve this.

Hot aggregate detection using F#

Sun, 16 Nov 2014 17:20:00 +0100

Last week, I wrote about splitting a hot aggregate. Discovering that specific hot aggregate was easy; it would cause transactional failures from time to time.

Long-lived hot aggregates often are an indication of a missing concept and an opportunity for teasing things apart. Last week, I took one long-lived hot aggregate and pulled smaller short-lived hot aggregates out, identifying two missing concepts.

Hunting for more hot aggregates, I could visualize event streams and use my eyes to detect bursts of activity, or I could have a little function analyze the event streams for me.

Looking at an event stream, we can identify a hot aggregate by having a lot of events in a short window of time.

Let’s say that when six events occur within five seconds from each other, we’re dealing with a hot aggregate.

[<Test>]
member this.``Given a hot aggregate, a hot aggregate will be detected``() = 
	let treshold = 4
	let secondsInWindow = 5		   
	let eventStream = createEventStream [0; 1; 2; 10; 11; 12; 13; 14; 60; 90]
	   
	streamIsHot eventStream treshold secondsInWindow |> should equal true

What I came up with is a function that folds over an event stream. It will walk over each event, maintaining the time window, allowing us to look back in time. When the window size exceeds the treshold, the event stream will be identified as hot. Once identified, the remaining events won’t be analyzed.

let streamIsHot (events : seq<Event>) treshold secondsInWindow =      
	let folder acc e =         
		let detect acc e = 
			let windowFilter = (fun x -> x >= e.Timestamp.AddSeconds(- float secondsInWindow))
			let window = e.Timestamp :: acc.Window |> List.filter windowFilter
			let hot = (window |> Seq.length) > treshold
			{ acc with Hot = hot; Window = window }

		match acc.Hot with 
			| true -> acc
			| false -> detect acc e   
			
	(Seq.fold folder initialDetectionResult events).Hot

Splitting hot aggregates

Sun, 09 Nov 2014 19:00:00 +0100

When you visit a real casino, the constant busy-ness is overwhelming; players spamming buttons, pulling levers, spinning the wheel, gambling on the outcome of sports games, playing cards, feeding the machine, cashing out, breaking bills, ordering drinks or even buying a souvenir. A single player will easily generate a thousand transactions in one sitting.

When you look at an online casino, this isn’t very different. In the system we inherited, the biggest and busiest aggregate by far is a user’s account. Basically every action that has money involved, leads to activity on this aggregate.
This makes sense. An account is an important consistency boundary, if not the most important one. Casino’s can’t afford to have people spend more than their account’s worth.

Since we’re applying optimistic concurrency, bursts of activity would occasionally lead to transactional failures. Looking at a real casino, it’s easy to see why they aren’t running into these types of issues.
In a physical casino, it’s only the owner of the wallet that gets to access it. Casino employees are not allowed to take a player’s wallet out of his pocket to make a transaction. There is no concurrent use of a player’s wallet: single spender principle.
Online on the other hand, we aren’t constrained by common courtesy and have no problem reaching into a user’s pocket. It’s common to have a user playing the slots, while we automatically try to pay out a sportsbetting win once the results of a game are in.

Mapping out an aggregate’s eventstream on a timeline is a great way to visualize its lifecycle and usage patterns. When we did this for an account, we came up with something that looked like this.

Activity peaks when a user starts a game. Each bet and each win drags in the account aggregate. When you know that some players make thirty bets per minute, it should be of no surprise that other processes accessing the account in the background might introduce transactional failures.

Inspired by a real casino, I wonder if users online would appreciate it if we stayed out of their pockets and let them do it for us instead.
Instead of paying out sportsbetting winnings automatically, we could notify a user that his bet was settled and that he can head over to his bet and cash out the winnings to his account any time.
The same goes for games; instead of cashing out wins to a player’s account after each bet, we could - like in a casino - cumulate all winnings in the slot machine itself, also known as a game session, for the player to cash out by pushing a button once he’s done playing. To reduce the amount of small bets taken from the account, we could also encourage users to feed the slot machine before they start playing.

In practice, we would extract behaviour out of the account aggregate and move it into the sportsbet and game session aggregates. It wouldn’t be until the end of their lifecycles, that we would involve the account aggregate to move money around.

By spreading activity to other and shorter lived aggregates, and having the player do a bit of our work, we could reduce the amount of concurrency on the account aggregate and end up with less transactional failures.

But can we really expect of users to cash out manually? Probably not, but we can still use most of the mechanics we just came up with, but cash out automatically. We can cash out winnings automatically when a user leaves a game session. We can queue up sportsbetting winnings and cash out when a user isn’t playing a game.

By exploring alternatives, we discovered that we can work the model to reduce activity and concurrency on the account aggregate, lowering the chances for transactional failures. Now, it’s only fair to say that there are other, more technical, options. The most obvious one would probably be making the existing transactions on the account aggregate shorter, also lowering the chance of concurrent use of the account.

I can’t help but guess the actor model might be a better fit for this type of problem.

Programmatically force create a new LocalDB database

Sun, 26 Oct 2014 16:59:00 +0100

I have spent the last week working in an integration test suite that seemed to be taking ages to run its first test. I ran a profiler on the setup, and noticed a few things that were cheap to improve. The first one was how a new LocalDB database was being created.
An empty database file was included into the project. When running the setup, this file would replace the existing test database. However, when there were open connections to the test database - SQL Server Management Studio for example - replacing it would fail. To avoid this, the SQL server process was being killed before copying the file, waiting for it to come back up.

Another way to create a new database, is by running a script on master. You can force close open connections to the existing database, by putting the database in single user mode and rolling back open transactions. You can also take advantage of creation by script to set up a sane size to avoid the database having to grow while running your tests. When you specify the database size, you need to also specify the filename; I’m using the default location.

using (var conn = new SqlConnection(@"Data Source=(LocalDb)\v11.0;Initial Catalog=Master;Integrated Security=True"))
{
    conn.Open();
    
    var cmd = new SqlCommand();
    cmd.Connection = conn;
    cmd.CommandText =  string.Format(@"
	IF EXISTS(SELECT * FROM sys.databases WHERE name='{0}')
	BEGIN
		ALTER DATABASE [{0}]
		SET SINGLE_USER
		WITH ROLLBACK IMMEDIATE
		DROP DATABASE [{0}]
	END
	DECLARE @FILENAME AS VARCHAR(255)
	SET @FILENAME = CONVERT(VARCHAR(255), SERVERPROPERTY('instancedefaultdatapath')) + '{0}';
	EXEC ('CREATE DATABASE [{0}] ON PRIMARY 
		(NAME = [{0}], 
		FILENAME =''' + @FILENAME + ''', 
		SIZE = 25MB, 
		MAXSIZE = 50MB, 
		FILEGROWTH = 5MB )')", 
	databaseName);

    cmd.ExecuteNonQuery();
}

Switching to this approach shaved seven seconds of database creation.

Print out the maximum depth of recursion allowed

Sun, 19 Oct 2014 17:33:00 +0200

Karl Seguin tweeted the following earlier this week: “An interview question I sometimes ask: Write code that prints out the maximum depth of recursion allowed.”

This question is interesting for a couple of reasons. First, it’s a shorter FizzBuzz; can the candidate open an IDE, write a few lines of code, compile and run them? And second, does he know what recursion is?

Now let’s say, the interviewee knows how to write code and is familiar with the concept of recursion. If he had to do this exercise in C#, he might come up with something along these lines.

static void Main()
{
    Recursive(0);
}

static void Recursive(int i)
{
    Console.WriteLine(i);
    Recursive(i + 1);
}

Before you let him run his code, you ask him to guess the output of this little program. If he’s smart, he won’t give you much of an answer. Instead he will point out that the result depends on the runtime, compiler, compiler switches, machine architecture, the amount of available memory and what not.

If he’s not familiar with the C# compiler and runtime, he might even say there’s a chance the integer will overflow before the call stack does.
The recursive method call is the last call in this method, making it tail-recursive. A smart compiler might detect the tail-recursion and convert the recursive call into a plain loop, avoiding recursion.

Running this program shows that the C# compiler isn’t that smart, and will yield the maximum depth of recursion just before crashing.

...
171424
171425
171426
171427
Process is terminated due to StackOverflowException.

If we were to port this snippet to F#, a functional language in which recursion is a first class citizen, the results are a bit different.

let rec recursiveFun i =
    Console.WriteLine(i.ToString()) 
    recursiveFun (i + 1)
 
recursiveFun 0

This just kept running until I killed it when the count was far over 171427. Looking at the generated IL, you can see that the compiler was smart enough to turn this recursive function into a loop.

If we want the F# implementation to behave more like the C# one, we need to make sure the compiler doesn’t optimize for tail recursion.

let rec recursiveFun i =
    Console.WriteLine(i.ToString()) 
    recursiveFun (i + 1)
 
recursiveFun 0

Running this also ends in a StackOverflowException pretty early on.

I love how this question seems shallow at the surface, but gives away more and more depth the harder you look.

Read an Event Store stream as a sequence of slices using F#

Sun, 05 Oct 2014 15:21:00 +0200

I’m slowly working on some ideas I’ve been playing around with whole Summer. Since that’s taking me to unknown territory, I guess I’ll be putting out more technical bits here in the next few weeks.

Using the Event Store, I tried to read all events of a specific event type. This stream turned out to be a tad too large to be sent over the wire in one piece, leading to a PackageFramingException: Package size is out of bounds.

[07,09:57:30.489,ERROR] TcpPackageConnection: [127.0.0.1:1113, L127.0.0.1:55697, {d9265236-f72b-4418-a470-780ab7ef2af9}]. Invalid TCP frame received.
EXCEPTION(S) OCCURRED:
EventStore.ClientAPI.Transport.Tcp.PackageFramingException: Package size is out of bounds: 186992564 (max: 67108864). 
   at EventStore.ClientAPI.Transport.Tcp.LengthPrefixMessageFramer.Parse(ArraySegment`1 bytes)
   at EventStore.ClientAPI.Transport.Tcp.LengthPrefixMessageFramer.UnFrameData(IEnumerable`1 data)
   at EventStore.ClientAPI.Transport.Tcp.TcpPackageConnection.OnRawDataReceived(ITcpConnection connection, IEnumerable`1 data)

The Event Store already has a concept of reading slices, allowing you to read a range of events in the stream, avoiding sending too much over the wire at once. This means that if you want to read all slices, you have to read from the start, moving up one slice at a time, until you’ve reached the end of the stream.

Avoiding mutability, I ended up with a recursive function that returns a sequence of slices.

let rec read stream startFrom (conn : IEventStoreConnection) = 
    seq {
        let size = 10000
        let slice = conn.ReadStreamEventsForwardAsync(stream, startFrom, size, true).Result

        if (slice.IsEndOfStream) then
           yield slice
        else
           yield slice
           yield! read stream (startFrom + size) conn 
    }

let slices = read "$et-event-name" 0 conn

The road to Großglockner

Sun, 14 Sep 2014 21:01:00 +0200

Sixteen days after leaving home, we’re now on our way back to Antwerp. After Croatia, we’ve driven through Slovenia, Italy, Austra and Switzerland, arriving in France to meet up with my parents for a few days.

France offered us the typical vineyards and chateaux. What, next to the good company, will stick with me the most is the High Alpine road in Austria. We paid 34 euros to be allowed on the road, but the surroundings of that piece of asphalt are extraordinary.
The drive is rough, maybe even more so coming down than going up. No wonder car manufacturers use it to test drive their close to production-ready prototypes.

It has been quite the trip. I had never expected to give my eyes such a show so close to home.

Northbound again

Wed, 10 Sep 2014 11:25:00 +0200

Leaving Plitvice behind us, we crossed the country heading for the sea. We didn’t need to cover a lot of distance, to discover how diverse the Croatian landscape and climate is. In only two hours we went from the cold, foggy lakes and waterfalls to green meadows to dusty mountains to sunbathing coasts.

We followed the coastline for a little while, strolling through old coast towns, eating seafood, drinking something cold. Driving up North and more inland, we spent two days in Motovun, a picturesque town high on a mountain.

We’ll be continuing up North, slowly making our way home.

Tolkien's inspiration

Sun, 07 Sep 2014 10:36:00 +0200

The next morning we woke up to another rainy day. We caught up with the rain in Austria and we have been following it ever since. Or the other way around. Even the locals can’t seem to stop talking about how much rain is falling.

Between showers, we were able to hike the Vintgar trail. The one hour hike takes you along dangerous rapid waters, bringing you to a big waterfall. If I ever wondered how J.R.R. Tolkien came up with inspiration for his famous novels, now I have a good idea.

On Tuesday, exploring the Postojna cave system by train and on foot offered similar enchanting images.

After exchanging the Slovenian scenery for that of Croatia - still raining - we visited the Plitvice Lakes on Thursday. The non-stop rain had caused a big part of the park to flood. If you didn’t mind getting wet up to your knees, wading through the lakes that were now on top of the trails was definitely an option - one that we took.

We’ll spend our last few days in Croatia along the coastline, hoping to at least warm up a little after our wet and cold experience in the national park.

Porsches and a setting that sells them

Tue, 02 Sep 2014 22:24:00 +0200

With the Belgian ‘Summer’ ending, we are exploring South Eastern Europe. This trip will take us through Germany, Austria, Slovenia and Croatia.

On Saturday, we stopped in Stuttgart to complete our list of visits to the three most popular European sports car museums, after visiting Lamborghini and Ferrari two years ago: the Porsche museum. In my teens, my affinity to fast cars was ignited by spending rainy afternoons at one of my best friends’s, attempting to crush each other’s track records playing Need For Speed Porsche Unleashed. The brand has succeeded in making its way into my subconscious early on. It would still be a Porsche I’d buy today, if I would have too much money to spend.

In the meanwhile, my interest for fast cars has tempered, making place for a more general interest in the combination of good design and raw engineering. There was certainly a place for this at the Porsche museum. The architecture sets high expectations. Every detail in the building met Porsche’s standards; from the toilet pictograms to the elevator interior.

Unfortunately, like in most car museums, they fail to really capture the driving experience. Not once did I hear the trademark roar of a Porsche engine, which I learned to love playing video games.

Leaving Stuttgart behind us, we set out for Bled, Slovenia. This took us through Austria, where we diverged far from the optimal route to visit Hallstatt. This small village, famous around the world, even coined as one of the most beautiful places on earth, is hidden deep into the inhospitable landscape of Austria. Being so out-of-the-way might just be what has made it possible to preserve the charm of this little place.

Getting back into the car, it started raining, and it didn’t stop until we crossed the Slovenian border. The type of rain that gives you that claustrophobic car wash feeling. We arrived in Bled well after dark, and went to bed curious of what our surroundings would look like in the morning sun.

Solomon, the architect

Tue, 26 Aug 2014 19:52:00 +0200

Two junior developers who started working for the company at the same time, had been quite competitive with each other from the get-go. They had once been assigned to the same team, but because of the constant bickering, which had put a serious amount of stress on the team, one of them was pulled off the project and reassigned.

A good year later, just the two of them were assigned to a new smallish, but interesting in-house project. When management assigned them to the same project again, they had just been shuffling resources around, and had no idea of the history these two had. An architect was also assigned to the project, but this was not more than a formality. As soon as the enterprise architecture diagram was updated and the paper work was out of the way, he would do an official hand over, but he would only occasionally check in on the project from then on.

The Friday morning after the hand over, the architect was making small talk with one of the project managers in the coffee corner. His agenda for the day was almost empty - exactly how he liked it on Fridays. He planned on making the rounds this morning to check in on projects he was involved with, to attend a meeting at noon, and to spend the afternoon reading up on micro services, for then to go home early. While he charged the coffee machine with another coffee pod, he heard an uproar that had to be coming from the other side of the floor. He couldn’t quite make out what it was about, but he recognized the voices immediately: the two juniors.

He hurried over to the source of the noise. Prying eyes looked curiously over their monitors, ignoring the architect as he passed by. He found the two juniors standing next to the white board, yelling at each other, gesturing vigorously, crossing out parts of the drawings. The architect broke up the fight, and commanded them to get in the meeting room right away. Startled at first, but red-faced just a few seconds later, they shuffled towards the meeting room, not saying one more word, staring at their shoes.

The architect closed the door behind him, and started questioning them. What was this all about? He learned they had a big disagreement on how they should design a part of the new system. The architect, after hearing them out, thought he understood both their solutions. Given the information he had at that time, he thought both solutions were good enough for now - each made trade-offs, and only time would tell them more.
“There are still dozens of possible alternatives out there. I should try to show them how to come to a consensus together,” he thought. “Before I do that, I’m curious to discover their incentives though, so let me try something here.”

He told them that he liked both of their solutions, but that he couldn’t decide which was the better one. Instead, he would take random parts of each solution and throw them together, to come up with a hybrid solution. This way, nobody loses.

Junior number one seemed relieved. He nodded and glanced at his partner. To his surprise, his partner didn’t look very happy. Junior number two blurted out that he’d rather see his own solution in the bin, than to give up conceptual integrity, just because two people can’t agree.

With that, the architect learned that junior number one had fallen into the trap of making it personal, and was trying to save face. Number two however, was wise enough to favor conceptual integrity over getting his way. The architect complimented the kid, and acknowledged that sometimes giving in might be the wise thing to do - not always, you have to pick your battles. Then he rolled up his sleeves, picked up a bunch of post-its and a pen for both of them and said that it was time for them to “fight the problem together instead of fighting each other”. But not before he made another coffee.

Thinking No Computers

Tue, 19 Aug 2014 20:09:00 +0200

The other day I happened to see a curious exchange in one of our businesses. The cashier swapped a torn, but carefully restored by taping it together again, Euro bill for a new one with the manager. Inquisitive, I asked the manager what he was planning to do with that Euro bill. “Once a month, I take all those ripped up or badly worn bills to the National Bank, and trade them for new ones. All you need is one piece of the bank note for them to give you a newly printed one.”

While he started taking care of some other paper work, my mind started racing towards how the National Bank made this system work. I had noticed before that bank notes have an identifier on each side. I figured the National Bank probably stores the identifier of each note they trade, so people don’t go ripping up bank notes with the intent of trading them twice. That seems easy enough, no?

Once the manager finished up his paperwork, I wanted to confirm my idea and asked if he knew how that system worked. How do they avoid people cheating? “It’s really simple actually, you need to own more than 50% of a bill for it to be tradable.”

Well… that’s a much simpler solution. No need to store all the traded notes somewhere, you probably don’t even need a computer at all.

I catch myself regularly defaulting to wanting to solve problems using computers. While taking a step back and thinking of how it would be done without, often exposes a simpler model. Sometimes you will realize that you don’t need a computer at all. If not that, you get to steal from models that have been molded and battle tested for years.

Look at existing organizational structures and search for boundaries. You might find that aligning your software with existing boundaries makes the pieces of the puzzle fit. Learn how departments communicate; passing forms, by phone or by email. Maybe it shows that you need synchronous communication with strong consistency based on a formal protocol or that you might just get away with asynchronous communication with a less strong schema. Go through paper work, formulas, legislation, research papers, books and what not, and that hard to crack nut might become a bit softer. Look at where people are making decisions and how they are deciding on them, do you spot the patterns?

People have been solving problems for hundreds of years, tinkering with and perfecting models, way before computers were a commodity. Natural selection made sure only the strongest made it this far. Those solutions didn’t stop working all of a sudden, nor should they be discarded as a whole. A great deal of them will likely survive another few hundred years, in one form or the other.

Paper notes: A Study and Toolkit for Asynchronous Programming in C#

Sun, 08 Jun 2014 18:33:00 +0200

The .NET framework mainly provides two models for asynchronous programming: (1) the Asynchronous Programming Model (APM), that uses callbacks, and (2) the Task Asynchronous Pattern (TAP), that uses Tasks, which are similar to the concept of futures.

The Task represents the operation in progress, and its future result. The Task can be (1) queried for the status of the operation, (2) synchronized upon to wait for the result of the operation, or (3) set up with a continuation that resumes in the background when the task completes.

When a method has the async keyword modifier in its signature, the await keyword can be used to define pausing points. The code following the await expression can be considered a continuation of the method, exactly like the callback that needs to be supplied explicitly when using APM or plain TAP.

Do Developers Misuse async/await?

One in five async methods violate the principle that an async method should be awaitable unless it is the top level event handler.
Adding the async modifier comes at a price: the compiler generates some code in every async method and generated code complicates the control flow which results in decreased performance. There is no need to use async/await in 14% of async methods.
1 out of 5 apps miss opportunities in at least one async method to increase asynchronicity.
99% of the time, developers did not use ConfigureAwait(false) where this was needed.

The async/await feature is a powerful abstraction. asynchronous methods are more complicated than regular methods in three ways. (1) Control flow of asynchronous methods. Control is returned to the caller when awaiting, and the continuation is resumed later on. (2) Exception handling. Exceptions thrown in asynchronous methods are automatically captured and returned through the Task. The exception is then re-thrown when the Task is awaited. (3) Non-trivial concurrent behavior.

Each of these is a leak in the abstraction, which requires an understanding of the underlying technology - which developers do not yet seem to grasp.

Another problem might simply be the naming of the feature: asynchronous methods. However, the first part of the method executes synchronously, and possible the continuations do as well. Therefore, the name asynchronous method might be misleading: the term pauseable could be more appropriate.

Source

Not about the UI and the database

Sun, 01 Jun 2014 17:55:00 +0200

When you ask an outsider which components an average application consists of, he will most likely be able to identify the user interface and the database. He will also recognize that there is something in between that takes the input from the user interface, applies some logic and persists the result in the database.

In the past, trying to make sense of what goes on the middle, we started

with the best intentions - layering things. Each layer had its own responsibility and would build upon previous layers. Although there was a layer for business logic, we never really succeeded in capturing the essence. In the end we would still be orchestrating database calls, but now we would be forced to go through a bunch of indirections in the form of anemic layers and objects.

Some people saw these designs for what they were, broke free and started optimizing for the shortest path - from user interface to database with the least amount of effort. By aiming to serve the common denominator and by putting their trust in dark magic, frameworks popped up that would allow you to slap together an application in a matter of hours.

The problem with these frameworks is that they leave you with very little room for your own, and you often end up jumping through hoops when you need to deviate from the path carved out for you.

That’s not the only problem though - applications are a lot more than a user interface and a database. What lives between those two is more than a technical necessity - it’s a place where you get to build a model of the problem you are solving. The model gives you an opportunity to learn from and to communicate with domain experts, peers and users. And that’s exactly where most businesses make the difference, not by having a fancy user interface or a carefully designed database schema, but by really understanding and by being absorbed by the problem they are solving. It’s the user interface and the database that are the necessary evil we bring upon ourselves by solving problems using computers.

The state that lives in your database is a side effect - the result of the model’s behavior. The user interface tries to make it as easy as possible for users to drive and use the model.

Although the user interface and the database are important, it’s the model that is the heart and soul of your application.

(*) Disclaimer: all these drawings are simplistic by design.

Eventual consistency in the Wild West

Sun, 25 May 2014 18:01:00 +0200

San Francisco, 1852. With the California Gold Rush at its peak, successful gold-seekers wanted to protect all their precious gold nuggets by storing them in a strong safe. At the time, it wasn’t that easy to have access to a safe though. At the very beginning, it were just a few local merchants that owned one. Not much later, bankers swamped the area hoping to get their piece of the pie - bringing the strongest safes money can buy.

James King of William - who had made a fortune himself mining gold - was one of the first to found a trustworthy bank in San Francisco.
With the city growing from a small 200 residents in 1846 to about 36.000 in 1852, it became harder and harder for the bank to accommodate all their customers in that one office. It needed to expand its operations.
Three months later a new branch opened up on the other side of town. James determined to build a strong brand, wanted to allow customers to go to any of the two branches to deposit and withdraw their money. This meant that the books of the two branches had to be kept consistent. To maintain the books, James commanded all the clerks to duplicate all new records. Two hired horsemen would then come in every few hours and bring those records to the other branch. Since the books were now the same in both branches, customers could deposit and withdraw money in both sides of town. For James, life was good, he was now raking in twice as much.

Pancho and Lefty, two bandits fled from Mexico, trying their luck in California, were instead of spending their Friday afternoon in search of gold, trying to forget their gold drought by playing cards and drinking cheap Whiskey in the local saloon. While Lefty kept rambling on about how unlucky they had been these last few weeks, Pancho paid closer attention to the conversation going on between the saloon keeper and this well-dressed rider that had just entered. The leather bound notebook the rider was carrying stood out immediately - it carried this familiar looking mark and looked expensive. Even though Lefty kept on rambling, Pancho could make up quite a bit eavesdropping on the saloon keeper and the rider’s conversation. That suddenly came to an end though, when the rider knocked back his drink and got ready to leave - “Well, I better get going before the boss man notices that his books are no longer up to date.”
Pancho watched the rider step outside and get back on his horse. Once the rider was out of sight, Pancho with an intense gleam in his eyes, cut off Lefty and ordered him to shut his mouth - “Shut it fool, I have a plan that’s going to make us some easy money.”

That same day, Pancho and Lefty gathered all their coins, opened an account for Pancho at James King’s bank and made their first deposit. Afterwards, they stuck around the bank’s side entrance just to make sure one of the horsemen came by to pick up the books. Twenty minutes later, the same rider they saw at the saloon showed up - Pancho and Lefty spontaneously looked the other way, avoiding to be seen by the rider. They hung around a little longer, to verify that when the driver exited the bank, he was carrying a notebook - a notebook containing the records of the account they just opened and their first deposit.

The next day, Pancho and Lefty got up early. They only got a few hours of shut-eye in, they had been up all night sneaking around, scouting the biggest ranch in town. To make their plan work, they needed to borrow a fast horse.
Impatiently waiting in front of the bank, Lefty breathed a sigh of relief when the clerk opened the bank at nine AM sharp. Before he entered, he looked one more time to the left, where Lefty was standing in the shadows holding the horse - ready to go. Being the first customer to enter the bank, he walked straight up to the counter, pulled out his token of authentication and told the clerk he wanted to make a withdrawal for all that was on his balance. The clerk looked at the token carefully, but didn’t ask any further questions - he still remembered seeing Pancho yesterday, it wasn’t like he owned a fortune.

Money in hand, Pancho firmly walked out of the bank. To make the plan work, he now needed to jump on that horse as quickly as possible and ride off to the branch on the other side of town - if he made it there before the hired horsemen had the chance to transfer the latest records, he might be able to double his assets in a single morning.

Pancho didn’t spare the horse at all, frantically digging his spurs into its sides. He made sure to take the dirt road through town, avoiding obstacles and people where he could.
Tying the horse down next to the bank, he watched one of the horsemen walk out of the bank. Damn. Had he been too slow, were the books already consistent again?
Pancho broke out in a cold sweat, but he had to try. He shuffled into the bank and got in line - there were three customers in front of him. Every time he looked up at the clock, he got more anxious. By the time his turn came, he was terrified. Without looking up, he pulled out his token of authentication once more, and mumbled to the clerk that he wanted to withdraw all of his money. The clerk noticed the drops of sweat on Pancho’s forehead and how Pancho’s hands were shaking. “Poor soul,” the clerk thought, “he must have caught cholera like many others, they’re falling like flies.” The clerk went into the back, looked into the books, opened the safe and handed Pancho everything he was worth.

James King spent very little of his time at one of his banks - he was too busy looking for new ventures to fund. When he was in town and had the time, he did make a habit of jumping by to look in the books right before closing time. Today, one of the clerks walked up to James King as soon as he arrived; “I’m afraid there has been made a mistake in the books, sir. The books show that one customer withdrew all of his money twice today. First here, and then in the other branch. Something must have gone wrong copying the records.” Going through the records, it was obvious to James King what had happened; they had been robbed.

“Thank the lord that it’s only for a small amount. Let’s make sure this doesn’t happen again.” James started by hiring two extra horsemen, but to be even more sure he also had to introduce some new rules. A customer could now only withdraw $5 a day, unless he had proven to be a good and reliable customer. If a customer needed to withdraw extraordinary large amounts, he needed to inform a specific branch one day in advance. This allowed both branches to keep serving all customers smoothly while not giving up on operating in a semi-autonomous fashion. James was even considering opening a new branch in the famous city of angels, Los Angeles.

Next time; isolation levels in the Wild West.

NCrafts Eventstorming slides

Fri, 16 May 2014 15:22:00 +0200

Me and Tom just finished doing our Event storming workshop at NCrafts Paris. Although we made a few mistakes along the way, feedback on the workshop was great. I hope to put something out about what we learned facilitating later this week. People talked, discovered and eventually learned a new domain in under two hours. The domain? Two minutes before the workshop we found a domain expert prepared to talk about his coupon start-up.

Modeling a business process is often associated with people in suits having long and dull meetings. Way too much time gets wasted for an outcome that’s far from reality and which will be obsolete in weeks. Event storming is a workshop format that brings modeling back to all stakeholders, and aims to create usable models.

In this workshop, you won’t be writing any code, but you will be using lots of paper, post-its and a marker. After going over a bit of theory and a small example, we will present you with a real business problem, and in a few short playful sessions you will experience how powerful event storming can be in helping a team gain insight into complex problems. The techniques you will learn in this workshop will pay off immediately.

The slides we used are now up on Slideshare.

To learn more, you can join the EventStormers community, but more importantly, you should start experimenting yourself!

What if we stored events instead of state? - slides

Mon, 12 May 2014 17:41:00 +0200

I just returned from Croatia, where I got to speak twice at the second edition of The Geek Gathering.

Being such a young conference, I had no idea what to expect. Turns out they have a good thing going on; a small, local and very personal approach to conferences. Speakers both local and international, covering topics that serve the community, not their employer.

Together with Tom, I preached Alberto’s Event Storming during a four hour long workshop. As always, people were impressed by how quick you can gain an understanding of a new domain using this technique. Slides of this workshop will be online after I make some tweaks, and try it in Paris on Friday.

In my second talk, I got to share what I’ve learned these last two years on event sourcing. You can find the slides of that talk on Slideshare. Thanks Tom, Mathias, Stijn, Yves and Bart for reviewing them!

Glueing the browser and POS devices together

Sun, 04 May 2014 18:23:00 +0200

I have been occupied building a modest Point of Sale system over these last few weeks. Looking at implementing the client, there were two constraints; it needed to run on Windows and it should be able to talk to devices such as a ticket printer and a card reader.

Although we could use any Windows client framework, we like building things in the browser better for a number of reasons; platform-independence, familiar user experience, JavaScript’s asynchronous programming model and its incredible rich ecosystem. Having to talk to devices ruled out leveraging the browser to deliver our application though - or didn’t it?

Most Windows client frameworks give you a browser component which can be used to host web applications inside of your application. We used this component to host our web application, which turned the hosting application into not much more than a bridge between our web application and the devices.

This bridge processes commands sent by the browser (or the application itself), and produces events which are returned to the browser. I ended up not needing much code to implement this.

I defined two thread-safe queues

one to put commands on, and one to put events on.

private readonly BlockingCollection<ICommand> _commandQueue = 
    new BlockingCollection<ICommand>(); 
private readonly BlockingCollection<IEvent> _eventQueue = 
    new BlockingCollection<IEvent>();

Then I start consuming the command queue in the background by turning it into an observable and subscribing to it. Processing commands in the background ensures that command processing never blocks the UI thread.

Task.Factory.StartNew(() =>
{
    var processor = new CommandProcessor(_eventQueue);

    _commandQueue
        .GetConsumingEnumerable()
        .ToObservable()
        .Subscribe(processor.Execute);
});

When a command is dequeued, the associated handler will be invoked. The handler then does its work while raising events when appropriate.

public class DoSomethingHandler : IHandle<DoSomething>
{
    private readonly BlockingCollection<IEvent> _eventQueue;

    public SleepCommandHandler(BlockingCollection<IEvent> eventQueue) 
    {
        _eventQueue = eventQueue;
    }

    public void Execute(DoSomething cmd)
    {
        _eventQueue.Add(new DoingSomething());

        // do work

        _eventQueue.Add(new FinishedDoingSomething());
    }
}

In the meanwhile the event queue is being processed in the background as well - sending events to the browser as fast as they can be dequeued.

Task.Factory.StartNew(() =>
{
    _eventQueue
        .GetConsumingEnumerable()
        .ToObservable()
        .Subscribe(SendToBrowser);
});

Sending events to the browser is done by invoking a script through the browser control.

private void SendToBrowser(IEvent @event)
{
    object[] args = { string.Format("app.bus.send({0})", EventSerializer.Serialize(@event)) };

    if (WebBrowser.InvokeRequired)
    {
        WebBrowser.BeginInvoke((MethodInvoker)delegate
        {
            if (WebBrowser.Document != null)
                WebBrowser.Document.InvokeScript("eval", args);
        });
    }
    else
    {
        if (WebBrowser.Document != null)
            WebBrowser.Document.InvokeScript("eval", args);
    }
}

In the browser, we can now transparently subscribe to these events. As an implementation detail on that side, we’re using Postman for pub-sub in the browser.

With this, we’ve come full circle; commands come in, they get processed, leading to events being produced, which eventually go out to the browser.

With this, we provide a consistent web experience for users and for developers, while not having to jump through too much hoops to make it work.

I also thought of hosting communication with the devices in a Windows service while having that component expose its functionalities over HTTP so that the browser could talk to a local endpoint instead of being hosted in an application. While this is a valid alternative, it raised some concerns towards deployment in our scenario (we can’t push changes towards these clients, they need to come get them). With the existing set-up, I think even if we would like to change to such a model, it wouldn’t be that much trouble.

If you’ve pieced together a similar solution, feel free to let me know what I’m getting myself into.

Solving Mavericks with VMware Fusion 6 and Windows 8.1 hangs

Mon, 21 Apr 2014 12:07:00 +0200

Since I intended to avoid Windows at home, I got a Macbook Pro starting out at my new job. Overall it has been a great machine for doing development; it’s fast, light enough to carry around, its battery life is outstanding, it has a screen that’s gentle to the eyes, and full screen apps together with powerful mouse gestures allow me to quickly shuffle between things not missing touch or a second monitor.

Most of my professinal work is still on the Microsoft stack though, so I’m running a Windows 8.1 VM on VMWare Fusion 6. Much to my frustration, this setup would gradually slow down my system until it would completely grind to a halt every few hours. After complaining about it on Twitter, people said that having 8GB of RAM with half of that allocated to the VM might not be enough.

However after applying some tweaks, I got my system to chug away for a week without any hangs.

Here is what I changed:

Turn off App Nap for VMware
Install Memory Clean
Disable Windows visual effects (Advanced System Settings - Visual Effects)
Turn off Resharper Solution-Wide Analysis
Turn off Visual Studio rich client visual experience

Hope it helps.

Rebinding a knockout view model

Sun, 06 Apr 2014 18:43:00 +0200

As you might have noticed reading my last two posts, I have been doing a bit of front-end work using knockout.js. Here is something that had me scratching my head for a little while..

In one of our pages we’re subscribing to a specific event. As soon as that event arrives, we need to reinitialize the model that is bound to our container element. Going through snippets earlier, I remembered seeing the cleanNode function being used a few times - which I thought would remove all knockout data and event handlers from an element. I used this function to clean the element the view model was bound to, for then to reapply the bindings to that same element.

This seemed to work fine, until I used a foreach binding. If you look at the snippet below, what is the result you would expect?

<div id="books">
    <ul data-bind="foreach: booksImReading">
        <li data-bind="text: name"></li>
    </ul>
</div>

var bookModel = {
    booksImReading: [
        { name: "Effective Akka" }, 
        { name: "Node.js the Right Way" }]
};
                            
ko.applyBindings(bookModel, el);

var bookModel2 = {
    booksImReading: [
        { name: "SQL Performance Explained" },
        { name: "Code Connected" }]
};

ko.cleanNode(books);
ko.applyBindings(bookModel2, books);

Two list-items? One for “SQL Performance Explained” and one for “Code Connected”? That’s what I would expect too. The actual result shows two list-items for “SQL Performance Explained” and two for “Code Connected”

four in total. The cleanNode function is apparently not cleaning the foreach binding completely.

Looking for documentation on the cleanNode function, I couldn’t find any. What I did find was a year old Stackoverflow answer advising against using this function - since it’s intended for internal use only.

I ended up making the book model itself an observable. The element is now being bound to a parent model that contains my original book model as an observable. When the event arrives now, I create a new book model and set it to that observable property. This results in my list being rerendered with just two items - like expected.

<div id="books">
    <ul data-bind="foreach: bookModel().booksImReading">
        <li data-bind="text: name"></li>
    </ul>
</div>

var page = {
    bookModel : ko.observable({
        booksImReading: [
            { name: "Effective Akka" }, 
            { name: "Node.js the Right Way" }]
    })
};
                            
ko.applyBindings(page, el);

page.bookModel({
    booksImReading: [
        { name: "SQL Performance Explained" },
        { name: "Code Connected" }]
});

Don’t use the cleanNode function to rebind a model - instead make the model an observable too.

Sending commands from a knockout.js view model

Sun, 23 Mar 2014 18:31:00 +0100

While I got to use angular.js for a good while last year, I found myself returning to knockout.js for the current application I’m working on. Where angular.js is a heavy, intrusive, opinionated, but also very complete framework, knockout.js is a small and lightweight library giving you not much more than a dynamic model binder. So instead of blindly following the angular-way, I’ll have to introduce my own set of abstractions and plumbing again; I assume that I’ll end up with a lot less.

Let’s say that I have a view model for making a deposit.

var DepositViewModel = function() {
    var self = this;

    self.account = ko.observable('');
    self.amount = ko.observable(0);

    self.depositEnabled = ko.computed(function() {
        return self.account() !== '' && self.amount() > 0;
    });
    
    self.deposit = function() {
        if (!self.depositEnabled()) {
            throw new Error('Deposit should be enabled.');
        }

        $.ajax({ 
            url: '/Commands/Deposit', 
            data: { account: self.account(), amount: self.amount() }, 
            success: function() { self.amount(0); }
            type: 'POST', 
            dataType: 'json' 
        });
    };
};

ko.applyBindings(new DepositViewModel());

Writing a test for this, it was obvious that I couldn’t have my deposit function make requests directly. An abstraction that has served me well in the past, is a command executor.

CommandExecutor = function() {
    this.execute = function(command, success) { };
};

We can have an implementation that handles each command individually, or we can have it send requests to our server by convention. The implementation below assumes that the name of our command has a corresponding endpoint on the server.

CommandExecutor = function() {

    this.execute = function(command, success) {

        if (console) {
            console.log('Executing command..');
            console.log(command);
        }

        $.ajax({ 
            url: '/Commands/' + command.name, data: command.data, 
            success: success
            type: 'POST', dataType: 'json' 
        });

    };
};

While angular.js has dependency management built in, we can get away by injecting dependencies manually and a bit of bootstrapping - it’s not that I often have large dependency graphs in the browser, or that I care much about the life cycles of my components.

var DepositViewModel = function(dependencies) {
    var self = this;

    self.account = ko.observable('');
    self.amount = ko.observable(0);

    self.depositEnabled = ko.computed(function() {
        return self.account() !== '' && self.amount() > 0;
    });
    
    self.deposit = function() {
        if (!self.depositEnabled()) {
            throw new Error('Deposit should be enabled.');
        }

        var command = { 
            name: 'Deposit', 
            data: { account: self.account(), amount: self.amount() } };
        var callback = function() { self.amount(0); };
        dependencies.commandExecutor.execute(command, callback);
    };
};

ko.applyBindings(new DepositViewModel({ commandExecutor: new CommandExecutor() }));

See, very little magic required.

Writing a test, we now only need to replace the command executor with an implementation that will record commands instead of actually sending them to the server.

var CommandExecutorMock = function () {

    var commands = [];

    this.execute = function (command, success) {
        commands.push(command);
        success();
    };
    this.verifyCommandWasExecuted = function(command) {
        for (var i = 0; i < commands.length; i++) {
            if (JSON.stringify(commands[i]) === JSON.stringify(command)) {
                return true;                        
            }
        }
        return false;
    };

};

describe("When a deposit is invoked", function () {

    var commandExecutor = new CommandExecutorMock();
    
    var model = new DepositViewModel({ commandExecutor: commandExecutor });
    model.account('MyAccount');
    model.amount(100);
    model.deposit();

    it("a deposit command is sent.", function() {
        var command = {
            name: 'Deposit', 
            data: { account: 'MyAccount', amount: 100 }
        };

        expect(commandExecutor.verifyCommandWasExecuted(command)).toBe(true);
    });  

});

I did something similar for queries, and ended up with not that much code, that didn’t even take that long to write. I’m curious to see how this application will evolve.

Building a live dashboard with some knockout

Sun, 16 Mar 2014 21:23:00 +0100

Last week, we added a dashboard to our back office application that shows some actionable data about what’s going on in our system. Although we have infrastructure in place to push changes to the browser, it seemed more reasonable to have the browser fetch fresh data every few minutes.

We split the dashboard up in a few functional cohesive widgets. On the server, we built a view-optimized read model for each widget. On the client, we wrote a generic view model that would fetch the raw read models periodically.

var ajaxWidgetModel = function (options) {
    var self = this;

    self.data = ko.observable();
    self.tick = function () {
        $.get(options.url, function (data) {
            self.data(ko.mapping.fromJS(data));
        });
    };

    self.tick();
    setInterval(self.tick, options.interval);
};

We then used knockout.js to bind the view models to the widgets.

ko.applyBindings(
    new ajaxWidgetModel({ url: "/api/dashboard/tickets", interval: 30000 }), 
    document.getElementById('widget_tickets'));

<div class="widget-title">
    <h5>Tickets</h5>
</div>
<div class="widget-content" id="widget_tickets" data-bind="with: data">
    <table class="table">
        ...
    </table>
</div>

The with data-binding ensures that the content container only gets shown when the read model data has been fetched from the server.

Building dumb view-optimized read models on the server, binding them to a widget with one line of code, and some templating, allowed us to quickly build a live dashboard in a straightforward fashion.

Tests as part of your code

Sun, 09 Mar 2014 18:13:00 +0100

In the last project I worked on - processing financial batches - we put a lot of effort in avoiding being silently wrong. The practice that contributed most was being religious about avoiding structures to ever be in an invalid state. Preconditions, invariants, value objects and immutability were key.

One of the things we had to do with these structures was writing them to disk in a specific banking format; all the accounts with their transactions for a specific day. To verify the outcome of these functions, we had a decent test suite in place. But still, we felt like we had to do more; the person on the team that had been working in this domain for thirthy years had been relentlessy empathizing - nagging - that bugs here would be disastrous, and would have us end up in the newspaper. That’s when we decided to add postconditions, putting the tests closer to the production code. These would make sure we crashed hard, instead of silently producing something that was wrong.

To make sure we correctly wrote all transactions for one account to disk, we added a postcondition that looked something like this.

Ensure.That(txSumWrittenToDisk.Equals(account.Balance.Difference()));

A few weeks later, running very large batches in test, we had this assertion fail randomly. An account can have hundred thousands of transactions a day. This is why the account structure did not contain its transactions - there were too many to hold them in memory. To make sure an account and its transactions added up, we did do set validations earlier on - no faulty state there. Since the assertion would only fail randomly, and the function had no dependencies on time or mutable state, the only culprit could be data feeded into the function. Since all transactions for one account wouldn’t fit in memory, we were streaming them in pages from the database, and this is where we forgot to sort the whole result first, resulting in random pages - doh.

Without this postcondition, we probably would have ended up in the newspaper. While putting your code under test is super valuable, having some crucial assertions as integral part of your code might strengthen it even more(*).

* This concept is central to the Eiffel programming language.

Alternatives to Udi's domain events

Sun, 02 Mar 2014 18:21:00 +0100

Almost four years ago Udi Dahan introduced an elegant technique that allows you to have your domain model dispatch events without injecting a dispatcher into the model - keeping your model focused on the business at hand.

This works by having a static DomainEvents class which dispatches raised events.

This customer aggregate raises an event when a customer moves to a new address.

public class Customer
{
    private readonly string _id;
    private Address _address;
    private Name _name;

    public Customer(string id, Name name, Address address)
    {
        Guard.ForNullOrEmpty(id, "id");
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");

        _id = id;
        _name = name;
        _address = address;
    }

    public void Move(Address newAddress)
    {
        Guard.ForNull(newAddress, "newAddress");

        _address = newAddress;

        DomainEvents.Raise(new CustomerMoved(_id));
    }
}

By having a dispatcher implementation that records the events instead of dispatching them, we can test whether the aggregate raised the correct domain event.

var recordingDispatcher = new RecordingDispatcher();
DomainEvents.Dispatcher = recordingDispatcher;

var customer = new Customer(
    "customer/1",
    new Name("Jef", "Claes"),
    new Address("Main Street", "114B", "Antwerp", "2018"));
customer.Move(new Address("Baker Street", "89", "Antwerp", "2018"));

recordingDispatcher.Raised(new CustomerMoved("customer/1")); // true

While this worked out great for a good while, I bumped into difficulties scoping my unit of work and such when I redid some of my infrastructure. While there are ways to have your container address these issues, getting rid of the static components is simpler throughout.

A popular event sourcing pattern is to have your aggregate record events. There is no reason why we couldn’t apply the same pattern here. Using this technique, we still avoid having to inject something into our models, plus we get rid of that static DomainEvents component. Reponsibility of dispatching the events is now delegated to an upper layer.

public class Customer : IRecordEvents
{
    private readonly EventRecorder _recorder = new EventRecorder();

    private readonly string _id;
    private Address _address;
    private Name _name;

    public Customer(string id, Name name, Address address)
    {
        Guard.ForNullOrEmpty(id, "id");
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");

        _id = id;
        _name = name;
        _address = address;
    }

    public EventStream RecordedEvents() 
    {
        return _recorder.RecordedEvents();
    }

    public void Move(Address newAddress)
    {
        Guard.ForNull(newAddress, "newAddress");

        _address = newAddress;

        _recorder.Record(new CustomerMoved(_id));
    }
}

var customer = new Customer(
    "customer/1",
    new Name("Jef", "Claes"),
    new Address("Main Street", "114B", "Antwerp", "2018"));
customer.Move(new Address("Baker Street", "89", "Antwerp", "2018"));

customer.RecordedEvents().Contains(new CustomerMoved("customer/1")); // true

Another altnernative is to return events from your methods. This technique puts the responsibility of aggregating all events on to a higher layer. Better to put that closer to the aggregate.

What patterns are you using?

Strategic DDD in a nutshell

Sun, 23 Feb 2014 18:21:00 +0100

There are two big parts to Domain Driven Design; strategy and tactics. Strategy helps setting out a high-level grand design, while tactics enable us to execute on the strategy.

Practicing strategic design, we generally first try to list all of the different parts that make our business a whole; these are sub-domains. When you look at a supermarket chain, you would find sub-domains like real estate management, advertising, suppliers, stock, sales, human resources, finance, security and so on. Sub-domains will often relate to existing structures like departments and functions.

Once you’ve defined your sub-domains, it’s useful to determine how important of a role they play. First of all, you should figure out which sub-domain is most important to your business; the core domain. This is the sub-domain that differentiates you from other businesses, or more bluntly put; this is where the money is at. For our super market chain, this might not be that obvious for an outsider. A first uneducated guess would be sales, but if you gave it some more thought, you would realize that sales are very similar for most supermarkets. Digging deeper, we would find that supermarkets really compete with each other by squeezing the last bit of value out of suppliers and by collecting data to use for targeted advertising. Supplier management and advertising can’t stand on their own though; they need other sub-domains like stock and sales. These are supporting sub-domains; they are not core, but our business couldn’t do without them either - they still add a bunch of value. Other sub-domains like property management, human resources or security are generic sub-domains; these problems have been widely addressed and solving them yourself won’t make you any money.

Having a map of which areas are most important to your business makes it easy to distribute brain power accordingly. Make sure your core domain gets the most capable team assigned, before any other supporting sub-domain. Try to buy solutions of the shelve for generic sub-domains.

The concept of sub-domains lives in the problem space. The solution space on the other hand is where bounded contexts are at. Domain Driven Design tries to define natural boundaries between parts of your solution by putting the language first. These boundaries allow us to keep a language and model consistent inside of them, protecting conceptual integrity.
If you would ask a marketer what a product is, he would talk about images, campaigns, weekly promotions and so on. If you’d ask sales on the other hand, they would only mention price, quantity and loyalty points. The same concept can turn into something completely different depending on how you look at it. Bounded contexts enable us to build a ubiquitous understanding of concepts in a clearly defined context.

Mapping a bounded context to exactly one sub-domain would be DDD nirvana; addressing one problem with one focused solution. In the real world, things are more messy though. There will always be systems out of our control; for example legacy and third party software. As our understanding of the business grows, keeping our software aligned can be hard too. If we would lay out a map of sub-domains and bounded contexts we would see lots of overlap.

Bounded contexts will often be worthless on their own though; most useful systems exist of interconnected parts. If you have worked in the enterprise, you know how complex communication between teams and departments can be. This isn’t very different while integrating bounded contexts; you need to consider politics. This is where concepts like up-stream, down-stream, bandwidth, partnership, shared kernel, customer-supplier, conformist, anti-corruption layer etc come into play. The activity of thinking about and capturing how all these systems play together is called context mapping.
In our example, we notice that supplier- and stock management would fail or succeed together; they have a partnership where the bandwidth is very high - the teams sit across the hall from each other. Human resources and security on the other hand have a very different relationship. A product was bought for human resources, while a solution for security was outsourced. Security relies quite heavily on what the human resources’ open host service is exposing. If a product version bump changes those exposed contracts, security needs to comply as soon as possible; security is down-stream from human resources - shit floats down-stream.

For me, strategic DDD in one sentence, is the constant exercise of trying to see and understand your business at large, and aligning your software as efficiently as possible.

DDDBE slides on the Ubiquitous Language

Sun, 16 Feb 2014 18:04:00 +0100

Monday, I and four others did a DDDBE session on the strategic side of Domain Driven Design.

My talk covered the Ubiquitous Language, and can be found on Slideshare. I might end up writing down the content of the talk as well - some images are meaningless without words.

Evaluating feedback, I think the biggest mistake we made was keeping some things too abstract - curse of knowledge at work. If we get the chance to repeat the session, we need to make sure to weave a practical story through our talks to make them stick.

Reading an EventStore stream using JavaScript

Sun, 09 Feb 2014 18:07:00 +0100

Over Christmas break, I set out three days to play with the EventStore. One of the things I wanted to do was visualize the timeline of a stream in the browser. Since the EventStore exposes its event streams over atom in JSON, I could directly consume them from JavaScript.

An event stream can contain quite a few events. Since caching parts of that stream benefits all components in the system, the atom feed is split in multiple pages - where all full pages are cacheable. Thus if you want to read the entire event stream, you should work your way through all pages. What confused me at first, but what actually is quite logical, is that the last entry on the last page contains the first event. If you want to read the entire stream, you need to start at the last page, and work your way forward following the link to the previous page until there are no pages left to read.

I came up with something like this.

StreamFeedReader : function(feedUri) {   

    if (!feedUri) {
        throw new Error('feedUri missing.');
    }        
    
    var readLastFromHead = function (streamName) {                                        
        var dfd = $.Deferred();

        $.ajax({
            url : feedUri + streamName + '?embed=body'
        }).done(function (data) {                
            var lastLinks = data.links.filter(function(link) { 
                return link.relation === 'last'; 
            });                       

            if (lastLinks.length > 0) {               
                dfd.resolve(lastLinks[0].uri);           
            } else {
                dfd.resolve(feedUri + streamName);
            }
        }).fail(function() {                           
            dfd.reject();
        });

        return dfd.promise();
    };              

    var traverseToFirst = function (uri, entries, dfd) {                                                       
        $.ajax({
            url: uri + '?embed=body'
        }).done( function (data) {       
            var reversedEntries = data.entries.reverse();

            for (var i = 0; i < reversedEntries.length; i++) {
                entries.push(reversedEntries[i]);
            }            

            var previousLinks = data.links.filter(function(link) { 
                return link.relation === 'previous'; 
            });            

            if (previousLinks.length === 1) {
                traverseToFirst(previousLinks[0].uri, entries, dfd);
            } else {                
                dfd.resolve(entries);
            }           
        }).fail(function() {
            dfd.reject();
        });                    
    };  

    this.read = function (streamName) {                   
        if (!streamName) {
            throw new Error('streamName missing.');
        }  

        var dfd = $.Deferred();                           
        
        readLastFromHead(streamName).done(function(lastUri) {
            var entries = [];                        
            traverseToFirst(lastUri, entries, dfd);                        
        }).fail(function() { 
            dfd.reject(); 
        });    

        return dfd.promise();              
    };

}

First read the link to the last page. From there, read the entries on that page, look at the links on that page and start making your way forward, traversing the pages to the first one. All events on the page should also be reversed before they get pushed to the result.

Using this snippet, you can read a stream and have all events returned in the sequence they were appended.

new es.StreamFeedReader('http://127.0.0.1:2113/streams/')
.read('account-35')
    .fail(function() {
        test.ok(false, 'reading the stream failed.');
        test.done();
    })
    .done(function(res) {
        var streamContainsAllEvents = function() {
            test.equal(651, res.length, 'expecting 651 events in stream.');
        };
        var eventsInStreamAreOrdered = function() {
            var ordered = true;
            for (var i = 1; i < res.length; i++) {
                if (res[i - 1].eventNumber > res[i].eventNumber) {                            
                    ordered = false;
                }
            }
            test.ok(ordered, 'event numbers out of order.');
        };

        streamContainsAllEvents();
        eventsInStreamAreOrdered();                                

        test.done();
    });

This code is also available on GitHub.

Thinking in Systems

Sun, 02 Feb 2014 18:19:00 +0100

We are surrounded by systems day in, day out. As software developers, we even get to spend a big portion of our day actively building and changing systems - be it software, teams, communities or businesses. Seeing the whole, but more importantly understanding how systems exist of inter-related parts that affect each other in all kinds of interesting ways, is crucial to growing sustainable systems.

Thinking in Systems is a primer on the subject of system thinking written by Donella H. Meadows. The book is divided into three parts. The first part explains the basic concepts of system thinking using everyday systems. The second part covers why systems work so well, why they still surprise us, how to avoid common system traps and how to take advantage of some of the opportunities they present. The last part, and for me the most interesting part, shows where to intervene in a system and how to live in a world of systems.

Although the subject is pretty involved, the material is presented in a very consumable fashion, preparing you for more academic material. This is probably the book to read when you want to be introduced to systems thinking.

Here are the sections I highlighted.

Everyone or everything in a system can act dutifully and rationally, yet all these well-meaning actions too often add up to a perfectly terrible result.

You think that because you understand “one” that you must therefore understand “two” because one and one make two. But you forget that you must also understand “and”.

Purposes are deducted from behavior, not from rhetoric or stated goals.

Enough central control to achieve coordination toward the large-system goal, and enough autonomy to keep all subsystems flourishing, functioning and self-organizing.

If you become a manager, you probably will stop seeing labor as a deserving partner in production, and start seeing it as a cost to be minimized.

The structure of a commons systems makes selfish behavior much more convenient and profitable than behavior that is responsible to the whole community and to the future.

If the desired system state is good education, measuring that goal by the amount of money spent per student will ensure money spent per student. If the quality of education is measured by performance on standardized tests, the system will produce performance on standardized tests. Whether either of these measures is correlated with good education is at least worth thinking about.

There is a systematic tendency on the part of human beings to avoid accountability for their own decisions.

If you want to understand the deepest malfuctions of systems, pay attention to the rules and who has power over them.

So how do you change paradigms? You keep pointing at the anomalies and failures in the old paradigms. You keep speaking and acting, loudly and with assurance, from the new one. You insert people with the new paradigm in places of public visibility and power. You don’t waste time with reactionaries; rather, you work with active change agents and with the vast middle ground of people who are open-minded.

If no paradigm is right, you can choose whatever one will help to achieve your purpose.

There’s something within the human mind that is attracted to straight lines and not curves, to whole numbers and not fractions, to uniformity and not diversity, and to certainties and not mystery.

Repositories, where did we go wrong?

Sun, 26 Jan 2014 18:00:00 +0100

In essence, repositories are a simple abstraction over aggregate storage. A repository will insert, update, delete or fetch an aggregate from the underlying persistence mechanism. This abstraction avoids that databases, SQL statements, Object Mappers and the like leak into your domain. Next to that, swapping out repositories for an in-memory version makes testing easier.

Recently, the use of repositories is being questioned again.

Why would we wrap Object Mappers in yet another abstraction? Aren’t Object Mappers already an implementation of the repository pattern? In a recent project, we left out repositories. In that project we’re using RavenDB, which already has an expressive API, and which can be configured to use an in-memory database for testing. Even though LINQ and indexes help make simple queries expressive, a lot of cruft still leaks in, not doing the language any justice. In other projects, we did make use of repositories over our ORM. Partly because setting up in-memory tests without was awkward at best, but also because it removed constraints trying to capture the language. Next to testing and expressiveness, you should also consider how comfortable you feel gluing everything to a library or framework. When it comes to aggregate storage, having those repositories is a small price to pay to keep technicalities out.

Another remark is that a repository makes it hard to control eager- and lazy loading, which is contextual. In general I think that lazy loading introduces unpredictable behaviour. Getting in trouble without lazy loading is a strong indication that your aggregates are just too big.

The last and loudest argument is that once you have a view heavy application things get dirty really fast. It starts by adding a few badly named query methods on your repositories. Then, you start to see use cases where you need to query over multiple aggregates and deal with projections or aggregations. In these situations repositories won’t help you.
Truth is that repositories were never intended for complex reads. Views on the data that your application needs rarely resemble the structure of your aggregates. Making your aggregates suited for querying inevitably steers away from behaviour thinking, back to data thinking. The trick is to separate read concerns from your domain. Instead of trying to use repositories for querying, make use of the best tool for the job, something as close to the database as possible. The implementation depends on your flavor, but what has worked for me is having use case optimized read models, a query object and a query handler that reads from the database and converts the result into a read model. The implementation of each query handler can differ; from raw SQL, to hibernate query language, to a micro ORM… whatever works best really.
Doing this, you allow your domain model to stay focused on the task at hand - handling complex business problems, staying far away from read concerns. Before you know it you’re successfully applying that popular four letter acronym, enabling you to even try other concepts without having to rewrite your model completely.

new YearPassed(2013);

Sun, 19 Jan 2014 17:54:00 +0100

I normally write this up at the end of the year, but circumstances made me push back this post for a few weeks. I use this annual post to look back at the year passed, and to look ahead to the year to come. I gave up on making resolutions in 2011. This year, I’m going to be naively ambitious again going into 2014.

Compared to previous years, I feel more confident that I have a pretty good idea of what I want next. Just before the holidays, my fiancée and I spent the weekend out of town where we took a whole evening to modelstorm 2013 and the future. We used green and red post-its to list all positive and negative memories of 2013. We then categorized those, and defined concrete actions to avoid repeating negative experiences, and to increase the number of positive ones in 2014. Next to this technique, we used a timeline to plan some key milestones a few years ahead of time.

My career

September 2013 marked the two year anniversary working for Euricom. It also marked the second year at my first client. There, I spent last year deeply immersed working on what has been coined by upper management as the most important project of 2014. Having invested much, and despite having a bunch of ingredients for failure, I’m relieved the project went out the door yesterday. I look back at the project with mixed feelings; there were several periods of intense learning and going fast, but way too much time and energy was wasted fighting the system. After spending more than five years in the bowels of the beast, I don’t feel as if the enterprise is a healthy place for me to live in, let alone a place capable of cultivating good software.

I’m going to try really hard to avoid these breeds of working environments for a while. Unfortunately, they make up a good portion of Euricom’s customer base. This led me to resign at Euricom last month. Four weeks from now, I will start at a much smaller gig where I hope to find more autonomy, skin in the game and entrepreneurship. I’m also extremely curious to discover and learn about the insides of the domain.

Blog

I published a bit less compared to previous years, but I found a sustainable pace for writing: one post a week. Like I said earlier, traffic has stagnated, but the number of subscriptions and the average time spent reading has increased.

The topics I write about have drifted away from mostly technical ones to Domain Driven Design, modeling, personal experiences, conferences and book notes.

Community

Although I only spoke twice myself, I got to meet up with inspiring people more than once; the IDDD Tour, DDD Exchange, CQRSBeers, BuildStuff and most importantly the birth of the Belgian DDD Community.

Meeting up with likeminded souls, exchanging ideas, war stories and hopes, sharing food and beer is something that has been a source of true bliss. There are too many people to thank, but I’d like to name a few in particular: Mathias, Alberto, Christophe, Bart, Stijn, Yves, Tom, Marco and Thomas.

Next month, I’m attending FOSDEM and I’ll be talking at DDDBE. I also already bought tickets for buildstuff later this year.

Travel

I got spoiled this year; I got to visit six different countries, spending more than five weeks abroad:

England
Spain
Czech Republic
Lithuania
USA
Luxembourg

We’re still indecisive about whether we should visit Asia or something closer this summer. Anyone that can recommend locations in Asia?

Projects

I killed one side project at the start of this year, and documented it here. I spent a big part of this year working on another project. While I’m not sure if we will get that ever of the ground, I learned a few lessons though. Contact your potential customers before you build a close to finished product. If you’re going to work on something in the weekends, don’t make it feel like work.

Sports/Health

The fiancée and I have built a habit of regularly doing long hikes - once every month. This is something we’re going to keep up, maybe even put it in the center of one of our holidays.

I ran 705km this year, which is 270km less than last year. After running three competitions early on in the year, I lost my hunger, and started slacking. To keep the momentum going and to have something to work towards, I should probably enroll in a competition every few months. I already subscribed for the Antwerp Urban Trail run and the Antwerp 10 miles this year. Anyone else subscribing? I haven’t found a partner yet.

I also didn’t cancel my gym subscription this year. I have put together a satisfying routine that mostly consists of lifting heavy things avoiding modern equipment. It’s been fairly successful, bulking up 9kg the last six months.

2014 side missions

Avoid Windows at home.
Learn a new (programming) language.
Build something just for me.
Spend less time in front of my machine.
Read more. Watch more conference videos.
Do nothing. It heals the soul.
Plan the weekend.
More dinner parties.
Plan the wedding.
Take more guitar lessons.

Thanks for everything, I’m very grateful.

Happiness before success

Sun, 12 Jan 2014 18:57:00 +0100

Somewhere in the beginning of last century, two shoes salesmen were sent to Africa hoping to expand their employer’s market. Both salesmen reported home within days of their arrival. The first salesman wrote: “This trip turned out to be a waste of time; the locals are not wearing any shoes.” The second salesman wrote something similar, yet very different: “This is looking very promising; people aren’t wearing any shoes yet!”

A group of seventy year olds was invited to spend a week in a remote location where everything was arranged to make it feel as if they went back in time twenty years; they would get to read old news papers, were shown recorded television shows etc… They were also asked to talk about their jobs and kids as if it were twenty years ago. Once the week was over, medical exams showed extraordinary results. Their health had improved noticeably; their eyesight improved, their blood pressure went down, their posture had changed…

In another experiment, a group of people was trained into using a new piece of software. Half of the class was taught to prevent errors from happening, while the other half was encouraged to make mistakes. A test afterwards showed that the group that had learned by making mistakes was a lot faster and efficient using the software than the group that was taught to avoid errors at all costs.

The mind is incredibly powerful; it’s in full control of how we perceive things. One might be poor, having to live off a few dollars a day, and be perfectly content with his situation. While someone on the other side of the world might have everything he can imagine at his fingertips, but be completely miserable.

Popular belief dictates that if you work hard, you will eventually become successful, and once you’re successful, then you’ll be happy. This recipe is elementally flawed. With each victory, we put our eyes on that next big thing, never arriving at happiness.

In The Happiness Advantage, Shawn Achor starts by claiming that we have it all wrong, and that it’s actually the other way round; happiness leads to success. Positive brains make us more efficient, creative, thoughtful and productive, boosting performance at work and at home. While the first few pages make a great sales pitch, the rest of the book differs from the usual life improvement charlatans backing up this hypothesis with narratives and DIY practices extracted from modern happiness research.

The book turned out to be a very casual read; the writing is easy and it only counts two hundred small pages - I finished it in a few hours. It has inspired me to try and be more conscious about how I deal with certain things, to always focus on the good, not allowing negativity to slowly poison me.

Quotes from the book

If all you strive is diminishing the bad, you’ll only attain the average and you’ll miss out entirely on the opportunity to exceed the average.

Happiness is not just a mood - it’s a work ethic.

It takes about three positive comments, experiences, or expressions to fend off the languishing effects of one negative.

Happiness is not about lying to ourselves, or turning a blind eye to the negative, but about adjusting our brain so that we see the ways to rise above our circumstances.
The fastest way to disengage an employee is to tell him his work is meaningful only because of the paycheck.

Constantly scanning the world for the negative comes with a great cost. It undercuts our creativity, raises our stress levels, and lowers our motivation and ability to accomplish goals.

Common sense is not common action.

Couldn’t the key to sustaining positive change be to turn each desired action into a habit, so that it would come automatically, without much effort, thought, or choice?

This invisible pull toward the path of least resistance can dictate more of our lives than we realize, creating an impassible barrier to change and positive growth.

Lower the activation energy for habits you want to adopt, and raise it for habits you want to avoid.

Command and event semantics

Sun, 05 Jan 2014 18:12:00 +0100

Yesterday, I read this blog post by Michael Feathers. In the post he goes over a pain point he has often found himself struggling with while breaking down a large method; conditional statements.

if (alarmEnabled) {
    var alarm = new Alarm();  
    ...
    alarm.Sound();
}

Should we extract the if and the associated block into a new method, or just the content of the block? Is the condition too important to hide in a method? How would we name the extracted method? How do we avoid the code from telling us lies?

To read up on all the nuances that the answers to these questions bring, you should read the full post. What’s important for this post - spoiler alert - is that he ends up with two strategies.

He either gives the method an event-ish name…

IntruderDetected();

…or raises the level of abstraction by giving the method a very general name, avoiding lies too.

PerformNotifications();

While both options solve the original problem, their semantics are very different. When I raise an event, I don’t care who is listening; one, multiple or no things might be listening. When I issue a command (the second strategy), I expect it to go to exactly one destination. Sending commands indicates a rather strong dependency; you expect something to happen because of it. Events are more loosely coupled; you broadcast something happened, and something might or might not happen because of it.

With this in mind, you can still make arguments for both options. The key here is whether notifications are considered to be an essential component of intrusion detection. Is intrusion detection still conceptually whole without notifications? Are notifications just a side effect of an intruder being detected?
Since the alarm is only sounded when it’s enabled, it doesn’t seem to be an indispensable part. Intrusion detection can live on its own, unaware of notifications. This makes a pretty strong case for having the detection component just raise an event - indirectly resulting in the notifications being sent.

In this example, I’d probably pull notifications out, add an event and a bit of infrastructure that dispatches events, making the separate concepts and messages between them explicit.

Events.Raise(new IntruderDetected());

public class NotificationService : IHandle<IntruderDetected>
{
    public void Handle(IntruderDetected event)    { ... }        
}

With those extra bits, we could also, instead of listen for the event and only sounding the alarm when it’s enabled, only subscribe to the event when the alarm is enabled.

2013's most read posts

Sun, 29 Dec 2013 17:05:00 +0100

This time of year, we’re all very busy honoring traditions. This post is no exception (2009, 2010, 2011, 2012).

Today was probably the first time this year that I took some time to study my blog’s Google Analytics metrics. If I keep last year’s black swan posts out of the equation, traffic seems to be comparable to last year’s. The most meaningful metrics to me are the number of subscriptions and the average time spent reading. Both of them increased - happy about that!

The actor model has been gaining in popularity more and more. Next to addressing more infrastructural concerns, it can also be used as a framework for modeling and reasoning about complex systems. Working with a team of mainframe programmers over the last year, I observed a few similarities in how they have been designing their systems. Actor Model in COBOL is the third most read post of 2013.

The second most read post was Not handling edge cases, making them explicit instead. Inspired by a talk with Greg Young at DDDX, I tried to create a narrative that uncovered an edge case in a green field project. Instead of handling the edge case, we use an event to make it explicit, allowing a human to intervene. This way we can go to market more quickly, with less code, and we might even end up with happier customers.

Number one is But I already wrote it. In this post I shared my motivation having an argument with a colleague about whether to dump some code that did too much. Don’t cherish your code. It’s nothing but a means to an end; the side product of creating a solution. Aim for simple and lean solutions; nobody likes bulky software, nobody likes fighting complexity all day. Don’t neglect the hidden cost of that extra small feature you’re throwing in.

Thank you for reading!

Databases are growing on me

Sun, 22 Dec 2013 21:53:00 +0100

I learned all about logical design of relational databases back in school; tables, columns, data types, views, normalization, constraints, primary keys, foreign keys… At the same time, I learned how to use SQL to put data in, and how to get it out again; INSERT INTO, SELECT, FROM, WHERE, JOIN, GROUP...

In the first project I worked on just out of school, we weren’t doing anything interesting with databases; we didn’t have that many users, or that much data. A database veteran on the team took it on him to maintain the schema and to provide stored procedures we could do work with.

All that time, I consciously was very ignorant of the database. I had no idea what was in the box, and I didn’t care either; databases were boring, applications were where the fun was at.

Since then, I have rarely worked with a team that had a dedicated role for database design. Why invest in another person when you can do without? Database basics are not rocket science; with 20% of the knowledge, you get very far. Definitely now that it’s probably easier and cheaper to throw hardware at the problem.

That being said, it’s a good idea to keep a DBA close. Time and time again I see them only being called in when it’s too late and much needed improvements are often too far-reaching and expensive. No wonder DBA’s are grumpy all the time.

Being exposed to databases more and more, I got to pick up a few things here and there - mostly cargo-cult best practices. It wasn’t until last year that I got really curious for what was in the box. Working on an application with a decent amount of data crunching for a year forced me to open up the lid. Also my ventures in NoSQL land, overhearing discussions on Twitter between kellabyte, ayende, gregyoung, pbailis and others had much to do with it.

On opening the lid, I found a lot more than I expected. It had never occurred to me how much interesting problems databases have to solve. Making a database execute a query and see results returned in milliseconds only looks easy on the surface. Memory, disk, CPU, caching, networking, protocols, concurrency, fault tolerance, data structures, transactions, compilation… it’s all in there.

The book Physical Database Design and the SQLite technical documentation were the first good reads that helped me understand what was going on closer to the metal. From there, I now try reading a paper (or a reference) from the Readings in Database Systems collection once in a while. This collection of papers is supposed to contain the most important papers in database research. Maybe academic, but delicious brain food nonetheless - stretching my mind in ways I’m not used to.

Buildstuff 2013

Sun, 15 Dec 2013 22:03:00 +0100

Last night, I returned from Vilnius, after seven intensive days of Buildstuff. After a few long months, I was looking forward to be influenced and inspired by concepts and experiences strange to my day-to-day job - surrounded by people hungry to build better software. Because of the diversity of the program, the high level of the speakers and the presence of some familiar DDD-community faces, my expectations were easily met. Also, the location and low rates for tickets, transport and beer make it a very reasonable investment.

I expect the videos to be uploaded over the next few weeks and thus I will not bore you with session summaries. Instead I picked out some quotes which stood out for me in one way or the other; because they were spot on, summarize a concept in just a few words, or because they challenge me and give me food for thought.

Hope to see you again next year, Buildstuff.

Bertrand Meyer

Code contracts help generating tests. Pre-condition failed? Not a useful test. Post -condition failed? We found a bug.

Documenting assumptions outside of code lead to rapid unscheduled disassembly of the Arianne 5.

Why separate tests from your program?

RBob Ashton

If I have to download the internet to use your module, I’m not going to use it.

Ian Cooper

The test is the unit, not the object under test.

If you use tests to drive the implementation that couple to the inside, remove them once you’re finished.

Devs often find outside-in testing too hard because they don’t master the good setup techniques.

Joe Armstrong

A program is the most precise description of the problem that we have.

Paper systems are far more fault tolerant than what we’re replacing them with.

Failure should not be handled in each piece of code but let it be handled externally.

Jonas Bonér

Isolate the failure, compartmentalize, manage failures locally, avoid cascading failure.

Synchronous RPC, distributed transactions and shared mutable state are the graveyard of distributed systems.

Synchronous RPC give you a very high abstraction that ignores all problems.

It’s all about message passing; let that be the programming model.

Simon Brown

You can have micro-services without them being different physical artifacts. Pull them out when it pays off.

Package by component instead of by layer. Components can have layering too.

If your code doesn’t match the architecture, it becomes very hard to do architectural refactoring.

Good architecture enables agility.

Torben Hoffmann

Technical arguments are never enough, you need economical ones.

You need to experience pain to have the willingness to change.

Aras Pranckevicius

Better hardware and software is the cheapest way to get more work done.

Stupidity adds up, intelligence multiplies.

Guidelines, not rules.

Alberto Brandolini

Watching the ceiling while making decisions is forbidden; visualize.

Time-box decisions. Be happy with a decision, although it’s probably not the best solution.

The easiest way to remove crap from your system is not to put crap in.

Greg Young

An append-only model can be cached forever.

Wrong models cause massive accidental complexity.

Pieter Hintjens

You are defined by what you say online, not by who you are.

You can’t avoid getting your information stolen, but we can make it extremely expensive.

As information centralizes, it gets easier to spy on us.

Book review: Antifragile

Sun, 08 Dec 2013 17:30:00 +0100

When things are fragile, they break easily. We often see fragility as a bad thing and design things to be robust. But this isn’t what we’re really after either; things that are robust might be hard to break, but they’re also hard to change, making them fail to adapt to new stressors over time. The model that we’re really after is antifragility; when something is antifragile it will benefit from stressors and get better over time.

A cup is fragile; drop it and it breaks. A skyscraper on the other hand is robust; it is designed to resist the forces of nature; from hurricanes to earthquakes. A perfect example of antifragility is the human body; go to the gym, work your ass off, and you will grow stronger.

In this book, Taleb takes this model and applies it to a wide variety of systems; biological, medical, economic and political. Most concepts are easily relatable to software too.

While the core concepts could be summarized in a few pages, Taleb uses anecdotes, ancient texts, narratives and formulas to prove his point - resulting in a 519 pages thick book. Not all passages are a smooth read though. Some parts read as if they were written in one sitting, dumping everything that had been building for years on paper, without being proofread after. Mixing that with Taleb’s extremely rich (= hard) vocabulary makes for reading sessions where your full concentration is a must - not suitable for after work commutes. It took me six weeks to finish the book - well worth it though.

There are a lot of things that stuck with me. Instead of sharing those in my own words, I revisited some quotes I highlighted and copied them below. I hope they give a better feel of what to expect from the book.

Remember that you need a name for the color blue when you build a narrative, but not in action - the thinker lacking a word for “blue” is handicapped; not the doer.

It would have taken a bit of heroic courage to justify inaction in a democracy where the incentive is to always promise a better promise than the other guy, regardless of the actual, delayed cost.

It’s much easier to sell “Look what I did for you” than “Look what I avoided for you.” Of course a bonus system based on “performance” exacerbates the problem.

The benefits of procrastination apply similarly to medical procedures: we saw that procrastination protects you from error as it gives nature a chance to do its job, given the inconvenient fact that nature is less error-prone than scientists.

People who build their strength using these modern expensive gym machines can lift extremely large weights, show great numbers and develop impressive looking muscles, but fail to lift a stone; they get completely hammered in a street fight by someone trained in more disorderly settings.

In project management, Bent Flyvbjerg has shown firm evidence that an increase in the size of projects maps to poor outcomes and higher and higher costs of delays as a proportion of the total budget. But there is a nuance: it is the size per segment of the project that matters, not the entire project.

What survives must be good at serving some purpose that time can see but our eyes and logical faculties can’t capture.

We notice what varies and changes more than what plays a large role but doesn’t change. We rely more on water than on cell phones but because water does not change and cell phones do, we are prone to thinking that cell phones play a larger role than they do.

Corporations that are large today should be gone, as they have always been weakened by what they think is their strength: size.

There are secrets to our world that only practice can reveal, and no opinion or analysis will ever capture in full.

Recall that under nonlinearities, the simple statements “harmful” or “beneficial” break down: it is all in the dosage.

When you think you have found a free lunch, say, steroids or trans fat, something that helps the healthy without visible downside, it is most likely that there is a concealed trap somewhere. Actually, my days in trading, it was called a “sucker’s trade.”

What made medicine mislead people for so long is that its successes were prominently displayed, and its mistakes literally buried - just like so many other interesting stories in the cemetery of history.

We are built to be dupes of theories. But theories come and go; experience stays.

The English went further and had the families of the engineers spend time with them under the bridge after it was built.

In the old days, privilege came with obligations. You want war? First in battle.
Never ask anyone for their opinion, forecast, or recommendation. Just ask them what they have - or don’t - in their portfolio.

Corporate managers have incentives without disincentives - something the general public doesn’t quite get, as they have the illusion that managers are properly “incentivized.”

Third layer, the even more serious violation: companies trying to misrepresent the product they sell by playing with our cognitive biases, our unconscious associations, and that’s sneaky. The latter is done by, say, showing a poetic picture of a sunset with a cowboy smoking and forcing an association between great romantic moments and some given product that, logically, has no possible connection to it. You seek a romantic moment and what you get is cancer.

Don’t be fooled by money. These are just numbers. Being self-owned is a state of mind.

Antifragile by Nassim Nicholas Taleb.

Observations over assumptions

Sun, 24 Nov 2013 18:08:00 +0100

I heard a story once about an engineer who worked on the Disneyland site when it opened in Anaheim, CA in 1955.

A month before the park opened, the new grass and sod were being applied to the grounds as one of the last items to be completed before the big grand opening. The parking lot was some distance from the park’s gates and required a lot of turf. However, the sidewalks had not been planned or constructed to accommodate patterns.

Before engineers would validate the proper placement of the sidewalks, a heated internal discussion grew among landscape designers and park developers over how and what to build. One engineer suggested they allow visitors to walk on the grass for months in order to observe the paths they created themselves. Then they would build over those paths that showed the most foot traffic.

Those sidewalks are still there today.

One engineer’s focus on meeting the guests’ needs saved the park millions of dollars’ worth of error and political positioning.

I found this story on Quora a few weeks ago, and thought it was a testament to how observing - instead of assuming - can save you a lot of effort, and will end up serving the users best.

In the first product I helped building, there had been lots and lots of requirements gathering before we got to build the first functionality. Once we rolled out the pieces to the first set of users, they were disappointed to say the least - enraged actually; “This not usable at all! Do you even know what you’re doing?”. Apparently requirements were made up by people higher in rank, without consulting those who would have to use the software on a day-to-day basis. Not the best move getting buy-in from actual users, but I can’t really blame them either; requirements are hard, often times you’re just making stuff up as you go along - with best intentions. Being in a crisis, we got to learn a lot the next few months. One of us got sent out to the customer location a few times, and got to observe how they were trying to use our software. That information proved to be invaluable, and gave us enough to start shipping better and more useful software. Users felt empowered by our software, eventually leading to earning their trust and approval.

Another example can be found in my current project - which is not that visible to users, but more behind the scenes. Instead of guessing performance targets, it’s by observing production metrics that we’re able to set realistic goals.

Observing instead of assuming usually leads to better results. Keep in mind that you can misinterpret what you’re observing too, and that there often is no other option than to assume. You can still avoid overly expensive mistakes, by validating these assumptions as early as possible.

Event storming workshop slides

Sun, 17 Nov 2013 19:56:00 +0100

At Euricom, we quarterly all retreat to headquarters for a day of sharing and learning. This time, I and others organized and facilitated an event storming workshop.

After a short introduction on event storming participants were initiated to the domain of Cambio CarSharing - which is packed with behaviour. After that, seven groups of five (+ one domain expert) spread out across the office, and spent two slots of twenty minutes modeling the domain - with two extra slots for feedback.

Even after an afternoon of taxing sessions, people were willing to tap out of their energy reserves, and ended up presenting great results.

You can find the slides (heavily based on Alberto’s material) I used here.

If you’re interested in running your first event storming workshop, I’d love to come over and help you get started.

An event store with optimistic concurrency

Sun, 10 Nov 2013 18:25:00 +0100

Like I mentioned last week - after only five posts on the subject - there still are a great deal of event sourcing nuances left to be discovered.

My current event store implementation only supports a single user. Due to an aggressive file lock, concurrently accessing an aggregate will throw an exception. Can we allow multiple users to write to and read from an event stream? Also, what can we do about users making changes to the same aggregate; can we somehow detect conflicts and avoid changes to be committed?

Multi-user

In the current version, concurrently appending to or reading from an aggregate’s event stream will throw since the file will already be locked.

Parallel.For(0, 1000, (i) =>
{    
    _eventStore.CreateOrAppend(aggregateId, new EventStream(new List<IEvent>() 
    { 
        new ConcurrencyTestEvent() 
    }));
    _eventStore.GetStream(aggregateId);    
});

The exception looks like this: "System.IO.IOException: The process cannot access the file 'C:\\EventStore\\92f42a08-8583-4dcf-98a5-440b06f34719.txt' because it is being used by another process."

To prevent concurrent file access, we can lock code accessing the aggregate’s event stream. Instead of using a global lock, we maintain a dictionary of lock objects; one lock object per aggregate.

lock (Lock.For(aggregateId))
{
    using (var stream = new FileStream(
        path, FileMode.Append, FileAccess.Write, FileShare.Read))
    {
        // Access the aggregate's event stream
    }
}

public class Lock
{
    private static ConcurrentDictionary<Guid, object> _locks = 
        new ConcurrentDictionary<Guid, object>();

    public static object For(Guid aggregateId)
    {
        var aggregateLock = _locks.GetOrAdd(aggregateId, new object());

        return aggregateLock;
    }
}

Optimistic concurrency

Before committing changes, we want to verify that no other changes have been committed in the meanwhile. These changes could have influenced the behaviour of our aggregate significantly. Appending the last changes without considering what might have happened in the meanwhile might corrupt our aggregate’s state.

One way to verify this is by using a number (or a timestamp - clocks, bah) to keep track of an aggregate’s version. It’s up to the client to tell us which version he expects when appending to a stream. To accommodate for this, we need to change the contract of our event store.

public interface IEventStore
{
    void Create(Guid aggregateId, EventStream eventStream);

    void Append(Guid aggregateId, EventStream eventStream, int expectedVersion);

    ReadEventStream GetStream(Guid aggregateId);
}

Clients now need to pass in the expected version when appending to a stream. The result of reading a stream will include the current version.

In the event store, we now store an index with every event.

If we append to an event stream, we will get the current version by reading the highest index - storing this in aggregate meta data would be faster for reading. If the current version doesn’t match the expected version, we throw an exception.

var currentVersion = GetCurrentVersion(path);

if (currentVersion != expectedVersion)
    throw new OptimisticConcurrencyException(expectedVersion, currentVersion);

using (var stream = new FileStream(
    path, FileMode.Append, FileAccess.Write, FileShare.Read))
{
    using (var streamWriter = new StreamWriter(stream))
    {
        foreach (var @event in eventStream)
        {
            currentVersion++;

            streamWriter.WriteLine(new Record(
                aggregateId, @event, currentVersion).Serialized());
        }
    }
}

A test for that looks something like this.

try
{
    GivenEventStore();
    GivenAggregateId();
    GivenEventStreamCreated();
    WhenAppendingTwoEventStreamsWithTheSameExpectedVersion();
}
catch (OptimisticConcurrencyException ocex) 
{
    _expectedConcurrencyException = ocex;
}

[TestMethod]
public void ThenTheConcurrencyExceptionHasANiceMessage()
{
    var expected = "Version found: 3, expected: 1";
    var actual = _expectedConcurrencyException.Message

    Assert.AreEqual(expected, actual);
}

Reading the event stream doesn’t change much; we now also read the current version, and return it with the event stream.

var lines = File.ReadAllLines(path);

if (lines.Any())
{
    var records = lines.Select(x => Record.Deserialize(x, _assembly));
    var currentVersion = records.Max(x => x.Version);
    var events = records.Select(x => x.Event).ToList();

    return new ReadEventStream(events, currentVersion);
}

return null;

And that’s one way to implement optimistic concurrency. The biggest bottleneck in this approach is how we read the current version; having to read all the events to find the current version isn’t very efficient.

Transactional behaviour is also missing. I’ve been thinking about adding a COMMIT flag after appending a set of events, and using that to resolve corruption on reads, or is this fundamentally flawed?

Event source all the things?

Sun, 03 Nov 2013 18:45:00 +0100

Having covered projections last week, I think I have come full circle in these posts that turned out to be a small preliminary series on event sourcing. Even though there are still a vast amount of nuances to discover, I think I’ve captured the gist of it. Even without running an event sourced system in production - I feel as if I somewhat have an idea of what event sourcing can bring to the table.

Event sourcing gives you a complete history of events that caused an aggregate to be in its current state. In some scenarios this will add an enormous amount of value, in other scenarios it will give you nothing - it might even steal time and effort.

The first thing you do - before even considering implementing event sourcing - is talking to your business. Do they feel as if events are a natural way to represent what’s going on in their domain? Event sourcing is a lot more than just a technical implementation detail, discovering and understanding all of what goes on in a domain is a big investment - from both sides. Is it worth the trouble?

In my first job I worked on software for fire departments. I just now realize in how many bits of our solution event sourcing could have helped us:

the life cycle of a vehicle assigned to an emergency: vehicle dispatched, vehicle left the station, vehicle en route, vehicle arrived on the scene, vehicle back in the station…
a person’s career: person was promoted, person was detached to another station, person learned a new skill…
a shift’s schedule: person attached to unit, person returned to person pool, unit dispatched…

This data had to be made available in a set of diverse read models. Getting the data out was complex at times, often even impossible. A lot of these changes had to be propagated to external systems; there was no way to get that info out in real-time, and external systems had no notion of what happened.

In one of the functionalities of a system I’m currently working on, users also wanted to know what happened in the past, but for completely different reasons. Being in a financial context, they wanted to know who was responsible for changing system settings. Here it’s not an event log they need, but a simple audit trail.

If it is just a passive log your business wants, you can get away with cheaper alternatives; a command journal, an audit trail and so on.

Benefits

Event sourcing goes hand-in-hand with Domain Driven Design. Events are a great tool to go from a structural model to a behavioural model, helping you to capture the true essence of a domain model.

Building and maintaining an event store should be doable. It’s an append-only data model, storing serialized DTO’s with some meta data. This makes - compared to ORM’s and relational databases - tooling easier as well.

In traditional systems, you have to keep a lot of things in your head at once; how do I write my data, but also how do I query my data, and more importantly how do I get my data out in all these different use cases without making things too hard. In event sourced systems, separating writes from reads makes for more granular bits, easing the cognitive load.

Events can be projected into anything: a relational database, a document store, memory, files… This allows you to build a read model for each separate use case, while also giving you a lot of freedom in how you’re going to persist them.

You can replay projections, rebuilding a read model from scratch. Forget about difficult data migrations.

Testing feels consistent and very complete. A test will assert if all the expected events were raised, but will also implicitly assert that unexpected events were not raised. Testing projections is also straight-forward.

Events provide a natural way of integrating with other systems. Committed events can be published to external subscribers.

Troubleshooting becomes easier since a developer can copy an event stream from production, and replay it locally - reproducing the exact issue without jumping through hoops getting the system in a specific state.

Instead of patching corrupted production data directly, you can send a compensating event or fix the projection and replay everything. This way nothing gets lost, and consistency between code and outcome is guaranteed.

Downsides

Defining events is hard. Defining good events takes a lot of practice and insight. If you’re forcing a structural model into a behavioural one, it might even be impossible. So don’t even consider turning CRUD into an event sourced model.

There are a few places you need to be on the look out for performance bottlenecks. Event streams of long lived aggregates might grow very big. Loading a giant event stream from a data store might take a while - snapshots can help here. Projecting giant event streams might get you into trouble too - how long will it take to rebuild your read model, will it even fit into memory? Making projections immediate consistent might become a problem if you do a lot of them. Parallelization or giving up on immediate consistency might bring solace.

Events don’t change, versioning might get awkward. Are you going to create a new event type for each change, or will you relax deserialization? Or maybe you want to implement event migrations?

Since you’re persisting multiple models; events and one or more read models, you’re going to consume more storage, which will cost you.

Adaptation in the wild

Although there are - from a a business and engineering perspective - some good arguments to be made for event sourcing, those arguments only apply to a modest percentage of projects. Even when there’s a strong case to be made for event sourcing, there are very few people with actual experience implementing an event sourced system and prescriptive frameworks that you can just drop into a project and feel good about, are lacking. Most won’t even care about event sourcing to start with, but even if they do, it’s a fight upstream; it introduces a risk most might not be comfortable with.

Having said that, there are some really good projects out there that are steadily gaining popularity and maturity. Pioneers in the field are sharing and documenting their experiences, lowering the barriers for others. Things are moving for sure.

As always, event sourcing is not a paradigm to blindly apply to each and every scenario, but definitely one worth considering.

Since I’m not running any of it in production, tell me what I’m missing, there must be more things that turn out to be harder than they sound at first right? If you’re not running it in production, but thinking about it, what are some of your concerns? What are your predictions for the future of event sourcing?

Event projections

Sun, 27 Oct 2013 17:43:00 +0100

In my first two posts on event sourcing, I implemented an event sourced aggregate from scratch. After being able to have an aggregate record and play events, I looked at persisting them in an event store. Logically, the next question is: how do I query my aggregates, how do I get my state out?

In traditional systems, write and read models are not separated, they are one and the same. Event sourced systems on the other hand have a write model - event streams, and a separate read model. The read model is built from events committed to the write model; events are projected into one or more read models.

An interface for a projection could look like this.

public interface IProjection {
    void Handle(EventStream eventStream);                     
}

A projection takes in an event stream, and projects it to some read model.

A read model can be anything; a cache, a document store, a key value store, a relational database, a file, or even some evil global state.

public class EvilStatisticsReadModel {
    public static int WithdrawalAmountExceededCount { get; set; }
    public static int AmountDepositedCount { get; set; }
}

In this model, we want to maintain statistics of events that happened. For that to happen, we need to define a projection of our event stream.

public class ProjectionsToEvilStaticsReadModel : IProjection {
    public void Handle(EventStream eventStream) {
        foreach (var @event in eventStream)
            When((dynamic)@event);
    }

    public void When(WithdrawalAmountExceeded @event) {
        EvilStatisticsReadModel.WithdrawalAmountExceededCount++;
    }

    public void When(AmountDeposited @event) {
        EvilStatisticsReadModel.AmountDepositedCount++;
    }    
}

If we now let this projection handle an event stream, our read model will be kept up-to-date.

[TestMethod]
public void ReadModelIsKeptUpToDateWhileProjectingTheEventStream() {
    var events = new List<IEvent>() {
        new WithdrawalAmountExceeded(new Amount(3000)),
        new AmountDeposited(new Amount(300)),
        new AmountDeposited(new Amount(500)),
        new AmountWithdrawn(new Amount(100))
    };
    var stream = new EventStream(events);

    new ProjectionsToEvilStaticsReadModel().Handle(stream);

    Assert.AreEqual(1, EvilStatisticsReadModel.WithdrawalAmountExceededCount);
    Assert.AreEqual(2, EvilStatisticsReadModel.AmountDepositedCount);    
}

One could argue that all of this is too much - not worth the effort. Where you first just persisted the structure of an aggregate, and could query that same structure, you now first have to persist events for then to write projections that maintain separate read models that can be queried.

You have to look beyond that though. Those that have done any serious work on a traditional stack have felt the pain of migrations, complex queries that take up three pages, obscure stored procedures that run for hours, optimizing while having to consider a handful of different use cases, finding the balance between write- and read performance, database servers that can’t handle the load on special events, expensive licenses and so on. While these first few concerns are mostly technical, personally I’m often overwhelmed by how much concepts these designs force you to keep in your head all at once.

Separating reads from writes using event sourcing might bring some relief. Reducing cognitive overload by separating responsibilities into smaller, more granular bits might be the only argument you need. However, there’s a lot more. Running an event store should be low-maintenance; it’s an append-only data model storing simple serialized DTO’s with some meta data - forget about big migrations (not completely though), schemas, indexes and so on. Even if you project into a relational database, being able to re-run projections should make migration scripts and versioning avoidable. An event can be projected into multiple read models, allowing you to optimize per use case, without having to take other use cases into account. Since it should be easy to rebuild read models, they can be stored in cheap and volatile storage - think key-value store, in-memory and so on, allowing for crazy fast reads.

Letting go of the single-model dogma seems to enable so much more, giving you a whole new set of possibilities. Another extremely useful use case that suddenly becomes a lot easier to support is business intelligence; when business experts think of new ways to look at the past, you just create a new projection and project events from day one. Getting statistics of how your users are using your system doesn’t sound that hard now, does it?

One of the obvious drawbacks next to writing a bit more, boring code is that storage costs will increase - you are now persisting the same data in multiple representations. But storage is cheap, right? Maybe money isn’t an issue, but what about performance? It’s slower to do three writes instead of one, right? For a lot of scenarios this won’t be much of an issue, but if it is, there is a lot of room for optimiziations doing projections; parallelization, eventual consistency and so on.

Next week: event source all the things?

An event store

Sun, 20 Oct 2013 17:30:00 +0200

Last week, I implemented an event sourced aggregate from scratch. There I learned, that there isn’t much to a naively implemented event sourced aggregate; it should be able to initialize itself from a stream of events, and it should be able to record all the events it raises.

public interface IEventSourcedAggregate : IAggregate {
    void Initialize(EventStream eventStream);

    EventStream RecordedEvents();
}

The question I want to answer today is: how do I persist those event sourced aggregates?

In traditional systems, aggregate persistence is not a trivial topic. Especially relational databases have the reputation to make things hard on us. Even though tools such as ORM’s have tried to help in making the gap between the relational and object oriented model as small as possible, there is still a lot of friction associated with the notorious impedance mismatch.
The last two years I have done some work using one of the popular NoSQL variants: a document store. In this paradigm, each aggregate materializes into a single document. Structure, constraints and referential integrity are not enforced by the database, but by code. The advantage of relaxing consistency at the database, is that it makes it easier to scale outside a single machine, and that developers feel more empowered. Giving in on consistency guarantees is not acceptable for each system though. Again, pick the right tool for the job.
What both paradigms have in common is that they both focus on structure instead of behaviour.

Event sourced systems on the other hand, don’t care about the structure of an aggregate, but about the events that caused the aggregate to be in its current state. Only having to store events - which are represented as DTO’s - makes persistence and tooling much easier compared to traditional systems.

There are three things a minimalistic event store should be able to do:

Store a new event stream
Append to an existing event stream
Retrieve an existing event stream

An interface for that could look like this.

public interface IEventStore {
    void CreateOrAppend(Guid aggregateId, EventStream eventStream);

    EventStream GetStream(Guid aggregateId);
}

Notice that there is no update or delete - events happen, we can’t jump in a time machine and alter the past. This allows us to get by with an append-only data model. Can you imagine how much easier to implement, optimize and distribute this must be compared to traditional models?

As an exercise, I took the interface I just defined and implemented a durable, non-transactional, non-scalable (up to 4294967295 streams), single-user event store that persists event streams in raw text files. Each record on disk represents a serialized event with a tiny bit of metadata.

public class FileEventStore : IEventStore {    
    private const string Dir = @"C:\EventStore";            

    public void CreateOrAppend(Guid aggregateId, EventStream eventStream) {
        EnsureDirectoryExists();

        var path = EventStoreFilePath.From(Dir, aggregateId).Value;

        using (var stream = new FileStream(
            path, FileMode.Append, FileAccess.Write, FileShare.None))
        {
            using (var streamWriter = new StreamWriter(stream))
            {
                streamWriter.AutoFlush = false;
                foreach (var @event in eventStream)
                    streamWriter.WriteLine(
                        new Record(aggregateId, @event).Serialized());
            }
        }
    }
    
    public EventStream GetStream(Guid aggregateId) {           
        var path = EventStoreFilePath.From(Dir, aggregateId).Value;

        if (!File.Exists(path))
            return null;

        var lines = File.ReadAllLines(path);
        var events = lines
            .Select(x => Record.Deserialize(x))
            .Select(x => x.Event)
            .ToList();

        if (events.Any())
            return new EventStream(events);

        return null;
    }

    private void EnsureDirectoryExists()
    {
        if (!Directory.Exists(Dir))
            Directory.CreateDirectory(Dir);
    }
}

A long-ish test proves that I can create a stream, append to it and read it again without losing any data.

[TestMethod]
public void EventStoreCanCreateAppendAndRetrieveEventStreams() 
{
    var eventStore = new FileEventStore();

    var aggregateId = Guid.NewGuid();
    var account = new Account(aggregateId);
    account.Deposit(new Amount(3000));
    account.Withdraw(new Amount(400));    
    
    Assert.AreEqual(2, account.RecordedEvents().Count());
    Assert.AreEqual(new Amount(2600), account.Amount);

    eventStore.CreateOrAppend(aggregateId, account.RecordedEvents());
    var eventStream = eventStore.GetStream(aggregateId);

    Assert.AreEqual(2, eventStream.Count());

    var anotherAccount = new Account(aggregateId);
    anotherAccount.Initialize(eventStream);

    Assert.AreEqual(new Amount(2600), anotherAccount.Amount);

    anotherAccount.Withdraw(new Amount(200));

    Assert.AreEqual(new Amount(2400), anotherAccount.Amount);
    Assert.AreEqual(1, anotherAccount.RecordedEvents().Count());

    eventStore.CreateOrAppend(aggregateId, anotherAccount.RecordedEvents());

    var finalEventStream = eventStore.GetStream(aggregateId);
    Assert.AreEqual(3, finalEventStream.Count());
}

This produced the following artifact on disk.

While this implementation is far from ideal - dangerous really, it does show that implementing a minimalistic event store is doable - especially if you can build on top of existing data stores.

Doable, but not trivial. Greg Young - having actually implemented an event store, on the CLR too - recently shared some invaluable insights into what it takes to build a real-world event store.

I have always said an event store is a fun project because you can go anywhere from an afternoon to years on an implementation.

I think there is a misunderstanding how people normally use an event stream for event sourcing. They read from it. Then they write to it. They expect optimistic concurrency from another thread having read from then written to the same stream. This is currently not handled. This could be handled as simply as checking the expected previous event but this wouldn’t work because the file could be scavenged in between. The way this is generally worked around is a monotonically increasing sequence that gets assigned to an event. This would be relatively trivial to add.

The next issue is that I can only read the stream from the beginning to the end or vice versa. If I have a stream with 20m records in it and I have read 14m of them and the power goes out; when I come back up I want to start from 14m (stream.Position = previous; is a Seek() and 14m can be very expensive if you happen to be working with files the OS has not cached for you). This is a hugely expensive operation to redo and the position I could have saved won’t help me as the file could get compacted in between. To allow arbitrary access to the stream is a bit more difficult. The naive way would be to use something like a sorted dictionary or dictionary of lists as an index but you will very quickly run out of memory. B+Trees/LSM are quite useful here.

Even with the current index (stream name to current position) there is a fairly large problem as it gets large. With 5m+ streams you will start seeing large pauses from the serializing out the dictionary. At around 50m your process will blow up due to 1gb object size limit in CLR
Similar to the index issue is that with a dictionary of all keys being stored in memory and taking large numbers of writes per second it is quite likely you will run out of memory if people are using small streams (say I have 10000 sensors and I do a stream every 5 seconds for their data to partition). Performance will also drastically decrease as you use more memory due to GC.
A more sinister problem is the scavenge / compaction. It stops the writer. When I have 100mb of events this may be a short pause. When I have 50gb of events this pause may very well turn into minutes.

There is also the problem of needing N * N/? disk space in order to do a scavenge (you need both files on disk). With write speeds of 10MB/second it obviously wouldn’t take long to make these kinds of huge files especially in a day where we consider a few TB to be small. The general way of handling this is the file gets broken into chunks then each chunk can be scavenged independently (while still allowing reads off it). Chunks can for instance be combined as well as they get smaller (or empty).

Another point to bring up is someone wanting to write N events together in a transactional fashion to a stream. This sounds like a trivial addition but its less than trivial to implement (especially with some of the other things discussed here). As was mentioned in a previous thread a transaction starts by definition when there is more than one thing to do.

There are decades worth of previous art in this space. It might be worth some time looking through it. LSM trees are a good starting point as is some of the older material on various ways of implementing transaction logs.

Playing with Greg’s event store is something that has been on my list for a long time.

Next week: but how do we query our aggregates now?

An event sourced aggregate

Sun, 13 Oct 2013 18:36:00 +0200

Last week I shared my theoretical understanding of event sourcing. Today, I want to make an attempt at making that theory tangible by implementing an event sourced aggregate.

In traditional systems, we only persist the current state of an object.

In event sourced systems, we don’t persist the current state of an object, but the sequence of events that caused the object to be in the current state.

If we want an aggregate to be event sourced, it should be able to rebuild itself from a stream of events, and it should be able to record all the events it raises.

public interface IEventSourcedAggregate : IAggregate
{
    void Initialize(EventStream eventStream);

    EventStream RecordedEvents();
}

Let’s implement the example aggregate we used last week: an account. An account owner can deposit and withdraw an amount from his account. There is a maximum amount policy for withdrawals though.

public class Account : IEventSourcedAggregate {
    private readonly Guid _id;

    public Account(Guid id) {
        _id = id;
    }

    public Guid Id { get { return _id; } }

    public void Initialize(EventStream eventStream) { 
        throw new NotImplementedException();
    }

    public EventStream RecordedEvents() { 
        throw new NotImplementedException(); 
    }
    
    public void Deposit(Amount amount) { }
    
    public void Withdraw(Amount amount) { }
}

Next to the Initialize and RecordedEvents method, our aggregate facade hasn’t changed. We still have a Deposit and a Withdraw operation like we would have in a traditional aggregate. How those two methods get implemented differs though.

When we deposit or withdraw an amount, we want to - instead of changing the state directly - apply events. When an event gets applied its handler will first be invoked, for the event then to be recorded.

public void Deposit(Amount amount) {
    Apply(new AmountDeposited(amount));
}

public void Withdraw(Amount amount) {
    if (amount.IsOver(AmountPolicy.Maximum))     {
        Apply(new WithdrawalAmountExceeded(amount));

        return;
    }

    Apply(new AmountWithdrawn(amount));
}

private void Apply(IEvent @event) {
    When((dynamic)@event);
    _eventRecorder.Record(@event);
}

An event recorder is a small object that keeps track of recorded events.

public class EventRecorder
{
    private readonly List<IEvent> _events = new List<IEvent>();

    public void Record(IEvent @event) {
        _events.Add(@event);
    }

    public EventStream RecordedEvents() {
        return new EventStream(_events);
    }
}

This object will be used to have our aggregate return a stream of recorded events.

public EventStream RecordedEvents() {
    return _eventRecorder.RecordedEvents();
}

We can now also implement initializing the aggregate from a stream of events.

public void Initialize(EventStream eventStream) {
    foreach (var @event in eventStream)
        When((dynamic)@event);
}

Here too, event handlers are invoked by using the dynamic run-time to find the best overload.

It’s the event handlers that will change the aggregate’s state. In this example, they can be implemented like this.

private void When(AmountWithdrawn @event) {
    _amount = _amount.Substract(@event.Amount);
}

private void When(AmountDeposited @event) {
    _amount = _amount.Add(@event.Amount);
}

private void When(WithdrawalAmountExceeded @event) { }

A test verifies that when I invoke operations on the aggregate, all the events are recorded, and the state has changed. When I use those recorded events to rebuild the same aggregate, we end up with the same state.

    [TestMethod]
    public void ICanReplayTheEventsAndHaveTheStateRebuilt() {
        var account = new Account(Guid.NewGuid());

        account.Deposit(new Amount(2500));
        account.Withdraw(new Amount(100));
        account.Withdraw(new Amount(200));

        Assert.AreEqual(3, account.RecordedEvents().Count());
        Assert.AreEqual(new Amount(2200), account.Amount);

        var events = account.RecordedEvents();

        var secondAccount = new Account(Guid.NewGuid());
        secondAccount.Initialize(events);

        Assert.AreEqual(new Amount(2200), secondAccount.Amount);
        Assert.AreEqual(0, secondAccount.RecordedEvents().Count());
    }

And this is all there is to an event sourced aggregate.

For this exercise I tried to keep the number of concepts low. Many will have noticed that extracting a few concepts would benefit re-use and explicitness.

Also using the DLR to invoke the correct event handlers might be frowned upon; it’s not the most performant method, each event must have a handler, and in case a handler is missing the exception is not pretty. Experienced readers will also have noticed concepts such as versioning and snapshots are not implemented yet. I hope limiting the amount of concepts and indirections made this blog post easier to read.

Next week: where do I persist these events?

My understanding of event sourcing

Sun, 06 Oct 2013 18:32:00 +0200

I’ve been studying event sourcing from a distance for little over a year now; reading online material and going through some of the excellent OS code. Unfortunately, there would be no value introducing it into my current project - it would even be a terrible idea, so I decided to satisfy my inquisitiveness by consolidating and sharing my understanding of the concept.

Domain events

An event is something that happened in the past.

Events are described as verbs in the past tense. For example; amount withdrawn, amount deposited, maximum withdrawal amount exceeded. Listen for them when talking to your domain experts; events are as much a part of the ubiquitous language as commands, aggregates, value objects etc…

Once you’ve captured a few events, you will notice how these concepts have always implicitly been there, but by making them explicit you introduce a whole new set of power tools to work with.

Event sourcing

Having defined domain events one more time, we can now look at event sourcing. By the name alone, it should be obvious events are going to play the lead role.

In traditional systems, we only persist the current state of an object. In event sourced systems, we don’t persist the current state of an object, but the sequence of events that caused the object to be in the current state.

In traditional systems, every time a change happens, we retrieve the old state, mutate it, and store the result as our current state. In this example, only the last column would be persisted.

Old amount      Command         Current amount
                CreateAccount   $0
$0	            Deposit $2000	$2000
$2000	        Withdraw $100	$1900
$1900	        Withdraw $500	$1400
$1400	        Withdraw $2000	$1400
$1400	        Withdraw $300	$1100

In event sourced systems on the other hand, we store the changes that happened - the second column, not the current state. To arrive at the current state again, we take all these events - and replay them.

Command	        Event	                                Current amount
CreateAccount	AccountCreated	                        $0
Deposit $2000	Deposited $2000	                        $2000
Withdraw $100	Withdrawn $100	                        $1900
Withdraw $500	Withdrawn $500	                        $1400
Withdraw $2000	Maximum withdrawal amount exceeded! 	$1400
Withdraw $300	Withdrawn $300	                        $1100

Notice how we already gain better insights into what’s happening by seeing an explicit maximum amount exceeded event.

Next time; what does this look like in code?

CZ The Trilogy

Sun, 29 Sep 2013 18:15:00 +0200

Over the weekend, we visited the Czech Republic for the third time (one and two). It’s mostly chance that sends us that way every time though. This time, we were invited by friends to accompany them in staying over at their family’s house - who made it their job to lead guided tours through Prague.

We left Thursday right after work, hoping to get there in eight hours. A decent traffic jam, a missed exit, and some bad map reading decided otherwise; it added three hours to the trip.

We stayed in a small town just twenty minutes outside of Prague, which has been able to preserve all of its rural character. We took one day to go out hiking, and were surprised by the local fauna; we were lucky enough to spot wild deer, wild boars and a viper (that last one already crushed to death though). Next to those species, the forest was heavily occupied by locals gathering mushrooms - must be the season?

We used the remaining two days to stroll through Prague, and to live like kings getting the most out of the favorable prices. Food and drinks cost less than half of what they do in Belgium. There is a zero-tolerance alcohol policy for drivers though - I got pulled over too, so they seem to be pretty serious about it.

Actor Model in COBOL

Sun, 22 Sep 2013 16:38:00 +0200

In an Actor system, each Actor acts as a self-contained and autonomous component. An Actor can only communicate with other Actors by exchanging messages - they are not allowed to share state. Messages are handled asynchronously, and are nondeterministic. The location of Actors should be transparent; they can either live on the same machine, or on a distributed system. These properties make the Actor Model a great fit for parallel and distributed computing.

Even without considering parallelism and distribution, the Actor Model appeals to me. If you take an existing system, and make each aggregate in that system an Actor, what would the impact be? You can get rid of all the messaging and queuing infrastructure; messages and asynchrony are now first class citizens. Where you had to have discipline abiding the aggregate rules of thumb - modifying one aggregate per transaction, no references to other aggregates, Tell Don’t Ask… - the very nature of Actors will guide you into doing the right thing.
Next to these implementation concerns, the model itself can be used as a framework for modeling and reasoning about complex systems. Once they are well educated on the constraints, it must come natural for domain experts as well.

Having worked with a team of mainframe programmers over the last year, it recently came to me that how they have designed their systems over the years is compatible with a good amount of Actor laws.

Composition

A good thing about COBOL seems to be that it’s nearly impossible to write maintainable big programs, so you’re forced to decompose your program into smallish autonomous components - into jobs.

Messages

Communication between these jobs happens by passing flat files around

the only format that’s supported out-of-the-box. Messages come in, and new messages go out. Jobs will never mutate the incoming payload, a new copy is created instead; pipes and filters. Folders serve as a queue, allowing files to be processed asynchronously and deterministic.

Staying clear of mutating messages makes debugging extremely easy; you’ll never hear someone on the team asking for reproduction steps, they just restore the production archives locally.

Addresses

Actors send messages to other Actors using their addresses. This can be a memory or disk address, a network address, email address, whatever really. In mainframe land, file system paths serve as addresses.

No shared state

In general they stay away from jobs sharing state; the default is to lock files exclusively, so sharing them is highly impractical. Even most static data gets synchronized instead of shared - banking reference data, customer addresses, configuration etc…

Scheduling

A scheduler sits on top of all these jobs. Its responsibility is to start a job when a new file arrives. If a job fails, the scheduler acts as a supervisor and will notify operations, which will investigate the issue - probably look at what’s on the file system, and use the same scheduler to restart the failed job. Notice that one failing job doesn’t impact other jobs.

All of this gives you an automated, highly observable and fault-tolerant system.

*Although COBOL remains to be a horrible language, mainframe systems do have their strengths. There must be some good reason a lot of core business functions are still running on mainframes, right? Maybe similarities with the Actor Model are far-fetched and merely a figment of my imagination. Feel free to share your thoughts. *

Slides from my talk on the Ubiquitous Language

Sun, 15 Sep 2013 16:34:00 +0200

I just returned from our yearly Euricom retreat. This year, all forty of us got to spend four days in the South of Spain. Where we had longish sessions and a few workshops last year, we experimented with shorter talks this year - a la lightning talks, TEDx style.

This format made it possible for everyone to speak, but also forced the speaker to keep the scope of the talk focused, and to organize the information in a way that attendees can get the gist of it in only twelve minutes. This makes for high-energy talks designed to peak one’s interest, to share useful tips, to plant a seed or to pitch an idea.
Covering more than just technical ground alone made topics extremely diverse; from query tuning to empathy, from automated testing to how to explain your kids what you do for a living, from personal kanban to juggling with a diabolo. Going back and forth between these technical and less technical sessions kept my brain from being oversaturated.

Definitely an experiment that only yielded positive results; we will be using this format more frequently when organizing internal events.

Initially, I planned on doing a session on the DDD strategic patterns; the ubiquitous language, subdomains, bounded contexts, context mapping… but I couldn’t capture all of that in a meaningful way in less than twelve minutes. That’s why I started over, focusing on the ubiquitous language alone.

You can find my slides on Slideshare.

The first DDDBE Modellathon

Fri, 06 Sep 2013 13:24:00 +0200

On our way back from DDD Exchange, heavily influenced by yet another immersive DDD experience, we searched for ways to keep the momentum going. Sure, we met up regularly for CQRS beers, but we felt that we could do more, better. That’s when we coined the term modellathon, something like a hackathon, but instead of writing code, we would build models.

Thanks to the effort of Mathias, Stijn and Yves, Tuesday marked the first get-together of the Belgian DDD user group in its official form. Combell was kind enough to provide us with a location, while Mathias fronted paper - lots of it too, post-its and markers.

Mathias and Stijn took the lead introducing themselves as domain experts of the day. The domain? The United Schools of Kazachstan.

We split up into groups of four, and used our first pomodoro trying to understand the domain. The second pomodoro, we threw everything away and started fresh.

The first modeling technique our group tried was Alberto Brandolini’s event storming. We took what we thought was the most important event report approved, wrote it on a post-it, and posted it on the center of our sheet of paper. Then we worked our way back to how we got there, but also looked at what happened next. This modeling approach yielded results very quickly; we all gained a decent understanding of everything what’s going on in the domain. Talking to the domain expert made it obvious what the hotspots were, he kept referring to two post-its in particular.

We might have zoomed in right there, but for the sake of experimenting with event storming, we stuck to events a little longer. We added commands, looked for clusters, made aggregate boundaries based on that, and looked where they were talking to each other.

We initially used a sheet of paper and stayed seated, but this was holding us back. Once we stood up and moved to the wall, our synergy increased. Space and blood circulation seem to be important.

Flow is important too. Since we only had two domain experts, we often had to make assumptions and come up with a name that made sense. This slowed us down. We should just write down whatever we come up with, and make doubts explicit on the post-it. You can always verify and fix the language consulting the domain experts later.

The next visualization technique, initiated by Yves, took a UI-first approach. While this quickly gives you something concrete to chat about with the domain expert, I learned it can also lead you to bounded context boundaries by helping you answer the question “Where is all this data coming from, which contexts does this data belong to?”

I thought this first experiment went really well - a lot better than I expected. It proves once again the value of visualization and collaboration. All models were probably wrong, but turned out to be useful. The end result probably doesn’t matter that much, discovery and learning along the way does.

Note to self: make pictures of the end results; they would help explaining some of the experiments.

Inheritance is like Jenga

Sun, 25 Aug 2013 18:18:00 +0200

These last few days, I have been working on a piece of our codebase that accidentally got very inheritance heavy.

When it comes to inheritance versus composition, there are a few widely accepted rules of thumb out there. While prefer composition over inheritance doesn’t cover the nuances, it’s not terrible advice; composition will statistically often be the better solution. Steve McConnell’s composition defines a ‘has a’- relationship while inheritance defines an ‘is a’-relationship, gives you a more nuanced and simple tool to apply to a scenario. The Liskov substitution principle which states that, if S is a subtype of T, then objects of T may be replaced with objects of type S without any of the desirable properties of that program, is probably the most complete advice.

Inheritance, when applied with the wrong motivations - reuse and such, often leads to fragile monster class hierarchies which are too big to wrap your head around and extremely hard to change.

When I was working on such a monstrous hierarchy, it reminded me of playing Jenga. Some time not that long ago, someone had built this tower from the ground, laying block over block, layer over layer. On the surface it appears to be stable and rigid, but as soon as someone wants to winkle out one block, it becomes obvious one block can bring down the whole structure. The lower the block in the structure, the more layers rest on it, the greater the risk of breaking everything on top. Even if you do succeed in pulling one block out, chances are you had to touch the surrounding blocks to prevent the tower from tumbling over.

Instead of Jenga, I’d prefer a puzzle made of just a few pieces - designed for toddlers. A puzzle is flat, you can see the big picture in one glance, while you can reason about each piece individually as well. As long as the edges of the pieces fit together, you can assemble whatever picture you want.

But I already wrote it

Sun, 18 Aug 2013 16:24:00 +0200

A few weeks ago, we set out to implement a feature that enabled back office users to set a new rate ahead of time. With our analyst and the involved user being out of the office for days, we had to solely rely on written requirements. Two of us skimmed the documents, but didn’t take the time to assure there wasn’t any ambiguity - it looked trivial really. I went back to what I was doing, while my colleague set out to implement this feature. Going over the implementation together the next day, he had built something a lot more advanced than I had anticipated. While I argued that this was a lot more than we needed, we agreed to wait for feedback from our analyst to return from her holiday.

When our analyst returned, she confirmed that the implementation did a lot more than we needed. I suggested removing what we didn’t really need. My colleague argued that he now already had put in the effort to write it, and we should just leave it as is.

I can relate to feeling good about freshly written code, but that shouldn’t stop you from throwing it away. Code is just a means to an end; the side product of creating a solution or learning about a problem. If you really can’t let go, treasure it in a gist.

In this particular scenario, one could argue that making the solution more advanced than it should be, isn’t strong enough of an argument to make a big deal out of it. We’re giving the users a little extra for free, right?

I cannot stress the importance of simplicity enough; to be simple is to be great, perfection is achieved not when there is nothing more to add, but when there’s is nothing left to take away, and all of that. Nobody likes bulky software. Nobody likes fighting complexity all day.

But by only considering the cost of initially writing it, you are also ignorant of the true cost of what appears to be a small little extra on the surface. Users, developers, designers and analysts alike have yet another thing to wrap their heads around. More code is not a good thing; more code to test, more code to maintain. Each feature, how small it may seem, needs to be taken into account when planning on new ones. Each feature, definitely an advanced one, makes the cost of training and support go up. The cost of implementing a feature is just a tiny portion of what it costs to support that feature through its entire lifetime.

Using this argument, I eventually succeeded in persuading my peer to dump the ballast. The real lesson for me however, is probably that how trivial it might have seemed, we could have ruled out any possible ambiguity in advance by using one of the various tools we have to our disposal; a smallish white board session or maybe pairing on some high level tests.

Eventual consistent domain events with RavenDB and IronMQ

Thu, 15 Aug 2013 14:03:00 +0200

Working on side projects, I often find myself using RavenDB for storage and IronMQ for queueing. I wrote about that last one before here and here.

One project I’m working on right now makes use of domain events. As an example, I’ll use the usual suspect: the BookingConfirmed event. When a booking has been confirmed, I want to notify my customer by sending him an email.

I want to avoid that persisting a booking fails because an eventhandler throws - the mail server is unavailable. I also don’t want that an eventhandler executes an operation that can’t be rolled back - sending out an email - without first making sure the booking was persisted succesfully. If an eventhandler fails, I want to give it the opportunity to fix what’s wrong and retry.

public void Confirm()
{
    Status = BookingStatus.Accepted;

    Events.Raise(new BookingConfirmed(Id));
}

Get in line

The idea is, instead of dealing with the domain events in memory, to push them out to a queue so that eventhandlers can deal with them asynchronously. If we’re trusting IronMQ with our queues, we get in trouble guaranteeing that the events aren’t sent out unless the booking is persisted succesfully; you can’t make IronMQ enlist in a transaction.

Avoiding false events

To avoid pushing out events, and alerting our customer, without having succesfully persisted the booking, I want to commit my events in the same transaction. Since IronMQ can’t be enlisted in a transaction, we have to take a detour; instead of publishing the event directly, we’re going to persist it as a RavenDB document. This guarantees the event is committed in the same transaction as the booking.

public class DomainEvent
{
    public DomainEvent(object body)
    {
        Guard.ForNull(body, "body");          
        
        Type = body.GetType();
        Body = body;
        Published = false;
        TimeStamp = DateTimeProvider.Now();
    }
    
    protected DomainEvent() { }

    public string Id { get; private set; }

    public DateTime TimeStamp { get; private set; }

    public Type Type { get; private set; }

    public object Body { get; private set; }

    public bool Published { get; private set; }

    public void MarkAsPublished()
    {
        Published = true;
    }
}

public class DomainEvents : IDomainEvents
{
    private IDocumentSession _session;

    public DomainEvents(IDocumentSession session)
    {
        _session = session;
    }

    public void Raise<T>(T args) where T : IDomainEvent
    {       
        _session.Store(new DomainEvent(args));
    }
}

Getting the events out

Now we still need to get the events out of RavenDB. Looking into this, I found this to be a very good use of the Changes API. Using the Changes API, you can subscribe to all changes made to a certain document. If you’re familiar with relation databases, the Changes API might remind you of triggers - except for that the Changes API doesn’t live in the database, nor does it run in the same transaction. In this scenario, I use it to listen for changes to the domain events collection. On every change, I’ll load the document, push the content out to IronMQ, and mark it as published.

public class DomainEventPublisher
{
    private readonly IQueueFactory _queueFactory;
    
    public DomainEventPublisher(IQueueFactory queueFactory)
    {           
        _queueFactory = queueFactory;
    }

    public void Start()
    {
        DocumentStore
            .Get()
            .Changes()
            .ForDocumentsStartingWith(typeof(DomainEvent).Name)
            .Subscribe(PublishDomainEvent);
    }

    private void PublishDomainEvent(DocumentChangeNotification change)
    {
        Task.Factory.StartNew(() =>
        {
            if (change.Type != DocumentChangeTypes.Put)
                return;

            using (var session = DocumentStore.Get().OpenSession())
            {
                var domainEvent = session.Load<DomainEvent>(change.Id);

                if (domainEvent.Published)
                    return;

                var queue = _queueFactory.CreateQueue(domainEvent.Type.Name);
                queue.Push(JsonConvert.SerializeObject(domainEvent.Body));

                domainEvent.MarkAsPublished();

                session.SaveChanges();
            }
        });
    }
}

I tested this by raising 10,000 events on my machine, and got up to an average of pushing out 7 events a second. With an average of 250ms per request, the major culprit is posting messages to IronMQ. Since I’m posting these messages over the Atlantic, IronMQ is not really to blame. Once you get closer, response times go down to the 10ms - 100ms range.

A back-up plan

If the subscriber goes down, events won’t be pushed out, so you need to have a back-up plan. I planned for missing events by scheduling a Quartz job that periodically queries for old unpublished domain events and publishes them.

In conclusion

You don’t need expensive infrastructure or a framework to enable handling domain events in an eventual consistent fashion. Using RavenDB as an event store, the Changes API as an event listener, and IronMQ for queuing, we landed a rather light-weight solution. It won’t scale endlessly, but it doesn’t have to either.

I’m interested in hearing which homegrown solutions you have come up with, or how I could improve mine.

When your commands spell CUD

Sun, 04 Aug 2013 19:08:00 +0200

A good while ago, I blogged on commands (and queries). After exploring various flavors, I eventually settled on this one; commands, handlers and an in-memory bus that serves as a command executor.

Commands help you in supporting the ubiquitous language by explicitly capturing user intent at the boundaries of your system - think use cases. You can look at them as messages that are being sent to your domain. In this regard, they also serve as a layer over your domain - decoupling the inside from the outside, allowing you to gradually introduce concepts on the inside, without breaking the outside. The command executor gives you a nice pipeline you can take advantage of to centralize security, performance metrics, logging, session management and so on.

We always need to be critical of abstractions though, and regularly assess their value. A smell that might indicate that commands might not be working for you, or are adding little value, is that the first letters of your commands spell CUD - Create Update Delete.

For example; CreateCarCommand, UpdateCarCommand and DeleteCarCommand.

The language needs attention

Possibly, your team doesn’t fully grasp the power of cultivating the ubiquitous language. If you start listening to your domain experts, you might end up with totally different command names; TakeInNewCarCommand, RepaintCarCommand, InstallOptionCommand and RemoveCarFromFleetCommand.

Maybe though, there is no language at all, and you’re really just doing CRUD. If the context you are working on is implementing a generic or supporting subdomain this might not be terrible.

If I’m doing CRUD, do I still need commands?

Commands help you decouple the inside from the outside. If there is no domain on the inside though, they can still help you decouple the application layer from other concerns. You might prefer to use another facade to separate concerns though, such as a thin service layer. I don’t think the service layer abstraction gives you anything commands don’t though. Maybe you don’t find any value in separating things at all, and just dump everything in the the application layer.

All of these approaches are valid. You just have to consider the trade-offs. With these last approaches, next to losing decoupling in from out, you also lose that central pipeline that a command executor gives you.

Doesn’t my application layer give me this pipeline for free?

It sure can. Looking at modern networking stacks, these all have interception points built in. For example; NancyFx allows you to hook in the request pipeline using before and after hooks; Web API gives your message handlers and action filters; and WCF has a concept of interceptors.

Not all application frameworks do though - think of frameworks targeting desktop software.

The advantage of having your own pipeline, instead of solely having to rely on your application framework’s interception points, is that you’re boss and don’t have to study the ins and outs of each framework, and hope for the framework to have thought of your needs. Also when you need to support multiple application layers, you don’t have to implement all features twice.

What about using aspects instead of a pipeline to centralize all these concerns?

You can - instead of a pipeline - use aspects to take care of croscutting concerns such as security, logging, session management.. and have them woven into your executable at compile- or runtime. I think of aspects as if they were macros, which save you on lines of code written, but also often conceal the real problem; missing concepts. While they try to sell you on separation of concerns, notice that you’re actually still producing procedural code, but instead of writing it by hand, you’re now letting an AOP framework do it for you. Add harder testing, debugging and readability to the bunch, and you can understand why I’m not a fan of AOP. There are scenarios where this all doesn’t matter much though, and to strengthen the cliche; granular logging is a good use case.

When your commands spell CUD, it might indicate you could do without them. Do realize what the consequences are of taking them away though;

you lose the opportunity to capture user intent at the boundaries of your system, to strengthen the ubiquitous language
you may need an alternative facade to decouple in from out
you lose that command executor serving as your own pipeline

The last cowboy

Tue, 16 Jul 2013 05:46:00 +0200

After seventeen days, 2000 miles (3220km), seven states, eight national parks and fifteen motels, our second USA road trip has come to an end. Inspired, and with new insights, we’re leaving for Belgium in the morning.

Along the way, we documented our journey in these posts:

Thanks for reading! When I return, I’ll get back to more technical writing.

The last cowboy song

The road goes on forever

Mon, 15 Jul 2013 04:51:00 +0200

We didn’t get to see a lot of Salt Lake City. Before heading to Idaho Falls, we hung around at the mall, feasted on Sushi, and bought new hiking shoes since ours literally fell apart.
Since Idaho Falls just served as a launching point to Yellowstone, we filled the hole in our schedule with a date night; we had BBQ and went to see WorldWar Z.

After another long drive the next day, we exchanged the dirt for greener scenery. We drove through Grand Teton, to get to of one our last and most anticipated destinations; Yellowstone National Park. We spent two days here, in which we drove across the extensive volcanic area, stopping at all popular attractions (geysers, pools, waterfalls…), spotting and stalking wildlife on the way.

Instead of a motel, we slept in the wild - in bear country, in a Tipi. Although our host armed us with bear spray, we got a good night’s sleep. Today however, we did spot a baby Grizzly bear from afar!

Tomorrow will be our last day, which we will spend in Cody, Wyoming.

The road goes on forever

The long highway

Fri, 12 Jul 2013 07:13:00 +0200

We’re past halfway our trip and we need to make headway if we want to stay on schedule and reach Yellowstone by Sunday. Long, but pleasant drives; if you’re used to Belgian traffic.

Yesterday we visited Capitol Reef National Park. We originally planned on doing the Hickman Bridge Trail, but apparently there had recently been a big rock fall, blocking access at the trailhead. As an alternative we did the Narrows trail at Grand Wash Road, which was carved by a stream ages ago and is now still sometimes victim to flashfloods. Afterwards, we followed the scenic route, which made us zigzag between big boulders, on- and off-road.

In the evening we arrived at touristy Moab, going to dinner felt a lot like home, being surrounded by Dutch-speaking people. The temperature and the view gave away our true location though.

This morning, instead of hiking, we chose another means of transportation: we rented a UTV (Utility Vehicle). This allowed us to explore the Sand Flats Recreation Area, maneuvering over slick rocks and small desert dunes, following a trail of black skid marks. We completed the Hell’s Revenge Trail, which lives up to its name; it was frighteningly exciting.

Since this already took up a big chunk of our day and we still had a four hour drive to Salt Lake City ahead of us, we had to do the Arches National Park scenic drive in the fast lane.

The long highway

Stairway to heaven

Wed, 10 Jul 2013 07:04:00 +0200

These last two days have been rather tiring. Yesterday we drove up to Bryce Canyon, where we did the Queen’s Garden and Navajo Loop hike, which was only 3 miles (5km) but was rather strenuous because the trail takes you all the way down and all the way up again.

Today we followed the advice of the kind lady at the visitor center and hiked along Calfs Creek. It was a 6 miles (10km) round trip through loose sand in over 100°F (38°C), but halfway you’re rewarded with a stunningly beautiful waterfall and a refreshing dip in the basin. Driving up there, we got to witness the grandeur of the Utah plateau. When we finally got back to the air conditioned car, we made a detour to see the Grand Staircase – Escalante National Monument.

Stairway to heaven

She said yes!

Tue, 09 Jul 2013 07:42:00 +0200

Since our stay in Vegas was rather short, we selected some things we failed to do last time. For me, shooting a gun was definitely on that list. Although I’m not that big on gun culture - seems to me that if you live by the gun, you’re more likely to die by the gun - it’s still a thrill holding such a powerful weapon. We headed to Vegas’s most popular shooting range: The Gun Store. At the entrance, I got to pick a gun and a target to point it at. The heavy Desert Eagle became my weapon of choice to blow a zombie’s head off. Being guided to the shooting range by a professional, he ensured that I didn’t shoot my foot, and helped me land five headshots.

In the evening we got a players card hoping Lady Luck would bend variance to our benefit for once - those hopes were in vain. It didn’t take too long before we busted our 20$ gambling budget on virtual Poker. It earned us a free buffet though. We decided to drown our losses at the lounge with beer and cocktails.

Checking out late, we set out for Cedar Breaks National Monument. We waited till the evening to drive up because we were hoping for a clear night sky to do some stargazing. When we drove off, it was pouring rain, so things weren’t looking up. Reaching the top of the mountain, skies cleared up and a rainbow appeared. There we were, just the two of us, standing on top of the world. Finally, after an overcrowded observatory and barely making the 4th of July fireworks, I got to complete my quest and be freed of the one ring that had been burning in my backpack for over a week. I went down on one knee and asked my girlfriend of seven years to marry me. And she said yes!
We spent the rest of our evening looking at the impressive stellar sky as a happy newly engaged couple.

Fireworks, Tequila and Silvertown Blues

Sun, 07 Jul 2013 06:30:00 +0200

After a strenuous drive from Los Angeles to San Diego, we took advantage of the mild climate to unwind at San Diego’s Aquatica.

Being the 4^th of July, we joined the festivities and watched the fireworks from the car. Sadly we didn’t get to see a redo of last year’s spectacle, when San Diego accidentally launched all its fireworks in 30 seconds.

Next we headed for Joshua Tree National Park. With temperatures flirting with 100°F (38°C), we took on one small hike and drove up to two viewpoints; Skull Rock and Key’s View. Driving through cities of the plain in the California desert, we stopped in Barstow for some Mexican food and Corona, to finally faceplant in our pillows.

This morning we got up early to hunt down some ghosts in the silver town of Calico. Too bad they couldn’t keep the ghost town more authentic.

Tonight: Sin City.

Fireworks, Tequila and Silvertown Blues

To Live & Die in L.A.

Thu, 04 Jul 2013 08:51:00 +0200

After exploring the USA’s West Coast two years ago, we felt like we had to pay a second visit to fill in some blanks.

We left Antwerp Sunday 7AM to touch down 25 hours later in LAX, after an eight hour cross-Atlantic flight to Newark and a six hour overlay waiting for a connecting flight.

We were welcomed by a rental car salesman who strongly advised us to upgrade to a bigger SUV which would give us tinted windows since it’s gonna be hot, you know. After politely blowing him off, we drove off to our first hotel. Arriving at an airport motel, we got to check off the clichés with a lost empty beer bottle in the room, and us playing ‘the stained carpet is lava’.

We started off our first day with a real American breakfast; eggs and steak. Saturated, we continued our path to the California Science Center, driving through LA’s colorful suburbs. Arriving there, we found the highly welcomed Space Shuttle named Endeavour in its retirement home. We also visited the IMAX theatre, which is always worth your while. This time we got to see Blue Planet. We followed this up with a small lunch at LA’s real farmer’s market.

Yesterday we took part in a Warner Bros VIP tour. With a group of only eleven, the guide did a great job making the tour a lot more intimate than the one Universal offers. Cruising through the outdoor sets, he emphasized once again how fake everything is. A small patch of grass serves as Central Park; outdoor stages can be stripped down and rebuilt to go from 18^thcentury San Francisco to nowadays New York in a couple of days; leaves are plucked from trees to make it Winter and sewed back on to fast-forward to Spring; cracks are drilled into new pavements, and patches of tar are put on, to make them look worn out. Besides the outdoor stages we got a close look at the Big Bang Theory set and the retired Friends props, we even got to sit on the famous couch. Our final look behind the scenes went to the retired car museum, which was a real treat for me as a Batman fan, since it housed all Batmobiles, even the Tumbler.

After that, we drove up to Hollywood Boulevard, to be washed over by thousands of tourists walking around Hollywood’s two most famous theatres - the Dolby (former Kodak) theatre, and the Chinese theatre – just to stand where the stars once stood.

After nightfall we drove up to the observatory, just to put the size of LA in perspective. We were also hoping for some romance, but so were hundreds of others apparently.

Today we left central LA for Santa Monica and Venice Beach. We started our boardwalk in Venice Beach, embracing beach culture: street performers, medical marihuana offices, hustlers and amateurs pumping iron at Muscle Beach.

Next up; San Diego.

To Live & Die in L.A.

Multiplayer Enterprise Architect

Sun, 30 Jun 2013 17:00:00 +0200

Hanging around in the pub after DDDX, I ended up talking to Alberto Brandolini. For those who have never met him; he’s very much into visualization. You will always see him carrying a drawing pad, with a dash of permanent marker on his cheek, and a few lost sticky notes on his back. I don’t know if it was the Italian accent and the strong gestures, or my mildly intoxicated condition, but the idea of visualization as an important tool grew on me even more that evening.

During the conversation Alberto shared a recent experience where he was brought in as a facilitator to assist in planning multiple projects spread over a bunch of teams spanning multiple months. The bottom line was that with a big wall of paper, some hand drawn boxes, and lots of sticky notes, they had enabled intense and efficient collaboration, achieving what upper management had expected to take weeks, in a single day.

In my current project, we also could have taken advantage of these tools. Early on in the project, a team member was charged with the task of documenting all existing payment file transfers in the company. This isn’t as trivial as it sounds though; these files come in through a handful of specialized secure gateways, are then transferred internally by a few different propriety transfer platforms - of which some still stem from the mainframe era, to be processed, and returned to the customer through one of the gateways in one of the available output formats. To make matters worse, all of these systems are owned by different teams.
We probably tackled this in the worst way possible; with a spreadsheet and meetings. We could have tried occupying a meeting room, plastering the wall with paper; drawing all the systems on, and inviting all involved teams to come in and help us visualize all the transfers between systems. We could draw arrows between the systems, and stick post-it notes on top that represent a file with its constraints (source and destination folders, schedules etc…).
Too late for that now, but I’m curious to set out such an experiment in the future.

As you’re reading this, I’m traveling for three weeks - which I’ll surely write about, and in my backpack are two books that should help keep me occupied during the flight, and long overlays: Visual Meetings, and The Back of the Napkin. I’m confident there’s a lot of value to be found in visualization, either to make sense of a complex problem, or as a tool to support collaboration. Going forward I plan to get a lot better at it.

Not handling edge cases, making them explicit instead

Sun, 23 Jun 2013 16:49:00 +0200

When I wrote about accidental entities earlier, we followed a consultant building software for a car rental company. In the meanwhile, he has finished implementing the registration of new cars. Next on the list is allowing customers to make a booking.

We managed to get the CEO to set a whole hour apart to walk us over how the booking system should work.

CEO: “I’m not sure this meeting is going to take a whole hour though. Making a booking is rather trivial. Do you have any idea on how a booking would work?”

Us: “Well, as far as I understand - and don’t be too hard on me - a customer makes a booking, we allocate the car for the requested period, send the customer a confirmation email and we’re done.”

CEO: “Hold your horses, it’s not that trivial. When we receive a booking, we first need to verify the customer’s credit card.”

Us: “How do I verify a credit card?”

CEO: “You don’t. We use a third party electronic verification system that does just that; they check if the credit card is valid and has sufficient credit.”

Us: “What happens if the credit card is declined?”

CEO: “That’s easy; we just cancel the booking, and inform the customer.”

Us: “And what happens when the credit card gets verified?”

CEO: “Then we approve the booking.”

Us: “Let me stop you right there. Should we allocate the car immediately as soon as the customer makes the booking?”

CEO: “Hell no! Verification of the credit card can take a while; we might lose business if the credit card turns out to get declined.”

Us: “Hmmm, what happens if someone else has made a booking for the same car in the meanwhile?”

CEO: “That’s a good question. In the past, we’ve hardly encountered this problem, but it does happen though; we should probably take care of this edge case.”

Us: “How do you want to take care of these double bookings? I can imagine you don’t just want to cancel the second booking, and lose business over this, right?”

CEO: “No, exactly; I like your thinking! There are a few options here; if they only overlap for 30 minutes or so, we don’t do anything. When the customer comes to get his car, we apologize for the small delay and offer him a frappuccino to soothe him. If the overlap is bigger, and we have bigger or more expensive cars just sitting in the parking lot for that period, we often give the customer a free upgrade. When we don’t have any expensive cars left, we let our best sales person call the customer, and try to work something out. Sometimes we give them a small discount, and they’ll happily reschedule their plans a few hours instead of having to look for another rental car. We try to avoid canceling a booking as much as possible; we want to build a reputation where we always deliver.”

Us: “Wow, interesting. It’s going to cost me some time to get that right though. You said that this almost never happens. Since deciding on a good solution for the double booking seems to be non-trivial, how about we don’t handle this edge case right now, but make it explicit instead? We can mark the booking as double, make the system raise its hand, and ask for human intervention. Backoffice users already have the tools, and have the experience to decide on the best solution for the double booking. Leaving this out for now would also make it possible to go to market a few weeks earlier.”

CEO: “I really like this idea; let’s keep our first version as lean as possible. It really doesn’t make much sense trying to automate this right off the bat. It’s not like you’re cheap, you know. It would be useful for the system to keep count of every double booking though. Can we do that?”

Us: “Yep, that shouldn’t be too hard.”

After this talk with the CEO, we set out to model the solution in code.

In our first iteration, a booking has multiple representations, a representation for each state. One of the advantages of modeling it as such, is that it allows us to make all possible state changes per representation explicit; you can’t accept a booking if the credit card hasn’t been approved etc… Alternatives could be the classic state pattern, or something workflowish, but that isn’t really the focus of this post.

var booking = new Booking(
    bookingId, carId,
    new Customer(new CreditCard("343705171682875"), new CustomerName("Jef", "Claes")), 
    new Period(DateTime.Now.AddDays(1), DateTime.Now.AddDays(4)));

var bookingWithVerifiedCreditCard = booking.CreditcardVerified();
var doubleBooking = bookingWithVerifiedCreditCard.Double();

Each state change triggers an event. If we output each event, and run the snippet above we end up with this.

(EVENT) BookingCreditCardVerificiationPending
(EVENT) BookingCreditCardVerified
(EVENT) DoubleBooked`

The BookingCreditCardVerified event triggers us to detect doubles.

public class BookingCreditCardVerifiedHandler : IHandle<BookingCreditCardVerified> 
{    
    private readonly IBus _bus;

    public BookingCreditCardVerifiedHandler(IBus bus)
    {
        _bus = bus;
    }

    public void Handle(BookingCreditCardVerified @event)
    {
        _bus.Send(new DetectDoubleBooking(@event.BookingId));
    }
}

If a double booking is detected, we want to make the system raise its hand, and notify a human. This can be done through an email, a notification in the backoffice portal, or whatever really.

public class DoubleBookedHandler : IHandle<DoubleBooked>
{
    public void Handle(DoubleBooked @event)
    {
        NotifyHumans();
    }

    private void NotifyHumans() { }
}

Although, the technical implementation isn’t very special, I think it does show how events can help support the language and distribute responsibility to where it belongs.

There is this misconception that because we now have computers, they should solve all our problems, even all the edge cases. Edge cases - by definition - only happen at extreme conditions, and are regularly hard to take care of in a satisfactory manner, without a considerable investment. By making edge cases explicit, we allow a human to intervene, and decide on the best solution for the problem. This way we can go to market more quickly, with less code, and we might ironically even end up with happier customers. By collecting statistics on how often these edge cases occur, we can make a better informed decision on whether it’s worth the investment.

Angular.js and IE8 caching

Sun, 09 Jun 2013 16:52:00 +0200

Older Internet Explorer versions are notorious for agressively caching AJAX requests. In this post, you’ll find two techniques that combat this behaviour.

The first option is to have your server explicitly set the caching headers.

Response.Cache.SetExpires(DateTime.UtcNow.AddDays(-1));
Response.Cache.SetValidUntilExpires(false);
Response.Cache.SetRevalidation(HttpCacheRevalidation.AllCaches);
Response.Cache.SetCacheability(HttpCacheability.NoCache);
Response.Cache.SetNoStore();

Since you don’t necessarily own the server, or clients might already have cached some requests, you can trick the browser into thinking each request is a fresh one by making each url unique. Our old pal jQuery already learned this trick years ago. Angular.js on the other hand seems to have forgotten. We can get around though.

If you merge this pull request (or wait for angular.js version 1.2), you will find angular’s HTTP provider augmented with request interceptors, enabling you to mold the request before it goes out.

The interceptor we’re adding to kill the cache only touches GET requests, appending a ‘cacheSlayer’ querystring parameter with a timestamp to each url, making it unique and thus bypassing the cache. A factory is responsible for creating it, while a config block pushes it into a collection of interceptors.

var AppInfrastructure = angular.module('App.Infrastructure', []);

AppInfrastructure
    .config(function ($httpProvider) {
        $httpProvider.requestInterceptors.push('httpRequestInterceptorCacheBuster');
    })    
    .factory('httpRequestInterceptorCacheBuster', function () {
        return function (promise) {
            return promise.then(function (request) {
                if (request.method === 'GET') {
                    var sep = request.url.indexOf('?') === -1 ? '?' : '&';
                    request.url = request.url + sep + 'cacheSlayer=' + new Date().getTime();
                }

                return request;
            });
        };
    });

I hope this helps someone spending time on more important matters.

Accidental entities - what about the UI?

Sun, 02 Jun 2013 16:24:00 +0200

This post is a follow-up to my previous blog post “Accidental entities - you don’t need that identity”.

In that post, we followed a consultant building an application for a car rental. One of the requirements was that the CEO could manage a collection of available colors. Although the tools at our disposal - a relational database and NHibernate - wanted to trick us into making a car reference one of these available colors by its identifier, we found out that the CEO really thinks of a car’s color as a value, and does not care about a color’s identity. This means that we didn’t make a car reference an available color, but we copied its value instead. This allows the CEO to remove available colors, without it having an impact on cars that already came in that color.

The solution we’re building contains a public facing web application that allows customers to make reservations online, and a backoffice web application - hosted on the intranet, that employees will use to manage the cars.

When a new car arrives at the car rental, a backoffice user will register it. Once the car is registered - and in use, backoffice users should still be able to change some of its characteristics; brand, model, color, engine size, etc…

Instinct tells us to add a page that enables editing all the car’s properties. Some of these properties are free text, some radio buttons, but for the car color, it’s a dropdownlist.
Halfway through implementing this new functionality, we notice that changing the color gets us into trouble. We populate the dropdownlist with all available colors, but when we want to bind the car’s current color as the selected value, it’s not in the list of available colors. The CEO has removed the car’s current color out of the list of available colors.
After a bit of tinkering, we come up with a workaround that adds the car’s current color to the available colors with a default value of -1. This allows us to determine if the color needs to be changed.

Relational databases, RAD tools, scaffolding and anemic models have poisoned our minds, making us throw up the database schema all over the UI. We can do a lot more though.
If we take a step back, and make an effort to discover what changing these properties really means to our business users, we might come up with a totally different user experience.

We leave the UI as is, and ping the CEO on Lync, inviting her for a coffee break.

After a bit of obligatory small talk, we start asking questions about what we’re really after.

Us: “Can all of the car’s characteristics change after the initial registration?”

CEO: “No, have you ever seen a car change brand, or model through its lifecycle?”

Us: “I can imagine the engine size also belongs to the list of characteristics that can’t be changed after registration?”

CEO: “Oh, it can! We sometimes have the engine of our sports models tuned to have an edge over the competition. Guys are crazy for horsepower.”

Us: “What about changing the color?”

CEO: “If a car has some nasty scratches on it, we sometimes get it repainted. If we do get it repainted, it’s always in one of the colors available at that moment.”

We return to our desk with a far better understanding of what it really means to change each of the car’s characteristics. We discovered a whole new language, with some important constraints. Neither brands nor models change after registration. Engines get tuned, increasing their engine size. A car’s color doesn’t change; a car gets repainted - always in one of the available colors.

After iterating on this feedback, we’re far more satisfied with the model; it captures the language far better. Our backend implementation is not the most important part of the solution though; it’s useful to invest in improving the UI.

After some experimentation we come up with a more task-based UI, something that looks like this.

The UI now does a far better job supporting the language and communicating its constraints. And with that, we also solved the original problem that motivated us to ask these extra questions. There is no value in having the current color in the dropdownlist; we are better off making the process of having a car repainted explicit.

When we make an effort to really capture the language, this will be reflected in our model, but also in the UI. This can add tremendous value. It’s not only about making the user experience more intuitive, but about making a language consistent for thousands of users, supporting processes and communication throughout the company.

As a disclaimer; sometimes it’s fine to just throw up your database. There’s a place and time for anything. But don’t let CRUD be the only tool in your toolbox; you can do much more. Make sure to invest this extra effort where it matters, where it makes a difference.

Accidental entities - you don't need that identity

Sun, 26 May 2013 16:27:00 +0200

An entity is identified by an identifier, while value objects are identified by their value.

If I make a living renting cars to tourists, I might not care the least about the identity of the colors the cars came in. I just care about their value; Rosso Corsa, Azurro Metallic… If I repaint the car, the color changes, and the previous color is abandoned as a whole.
However, if I were a car paint manufacturer, I would care a great deal about the identity of a color. My first action might be to make up a marketable name for the color, something that I can identify it with - a la Burnt Sienna or Iceberg Blue. The color might have a certain structure from the get-go, but I might experiment with the structure along the way, while I’m still referring to it as the same color.

Imagine that I’m implementing a tool to manage the car rental’s fleet, and that the CEO told me that color is one of the specifications that seems to matter a lot to their customers. The list of available colors is rather limited though; black, dark gray, and blue. Yet the CEO insists on managing this collection by herself; this should avoid having to call in another expensive consultant a few years down the road.

Color as an entity

So we define a collection of colors that will be persisted by NHibernate. Since NHibernate, and our relational database don’t play nice without a primary key, we add an identifier to the colors.

We end up with two classes; one entity that defines a color, and another entity that defines a car. A car references a color.

public class Car
{
    public Car(Color color) 
    {
        if (color == null)
            throw new ArgumentNullException("color");

        Color = color;
    }

    protected Car() { }

    public virtual int Id { get; protected set; }

    public virtual Color Color { get; protected set; }

    public override bool Equals(object obj)
    {
        if (obj == null)
            return false;

        var car = obj as Car;

        if (car == null)
            return false;

        return car.Id == Id;
    }

    public override int GetHashCode()
    {
        return Id.GetHashCode();
    }
}

public class Color
{
    public Color(string name, string hexadecimalNotation)
    {
        if (string.IsNullOrEmpty(name))
            throw new ArgumentNullException("name");
        if (string.IsNullOrEmpty(hexadecimalNotation))
            throw new ArgumentNullException("hexadecimalNotation");

        Name = name;
        HexadecimalNotation = hexadecimalNotation;
    }

    protected Color() { }

    public virtual int Id { get; protected set; }

    public virtual string Name { get; protected set; }

    public virtual string HexadecimalNotation { get; protected set; }

    public override bool Equals(object obj)
    {
        if (obj == null)
            return false;

        var color = obj as Color;

        if (color == null)
            return false;

        return color.Id == Id;
    }

    public override int GetHashCode()
    {
        return Id.GetHashCode();
    }
}

public class CarClassMap : ClassMap<Car>
{
    public CarClassMap()
    {
        Id(x => x.Id).GeneratedBy.HiLo("10");

        References(x => x.Color);          
    }
}

public class ColorClassMap : ClassMap<Color>
{
    public ColorClassMap()
    {
        Id(x => x.Id).GeneratedBy.HiLo("10");

        Map(x => x.Name).Length(30).Not.Nullable();

        Map(x => x.HexadecimalNotation).Length(7).Not.Nullable();
    }
}

The generated schema looks like this.

And while this looks innocent at first, accidentally creating an entity raises a bunch of new concerns and questions. What happens if a color is no longer available, and the CEO wants to remove it from the collection? Does that mean we should delete all models that came in this color? No, those colors still exist, we’re not going to repaint all the vehicles; those colors just aren’t available anymore. This hints towards a concept that might be missing.

Fighting symptoms

We see the CEO heading over to the cafeteria, so we jump up, and ask her whether it makes sense for her to mark those colors as unavailable, instead of deleting them. After a short delay she shrugs and replies: “Well, I could do that if that makes things easier for you.” We go ahead and model our solution to reflect this new information.

public class Color
{
    public Color(string name, string hexadecimalNotation)
    {
        if (string.IsNullOrEmpty(name))
            throw new ArgumentNullException("name");
        if (string.IsNullOrEmpty(hexadecimalNotation))
            throw new ArgumentNullException("hexadecimalNotation");

        Name = name;
        HexadecimalNotation = hexadecimalNotation;
        Available = true;
    }

    protected Color() { }

    public virtual int Id { get; protected set; }

    public virtual string Name { get; protected set; }

    public virtual string HexadecimalNotation { get; protected set; }
    
    public virtual bool Available { get; protected set; }
    
    public virtual void MakeUnavailable() 
    {
        Available = false;
    }

    public override bool Equals(object obj)
    {
        if (obj == null)
            return false;

        var color = obj as Color;

        if (color == null)
            return false;

        return color.Id == Id;
    }

    public override int GetHashCode()
    {
        return Id.GetHashCode();
    }
}

A few days later we show off what we came up with to the CEO. She looks content with what we built over the last few days, until we show her the user interface that manages the colors. “I’m rather busy so I often make mistakes when I take care of these administrative tasks, could you add a button to really delete a color from this list anyway?” This brings us back to square one. First thing we think about is soft deleting the colors. We could also only make it possible to remove a color if it hasn’t been referenced yet. A voice in the back of our heads keeps telling us that we must be missing something though, and that this seems to be harder than it should be. A few hours later, driving home after a tough day, it becomes obvious that the CEO really thinks of a color as a value instead of an entity, so we should really be modeling it as such.

Color as a component

Luckily, NHibernate makes this pretty simple. The next day, we arrive early at the office, and change our mapping to use a component, so that instead of the car referencing a color, we store the value, and lose the id.

public class CarClassMap : ClassMap<Car>
{
    public CarClassMap()
    {
        Id(x => x.Id).GeneratedBy.HiLo("10");

        Component(
            x => x.Color,
            m =>
            {
                m.Map(x => x.Name).Column("ColorName").Length(30).Not.Nullable();
                m.Map(x => x.HexadecimalNotation).Column("ColorHex").Length(7).Not.Nullable();
            });
    }
}

The generated schema now looks like this; we’re no longer referencing the Color table.

When we store a color now, it’s an entity. But as soon as we put it on a car, it’s a value object. When we pull the car back out of our persistence store, we have lost the identity. We should modify our code so that the color’s behaviour reflects these changes. We modify the default constructor so that the object gets initialized with a default identifier. The default constructor will get used by NHibernate when it hydrates the object after getting it out the persistence store. We override the Equals and GetHashCode methods so that the identifiers are compared when there’s an identity, but when the identifier isn’t hydrated, the values are compared.

public class Color
{
    public Color(string name, string hexadecimalNotation)
    {
        if (string.IsNullOrEmpty(name))
            throw new ArgumentNullException("name");
        if (string.IsNullOrEmpty(hexadecimalNotation))
            throw new ArgumentNullException("hexadecimalNotation");

        Name = name;
        HexadecimalNotation = hexadecimalNotation;
    }

    protected Color() 
    {
        Id = -1;
    }

    public virtual int Id { get; protected set; }

    public virtual string Name { get; protected set; }

    public virtual string HexadecimalNotation { get; protected set; }

    public override bool Equals(object obj)
    {
        if (obj == null)
            return false;

        var color = obj as Color;

        if (color == null)
            return false;

        if (Id != -1)
            return Id == color.Id;

        return Name == color.Name && HexadecimalNotation == color.HexadecimalNotation;
    }

    public override int GetHashCode()
    {
        if (Id != -1)
            return Id.GetHashCode();

        return Name.GetHashCode() & HexadecimalNotation.GetHashCode();
    }
}

This feels off though; using one concept in two different contexts makes things rather confusing. Is color an entity, or value object? Or does it depend?

Separate concepts

We extract two explicit concepts instead; a color as a value object, and an available color as an entity.

public class AvailableColor 
{
    public AvailableColor(string name, string hexadecimalNotation)            
    {
        if (string.IsNullOrEmpty(name))
            throw new ArgumentNullException("name");
        if (string.IsNullOrEmpty(hexadecimalNotation))
            throw new ArgumentNullException("hexadecimalNotation");

        Name = name;
        HexadecimalNotation = hexadecimalNotation;
    }

    protected AvailableColor()
    {
    }

    public virtual int Id { get; protected set; }

    public virtual string Name { get; protected set; }

    public virtual string HexadecimalNotation { get; protected set; }

    public static explicit operator Color(AvailableColor value)
    {
        return new Color(value.Name, value.HexadecimalNotation);
    }  

    public override bool Equals(object obj)
    {
        if (obj == null)
            return false;

        var color = obj as AvailableColor;

        if (color == null)
            return false;

        return color.Id == Id;
    }

    public override int GetHashCode()
    {
        return Id.GetHashCode();
    }
}

public class Color
{
    public Color(string name, string hexadecimalNotation)
    {
        if (string.IsNullOrEmpty(name))
            throw new ArgumentNullException("name");
        if (string.IsNullOrEmpty(hexadecimalNotation))
            throw new ArgumentNullException("hexadecimalNotation");

        Name = name;
        HexadecimalNotation = hexadecimalNotation;
    }

    protected Color()
    {
    }

    public virtual string Name { get; protected set; }

    public virtual string HexadecimalNotation { get; protected set; }

    public override bool Equals(object obj)
    {
        if (obj == null)
            return false;

        var color = obj as Color;

        if (color == null)
            return false;

        return Name == color.Name && HexadecimalNotation == color.HexadecimalNotation;
    }

    public override int GetHashCode()
    {
        return Name.GetHashCode() & HexadecimalNotation.GetHashCode();
    }
}

This looks better. We now have two explicit concepts. An explicit conversion allows you to get a color out of an available color, losing the identifier.

var availableColorOrange = new AvailableColor("Orange", "#CC3232");
var car = new Car((Color)availableColorOrange);                    
                    
Console.WriteLine(car.Color.Equals(new Color("Orange", "#CC3232"))); // true

Conclusion

We meet up with the CEO one last time, and show her what we reworked. When we demo how she can manage the collection of available colors by just adding and deleting them - without caring whether one of the cars came in that color, a smile shows up on her face; “This is exactly what I needed, it really shouldn’t be harder than this.”

Tools often trick us into creating entities. These accidental entities then go on to introduce expensive coupling, introducing questions and problems that could easily be avoided by copying values around instead.

Next week: but what about the UI?

Validating composite models with knockout validation

Mon, 20 May 2013 17:02:00 +0200

When you use knockout validation to extend observables with validation rules, it will add a few functions to these observables - the most important ones being; error and isValid. You can use these functions to verify if any of the validation rules were violated, and to extract an error message.

To extract all of the error messages out of a composite model, you can use a grouping function.

function BookingModel() {      
    var self = this;
    
    self.contact = new ContactModel();
    self.departure = new DepartureModel();
    
    self.isValid = function() {
        return self.contact.isValid() && self.departure.isValid();
    };
    
    self.validate = function() {                           
        if (!self.isValid()) {
            var errors = ko.validation.group(self);                           
            errors.showAllMessages();
        
            return false;          
        }

        return true;
    };                    
}

function DepartureModel() {
    this.street = ko.observable('').extend({ required: true });
    this.houseNumber = ko.observable('').extend({ required: true });
    this.city = ko.observable('').extend({ required: true });
    this.time = ko.observable('').extend({ required: true });            
    
    this.isValid = function() {
        return 
            this.street.isValid() &&
            this.houseNumber.isValid() &&
            this.city.isValid() &&
            this.time.isValid();                
    };
}

function ContactModel() {
    this.firstName = ko.observable('').extend({ required: true });
    this.lastName = ko.observable('').extend({ required: true });
    this.phoneNumber = this.firstName = ko.observable('').extend({ required: true });
    this.email = ko.observable('').extend({ required: true });   

    this.isValid = function() {
        return 
            this.firstName.isValid() &&
            this.lastName.isValid() &&
            this.phoneNumber.isValid() &&
            this.email.isValid();                
    };    
}

This is what my first attempt looked like. A little later I discovered that you can get rid of these boilerplate functions on the composite model by applying the validatedObservable function instead.

ko.applyBindings(ko.validatedObservable(new BookingModel()));

ko.validatedObservable = function (initialValue) {
    if (!exports.utils.isObject(initialValue)) { 
        return ko.observable(initialValue).extend({ validatable: true }); }

    var obsv = ko.observable(initialValue);
    obsv.errors = exports.group(initialValue);
    obsv.isValid = ko.computed(function () {
        return obsv.errors().length === 0;
    });

    return obsv;
};

The validatedObservable function will add an errors function to the composite model which traverses the object graph and validates each eligible property. The errors function also has a showAllMessages function that will display an error message next to each invalid element. The isValid function only asserts if there are any errors.

function BookingModel() {      
    var self = this;
    
    self.contact = new ContactModel();
    self.departure = new DepartureModel();

    self.validate = function() {                           
        if (!self.isValid()) {                                         
            self.errors.showAllMessages();
        
            return false;          
        }

        return true;
    };             
}   

function DepartureModel() {
    this.street = ko.observable('').extend({ required: true });
    this.houseNumber = ko.observable('').extend({ required: true });
    this.city = ko.observable('').extend({ required: true });
    this.time = ko.observable('').extend({ required: true });            
}

function ContactModel() {
    this.firstName = ko.observable('').extend({ required: true });
    this.lastName = ko.observable('').extend({ required: true });
    this.phoneNumber = this.firstName = ko.observable('').extend({ required: true });
    this.email = ko.observable('').extend({ required: true });        
}

Removing that cruft results in less bulky, cheaper models.

If you try this example, you will notice that the model appears to be valid even though the validation rules are clearly violated. It took me a few minutes of browsing the source to figure out why this was happening. When you use the group functions to validate your model, they will by default only look at first level properties. So if you have a composite model, you need to modify the grouping validation configuration, and set the deep property to true.

ko.validation.init({ grouping : { deep: true, observable: true } });

IDDD Tour notes (2/2)

Sun, 12 May 2013 15:44:00 +0200

This is the second and last part of my notes I scribbled down attending the IDDD Tour. The first part was published last week.

A better model

Even if you come up with a better model, the fact that it has been the ubiquitous language of the domain for decades proves that it works for them.

This quote bothers me a bit. There definitely is truth to this, but modeling an existing process often presents such a great opportunity to revise and improve it. Naked models don’t conceal deficiencies, inefficiencies and aberrations. Exploring alternative models free of habituation, politics and legacy is dirt cheap, while the outcome could considerably benefit all. It seems such a shame not to take advantage of this. As with most things, know when to pick your fights.

Elegance

Elegance is for dressing, not for delivering software.

This is one to remember; I’ll be using this one next time someone uses elegance as an argument for gold plating.

Cultivating models

Models grow; you will never have the best model from the start. Improve them every time you pass by.

Your first attempt at it is hardly ever right. Don’t beat yourself up over it. The best models are the result of multiple iterations. In general I consider dwelling on one problem too long to be a waste of time. Settle for good enough, and allow the better solution to emerge by itself over time. When I feel a design is mighty important, I might accelerate this process by iterating over it multiple times in just a few days; asking for constant feedback, carrying the problem with me everywhere I go, always challenging it, and molding it in different shapes until one sticks.

Reuse

Lots of people use a shared kernel just for reuse. It’s often not worth it.

People are so obsessed with the DRY principle, and the dogma of avoiding duplication, it often does more harm than good. Nonexistent concepts are introduced just to spare a few duplicate lines of code, while they will hinder and complicate autonomous evolution.

REST and DDD

Don’t expose your domain model over REST: expose use cases.

You want to enable relentlessly evolving your domain model without breaking clients. In practice you would post intent-revealing command resources, while hypermedia guides you in navigating to subsequent commands.

Published language

How is the language agreed upon; over lunch or by a committee?

A great question which reveals if a language should be really considered as published.

And that’s the last of it. If you thought some of these were interesting, you should probably get the book. I finished Growing Object-Oriented Software Guided by Tests last week, so I finally get to read it myself after having it collect dust for two months.

IDDD Tour notes (1/2)

Sun, 05 May 2013 17:10:00 +0200

Two weeks ago I got to spend four days attending the IDDD Tour by Vaughn Vernon. Although my book queue has only allowed me to shallowly browse the book, I had high hopes for this course. I anticipated a week of getting lectured on DDD with a few practical exercises, but was blown away by the openness and interaction promoted by Vaughn and his associate Alberto Brandolini. A passionate group, engaging workshops, long days and lots of sharing made these few days exceptionally satisfying and inspirational. I’m grateful to those who got this show on the road; it was more than worth your trouble.

Over these few days I scribbled down some things I thought were worth remembering or worth some more thought. Most of these are aimed at the strategic side of DDD and the softer side of software since that’s where I acquired most new insights.

Complexity

If we really understood the complexity of a system, we would very often make different decisions from when designing new systems.

Understanding the complexity of a new system is hard. Modeling can serve as the perfect tool to help you expose the hidden complexities as early as possible.
Often a system appears to be not much more than CRUD, to quickly evolve into something with a lot more behaviour than anticipated. Make sure you don’t sabotage growing your design by picking an architecture or framework that doesn’t allow you to.
I often start with just exposing commands and queries to my application, which encapsulate all domain logic. Behind these command- and queryhandlers is very little abstraction, just some basic building blocks; aggregates, entities and value objects. Later on I can relentlessly refactor everything behind these commands and queries without breaking application code: introduce domain services, events, separate the write- from the read model etc…

Change

How do you bring in DDD? Gather a small cluster of motivated silent people and work on a project that has lots of potential value.

This is my recipe for change as well. Find a few like-minded people that are confident drawing outside the lines set out by the status quo, work in the shades, working your way towards the light.

Brown

There is always brown, even when it’s under the green.

Every application has parts which are less pretty, even green fields; use proper encapsulation to contain them, and avoid them corrupting other parts of the system.

Another brick in the wall

Break down that paper wall between domain experts and developers. Learn the favorite coffee of the domain expert.

Lots of organizations still build this paper wall between domain experts and developers. But paper doesn’t answer to questions, nor does it carry nuances very well. Communicating on a more personal level is key to tearing down that wall. Breaking the ice can be as easy as treating your domain expert to coffee.

One language

Which language do you use when naming things in code? Yours.

I don’t think native English speakers understand just how awkward it is to write code in something other than English. Keywords, API’s, documentation… it’s all in English. I feel that putting these together with your own language results in a cacophony of words.
In my current project the business is bilingual; they use Dutch and French names for the same concept interchangeably. We tried to find middle ground by introducing English names, and have a map between these languages. Ideally there would only be one language though.
Someone mentioned that they use English for all globally unambiguous concepts such as account number, name… but use their mother tongue in scenarios where nuances would be lost in translation. This seems to be an attractive compromise.

Defining a bounded context

A bounded context is where one ubiquitous language is consistent.

This is by far the simplest definition of a bounded context.

Bounded context relations

You say please, so I am upstream.

The relationship between bounded contexts is expressed by saying one is upstream or downstream. I thought of the quote above to be a creative expression that can aid in explaining these two concepts.

Rewrites

When you rewrite an application, you have to look at the existing forces in an organization; chances are the previous team wasn’t terrible either.

Tons of factors influence the outcome of a project; it’s not just about teams and their skillset. Some forces in an organization can make it close to impossible to ship good software; fear of change, lack of vision, bone-dry information streams… So don’t assume your rewrite will fix everything.

Surprise

Adding surprise value is often not that valuable.

This seems to be related to the mantra of ‘good enough’. Sometimes we - with the best intentions - invest more time and money in solving a problem in the best possible way, than it will ever return.

These were some notes I wrote down the first two days. I’ll try to go through the other half over these next few days. I hope you found some of them valuable too.

Designing entities: immutability first

Sun, 07 Apr 2013 17:35:00 +0200

The first year I wrote software for a living I spent my days mostly writing forms over data applications; most of my efforts were wasted just trying to make things work using ASP.NET and the Webforms engine. It was only after a year and graduating from the School of Hard Knocks that I learned there is a lot more to building clean and maintainable software than knowing the ins’ and outs’ of a proprietary UI technology.

One of the habits I have adapted over time, is designing new entities as immutable objects first, and to go from there. I believe this helps me to make more deliberate design decisions, leading to better designs.

Allow me to demonstrate this using a customer entity - I know this is cliché, but I have a hard time coming up with original examples.

A constructor

This one might sound rather obvious, but you would be surprised how many classes don’t have the decency of a simple constructor, especially since C# introduced object initializers. While object initializers definitely look good, they make your code harder to maintain; classes don’t communicate what they need to get by, nor do they get the opportunity to protect their invariants as early as possible.

Like I said, I design my classes to be immutable from the start; after initialization they can’t be modified. I’m not even exposing any state for now.

public class Customer
{
    public Customer(
        Guid id,
        string firstName,
        string lastName,
        string street,
        string city,
        string country,
        DateTime birthdate,
        decimal balance)
    {
        FirstName = firstName;
        LastName = lastName;
        Street = street;
        City = city;
        Country = country;
        BirthDate = birthdate;
        Balance = balance;
    }

    public Guid Id { get; private set; }
    private string FirstName { get; set; }
    private string LastName { get; set; }
    private string Street { get; set; }
    private string City { get; set; }
    private string Country { get; set; }
    private DateTime BirthDate { get; set; }
    private decimal Balance { get; set; }
}

Comparing identities

An entity is always unique within a system; it has an identity. To encapsulate comparing two entities, and to avoid repeating myself, I always override the Equals and GetHashCode methods. We’re not guided into doing this by our object’s immutability though, but by common sense.

public override bool Equals(object obj)
{
    if (obj == null || GetType() != obj.GetType())
        return false;            

    var other = obj as Customer;
        
    return Id == other.Id;
}

public override int GetHashCode()
{
    return this.Id.GetHashCode();
}

Defaults

When a new customer is created, its balance is still empty. The constructor is the perfect place to initialize this default.

public Customer(
    Guid id,
    string firstName,
    string lastName,
    string street,
    string city,
    string country,
    DateTime birthdate)
{
    FirstName = firstName;
    LastName = lastName;
    Street = street;
    City = city;
    Country = country;
    BirthDate = birthdate;
    Balance = 0;
}

Protecting invariants

The next thing I’ll do is use the constructor to guard for arguments that don’t satisfy my objects invariants. This way we guarantee that the object can never be initialized with an invalid state; avoiding a bunch of trouble along the way.

We expect a customer to have a first name, last name, street, city and country which isn’t empty. We also expect its birth date not to be in the future.

public Customer(
    Guid id,
    string firstName,
    string lastName,
    string street,
    string city,
    string country,
    DateTime birthdate)
{
    Guard.ForNullOrEmpty(firstName, "first name");
    Guard.ForNullOrEmpty(lastName, "last name");
    Guard.ForNullOrEmpty(street, "street");
    Guard.ForNullOrEmpty(city, "city");
    Guard.ForNullOrEmpty(country, "country");
    Guard.ForDatesInTheFuture(birthdate, "birthdate");

    Id = id;
    FirstName = firstName;
    LastName = lastName;
    Street = street;
    City = city;
    Country = country;
    BirthDate = birthdate;            
    Balance = 0;
}

public class Guard
{
    public static void ForNullOrEmpty(string value, string desc)
    {
        if (string.IsNullOrEmpty(value))
            throw new NullReferenceException(desc);
    }

    public static void ForDateInTheFuture(DateTime value, string desc)
    {
        if (value > DateTimeProvider.Now())
            throw new ArgumentException(
                string.Format("{0} can't be a date in the future.", desc))
    }
}

Composition and extracting value objects

I’m fairly confident that every self-respecting programmer is annoyed by the fact that the obvious concepts name and address haven’t been extracted yet. Especially since that bloated constructor is all up in your face. Let’s go ahead and extract two value objects: name and address.

public class Customer
{
    public Customer(
        Guid id,
        Name name,
        Address address,
        DateTime birthdate)
    {
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");
        Guard.ForDateInTheFuture(birthdate, "birthdate");

        Id = id;
        Name = name;
        Address = address;
        Balance = 0;
    }

    public Guid Id { get; private set; }
    private Name Name { get; set; }
    private Address Address { get; set; }
    private DateTime BirthDate { get; set; }
    private decimal Balance { get; set; }
}

public class Name
{
    public Name(string first, string last)
    {
        Guard.ForNullOrEmpty(first, "first name");
        Guard.ForNullOrEmpty(last, "last name");

        First = first;
        Last = last;
    }

    private string First { get; set; }
    private string Last { get; set; }

    public string Full 
    {
        get
        {
            return string.Format("{0} {1}", First, Last);
        }
    }
    
    public override bool Equals(Object obj)
    {
        ...
    }
    
    public override int GetHashCode()
    {
        ...
    }
}

public class Address
{
    public Address(string street, string city, string country)
    {
        Guard.ForNullOrEmpty(street, "street");
        Guard.ForNullOrEmpty(city, "city");
        Guard.ForNullOrEmpty(country, "country");

        Street = street;
        City = city;
        Country = country;
    }

    private string Street { get; set; }
    private string City { get; set; }
    private string Country { get; set; }
    
    public override bool Equals(Object obj)
    {
        ...
    }
    
    public override int GetHashCode()
    {
        ...
    }
}

There, that looks better.

Tell, don’t ask

In good OOP we tell objects what to do, we don’t want to query an object for its state and do something based on that; that would lead to broken encapsulation and thereby more brittle code.

Since our immutable object isn’t exposing any state yet, we are forced to ask ourselves why we want to expose something. This way, disregarding “tell don’t ask” becomes a deliberate choice.

Let’s say we want to decrease the balance, and throw an exception when funds are insufficient. Instead of asking the customer instance and acting on that..

if (!customer.HasSufficientFunds())
    throw new InsufficientFundsException();

It’s better to encapsulate data and behaviour, and tell it what to do instead.

public class Customer
{
    public Customer(
        Guid id,
        Name name,
        Address address,
        DateTime birthdate)
    {
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");
        Guard.ForDateInTheFuture(birthdate, "birthdate");

        Id = id;
        Name = name;
        Address = address;
        Balance = 0;
    }

    public Guid Id { get; private set; }        
    private Name Name { get; set; }
    private Address Address { get; set; }
    private DateTime BirthDate { get; set; }
    private decimal Balance { get; set; }

    private bool HasSufficientFunds(decimal value) 
    {
        return Balance - value >= 0;
    }

    public void DecreaseBalance(decimal value)
    {
        if (HasSufficientFunds(value))
            throw new InsufficientFundsException();

        Balance -= value;
    }
}

Making operations explicit

Next thing we want to do is change the address of a customer. Since we’re not exposing the address property directly, we’re also forced to think about a useful name for the operation; I decided to name it MoveTo. With this we make things more explicit, and thus more evident for people who are new to the domain.

public class Customer
{
    public Customer(
        Guid id,
        Name name,
        Address address,
        DateTime birthdate)
    {
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");
        Guard.ForDateInTheFuture(birthdate, "birthdate");

        Id = id;
        Name = name;
        Address = address;
        Balance = 0;
    }

    public Guid Id { get; private set; }    
    private Name Name { get; set; }
    private Address Address { get; set; }
    private DateTime BirthDate { get; set; }
    private decimal Balance { get; set; }

    ...

    public void MoveTo(Address newAddress)
    {
        Guard.ForNull(newAddress, "new address");

        Address = newAddress;
    }
}

var customer = new Customer(
    Guid.NewGuid(),
    new Name("Jef", "Claes"),
    new Address("Ergens", "Antwerp", "Belgium"),
    new DateTime(1987, 10, 18));

customer.MoveTo(new Address("Quelque part", "Brussels", "Belgium"));

Opening up bit by bit

In the meanwhile our entity is no longer immutable, but we’re doing a good job of not exposing any state. If you’re using the same model for reads as you do for writes, you’re going to need to expose some properties eventually; for querying, but also for assertions. Make sure to only expose the getters though, and to avoid violating “Tell, don’t ask” since that became a lot easier now.

public class Customer
{
    public Customer(
        Guid id,
        Name name,
        Address address,
        DateTime birthdate)
    {
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");
        Guard.ForDateInTheFuture(birthdate, "birthdate");

        Id = id;
        Name = name;
        Address = address;
        Balance = 0;
    }

    public Guid Id { get; private set; }    
    public Name Name { get; private set; }
    public Address Address { get; private set; }
    public DateTime BirthDate { get; private set; }
    private decimal Balance { get; set; }

    private bool HasSufficientFunds(decimal value) 
    {
        return Balance - value >= 0;
    }

    public void DecreaseBalance(decimal value)
    {
        if (HasSufficientFunds(value))
            throw new InsufficientFundsException();

        Balance -= value;
    }

    public void MoveTo(Address newAddress)
    {
        Guard.ForNull(newAddress, "new address");

        Address = newAddress;
    }
}

Summarized

By making my entities immutable from the start I am guided into adhering to a bunch of good practices:

A constructor
Comparing identities
Defaults
Protecting invariants
Composition and extracting value objects
Tell, don’t ask
Making operations explicit
Opening up bit by bit

Reading large files in chunks with proper encapsulation

Sun, 24 Mar 2013 18:16:00 +0100

I’ve been doing some work lately which involves sequentially reading large files (> 2 to 5GB). This entails that it’s not an option to read the whole structure in memory; it’s more reliable to process the file in chunks. I occasionally come across legacy that solves exactly this problem, but in a procedural way, resulting in tangled spaghetti. To be honest, the first piece of software I ever wrote in a professional setting also went at it in the wrong way.

There is no reason to let it come to this though; you can use the often overlooked yield return keyword to improve encapsulation.

When you use the yield keyword in a statement, you indicate that the method, operator, or get accessor in which it appears is an iterator. You consume an iterator method by using a foreach statement or LINQ query. Each iteration of the foreach loop calls the iterator method. When a yield return statement is reached in the iterator method, expression is returned, and the current location in code is retained. Execution is restarted from that location the next time that the iterator function is called.

Have a look at the following Reader class which takes advantage of yield returning. This class reads from file, line by line, building a chunk, to return it when the desired chunk size is attained. In the next iteration, the call will continue by clearing the lines - thereby releasing memory, and rebuilding the next chunk.

public class Reader
{
    private int _chunkSize;

    public Reader(int chunkSize) 
    {
        _chunkSize = chunkSize;
    }

    public IEnumerable<Chunk> Read(string path)
    {
        if (string.IsNullOrEmpty(path))
            throw new NullReferenceException("path");

        var lines = new List<string>();

        using (var reader = new StreamReader(path))
        {
            string line;
            while ((line = reader.ReadLine()) != null)
            {
                lines.Add(line);

                if (lines.Count == _chunkSize)
                {
                    yield return new Chunk(lines);

                    lines.Clear();
                }
            }                
        }

        yield return new Chunk(lines);
    }
}

public class Chunk
{
    public Chunk(List<string> lines) 
    {
        Lines = lines;
    }

    public List<string> Lines { get; private set; }
}

And that’s one way to achieve clean encapsulation without starving your machine’s memory.

var reader = new Reader(chunkSize: 1000);
var chunks = reader.Read(@"C:\big_file.txt");

foreach (var chunk in chunks)            
    Console.WriteLine(chunk.Lines.Count);

Putting my IronMQ experiment under stress

Sun, 17 Mar 2013 16:10:00 +0100

Two weeks ago, I shared my first impressions of IronMQ. Last week, I looked at some infrastructure to facilitate pulling from IronMQ. This implementation worked, but I hadn’t put it under stress yet; “First make it work, then make it fast”, and all of that.

I arranged a simple scenario for testing: one message type - thus one queue, where there are eight queue consumers that simultaneously pull messages from that queue, and dispatch them to a handler which sleeps for one second.

public class MessageSleepForOneSecond { }

public class MessageSleepForOneSecondHandler : IMessageHandler<MessageSleepForOneSecond>
{
    public void Handle(MessageSleepForOneSecond message)
    {
        Thread.Sleep(1000);
    }
}

To establish a baseline, I foolishly set the polling interval to only 100ms, and pulled 2000 messages from the queue one at a time. With this configuration I processed all 2000 messages in 2 minutes and 20 seconds, with an average throughput of 14.3 messages per second. In theory you would expect the throughput to be higher though.

The constraint in this story is the CLR’s thread pool. Every time a queue consumer’s internal timer ticks, the callback which pulls from the queue and invokes the messagehandler, takes up a new thread on the thread pool. The thread pool makes a few threads available when you start your application, but once they’re all in use, it will have to start new ones, which is rather expensive. More importantly though, when you’re queuing too many tasks on the thread pool, and the number of active threads is higher than the number of processors, it will slow down, and wait 500ms to see if it can reuse the existing threads, before creating a new one. When the maximum number of threads is reached, the thread pool will still enlist your tasks in its queue, but only start processing them once threads become available again. In short, the thread pool has a few tricks up its sleeve to protect you from saturating your resources. Remember that too much parallelization and its corresponding context switches won’t do you any good.

Having established a baseline, and having learned a bit more on how the thread pool behaves, I tried one of the first optimizations I already had in mind last week; pulling batches instead of single messages. This reduces the number of necessary HTTP requests, and the number of threads needed to do work on. To support this, I extended the queue consumer configuration with a new property, and changed the queue consumer to take the batch size into account.

public interface IQueueConsumerConfiguration<T>
{
    int PollingInterval { get; }
    int BatchSize { get; }
}

try
{
    var messages = (IEnumerable<Message>)null;

    if (!_queue.TryGet(out messages, _queueConsumerConfiguration.BatchSize))
        return;

    foreach (var message in messages)
    {
        try
        {
            var messageBody = (T)JsonConvert.DeserializeObject(message.Body, typeof(T));

            _messageDispatcher.Dispatch<T>(messageBody);

            _queue.Delete(message.Id);
        }
        catch (Exception ex)
        {
            _errorHandler.Handle(ex, message);
        }
    }
}
catch (Exception ex)
{
    _errorHandler.Handle(ex, null);
}

On repeating the test with 2000 messages, the same polling interval of 100ms, but with a batch size of 30, the messages were now all processed in one minute and fifteen seconds, resulting in a throughput of 26 messages per second. That’s almost an improvement of 100%.

This throughput isn’t sustainable though if we had a lot more messages to process. We’re starting a new thread every 100ms or 500ms, while the work we are doing on it only finishes after a rough 30 seconds (it’s not only invoking the handlers, but the HTTP requests also take time). We’re burning through threads quicker than we’re releasing them. If we would run out of threads on the thread pool, it would just stop starting new ones, and queue the tasks until other threads are done doing work.

In my previous post I also considered a smart polling algorithm, but I haven’t looked at that yet, what’s in place is more than good enough for me at the moment.

Be sure to take these numbers with a grain of salt. I would have to test my infrastructure with millions of messages on the queue instead of just 2000 to get trustworthy results. I feel I can predict fairly well how the system will behave when put under load for a longer amount of time though; it would grind to a halt. As mentioned before, we would run out of threads to do work on. I simulated this by lowering the thread pool’s maximum number of threads. Other parameters that influence the numbers in this test are: size of the messages, version of the runtime, the operating system, the amount of processors, latency of the network… I ran these tests with empty messages, .NET 4 installed on my own Windows 7 box with an Intel i7 on board.

It comes down to cherry picking a configuration per queue consumer that will be sustainable based on the amount of messages you expect, the desired throughput, and the time it takes to process a single message.

Some experimental infrastructure for IronMQ pull

Sun, 10 Mar 2013 17:29:00 +0100

I wrote about using IronMQ as a cloud-based message queue last week. In that post I explained that you can go at using IronMQ in two ways; either you pull from the queue yourself, or you let IronMQ push messages from the queue to your HTTP endpoints. At first sight, the latter allows you to outsource more infrastructure to their side, but upon closer inspection it also introduces other concerns: security, local debugging and scalability.

Out-of-the-box there is no infrastructure in the client libraries to facilitate periodic pull - polling, that’s why I took a stab at doing it myself. It’s still rough, not production tested, and hasn’t considered a bunch of niche scenarios, but it should give you an idea of the direction it’s going.

A high-level overview of the components looks like this.

A queue host hosts multiple queue consumers. Each queue consumer will poll a queue for one type of message on a configurable interval, and dispatch dequeued messages to relevant message handlers. After handling the message, the message will be deleted from the queue. If something happened to go wrong, the error handler for this type of message will be invoked, and the message will automatically return to the queue.

I’m going to look at each component, starting with the smallest, and slowly assemble them into bigger components.

Each queue consumer can be configured independently. For now, only the polling interval can be changed. By default it’s one second, or 1000 milliseconds.

public interface IQueueConsumerConfiguration<T>
{
    int PollingInterval { get; }
}

public class QueueConsumerConfiguration<T> : IQueueConsumerConfiguration<T>
{     
    public int PollingInterval
    {
        get { return 1000; }
    }       
}

A queue can push messages, get raw messages, and delete them. The implementation makes use of the OSS IronIO client libraries.

public interface IQueue<T>
{
    bool TryGet(out Message message);
    void Delete(string messageId);
    void Push(T message);
}

public class Queue<T> : IQueue<T>
{
    private readonly IronIO.IronMQ _queue;

    public Queue(string projectId, string token)
    {
        Guard.ForEmptyString(projectId, "projectId");
        Guard.ForEmptyString(token, "token");

        var queueName = typeof(T).Name; 

        _queue = new IronMQ(queueName, projectId, token);
    }

    public bool TryGet(out Message message) 
    {
        message = _queue.Get();            

        return message != null;
    }

    public void Delete(string messageId)
    {
        Guard.ForNull(messageId, "messageId");

        _queue.Delete(messageId);
    }
    
    public void Push(T message)
    {
        Guard.ForNull(message, "message");

        _queue.Push(JsonConvert.SerializeObject(message));
    }
}

A message dispatcher dispatches messages to the relevant handlers.

public interface IMessageDispatcher
{
    void Dispatch<T>(T message);
}

public class MessageDispatcher : IMessageDispatcher
{
    private readonly IKernel _kernel;

    public MessageDispatcher(IKernel kernel)
    {
        _kernel = kernel;            
    }

    public void Dispatch<T>(T message)
    {
        var handlers = _kernel.GetAll<IMessageHandler<T>>();
        
        foreach (var handler in handlers)
            handler.Handle(message);                        
    }
}

If something goes wrong pulling the message from the queue or handling it, the error handler will be invoked passing in the exception and the raw message. Since it’s possible that something is wrong with the message in itself, I pass in the raw message with the serialized message and all its meta data like id, delay and expiration date.

public interface IErrorHandler<T>
{
    void Handle(Exception exception, Message message);
}

public class ErrorHandler<T> : IErrorHandler<T>
{       
    public void Handle(Exception exception, Message message)
    {
        throw exception;
    }     
}

Putting all these components to work together, we end up with a queue consumer. When a queue consumer is started, it instantiates a timer which will try to get a raw message from the queue on each tick. If there’s a raw message, it will extract the body, deserialize it into the message, dispatch it, and finally delete the raw message from the queue. If something goes wrong here, the error handler will be invoked, and the message will automatically return back to the queue.

public interface IQueueConsumer<T> : IQueueConsumer where T : class
{
}

public interface IQueueConsumer : IDisposable
{
    void Start();
}

public class QueueConsumer<T> : IQueueConsumer<T> where T : class
{
    private readonly IQueue<T> _queue;
    private readonly IErrorHandler<T> _errorHandler;
    private readonly IQueueConsumerConfiguration<T> _queueConsumerConfiguration;
    private readonly IMessageDispatcher _messageDispatcher;
    private Timer _timer;        

    public QueueConsumer(
        IQueue<T> queue, 
        IErrorHandler<T> errorHandler,
        IQueueConsumerConfiguration<T> queueConsumerConfiguration,
        IMessageDispatcher messageDispatcher)
    {
        _queue = queue;
        _errorHandler = errorHandler;
        _queueConsumerConfiguration = queueConsumerConfiguration;
        _messageDispatcher = messageDispatcher;
    }    

    public void Start()
    {
        _timer = new Timer((x) =>
        {
            var message = (Message)null;
            var messageBody = (T)null;

            try
            {
                if (!_queue.TryGet(out message))
                    return;

                messageBody = (T)JsonConvert.DeserializeObject(
                    message.Body, typeof(T));

                _messageDispatcher.Dispatch<T>(messageBody);

                _queue.Delete(message.Id);
            }
            catch (Exception ex)
            {
                _errorHandler.Handle(ex, message);
            }                
        }, null, 0, _queueConsumerConfiguration.PollingInterval);            
    }

    public void Dispose()
    {
        if (_timer == null)
            return;

        _timer.Dispose();
    }
}

Since we have multiple queues to pull from, we can use a queue host to control multiple queue consumers at once. The queue host configuration decides which queue consumer to instantiate and start.

public class QueueHostConfiguration
{
    public QueueHostConfiguration(IEnumerable<Type> messageTypes)
    {
        Guard.ForNull(messageTypes, "messageTypes");

        MessageTypes = messageTypes;
    }

    public IEnumerable<Type> MessageTypes { get; private set; }
}

public class QueueHost : IDisposable
{
    private readonly IKernel _kernel;
    private readonly QueueHostConfiguration _configuration;

    private readonly List<IQueueConsumer> _consumers;

    public QueueHost(IKernel kernel, QueueHostConfiguration configuration)
    {
        _kernel = kernel;
        _configuration = configuration;

        _consumers = new List<IQueueConsumer>();
    }

    public void Start()
    {
        foreach (var messageType in _configuration.MessageTypes)
        {
            var queueConsumerType = typeof(IQueueConsumer<>).MakeGenericType(messageType);
            var queueConsumer = (IQueueConsumer)_kernel.Get(queueConsumerType);

            _consumers.Add(queueConsumer);

            queueConsumer.Start();
        }
    }

    public void Dispose()
    {
        if (_consumers == null)
            return;

        foreach (var consumer in _consumers)
            consumer.Dispose();
    }
}

In your application, you’ll end up doing something like this to start the queue host.

using (var host = new QueueHost(kernel, new QueueHostConfiguration(
    new[] { typeof(MyMessage) })))
{
    host.Start();

    Console.ReadLine();
}

All the components are glued together using Ninject and some conventions.

public class Bootstrapper
{
    public void Run(IKernel kernel)
    {
        kernel.Bind(x => x
            .FromThisAssembly()
            .SelectAllClasses()
            .InheritedFrom(typeof(IMessageHandler<>))
            .BindAllInterfaces());

        kernel.Bind<Infrastructure.IMessageDispatcher>()
            .To<Infrastructure.MessageDispatcher>()
            .InTransientScope();
        kernel.Bind(x => x
            .FromAssemblyContaining(typeof(IQueue<>))
            .SelectAllClasses()
            .InheritedFrom(typeof(IQueue<>))
            .BindAllInterfaces()
            .Configure(y =>
            {
                y.WithConstructorArgument("projectId", z => { return "your_project_id"; });
                y.WithConstructorArgument("token", z => { return "your_token"; });
            }));
        kernel
            .Bind(typeof(Infrastructure.IQueueConsumer<>))
            .To(typeof(Infrastructure.QueueConsumer<>));
        kernel
            .Bind(typeof(Infrastructure.IQueueConsumerConfiguration<>))
            .To(typeof(Infrastructure.QueueConsumerConfiguration<>));
        kernel
            .Bind(typeof(Infrastructure.IErrorHandler<>))
            .To(typeof(MyErrorHandler<>));    
    }
}

This is what I got for now. Next step is to do some more serious integration testing, and see what gives. There are two things I already kind of expect to run into; the maximum number of concurrent connections, and thread starvation (each timer tick starts a new thread). Anything else I’m going to run into?

The biggest disadvantage of opting for pull that is already obvious now, is the possible number of wasted HTTP requests. You could increase the polling interval, and thereby lower the number of requests, but this would harm the throughput of message bursts. Something I’m considering right now, is introducing a smart polling algorithm. Another option that will lower the number of requests, is to pull batches instead of single messages from the queue. Implementing this one will be rather straightforward, yet improve things considerably.

First IronMQ impressions

Sun, 03 Mar 2013 16:45:00 +0100

First time I touched messaging was in the first few years of my professional life working on software that supported fire departments in their day-to-day activities. The dispatching software would send messages to a proprietary broker, which in its turn would forward them to interested subscribers; other dispatching clients, or services. To ensure availability, the broker component could failover to a different machine, but that was about it. It didn’t allow you to queue or retry messages; if you weren’t up when the messages were forwarded, you would never receive them. When the brokers were both down, all messages would be lost; the clients didn’t have infrastructure out-of-the-box that could queue the messages locally until it came back up again. When things went haywire, and they occasionally did, these missing features would often leave us with an inconsistent state. More modern messaging software has solved these concerns though.

Although quite a few systems would benefit from asynchronous and loosely coupled messaging - especially to improve reliability and (perceived) performance, but also scalability, I still too seldom see or hear about projects that get to go that extra mile. Solutions often end up compromising in quality to avoid introducing that extra component and those unconventional questions. And this decision might be perfectly sound, because lots of factors are at play, not just technical ones. It’s still a pity when you see solutions struggle to solve a problem in a decent way because they’re stuck with synchronous communication.

Imagine a public website that’s in the business of booking hotels. The offers they show to their customers are all based on data provided by third parties. Because it’s so expensive to fetch this data, it’s being cached, and as a result, it’s stale seconds after fetching it. The moment a user confirms, they could fetch fresh data from the relevant third party to make sure the room is still available, but this process is error prone: the third party might be down, fetching the data is really slow, one of our their own components might be down, on conflicts they might want to compile a list of some decent alternatives, which in its turn might also be too slow if done on demand. One alternative could be to queue the booking, and process it in the back-end. Once they’re done processing it, they can mail the user a confirmation or an apology with a list of alternatives attached. By making this process asynchronous, they avoid the risk of a slow user experience and clumsy failures they can’t recover from, making them lose business in the long run. But then again, they also take the burden of extra infrastructure, new operational concerns and different questions. Trade-offs.

Anyways, in this blog post, I wanted to share some first impressions on IronMQ. I admittedly kind of accidentally discovered this service browsing through AppHarbor’s add-ons. Here it is described as “a scalable cloud-based message queue”.

Basically IronMQ gives you a REST enabled queue in the cloud. After authenticating, you can POST a new message to the queue. If you are unable to POST the message, you’ll lose it, since there is no support out-of-the-box for retrying or persisting the message on the client. Once the message is posted, another process can GET the message, and DELETE it after successfully processing it. If something happened to go wrong while processing the message, the message will return to the queue, and be retried one minute later. If the message processing keeps failing on subsequent retries, the retries won’t repeat infinitely though; messages expire (the default is 7 days, and the maximum is 30 days). This is exactly the kind of infrastructure we need to support the asynchronous booking scenario: have the customer put its booking on the queue, and one of our background processes will try to process it; if something goes wrong, we’ll just keep retrying for a while.

The REST API is simple, yet there are client libraries available for most popular languages. They don’t provide that much extra functionality though. Here’s the gist.

var queue = new IronMQ(queueName, projectId, token);
queue.Push("{ hello: world }");
var message = queue.Get();
queue.Delete(message.Id);

In this scenario you’re responsible for pulling data from the queue. This is just one way to go at things though; another option is to let IronMQ push messages to your HTTP endpoints. While this allows you to outsource some infrastructure to their side, it raises other concerns:

Security: you might need to enable HTTPS, and provide an authentication mechanism.
Debugging: if you want to do some end-to-end integration testing on your local machine, you’ll need to give your machine a public IP and set up something like dyndns.
Scalability: depending on the expected message volume, and the web stack you’re rolling with, it might be more expensive to have to set up all these web servers, instead of a few background workers.

Errors are handled quite elegantly; once you processed the message successfully, make your endpoint return HTTP status code 200 or 202, and the message will be removed from the queue. HTTP status code 202 is used for long running processes. If the response code is in the 400 or 500 range, the message will return to the queue to be retried later.

When the expected volume of messages is rather small, it makes more sense to opt for push; you don’t waste that many HTTP requests.

IronMQ makes it extremely simple to get started; go to their site, get a project id and a token, and start making HTTP calls. Do them yourself, or use one of the client API’s. But it also seems to be all you’re going to get; there is no infrastructure that addresses operational concerns, error queues, retry strategies, local queues,… IronMQ provides you with raw queueing infrastructure, not a framework.

Their site does give you a look into your queues though; you can’t look at the messages, but you do get a nice overview.

I don’t mind that it’s not a turnkey solution though; I learn a lot from tinkering with this stuff. Solving problems and considering trade-offs for yourself is priceless.

Using HTTP for messaging still feels a bit quirky to me. As long as I can remember people have been making me believe HTTP is not the best fit for high-throughput messaging scenarios, and I do understand their motivations somewhat. But when even databases start to embrace HTTP, it’s probably time to shake off the doctrine. It’s so comfy to not have to understand a new protocol, and HTTP just seems such a sensible thing to do when you’re off-premise.

I’m going to conduct some more experiments this week, I’ll see what gives.

My Christmas holiday project postmortem

Sun, 24 Feb 2013 16:42:00 +0100

Somewhere over a year and a half ago I discovered the music of Dire Straits, which has sparked a fanatical love and fascination for the guitar in me, and basically for every piece of music Mark Knopfler has ever touched (*). A year ago, I finally had the courage to pick up the guitar myself. Not sure if I’d stick with it, I made an uninformed purchase of a rather inexpensive Squier Jazzmaster, just because it somewhat resembled the real object of desire, a Fender Stratocaster. Three months ago, I got rid of the Jazzmaster, and bought myself a tango red Mexican Stratocaster, and I’m in love with it. Yet, I have also become very fond of the sound of a Les Paul these days. Having just bought the Strat, I thought I could maybe find a cheap used Les Paul online.

I set out to find one on the most popular Belgian online secondhand marketplace (600k visitors daily), so I started browsing their listings daily. This turned out to be rather cumbersome and inefficient: constantly repeating the same process, items were already sold before I could make a bid, there were no new items since the last time I visited… It didn’t take me long before I started thinking of a way to automate this dull process. Looking at other marketplaces, I found that some unburden their users by providing notifications; push instead of pull. Maybe I could build something similar?

Doing a bit of research over the weekend, I found out that they expose, what I call, an accidental API; written in first place to provide a snappy mobile user experience, not to expose all their data to third parties. Having an uncomplicated way of searching their data, I built a bit of code on top of it. Nothing all too fancy though; a Quartz job which periodically queries their service, parses the results and stores them in a RavenDB database. A few times a day, these search results are then compiled, and sent to my inbox using MailGun. All of this running for free on AppHarbor.

Showing all of this to the girlfriend, she asked if I could set up the same thing for her, but then for a specific type of camera. This is when I decided it could be useful enough to make it publicly available. I thought of a few ways to support the costs of hosting (which are extremely low): embed ridiculously relevant ads in the mails, make people pay for more frequent polling, or sell it (although very unlikely).

Two weeks later, I had something working online.

But how do I inform people of its existence? I first thought of using Twitter to monitor for relevant tweets where people ask for something secondhand. I started out by using a new dedicated generic account, but this got suspended rather quickly. In the second iteration, I used my personal account. This yielded better results; people saw I was human, and regularly thanked me for the tip. They didn’t sign up too often though..

Not willing to give up on the idea already, I used Google Adwords to advertise on the online marketplace directly. The results of this campaign were sobering, but extremely valuable; people just didn’t care. As a side note; secondhand seems to be quite an expensive keyword!

In hindsight, I can think of plenty of reasons why this somewhat useful project has no success:

The problem it is trying to solve obviously isn’t enough of a pain!
People don’t Google for it, and even if they did, this site had hardly any chance in making it to the first page.
In general the secondhand offering is enormous, and people often quickly settle for less.
Specialized markets aren’t situated on these sites.
People don’t like giving away their email address, definitely not to strangers they don’t trust.
Everyone struggles to keep their inboxes clean, receiving mail puts off people.

Damn, it’s always evident in hindsight.

Here are a few things I learned/got confirmed:

Being your own customer is priceless; you know exactly where the value is at.
Chances are you will never see a user; don’t spend too much time optimizing for scalability and reliability.
Some Google ads plus a website with a simple form can be enough to let you cheaply validate the worthiness of pursuing an idea.

(*) Take some time to explore his work: brothers in arms, postcards from Paraguay, song for Sonny Liston, you and your friend, and so many more are worth a listen.

Adding the R to CQS: some storage options

Sun, 17 Feb 2013 17:57:00 +0100

I’ve been writing quite a bit about CQS (or command and query separation) lately. In my last post on using events, I already hinted towards bringing in the R; command and query responsibility separation.

With CQS, commands can mutate data, while queries can only read that data. CQRS takes this one step further, and assigns commands and queries each a dedicated model; we now talk of a write side, and a read side.

I like Clemens Vasters definition best.

CQRS is a simple pattern that strictly segregates the responsibility of handling command input into an autonomous system from the responsibility of handling side-effect-free query/read access on the same system. Consequently, the decoupling allows for any number of homogeneous or heterogeneous query/read modules to be paired with a command processor. This principle presents a very suitable foundation for event sourcing, eventual-consistency state replication/fan-out and, thus, high-scale read access. In simple terms, you don’t service queries via the same module of a service that you process commands through. In REST terminology, GET requests wire up to a different thing from what PUT, POST, and DELETE requests wire up to.

A nice drawing also helps in understanding CQRS (from the CQRS journey material).

Although scalability seems to be one of the big selling points of CQRS, there are still some valid arguments applicable to my world; the strongest one being able to avoid the discrepancy which exists while you use the same model for reading and writing. I think everyone suffers from this one regularly. A popular and realistic example is this one; an ORM is used to map our domain model to a relational database, the tables are mapped very closely to the structure of the domain model. Not long after, it becomes evident that it’s impossible to write simple and performant queries targeting this datastructure. We could optimize for reads, but this would impact the complexity and performance of writing. With CQRS, reads and writes are segregated; we can now optimize both parts independently. And this doesn’t only result in being able to show a list faster on a user’s screen, but interesting things can also be done to empower reporting and data mining; think of how often using the same database for these tasks makes it hard and expensive to change things.

While I still have done very little with CQRS, I have been looking at more and more real world examples, trying to fill in the blanks. What always has been kind of vague to me, is how you go at storing your domain model, and your read models in practice. Here are a few possible techniques - these are some proven techniques, and partially my own presumptions (you hardly find any OSS brown field examples).

The compromise

CQRS doesn’t have to be an application-wide architecture necessarily; nothing stops you from introducing it gently, and just applying it to parts of your application where the added value is over-obvious. This could mean that you use a conventional architecture; a relational database with an ORM, or a document store, not distinguishing the write side from the read side. Yet for certain scenarios, you could introduce a specialized read or write side. For example; update the statistics read model on every relevant write, update a denormalized optimized read model for searches, etc..

NORM

While the relational paradigm definitely has its place, mapping your domain to the database can get complex, and require much maintenance. If you don’t expect of your write side to be queryable, you can take advantage of less cumbersome techniques such as a key value store to store your domain model. This does force you to completely separate reads from writes though.

Event Sourcing

When you look at most OSS CQRS implementations, Event Sourcing and CQRS go hand in hand. With Event Sourcing, you capture all application state changes as a sequence of events. I’m really fond of the theory behind this pattern, and I can imagine the added operational value of having a log of each change. Yet, I also think you could largely achieve the same result by enabling journaling and adding some interception. Storage wise, you store all the event streams in an event store, which is optimized for such a task. Your read side can again be whatever you fancy.

These three techniques aren’t mutually exclusive. There are a bunch of arguments to consider, and everything is highly dependent on your technical and operational requirements.

What is your experience with CQRS? Which techniques have you applied in practice?

Premature judgement

Sun, 10 Feb 2013 16:46:00 +0100

When I started my first job, I hardly ever judged my peers. After all, how could I? Everything was unknown for me; I couldn’t differentiate good from bad. Over the years that has changed a bit, but with that, I’ve also slowly become more judgemental towards peers, often prematurely, and not always deservedly.

The first few months of last year, I found myself doing maintenance on a legacy code base between projects. While I worked my way through layer after layer, I pointed my frustration towards those that had come before me; they were responsible for putting me in this mess. With half the office having touched the code base, that didn’t really add up though. When I looked at the commit history of some of the offending modules, I found names that I didn’t expect; those people were still around, and I actually thought of them pretty highly.

Judging someone’s competence solely by code he has written in the past is a flaw. There are very little pieces of code I have written over the years where I still feel comfortable about today. When I reflect on what made it go wrong, I don’t have to look far to find a bunch of reasons to blame it on; consistency was favored over common sense, major breakthroughs occurred only after the project was already in maintenance mode, people inexperienced with the domain and infrastructure were dumped on the project last minute to make up for bad planning, knowledge of the technology stack hadn’t matured, some patterns and practices weren’t commonplace yet, etc… I always find plenty of reasons to shift blame, but when I look at code written by someone else, it has to be their own fault; they must not be very good at building software. And this is unfair; I have no way of knowing the constraints they had to deal with, nor the context they had to work in. None of these justify neglecting basic hygiene though!

I tried to come up with other things that influence my opinion on someone before having actually worked with them; I found two.

The biggest influencer is word of mouth. I try to surround myself with people that share a similar thinking, and if that trusted circle has a strong opinion on someone else, I take note.
The last influencer is someone’s online presence. When I learn of someone new joining ranks, I can’t resist to look up what he’s doing online. Twitter, Facebook or a blog can give away quite a bit.

All in all, I think some preconception is human, and might be the result of subconsciously protecting your work. You only want to involve those with whom you will enjoy working towards your shared goal.

Have you experienced similar behavior? When do you judge prematurely?

Raising events in commandhandlers

Sun, 03 Feb 2013 17:49:00 +0100

I’ve explored quite a few options on how to handle commands and queries in the last few posts. I finally settled on this approach. The example used in that post looked like this.

public class CreateSubscriptionCommandHandler : ICommandHandler<CreateSubscriptionCommand>
{    
    private IDocumentSession _session;

    public CreateSubscriptionCommandHandler(IDocumentSession session)
    {
        _session = session;
    }

    public void Handle(CreateSubscriptionCommand command)
    {
        var subscription = new Documents.Subscription(
            command.Value, command.Category, command.EmailAddress);

        _session.Store(subscription);    
    }
}

Now imagine I would want to do some extra stuff after creating the subscription; update the sales statistics, append the email address to a mailing list, send out a confirmation email, etc..

You could go at this by simply extending the commandhandler, but the problem here is that you quickly end up with a bulky and dependency-heavy commandhandler, which will quickly fail to communicate its intent.

One solution could be to introduce events to decouple things in smaller pieces, and to help communicate intent more clearly.

The infrastructure to handle events is rather straightforward, and can be based on Udi Dahan’s Domain Events Salvation.

public class Events : IEvents
{
    private readonly IKernel _kernel;

    public Events(IKernel kernel)
    {
        _kernel = kernel;
    }

    public void Raise<T>(T @event) where T : IEvent
    {
        var handlers = _kernel.GetAll<IEventHandler<T>>();

        foreach (var handler in handlers)        
            handler.Handle(@event);        
    }     
}

When an event is raised, the eventing infrastructure will look in the container for implementations that can handle the event, and invoke them in a random order.

Raising an event from the commandhandler can be done by injecting this extra piece of infrastructure.

public class CreateSubscriptionCommandHandler : ICommandHandler<CreateSubscriptionCommand>
{    
    private IDocumentSession _session;
    private readonly IEvents _events;

    public CreateSubscriptionCommandHandler(IDocumentSession session, IEvent events)
    {
        _session = session;
        _events = events;
    }

    public void Handle(CreateSubscriptionCommand command)
    {
        var subscription = new Documents.Subscription(
            command.Value, command.Category, command.EmailAddress);

        _session.Store(subscription);    
        
        _events.Raise(new SubscriptionCreatedEvent(query.Id));
    }
}

The SubscriptionCreatedEvent class is a simple value object, which exposes the subscription identifier.

public class SubscriptionCreatedEvent : IEvent
{
    public SubscriptionCreatedEvent(string subscriptionId)
    {
        SubscriptionId = subscriptionId;
    }

    public string SubscriptionId { get; private set; }

    public override bool Equals(Object other)
    {
        if (other == null)
            return false;

        var otherEvent = other as SubscriptionCreatedEvent;
        if (otherEvent == null)
            return false;

        return otherEvent.SubscriptionId == SubscriptionId;
    }

    public override int GetHashCode()
    {
        return SubscriptionId.GetHashCode();
    }
}

To subscribe to this event, implement the IEventHandler interface, and register the implemenation in the container.

public interface IEventHandler<T> where T : IEvent
{
    void Handle(T @event);
}

public class SendConfirmationMailOnSubscriptionCreated : IEventHandler<SubscriptionCreatedEvent>
{    
    public void  Handle(SubscriptionCreatedEvent @event)
    {
        ...
    }
}

public class UpdateSalesStatisticsOnSubscriptionCreated : IEventHandler<SubscriptionCreatedEvent>
{    
    public void  Handle(SubscriptionCreatedEvent @event)
    {
        ...
    }
}

Eventhandlers are invoked synchronously, and participate in the commandhandler’s unit of work, so if something goes haywire in one of the eventhandlers, nothing gets committed, not even what happened in the original commandhandler. Depending on your requirements, you might want to handle this differently though.

With this approach, tests also become more compact. Commandhandler tests now only need to assert that the event gets raised, and all the other logic gets offloaded to separate tests per eventhandler.

Summary

By introducing events, you can decouple commandhandlers into more focused, and intent-revealing bits. Your tests are the perfect proof of how much cleaner things get. One of the cues to listen for is when you do x or on doing y, also do z.

Are you using events? If so, domain events, or its big brother Event Sourcing?

Organizing commands and queries

Sun, 27 Jan 2013 18:23:00 +0100

In the last few posts I settled on an architecture for handling commands and queries. A byproduct of the described approach, is that your codebase quickly racks up plentiful little classes; a class to hold data, and a handler to act on that data, for each use case.

There are a few ways you can go at organizing things.

Everything in one location

When there is very little going on in your application, you can just dump everything in one location without getting hurt too much.

Folder per functionality

When your application grows, and the coherency between different use cases are obvious, you can just use folders - and corresponding namespaces, to organize your commands and queries, and to draw their functional boundaries. Uncle Bob and Mark Needham have sold me on structuring my code based on functionality instead of technical concepts.

Keeping each commandhandler in a separate class is especially interesting when they are rather bulky and contain a good amount of logic. You can think of each class as a little piece of functionality in itself. Take a look at the RavenDB codebase to get an idea of what that could look like.

It also feels like this way tends to bring your code closer to the Solution Explorer; just one double-click and you are looking at your implementation; no scrolling or searching between method definitions necessary. Maybe the problem is now just shifted a level higher in the hierarchy though.

Composing a service class

You can also opt to group commandhandler implementations in one service class. This variation might make more sense when your implementations are rather skinny, and don’t do a whole lot but translating and forwarding your invocation.

public class SubscriptionService : 
        ICommandHandler<SubscribeCommand>,
        ICommandHandler<UnsubscribeCommand>
{
    public void Handle(SubscibeCommand command)
    {
        throw new NotImplementedException();
    }

    public void Handle(UnsubscribeCommand command)
    {
        throw new NotImplementedException();
    }
}

Use the hints your dependency graph gives you to find a composition that makes sense.

How do you go at organizing commands and queries?

RavenDB: Drop all collections

Thu, 24 Jan 2013 20:08:00 +0100

I never stub or mock the database when I’m using RavenDB. Generally, I use an embeddable documentstore running in memory, and initialize a new instance on every test. However, I like to run some stress tests against a real instance, and here I found myself wanting to wipe clean the state of previous tests, without having to create a new database (which is rather slow).

First I create the default DocumentsByEntityName index to make sure it’s there - it normally gets created when you open the studio for the first time. Then I use one of the advanced database commands: DeleteByIndex, and query all the tags.

using (var session = _documentStore.OpenSession())
{
        new RavenDocumentsByEntityName().Execute(_documentStore);
        session.Advanced.DatabaseCommands.DeleteByIndex(
                "Raven/DocumentsByEntityName",
                new IndexQuery { Query = "Tag: *" });                
}

This technique doesn’t seem to be widely used judging by the first page of Google search results. If there is a reason for that though, let me know!

Separating command data from logic and sending it on a bus

Sun, 20 Jan 2013 22:03:00 +0100

In my first post on this topic, I started out with an attempt to limit abstractions to solely commands and queries. Commands and queries were self-contained and could be invoked by passing them to a context-providing generic handler. The drawback of this approach was that it made constructor dependency injection impossible. In a next post, I separated data from logic, but never got around to writing a dispatcher that associates command data with their handlers. Last week, I revisited the first approach, and added an unconventional implementation of injecting dependencies by an Inject method convention.

I still believe that last approach is very simple and works fine if extra dependencies are exceptional. I do admit that it will make architectural shoots harder to handle; everything is rather tightly coupled. So let’s look at an alternative architecture which pulls all the bits apart, and should be better equipped to handle change.

Let’s first separate command data from logic.

public class CreateSubscriptionCommand 
{     
    public CreateSubscriptionCommand(string value, string category, string emailAddress)
    {
        Guard.StringIsNullOrEmpty(value, "value");
        Guard.StringIsNullOrEmpty(category, "category");
        Guard.StringIsNullOrEmpty(emailAddress, "emailAddress");

        Value = value;
        Category = category;
        EmailAddress = emailAddress;
    }

    public string Value { get; private set; }
    public string Category { get; private set; }
    public string EmailAddress { get; private set; }

    public override bool Equals(Object other)
    {
        if (other == null)
            return false;

        var otherCommand = other as CreateSubscriptionCommand;
        if (otherCommand == null)
            return false;

        return otherCommand.Value == Value && 
            otherCommand.Category == Category && 
            otherCommand.EmailAddress == EmailAddress;
    }    

    public override int GetHashCode()
    {
        return Value.GetHashCode() ^ 
            Category.GetHashCode() ^ 
            EmailAddress.GetHashCode();
    }
}

The data is just a POCO. Notice the equality overrides; this comes in handy when you’re testing.

The class that handles on the data needs to implement the ICommandHandler interface.

public class CreateSubscriptionCommandHandler 
    : ICommandHandler<CreateSubscriptionCommand>
{    
    private IDocumentSession _session;

    public CreateSubscriptionCommandHandler(IDocumentSession session)
    {
        _session = session;
    }

    public void Handle(CreateSubscriptionCommand command)
    {
        var subscription = new Documents.Subscription(
            command.Value, command.Category, command.EmailAddress);

        _session.Store(subscription);    
    }
}

Compared to the previous approach, we’re now injecting the session instead of having it handy as a property; that coupling is now completely gone.

Last thing left to do is create an interface that consumers can use to send commands on: a bus.

public class Bus : IBus
{
    private readonly IKernel _kernel;
    private readonly IDocumentSession _session;

    public Bus(IKernel kernel, IDocumentSession session)
    {
        _kernel = kernel;
        _session = session;
    }

    public void ExecuteCommand<T>(T command) where T : class
    {
        var handler = _kernel.Get<ICommandHandler<T>>();

        handler.Handle(command);

        _session.SaveChanges();
    }
}

The ExecuteCommand method dispatches data to the correct handler by resolving it from the container, and also commits the unit of work.

The consumer can execute commands like this.

bus.ExecuteCommand(new CreateQueryCommand(queryValue, category, emailAddress));

With this approach having all the bits spread, we have a bit more work gluing all the pieces together. The session is now known in the container, and is request scoped. The commandhandlers are also all registered in the container.

protected override void ConfigureRequestContainer(IKernel container, NancyContext context)
{        
        // you don't want to register them all individually
        container
            .Bind<ICommandHandler<CreateSubscriptionCommand>>()
            .To<CreateSubscriptionCommandHandler>();        
        // snip..
        container.Bind<IDocumentSession>()
            .ToMethod((ctx) => { 
                return ctx.Kernel.Get<IDocumentStore>().OpenSession();    
            })
            .InSingletonScope();                          
}

I think this approach might suit a lot of projects better if your commands are dependency heavy.
What I like most is that handling architectural shoots will be easier. For example: right now, all behaviour is in my entities and in my commands; the segregation of application services and domain services is non-existent. And this works fine so far; I yet have to find a use case where I would benefit from more separation. If that would change in the future though, I can introduce abstractions, and concepts, without breaking consumer code, and without having to do awkward stuff managing the newly introduced dependencies.

Self-contained commands with dependencies

Sun, 13 Jan 2013 18:17:00 +0100

Also read: separating command data from logic and sending it on a bus

In October I looked at an architecture that limits abstractions to solely commands and queries. In that post, I had some infrastructure that looked like this.

public abstract class Command
{
    public abstract void Execute();
}

public abstract class Query<T>
{
    public abstract T Execute();
}

public interface ICommandHandler
{
    void Execute(Command command);
}

public class CommandHandler : ICommandHandler
{
    public void Execute(Command command)
    {
        command.Execute();
    }
}

public interface IQueryHandler 
{
    T Execute<T>(Query<T> query);
}

public class QueryHandler : IQueryHandler
{
    public T Execute<T>(Query<T> query)
    {
        return query.Execute();
    }
}

Commands and queries are both accompanied by their specific handler. In this example, the handler does nothing but invoking the command or query. In reality, you want your handlers to do a little more. For example: provide the commands and queries with a context to work with, add logging, handle your unit of work and all of that good stuff.

Let’s look at the infrastructure, and particularly the handlers of a project that uses RavenDB to store its data.

public abstract class Command
{
    public IDocumentSession Session { get; set; }

    public abstract void Execute();           
}

public abstract class Query<T>
{
    public IDocumentSession Session { get; set; }

    public abstract T Execute();
}

public class CommandHandler : ICommandHandler
{
    public void Execute(Command command)
    {            
        var store = DocumentStore.Get();
        using (var session = store.OpenSession())
        {
            command.Session = session;
            command.Execute();

            session.SaveChanges();
        }
    }
}
    
public class QueryHandler : IQueryHandler
{    
    public T Execute<T>(Query<T> query)
    {
        var store = DocumentStore.Get();
        using (var session = store.OpenSession())
        {
            query.Session = session;

            return query.Execute();
        }
    }
}

The handlers take care of creating and managing the session, but also provide the commands and queries with a reference to the session.

An actual command could look like this.

public class ConfirmOrderCommand : Command
{
    private Guid _token;

    public ConfirmOrderCommand(Guid token)
    {
        _token = token;
    }

    public override void Execute()
    {
        var order = Session.Query<Documents.Order>().Where(x => x.Token == _token).First();

        order.ChangeStatus(Documents.Status.Confirmed);
    }
}

I really like this style of command and queries; very little ceremony. The downside though, is that you can’t use constructor dependency injection. Like discussed in this post, you could split your classes in two parts: the handler and the data, and you would solve that problem.

I wasn’t that keen on that approach. Also, now that I’m so accustomed to having fast in-memory integration tests with RavenDB, it’s exceptional that I have the need to inject dependencies. I worked out an alternative which allows me to inject dependencies without having to put my data somewhere else.

Instead of injecting the dependencies through the constructor, we’re going to use an Inject method. This is a convention; there is no interface that enforces this. Returning to our ConfirmOrderCommand, we’ll add support for creating a new folder on the file system.

public class ConfirmOrderCommand : Command
{
    private IFileSystem _fileSystem;

    private Guid _token;

    public ConfirmOrderCommand(Guid token)
    {
        _token = token;
    }

    public override void Execute()
    {    
        var order = Session.Query<Documents.Order>().Where(x => x.Token == _token).First();
        
        _fileSystem.CreateDirectory(Path.Combine("D:\", order.Customer.Id));

        order.ChangeStatus(Documents.Status.Confirmed);
    }
    
    public void Inject(IFileSystem fileSystem) 
    {
        _fileSystem = fileSystem;
    }
}

We can now amplify the commandhandler to automatically resolve and inject the dependencies into our commands. The handler uses reflection to look for a method named Inject on the type. If this method exists, it will inspect the method for its expected arguments and try to resolve those, to finally invoke the Inject method with its resolved arguments.

public class CommandHandler : ICommandHandler
{
    private IKernel _kernel;

    public CommandHandler() { }

    public CommandHandler(IKernel kernel)
    {
        _kernel = kernel;
    }

    public void Execute(Command command)
    {
        if (_kernel != null)
            ResolveDependenciesIfNeeded(command);
            
        var store = DocumentStore.Get();
        using (var session = store.OpenSession())
        {
            command.Session = session;
            command.Execute();

            session.SaveChanges();
        }
    }

    private void ResolveDependenciesIfNeeded(Command command)
    {
        var method = command.GetType().GetMethod("Inject");
        if (method != null)
        {
            var parameters = method.GetParameters();
            var parameterInstances = new List<object>();

            foreach (var parameter in parameters)
            {
                var type = parameter.ParameterType;
                var instance = _kernel.Get(type);

                parameterInstances.Add(instance);
            }

            method.Invoke(command, parameterInstances.ToArray());
        }
    }
}

Consumers now don’t have to care about the dependencies; they just have to be registered in the container. In the tests however, we can now explicitly inject mocks or stubs, and take advantage of having discoverable dependencies though the Inject convention.

[TestClass]
public class When_confirming_an_order
{
    private Mock<IFileSystem> _fileSystem;

    [TestInitialize]
    public void When()
    {
        DocumentStore.InitializeEmbedded();

        var cmd = new ConfirmOrderCommand("_token_");
        
        // Inject mock
        _fileSystem = new Mock<IFileSystem>();    
        cmd.Inject(_fileSystem.Object);

        new CommandHandler().Execute(cmd);
    }

    [TestMethod]
    public void the_status_is_changed_to_confirmed()
    {
        ...
    }
    
    [TestMethod]
    public void a_new_folder_is_created()
    {
        _fileSystem.Verify(...);
    }
}

Although I haven’t really gone the distance with this implementation - I only have one command that has extra dependencies, I find this technique showing lots of promise. You get the leanness of self-contained commands and queries, while you still allow discoverable dependency injection by convention, supported by a tiny bit of infrastructure in the handlers.

I’d like to hear your opinion.

2012 Annual Review

Mon, 31 Dec 2012 15:58:00 +0100

It’s that time of the year again where I take some time to look back on the year passed and peek at the year ahead. This is mostly a post I like to write up for myself, forcing reflection.

The biggest personal changes this year were moving out of my parents’ place, and my girlfriend’s decision to study medicine after graduating as a Master of Arts earlier this year. I had no idea how things would turn out once we lived together, but so far we have been doing great.

My career

September marked one year working for Euricom. I’m still at my first client, and while I definitely regularly have my share of the enterprise blues, I also learned a lot working on several projects: from maintaining legacy to leading green fields. Fairly, I often doubt if traditional enterprise life is it for me though.

Blog

I published 66 posts this year, which seems to be consistent with previous years.

While writing has turned into a habit over the years, I often struggle with deciding what I should write about. Any reason is probably a good one though; documenting, sharing opinions and experiences, or just writing for the sake of writing… But I still feel like I should add value and contribute something worthwhile when I spend time on this blog.

The topics haven’t changed that much this year. I still write often on Web technologies (ASP.NET, NancyFx and REST) - something I didn’t get to spend much time on at work this year, but I also wrote about architecture, testing and NoSQL. Next to those topics, I also penned some of my experiences gained doing experimental side projects.
In 2013, I intent to get more out of my comfort zone, and to also write down thoughts which are not that strongly held.

Page views have more than doubled (173k) compared to 2011, which I’m humbled by, but I seem to care less about that than a little while ago. It had even been over a month since I logged into Google Analytics.

Community

It very much depends on what I’m doing that day, but I still spend quite some time on Twitter. I often blurt out thoughts on software, and regularly, someone who knows better than me replies, and challenges my looks on software. That’s how I get value out of Twitter. That, and animated GIFs of course.

In real life, I went to see a few interesting speakers at local user groups, and gave a talk myself at Web.NET Europe Milan - international speaker, wooh. I want to applaud the organization of Web.NET Europe once more; they did a great job.

In general, I started to value artificial sponsored user groups less. They have their place, and they do add value, but it’s not something that I feel I need to be a part of. Having some beers with a few likeminded developers suits me better.

Travelling

Buying an apartment and all of that, we didn’t get to make any epic travels this year. I got to visit three different countries though: Italy, Tunisia, Italy again, and France. Two out of four were sponsored by my job; not bad.

We’re setting out for a big trip again in 2013. We fell in love with America two years ago, so we’ve been researching the South-West again - but closer to the Mexican border this time. Other destinations are still open for discussion though.

Projects

I spent a considerate part of 2012 building small things on the side. This didn’t result in anything useful, but most importantly, it got me in the habit of working on my own things, and I put together a toolset which enables me to ship things fast (NancyFx, Twitter Bootstrap, MongoDB, AppHarbor, …).

The first project I did in 2012 was Docary. This was supposed to be something between a timesheet and a diary. This idea quickly died after I couldn’t commit to using it myself. Technically I played with early versions of jQuery mobile and EF, and learned why you shouldn’t overlayer your MVC application - or any web application for the matter.

For the second project, I and Davy Brion worked on TopDevLinks. This was supposed to be a linkblog implementation on top of ASP.NET MVC, CQS and MongoDB. We both lost interest in the concept eventually.

I made my girlfriend a petite portfolio site built on top of NancyFx, the FlickR API, and some jQuery plug-ins. Right now, it’s actually running as a static site on GitHub pages though.

I already blogged about the next project: Kill Long Meetings. This one is still around, but quickly floundered into forgetfulness after its five minutes of Twitter fame.

Now that I look back at it, I did write quite some code after my hours this year… Next to being in the moment of building and shipping stuff, I genuinely believe it makes me better at my job.

I hope to be able to do as much in 2013, but to also let it be something worthwhile. I’ll be actually already shipping something new in a few weeks if all goes well.

Sport/Health

The biggest gift I gave myself health wise this year, was giving up on smoking; I lit my last one over six months ago!

Moving to Antwerp, and not having any area to put my weights, I’ve joined a gym again, where I’m doing a two times a week compound routine. That should make me stronger again; I’ve lost a lot of beef since picking up running three years ago.

I haven’t given up on running - living close to the city park helps. I just missed the 1000km (or 621 miles) milestone this year, which is kind of lame, but I’m far from complaining.

I don’t know what I hope for in this area in 2013. It’s starting to dawn on me that, even though those goals I set for myself have pushed me further than I thought I was capable of - I couldn’t jog 1km without a break three years ago, they are not that important. It eventually comes down to nourishing your body, and making use of it in every way possible while you can.

Conclusion

Earlier this week, I strolled down the main street of Brussels and crossed a beggar with one leg and no arms. Next to feeling extremely sorry for that man, I felt truly blessed; I’m healthy, loved, have more than I need, and I get to do what I like for a living. I need to remind myself more of that in 2013 - I can be a whiny bitch.

I plan to keep course in 2013, but as a better man.

How was your year? What do you plan for in 2013?

Now, if you’ll excuse me, I’m going to help a hand in preparing the festivities before the girlfriend really gets in a faul mood.

*Holy crap, when I sat down to write this, I didn’t expect to write a whole book. I’m wondering how many people actually made it down here. *

2012's most read posts

Sun, 23 Dec 2012 15:53:00 +0100

I look forward to writing this post each year; it’s by far the easiest one to write, and it provides me with an occasion to look back on previous years (2009, 2010, 2011). It’s entertaining - and shameful too - to see what kept me busy back then, which opinions I held, and which technologies and techniques are still relevant in the meanwhile. Anyways, here it goes.

The most read post this year, with 37.797 page views, is How a web application can download and store over 2GB without you even knowing it. I just ran this experiment again with the latest Chrome build, and apparently the Chrome team hasn’t addressed these concerns in the meanwhile; the browser still behaves exactly the same. I’m guessing usage stats might prove that it’s not worth the effort.

The second most read post, with 14.371 page views, is Learning: the Hacker Way. I still firmly stand by this one today. Passive learning can only take you so far; it’s crucial to supplement passive learning with a high volume of getting your hands dirty.

With 14.085 page views, Commuting: have you done the math? comes in third. Commuting wastes a good part of your life; optimizing your commute can significantly improve the quality of your average day. Readers shared some interesting experiences in the comments on this one.

Having 5.215 page views, the post Why I will always love RSS is the penultimate item in this list. This writing is basically a rant on how, although I somewhat understand the motivation behind public companies fencing their gardens, it is sad to see that even peers seem to advertise the death of RSS, while it’s RSS that has enabled me to follow my favorite sources of information without having to rely on others to tell me what to read, and without having to be connected the whole damn time.

With 1000 views less, making it the last item in the list, is the post HTML5 Offline Web applications as an afterthought in ASP.NET MVC. It was this post that landed me a paid article on InfoQ. I should revisit this topic again, I’m curious to see how many websites are already making use of this HTML5 super cache to improve performance, or to truly provide offline access.

Thank you for reading in 2012!

It's not cake we are baking

Sun, 09 Dec 2012 16:08:00 +0100

I recently watched a talk on Vimeo where Christin Gorman talks about how cookie dough relates to Hibernate; why use the generic, bloated and one-fits-all solution when you can mix together your own yummy cookie dough? We should aspire to be the Gordon Ramsey of software, not the college student who can only cook Ramen noodles. If you haven’t watched or listened to her talk, you should; it’s only a few minutes long, and she brings it really well. Go ahead, I’ll wait.

When I first watched her talk, it rubbed me the wrong way, but I couldn’t yet figure out the flaw in her plea. It recently dawned upon me that we are not in the business of baking cakes, but we are in the business of serving hungry people. And those people really don’t care which ingredients were used to bake your cake; they only care about how it tastes, how fast it’s served, and how much they’re paying for it. And although the romantic in me really sympathizes with her view, the realist just can’t. Over these few years only, each and every attempt of a hand-rolled ORM I’ve seen, ended with a horrible aftertaste. There is no way that you spend as much time building a good API from scratch as you do setting up an existing one. Writing and maintaining your own ORM is not a trivial thing to do, and there are already so many flavors* - from grandma’s recipe to decadent supreme deluxe - to pick from, that it’s hardly ever a justified decision. Read the package and choose the option that satisfies your daily caloric needs, and go from there. You can still sweeten, or swap some ingredients, before serving. And if you really can’t resist - which is perfectly healthy, bake all the cake you want, at home. Maybe, one day, you can bring a successful one to work.

* NHibernate, Entity Framework, Dapper, Massive, Simple.Data, LLBLGen Pro, OpenAccess, SubSonic …

Some notes on performance tuning with NHibernate

Sun, 02 Dec 2012 17:36:00 +0100

A few weeks back, I spent an intensive day performance tuning parts of a, to me, relatively unfamiliar part of our codebase. Like it often is, the biggest optimizations were to be found in how we work with the database. Now, I don’t consider myself to be an NHibernate expert; I read this book and have used it on two projects, but in the end I just do my best to avoid doing stupid things with it. The topics discussed below are mostly common knowledge for long time NHibernate users, but I thought it might be convenient for others to just summarize them, and add references to other, more in-detail, posts.

Under the covers

When you’re looking into optimizing, you probably want to have a look at what’s really going on. You could do this by turning on NHibernate’s log4net debug logging. This might be good enough for some scenarios, but it’s not really convenient for when there is lots and lots of stuff happening. Instead, you might want to look into NHibernate Profiler. It’s trivial to get started with, yet the feedback it provides is very powerful: next to session statistics and executed queries, you also get alerts which suggest techniques to improve your code. I need to use this tool more often just to get a better grip of the NHibernate internals.

The fastest query is the one that isn’t

Going to the database is probably one of the slowest things your application is going to do. If you can avoid it, do so.

If you’re using a stateful session, NHibernate will track your entities, and store them in its first level cache. When you get these items by id later on, you avoid going to the database. So instead of querying the database for the same entity multiple times in the same session, do it once, and get the entity by its id on subsequent calls.

When you’re having a hard time keeping those ids around, consider introducing a light-weight datastructure, such as a dictionary, which can help you build a small look-up cache. This could also be a sign that you might want to reconsider your identity strategy though.

Working with batches

An ORM really isn’t the best tool to do bulk inserts or updates; look at SqlBulkCopy instead.

If your batches are still relatively small, and you opt to stay with NHibernate anyways, there are two things you can do which will improve performance tremendously: use a stateless session and configure batching.

Switching to a stateless session is simple enough. Do take into account that some features won’t work anymore: lazy loading, caching, cascading and implicit updates. Setting up batching is also just a matter of configuration. The most important thing to remember is to use an appropriate identity generator; batching will only work when the application is responsible for generating the ids. I’m using a [HiLo generator](http://stackoverflow.com/questions/282099/whats-the-hi-lo-algorithm, but GUIDs or assigned ids will work too.

Released: Kill long meetings

Sun, 25 Nov 2012 17:45:00 +0100

A lot has already been said and written about meetings, and some have carried the message above par; ‘Meetings: where work goes to die’. Today, I’m not going to foul the internet with another rant, but I’d like to show you a small application built over the last few weeks after work.

I regularly find myself building small things as an antitoxin to the regular periods of not writing and shipping code at work. This time, me and @cgeers, built a not-so-serious application that aims to kill long meetings by visualizing the amount of time and money burned in a meeting.

While the application in itself probably will never attract a sustainable user base, development sprouted two useful Twitter Bootstrap plug-ins: a spin edit control and a multi-color progressbar.

Used technology stack

On the server, we’re using Nancy on an ASP.NET host with Razor views. There is hardly anything going on at the server, so we could have picked any server-side framework I guess. We might just be attracted to the as ’little friction as possible’ Nancy ethos though. Returning a view, and setting up a route for it, takes just a few LOC.

namespace KillLongMeetings
{
    public class RootModule : NancyModule
    {
        public RootModule()
        {
            Get["/"] = p => View["HomeView"];
        }
    }
}

Next to returning a correct view, we’re also making bundles on the server. For that, we’re using Cassette for Nancy.

At the client, we chose to use Twitter Bootstrap, because getting a lay-out right that works everywhere these days is hard; we like to spend that time on other things. Next to jQuery - duh, we’re using knockout.js as our client-side MVVM framework. This worked out lovely for our scenario, but overall, I’m distancing myself a bit from this framework. It’s simple, and extremely easy to get started with, but lots of companies I heard of seem to adapt this as an application framework, which it is not. It’s just two-way model binding in the browser. More complex scenarios benefit from a cleaner separation of concerns for validation, working with remote data, application composition and testing. Right now, I’m doing something with angular.js, and it seems very promising so far. It’s also a lot less intrusive than knockout.js.

We’re using AppHarbor and GitHub for continuous deployment. The production site however is a static site hosted on GitHub pages. Next to serving everything really fast, we’re now not paying for anything except the domain name.

The source can be found on GitHub. Let us know what you think!

Commuting? Have you done the math?

Sun, 04 Nov 2012 20:24:00 +0100

On my first job interview, over four years ago, I was asked whether I would relocate if I was hired. Back then, I still lived in the Campine region with my parents, while the Ferranti Computer Systems headquarters are in Antwerp. I thought about it for a few seconds and told the interviewer that I didn’t plan on moving out of my parents’ place in the first few years. Besides, the distance isn’t that great; it’s only 60km (=37 miles) of highway, how bad could it be? Apparently that reply was good enough since I was given the job a few days later.

Well, that commute grew old rather quickly though. Turns out that those 60km of highway are one of the most saturated pieces of asphalt in Belgium, with up to 20 to 30km (12 to 18 miles) of systematic traffic jams every damn single day.

And when you have to take the car to work, the options you have to spend that time useful are rather limited; the only things I could think of at the time were listening to podcasts and reflecting on the things of life. The latter made me hate my situation even more, so that wasn’t an optimal pastime.

I actually never had the courage to calculate how much time I wasted in traffic those three years. Today, I’m in a far more comfortable situation, so plucking up courage wasn’t that hard anymore. With 52 weeks in one year, and 5 days in a week, there are 260 business days in one year. In Belgium, we don’t work on all of those though, so I had to subtract 25 vacation days, and 6 holidays; bringing down the number of working days to 229. My rather optimistic guess is that I was on the road for somewhere around 140 minutes each day. So 229 days multiplied by 140 minutes comes down to a total of 32060 minutes, or 534 hours, or 66 working days. Gasp! 66 working days per year gone to waste. I knew the numbers were going to be bad, but this is even a lot worse than I expected.

Easing this burden isn’t trivial though, and I can only think of a few feasible options, excluded changing jobs:

Travel outside rush hours: leave for work very early or really late. I experimented with the former for quite some time, but I never really got used to it. It also seems a bit counterproductive to have your rhythm be too different from that of your team members. I’m guessing your family life will suffer as well, but I can’t be the judge of that.
Bring the office to you. In this technological day and age there are hardly any sound arguments not to encourage working from home. Yet, the classical enterprise shies away from embracing it and seems to be more comfortable sticking with the status quo than improving working conditions for their employees.
Relocate. This option might be drastic, yet it’s the one with the biggest probability of success.
If possible, use public transport. Even if your total time enroute increases, you will have more time to do something productive while you take away some of the frustration that comes along with commuting by car.

Earlier this year, I moved from one of the (work-wise) more remote corners of Belgium to a far better located area; the heart of Antwerp. To top that, I’m now living and working within walking distance of the train station, so I’m taking the train on a daily basis. In total I’m still on the go more than two hours, but I can now spend 75% of that time usefully. That’s 43 working days extra to spend each year! I’ve made a habit out of using that commute time for self-study; reading, working on side projects and writing. This seems to have considerably affected my state of mind for the better. I get home at night, and I’ve already spent a considerable amount of time challenging myself intellectually. I now no longer stress about practicing less ‘productive’, but very enjoyable activities; such as playing the guitar, working out or doing stuff around the house. I get to have both now.

There are only so few hours in a day, having to spend a considerable amount of that time just to get somewhere seems such a waste. Every day we try tools and techniques which save us a few minutes, and should help us improve the quality of our day-to-day lives, yet we often disregard optimization of the biggest time hog of them all. It goes without saying that those numbing long commutes by car aren’t something that I’ll ever decide to go back to lightly.

When I read my Twitter feed in the morning, I see a lot of other Belgian people complain about their commute. Is this a Belgian thing only? Which means of transport do you use for the commute? How long are you on the road each day? **Have you done the math? **

More comments on Hacker News.

Post Web.NET Europe

Mon, 22 Oct 2012 20:35:00 +0200

I attended and spoke at Web.NET Europe in Milan over the weekend. This was only my fourth full day- or more conference (Techdays Belgium, TechEd Berlin and HTML5 WebCamps), but it was undoubtedly the best one so far.

The quality of the sessions was definitely not inferior to those of bigger conferences. I especially enjoyed the talks on SignalR, OAuth and scaling data (I included some of my notes below). The strength of this conference doesn’t lie in the exceptional speakers or sessions though, but in its cozy size and the type of attendees it attracts. Being hosted on a Saturday, you already preclude all the developers who merely think of technology as a job. And when you put together those who care about what they do, and want to get better at it, good things happen. This was the first conference where I was able to talk to such a wide range of people - I guess I even spoke to more than six different nationalities - and where it didn’t feel awkward one bit. What helps in attracting such a variety of people, is that the conference is practically free and survives on donations from sponsors and attendees, making it very affordable even if you fly in from outside of Italy. Freelancers also seemed to appreciate that it was on a non-billable day.

In short, I really enjoyed the experience, and it might be just so that weekend conferences make for better conferences. Congratulazioni a Simone Chiaretta and Ugo Lattanzi for making this happen. I’m already looking forward to the next edition.

“Real Time” Web Applications with SignalR in ASP.NET (@A_Giorgetti)

Last year I did a talk on WebSockets at HTML5 WebCamps and although I built a few things that worked, the real-time web in the wild was still very much a mess. SignalR now abstracts all that clutter for you, and provides you with a seemingly clean infrastructure and simple API. Too bad the use for real-time web applications is rather limited in my world - stock ticker or chat application anyone?

OAuth-as-a-service using ASP.NET Web API and Windows Azure Access Control (@maartenballiauw)

I had my first serious look at OAuth in this session, and while it’s probably indispensable for public API’s, it doesn’t seem that trivial to implement. Azure ACS could make this easier though.

Slides

Scaling without going crazy (@Ayende)

This was one of the talks I really looked forward to, and it didn’t disappoint one bit. I never got to do anything with big data (data that can’t fit on one machine), but there are really interesting problems and trade-offs in that space - CAP theorem etc.

Some quotes worth giving more thought:

Caching often just hides a problem.
How is Facebook consistent? It’s personally consistent.

Slides and code from my Web.NET Europe REST and ASP.NET Web API session

Sun, 21 Oct 2012 00:16:00 +0200

I just returned to the hotel after attending and speaking at Web.NET Europe 2012. There were multiple sessions on ASP.NET Web API today, and I was responsible for delivering the first one of the day. Being first, and to avoid overlap with others, I tried to lay a solid foundation by focusing on the REST and ASP.NET Web API basics: resources, identifiers, representations, verbs and hypermedia.

More people than I expected showed up for my talk; somewhere around 50 to 70 attendees, I guess. I think it was fairly well received; after 25 votes, the session rating stands at 90%. It’s great to see your preparations have such a fruitful outcome.

I still have a few thoughts on the conference to materialize, but for now, I uploaded the slides and hosted the source code on GitHub.

Commands with dependencies

Mon, 15 Oct 2012 16:57:00 +0200

Also read: Separating command data from logic and sending it on a bus

Yesterday I wrote about an architecture which limits abstractions by solely introducing commands and queries. I shared a dead simple variation of this pattern, the advantages I experienced, and how I could still unit test the controller if I wanted to. At the end of that post I wondered how I would be able to test commands in isolation; suppose the implementation doesn’t use a database this time, but a hairy, too low-level, third party webservice.

Right now, the input arguments are inserted via the constructor, but this leaves no room to inject dependencies. That is, if we don’t want to do awkward stuff with our Dependency Injection framework, and don’t want to resort to property- or method injection.

public class WithdrawAmountCommand : Command
{
    public WithdrawAmountCommand(string user, int amount)
    {
        if (string.IsNullOrEmpty(user))
            throw new NullReferenceException("user");

        User = user;
        Amount = amount;
    }

    public string User { get; private set; }
    public int Amount { get; private set; }

    public override void Execute()
    {
        // Implementation
    }
}

One way to enable injecting dependencies could be to separate the command in two parts; the handler, and the data.

public interface ICommandHandler<in TCommandData> 
{
    void Execute(TCommandData commandData);
}

Now we could rewrite the WithdrawAmountCommand. One class contains and verifies the input data…

public class WithdrawAmountCommandData
{
    public WithdrawAmountCommandData(string username, int amount)
    {
        if (string.IsNullOrEmpty(username))
            throw new NullReferenceException("username");

        Username = username;
        Amount = amount;
    }

    public string Username { get; private set; }
    public int Amount { get; private set; }
}

…while the second class actually acts on the data.

public class WithdrawAmountCommandHandler : ICommandHandler<WithdrawAmountCommandData>
{
    private readonly IAccountWebservice _accountWebservice;

    public WithdrawAmountCommandHandler(
        IAccountWebservice accountWebservice)
    {
        _accountWebservice = accountWebservice;
    }

    public void Execute(WithdrawAmountCommandData data)
    {
        _accountWebservice.Invoke(data.Username, data.Amount);
    }
}

Testing commands in isolation is now straight-forward.

[TestMethod()]
public void Execute_should_invoke_webservice_with_correct_arguments()
{
    var accountWebService = new Mock<IAccountWebservice>();

    var command = new WithdrawAmountCommandHandler(accountWebService.Object); 
    command.Execute(new WithdrawAmountCommandData("JefClaes", 25));

    accountWebService.Verify(aws => aws.Invoke("JefClaes", 25));
}

Don’t forget that the controller also needs to accommodate this change.

public class HomeController : Controller
{
    private readonly IQueryHandler _qryHandler;
    private readonly ICommandHandler<WithdrawAmountCommandData> _withdrawAmountCommandHandler; 

    public HomeController()
    {
        _qryHandler = new QueryHandler();
        _withdrawAmountCommandHandler = new WithdrawAmountCommandHandler(new AccountWebservice());
    }

    public HomeController(
        IQueryHandler qryHandler,
        ICommandHandler<WithdrawAmountCommandData> withdrawAmountCommandHandler)
    {
        _qryHandler = qryHandler;
        _withdrawAmountCommandHandler = withdrawAmountCommandHandler;
    }

    [HttpPost]
    public ActionResult Foo(string user)
    {
        _withdrawAmountCommandHandler.Execute(new WithdrawAmountCommandData(user, 25));

        var totalAmount = _qryHandler.Execute<int>(new TotalAmountQuery(user));

        if (totalAmount < 0)
            return RedirectToAction("Warning");

        return RedirectToAction("Index");
    }
}

I’m not all that happy with how this approach will bloat my controller in the future to be honest. To smooth out this friction, I could - like in the previous post - introduce a dispatcher, which serves as an intermediary for the commandhandlers.

Integration testing wasn’t an option in this scenario, so I had to introduce some extra abstractions just to facilitate unit testing. Although this particular variation still holds lots of advantages I talked about in my previous post, there’s already a lot more boilerplate and some more complexity.

I’m going to give this topic some more thought. Care to share yours?

Commands, queries and testing

Sun, 14 Oct 2012 17:45:00 +0200

Also read:

We need abstraction, but the amount of abstraction we really need depends, and should be assessed on a case-by-case basis. It seems advisable to grow abstractions, and to introduce them gradually.

That being said, in this post I want to talk about an architecture that tries to limit abstractions to solely commands and queries.

It all starts with two small pieces of infrastructure: a command and a query. A command performs an action, and can change state, while a query should only return data, and not alter any state; basic command and query separation.

public abstract class Command
{
    public abstract void Execute();
}

public abstract class Query<T>
{
    public abstract T Execute();
}

Now imagine, we are doing something with accounts, and we want to have a command that can withdraw money from an account, and a query that returns the total amount available on an account. The assumption is that we’re only talking with a database.

public class WithdrawAmountCommand : Command
{
    public WithdrawAmountCommand(string user, int amount)
    {
        if (string.IsNullOrEmpty(user))
            throw new NullReferenceException("user");                
    
        User = user;
        Amount = amount;
    }

    public string User { get; private set; }
    public int Amount { get; private set; }

    public override void Execute()
    {
        // Implementation
    }
}

We inherit from the Command class, and pass in its input arguments via the constructor. We put our actual implementation in the Execute method override. For the query, we do something very similar.

public class TotalAmountQuery : Query<int>
{
    public TotalAmountQuery(string user)
    {
        if (string.IsNullOrEmpty(user))
            throw new NullReferenceException("user");
    
        User = user;
    }

    public string User { get; private set; }

    public override int Execute()
    {
        return 100; // Should be an actual implementation
    }
}

Now, if we would use this in ASP.NET MVC, we would end up with clean and compact controllers. Have a look at the action Foo, which first withdraws 25 euros, to then redirect to a warning action if the balance is negative, or to redirect back to the index action when it’s positive. Very readable, right?

[HttpPost]
public ActionResult Foo(string user)
{
    new WithdrawAmountCommand(user, 25).Execute();

    var totalAmount = new TotalAmountQuery(user).Execute();

    if (totalAmount < 0)
        return RedirectToAction("Warning");

    return RedirectToAction("Index");
}

There are several advantages I experienced while applying this pattern. First, there are less layers of abstraction, while not neglecting readability nor maintainability. In a more conservative approach, I would end up with an account repository, abstracting my data access, and probably also an account service, abstracting my business logic. This repository, and service, would have multiple methods, which probably solve related, yet different problems. I would have to wade through all the methods in these classes to find the operation that is relevant for me.

When you model each operation as a class on its own, you define it more explicitly. And you’re able to adhere to the Single Responsibility principle even more. Practically, it’s now also easier to locate an operation in the codebase by just looking at the solution explorer.

One last, but not unimportant, advantage I see, is that it gets easier to group queries and commands per functionality and context.

We could test the example above against the actual implementation, and I don’t see anything wrong with that per se, but it isn’t always that practical; the setup of your queries can be heavy and complex, performance of your tests might be suffering etc.. One technique would be to create an interface per query and command and inject them, but that seems cumbersome, and ceremony heavy, and that isn’t something I’m willing to do.

Something in between could be introducing a Command- and QueryHandler.

public interface ICommandHandler
{
    void Execute(Command command);
}

public class CommandHandler : ICommandHandler
{
    public void Execute(Command command)
    {
        command.Execute();
    }
}

public interface IQueryHandler 
{
    T Execute<T>(Query<T> query);
}

public class QueryHandler : IQueryHandler
{
    public T Execute<T>(Query<T> query)
    {
        return query.Execute();
    }
}

These could be injected into the controller, and be used as an intermediary to execute queries and commands for us.

public class HomeController : Controller
{
    private readonly IQueryHandler _qryHandler;
    private readonly ICommandHandler _cmdHandler;

    public HomeController()
    {
        _qryHandler = new QueryHandler();
        _cmdHandler = new CommandHandler();
    }

    public HomeController(
        ICommandHandler cmdHandler, IQueryHandler qryHandler)
    {
        _cmdHandler = cmdHandler;
        _qryHandler = qryHandler;
    }

    [HttpPost]
    public ActionResult Foo(string user)
    {
        _cmdHandler.Execute(new WithdrawAmountCommand(user, 25));

        var totalAmount = _qryHandler.Execute<int>(new TotalAmountQuery(user));

        if (totalAmount < 0)
            return RedirectToAction("Warning");

        return RedirectToAction("Index");
    }
}

This technique would make it possible, and easy, to use stubs or mocks for queries and commands.

[TestMethod()]
public void Foo_should_withdraw_25_euros_from_the_account_of_jef_claes()
{
    var cmdHandler = new Mock<ICommandHandler>();
    var qryHandler = new Mock<IQueryHandler>();

    var controller = new HomeController(
        cmdHandler.Object, qryHandler.Object);

    controller.Foo("JefClaes");

    cmdHandler.Verify(
        h => h.Execute(It.Is<WithdrawAmountCommand>(
            c => c.Amount == 25 && c.User == "JefClaes")));
}

[TestMethod()]
public void Foo_should_redirect_to_a_warning_page_when_amount_negative()
{
    var cmdHandler = new Mock<ICommandHandler>();
    var qryHandler = new Mock<IQueryHandler>();

    qryHandler
        .Setup(h => h.Execute<int>(It.IsAny<TotalAmountQuery>()))
        .Returns(-1);

    var controller = new HomeController(cmdHandler.Object, qryHandler.Object);
    var result = (RedirectToRouteResult)controller.Foo("JefClaes");

    Assert.AreEqual("Warning", result.RouteValues["action"]);
}

I don’t think this should be introduced as an application-wide thing necessarily, but it could only be used when testing the actual implementation becomes hard or awkward.

But what if I need to inject dependencies into my commands or queries for testing - there is more to it than a database? I gave that some thought as well, and it seems inevitable to introduce extra abstractions, and make changes to the concepts discussed above; I’ll publish these tomorrow.

I’ve been entertaining lots of ideas on this topic lately, and I’m curious to hear yours.

Côte d'Opale

Mon, 08 Oct 2012 21:51:00 +0200

The girlfriend, me and another couple, returned from a short weekend at the French Opal Coast yesterday evening. Although we only planned this trip a week in advance, we still failed to anticipate how erratic the weather can be in October; setting up camp in the dark, on a sloughy underground, with a soft drizzle on your back, isn’t my idea of fun after a long day at work. The next morning, with destination Cape Blanc Nez, we drove over coastal routes between vast green fields, towards clearer skies. Passing white cliffs of clay and untouched beaches, we eventually halted at the obelisk commemorating the Dover Patrol for a while. Here you can clearly see the English coast on the other side of the canal; no wonder this spot was strategically ideal to safeguard the canal during World War I. Differing from the main route, we discovered the ramshackle remainder of a bunker. I think it’s fascinating how quickly something what once played an important role in our history, can lose its prestige, and be seen for what it is: a massive block of concrete.

I honestly was taken aback by the beauty of the Northern French coast. I had always unjustifiably ignored this area, and now I will definitely have to revisit.

Sunday, before leaving, we visited a local bric-à-brac. Just before returning to the car, my girlfriend spotted an antique camera satchel, which she thought would make a unique accessory - you know how girls are, right? The woman behind the stand wouldn’t let it go without the camera though, and asked too much for both, so we strolled on. Passing the same spot a few minutes later, she inquired how much we wanted to give her for it. Mistakenly, my girlfriend said dix-cinq (ten five) instead of using the more correct quinze (fifteen). The woman understood this as dis cinq (how about five), and doubled it to ten. Sold! And that’s how my girlfriend saved herself five euros without even having to haggle.

Thanks to our companion Lars Oliviers for letting me put his pictures in this post.

On job titles

Sun, 07 Oct 2012 19:22:00 +0200

It didn’t take long before I noticed how little job titles mean. In my first job, you were assured to be granted a fancy title after only having acquired a minimum seniority, if you knew how to play the game. A more important sounding job title was HR’s default bribe that often kept people from leaving for greener pastures, for a short while. But even after being upgraded from a stable cleaner to a Senior Barn Hygiene Technician, you’re still cleaning shit though.

It goes without saying that I grew a healthy aversion from the hypocrisy of job titles pretty early on. They are nothing but easy incentives to provide volatile satisfaction, and are mostly worth just as much as it costs your company to grant them.

I always believed that the title you carry is rather meaningless. Everybody knows of a senior developer, technical lead, or architect, you wouldn’t go to with any remotely technical question. Then again, I also know of people who can’t be bothered to bargain another title, but have an extensive skill set, and can carry a project to the finish line. Without them, projects would have crashed and burned before lift off.
How people perceive you isn’t something that can be enforced by a title, it’s the result of your everyday actions. When you display mastery in what you love doing, people will take notice, and the word will spread like a wildfire. Eventually, work that you love doing will come to you, regardless of your job title. People want you to be on their team, because they care about what you bring to the table, not about what your job title is.

While I still stand by this view, it might be a bit too naive. Job title dynamics differ from company to company, and need to be examined carefully on a case-by-case basis. For example, in the classic enterprise, a title can influence your job substantially. There, when assembling a team, management often doesn’t see an individual, but they see a resource playing a role. There, your reputation doesn’t always have the chance to precede itself, but your title does. It would be a damn shame if such a small detail, a few words on a piece of paper, would regularly put you off to a bad start. Each organization also has a distinct opinion on what a certain title entails. It’s worth studying these subtleties to be able to pick one that comes closest to what you want to do.

It’s what you do that defines you, not your title. Yet, it’s often indivertible to play along; titles can serve as a means to an end.

I’m interested in hearing your opinion. Do you think they are important, or do you just play along? Have you ever regretted not caring?

Bonus: relevant Futurama scene.

The 7 R's of Hypermedia

Mon, 17 Sep 2012 21:42:00 +0200

While most REST concepts are rather easy to grok, there is one concept which I found harder to understand at first: Hypermedia. Let it be that without this concept, you’re missing out on an extremely important strength of REST. Hypermedia enables you to build dumb - or smart, depending on your perspective - clients, which are mostly driven by the server. Practically, this is implemented as resources embedding links which allow the client to discover and navigate through your RESTful service.

Accidentally actually, I saw someone tweet a link over the weekend to a Deep Fried Bytes episode, featuring Darrel Miller, where he talks about the seven R’s of Hypermedia. I listened to it on my morning commute, and the 56 minutes long show contained one of the best summaries on the uses of Hypermedia I’ve heard or read so far.
Here are some of my notes…

Relations: Make your service discoverable and self-documenting by embedding links to related concepts.
Embedded resources: Instead of embedding resources (for example: a company logo) into your client, and having to deal with specific storage techniques, make them available through your service and use built-in HTTP caching.
Reference data: Provide links to where your client can find optional or required reference data. Populating a dropdownlist is a good example.
Redistribution of effort: Instead of putting a dumb load balancer in front of your machines, you can use your links to refer certain functionality to a specific machine.
Reduction of payload size: Show clients where they can get extra data. Think endless scrolling, or more detailed resources.
Reflow: Allow the server to control the flow. The server can use links to dictate the next available steps in a business process.
Restriction of functionality: The client can derive from the presence of a certain link whether a certain functionality is enabled or disabled.

You can find the full show here.

Slides and code from my Tunisia REST and ASP.NET Web API session

Thu, 13 Sep 2012 08:55:00 +0200

I just returned from a four day trip to Tunisia with Euricom. Next to indulging on the sun, food and all-inclusive cocktails, getting to know each other in a less professional setting, we spent somewhere around half of our days - and this is a rather generous estimate - doing technical sessions and workshops.

I gave a 90 minute long crash course on REST and ASP.NET Web API.

Content

REST:

Concepts
Architectural values
Left overs

ASP.NET Web API

Server/client
Exception handling
Content negotiation
Message handlers
IoC
Testing

You can find the slides on SlideShare. The code is hosted on GitHub.

Supporting the OPTIONS verb in ASP.NET Web API

Sun, 02 Sep 2012 18:27:00 +0200

ASP.NET Web API controllers support only four HTTP verbs by convention: GET, PUT, POST and DELETE. The full list of existing HTTP verbs is more extensive though. One of those unsupported verbs which can be particularly useful for API discovery and documentation is the OPTIONS verb.

The OPTIONS method represents a request for information about the communication options available on the request/response chain identified by the Request-URI. This method allows the client to determine the options and/or requirements associated with a resource, or the capabilities of a server, without implying a resource action or initiating a resource retrieval.

If we wanted to implement controller support for the OPTIONS verb, we could manually map an action to the verb by decorating it with an AcceptVerbs attribute, and making the action return a response with a relevant Access-Control-Allow-Methods header.

For a values controller, with a GET and DELETE action, it could look like this.

public class ValuesController : ApiController
{        
    public IEnumerable<string> Get()
    {
        return new string[] { "value1", "value2" };
    }                 
    
    public void Delete(int id) { }

    [AcceptVerbs("OPTIONS")]
    public HttpResponseMessage Options()
    {
        var resp = new HttpResponseMessage(HttpStatusCode.OK);
        resp.Headers.Add("Access-Control-Allow-Origin", "*");
        resp.Headers.Add("Access-Control-Allow-Methods", "GET,DELETE");

        return resp;
    }
}

If we now make an OPTIONS request to http://localhost:53314/api/values..

OPTIONS http://localhost:53314/api/values HTTP/1.1
User-Agent: Fiddler
Host: localhost:53314

..we receive following response.

HTTP/1.1 200 OK
Server: ASP.NET Development Server/10.0.0.0
Date: Sun, 02 Sep 2012 13:46:21 GMT
X-AspNet-Version: 4.0.30319
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,DELETE
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Content-Length: 0
Connection: Close

While this works, it’s not a great solution. We don’t want to maintain the list of supported verbs manually, and we certainly don’t want to repeat this for each controller.

ASP.NET Web API provides a nice way to have a more high level solution: HTTP message handlers. These basically behave as HTTP intermediaries, meaning we can intercept each OPTIONS request early, and bypass the whole request chain. Another useful component we can make good use of is the default ApiExplorer; this abstraction allows us to obtain metadata of our API’s stucture.

To create a new HTTP message handler, I inherited from the DelegatingHandler class, and overwrote the SendAsync method. Here we’ll intercept the request when the verb equals OPTIONS. An instance of the Api explorer can be resolved through the global configuration. I grab the requested controller from the request route data, and use that to search the Api explorer’s ApiDescriptions collection for its associated actions, and its associated verbs. Once I get the outcome of this lookup, and at least one verb is supported (*), I’ll return an OK response with the relevant headers, and thus short circuit the request.

public class OptionsHttpMessageHandler : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        if (request.Method == HttpMethod.Options)
        {
            var apiExplorer = GlobalConfiguration.Configuration.Services.GetApiExplorer();

            var controllerRequested = request.GetRouteData().Values["controller"] as string;              
            var supportedMethods = apiExplorer.ApiDescriptions
                .Where(d => 
                {  
                    var controller = d.ActionDescriptor.ControllerDescriptor.ControllerName;
                    return string.Equals(
                        controller, controllerRequested, StringComparison.OrdinalIgnoreCase);
                })
                .Select(d => d.HttpMethod.Method)
                .Distinct();

            if (!supportedMethods.Any())
                return Task.Factory.StartNew(
                    () => request.CreateResponse(HttpStatusCode.NotFound));

            return Task.Factory.StartNew(() =>
            {
                var resp = new HttpResponseMessage(HttpStatusCode.OK);
                resp.Headers.Add("Access-Control-Allow-Origin", "*");
                resp.Headers.Add(
                    "Access-Control-Allow-Methods", string.Join(",", supportedMethods));

                return resp;
            });
    }

    return base.SendAsync(request, cancellationToken);
    }
}

GlobalConfiguration.Configuration.MessageHandlers.Add(new OptionsHttpMessageHandler());

This second solution still yields the same result, but is far more scalable.

(*) When a resource can’t be read nor manipulated, it must not exist right?

Is serialization really that expensive?

Wed, 22 Aug 2012 13:19:00 +0200

While wading through an exotic codebase, I stumbled upon a static class named Convert. This class contained somewhere around 2700 (non-generated) lines of code, where each method manually converted some object to a simple textual representation. These methods were then used to convert requests and reponses to and from a remote third party service before logging them to the database for auditing reasons.

public static class Convert
{
    public static string PaymentRequest(PaymentRequest req)
    {
        var sb = new StringBuilder();

        sb.Append("Reference: " + req.Reference + " - ");
        sb.Append("NumberOfLicenses: " + req.NumberOfLicenses + " - ");
        sb.Append("PricePerLicense: " + req.PricePerLicense + " - ");
        sb.Append("CardNumber: " + req.CardNumber + " - ");
        sb.Append("Address: " + req.Address);

        return sb.ToString();
    }
}

My first thoughts were something along of the lines of “What the.. this is insanely stupid code.” This must be a PITA to maintain and be extremely error-prone. Looking at the solution now, it looks simple enough to move that to some infrastructure and have the conversion done by something more generic. Serializing to JSON comes to mind; interpretable by man ánd machine.

Trying not to jump to conclusions, I looked for one of the remaining team members, and asked why they made that decision. “Well”, he said, “Those remote service calls are expensive as is; it’s a slow connection, we have to encrypt everything going over the wire, and we can’t make them asynchronously. We optimized where we could. Including logging.”
I asked if they found serialization to be so expensive that it could warrant all the monkey code. He said yes, but that he couldn’t vouch for the decision since they never measured.

Later that day, I took five minutes to see how the two really compare. I have this code snippet lying around if I quickly want to profile something.

static void Profile(string description, int iterations, Action func)
{
    // clean up
    GC.Collect();
    GC.WaitForPendingFinalizers();
    GC.Collect();

    // warm up 
    func();

    var watch = Stopwatch.StartNew();
    for (int i = 0; i < iterations; i++)    
        func();    
    watch.Stop();
    
    Console.Write(description);
    Console.WriteLine("Time Elapsed {0} ms", watch.ElapsedMilliseconds);
}

I picked an average sized object graph and ran the benchmark.

var req = new PaymentRequest()
{
    Reference = "ABC123",
    NumberOfLicenses = 3,
    PricePerLicense = 15.99,
    CardNumber = "123456",
    Address = "Sunset Boulevard"
};

Profile("Serializing a request.", 1, () =>       Newtonsoft.Json.JsonConvert.SerializeObject(req));
Profile("Doing it manually.", 1, () => Convert.PaymentRequest(req));

This yielded following results.

Serializing a request. Time Elapsed 0 ms
Doing it manually. Time Elapsed 0 ms

Neglectable.

Turning up the number of iterations to 100 produces following results.

Serializing a request. Time Elapsed 9 ms
Doing it manually. Time Elapsed 1 ms

This time around, we see a huge relative difference; doing it manually is 9 times as fast. The absolute difference is still neglectable though.

As it turns out, for this specific scenario, with this specific serialization library, the overhead of serialization would be very tolerable. Other serialization libraries might produce less tolerable results though. It’s important to measure this stuff; I’m (re)learning almost daily that assuming is a mistake.

Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it. - H. James Harrington

My learning resources distilled

Sun, 19 Aug 2012 16:10:00 +0200

I have picked up a few new tools this summer (MongoDB, NancyFx and WebAPI), and it occurred to me that I’ve built certain habits these last few years in how I make use of all the learning resources out there.

I tried to identify all of them, to then categorize them, to finally order them according to in which phase of my study process I use them.

The written and spoken word

The first thing I look for online is documentation. It might be a coincidence, but all the documentation I found for these three tools was excellent. It seems obvious, now more than ever, that the early adaptation of new tools can be proportional to the quality of their documentation.

When there is no documentation - studying a concept instead of a technology, when the documentation is too dispersed, or when the quality is too low, I might order a book. The problem with technical books though, is that they are often too lengthy. I think the first publisher that cuts down on the number of pages, while guarding quality, will be pleasantly surprised by demand. I really like the “Little book” format Karl Seguin has been experimenting with.

When there is no documentation, and there are no books, I find myself turning to the spoken word. In general, I prefer reading over watching and listening because I can do that at my own pace, skip chapters, or quickly skim back for a look-up.

Although often on the slow side, the videocasts from Pluralsight are not bad. If you don’t want to subscribe to one of the paid videocast services, you can also watch recorded conference sessions instead. Before I commuted by train, I also listened to relevant podcasts.

Getting intimate

As I wrote earlier, I am a strong advocate of learning by doing. Find something small, and build it.

When I’m using a new tool, I also tend to regularly peek at its internals to improve my understanding of how the tool works under the covers. It’s never a bad idea to get intimate with the tools you rely on.

This becomes extremely easy when the tool you are using is OSS and uses a source control provider that makes it easy to browse or fork the source - read GitHub. If the tool isn’t OSS, you can still decompile the sources using one of the free decompilation options out there. If you have a Resharper 6 license, you can even decompile straight from Visual Studio.

Sipping from the fire hose

Once I start building something, I crank open the fire hose - Twitter Search, and take a sip. Although trying to consume all the information on a subject is hardly bearable, and feels as being waterboarded at times, I think it’s a necessary evil. Until you have assembled your personal digest, that is.

I’ll do my best to keep up with the Twitter Search feed for a while, and try to filter knowledgeable people to follow, blogs to subscribe to, and repositories to watch. Once I have a qualitative list of subscriptions, I’ll shut off the fire hose, and limit my consummation to my personal high quality-to-noise digest. This digest will serve as one of the biggest aids in continuously improving my knowledge of said tool.

Free pop quizzes

I always thought that if you really want to become good at something - become an expert, you must have spent time in the trenches, knees deep in shit. Experiences like this force you to broaden your understanding of a technology.

Next to problems you encounter producing things, you can also simulate this experience by following Stackoverflow questions and commit to help solving them. The less ambitious can also wait a while, and just skim over the questions and their answers. The same goes for mailing lists obviously.

I’m very curious to hear about the techniques you use to unravel all the information out there.

Finito

Mon, 30 Jul 2012 22:23:00 +0200

With this last post, our trip to Italy has come to an end. We have spent these last few days practicing several indispensable tourist activities: testing the ice cold lake water, to wriggle between other sunbathers on the packed beach right after, exuberant eating and drinking, reading novels, and sauntering between prullaria shops.

Fortuitously, one of our last evenings here was upgraded by fireworks over an already moonlit Lake Garda. This outlying location gave the fireworks show an extra notch. With each firework rocket, the pitch dark beach was illuminated for a few seconds. With each bang, the sound echoed over the mountains, making the water ripple to strand in the pebble beach eventually.

Milano

Fri, 27 Jul 2012 22:25:00 +0200

After Venice on Wednesday, we decided to stay in Torbole for a relaxing hike yesterday. As it turned out, the hiking trail that was described in the brochures as suitable for all and with a duration of 1h20, in reality took us 2h30 and sour calves to complete. The views of the lake’s valley and its school of windsurfers defying the wind were worth the sweat though. Arriving at the endpoint, and discovering that we had to wait over an hour for the bus to take us back, we stumbled upon one of the best hidden lake side terraces we’ve seen yet.

Since the days of our dolce vita are numbered, we made our last big trip today: Milano. We parked our car in the suburbs, where you pay only a quarter an hour and have a direct metro-connection to the Duomo. While we’ve had it with churches in general, we couldn’t let this one slip. The Milanese Duomo is the third biggest church in all of Christendom and could easily host a Metallica concert. The entrance to the church is free, but if you want to visit the rooftop, you pay 7€ on foot (250 stairs) or 12€ if you want to take the elevator. Although the panorama wasn’t as striking as the one we had seen yesterday, the architecture and craftsmanship of the rooftop details are worth to see up close. After that, we continued the ‘Old City’ itinerary of our travel guide, which includes the Galleria Vittorio Emanuele II and the Teatro alla Scala.

The Floating City

Thu, 26 Jul 2012 00:38:00 +0200

Yesterday we cruised along the eastern coastline of Lake Garda. Plan A was to visit Malcesine and take the cableway up to the top of Monte Baldo, but since the line at the ticket center resembled one of a Justin Bieber concert, we skipped the cableway. Maybe we’ll give it another try later this week, but then earlier in the day. We continued our journey along the coast, passing past several genuinely picturesque villages. We stopped in Garda for an elaborate lake side lunch, before continuing to our last destination that day, Sirmione. There we visited the medieval castello Scaligero and after climbing the numerous steps inside the castle, we rewarded ourselves with an oversized typical Italian gelato.

Today we made a trip to Venice. After a 2 hour drive, we parked our car at Venezia Mestre train station, to enter the Floating City by train. We soon found ourselves lost in a maze of fairy alleys and waterways, searching for the main tourist attractions. First we headed for the Mercato Rialto, which is probably the best market in the world, but we can’t confirm that since it was already cleared out when we got there at 2PM. After having some delicious fresh fruit at the foot of the ponte di Rialto, we crossed the famous bridge stacked with souvenir shops. Arriving at Piazza San Marco, we took the time to visit the impressive basilica San Marco and the gigantesque Doge’s palace. While these classics were definitely worthwhile, you can’t miss out on strolling around the less tourist neighborhoods to experience the authentic Venice.

Ferrari red. Lamborghini yellow.

Tue, 24 Jul 2012 00:56:00 +0200

After 14 hours and 34 minutes in the car, driving 1148km, over the German autobahn, swerving Austrian roads, and the Italian autostrade, defying overwhelming hail storms, roadwork and miles long tunnel congestions, we arrived at Torbole, Italy.

Torbole is a small village, nestled between the mountains, right at the edge of Lake Garda. While the lakeside view is astonishing, you have to pick the right time to truly experience it. In the daytime, the village is overwhelmed by tourists; miles long rows of cars queue to enter the village, pizzerias and ristorantes on every corner, Germans galore. This doesn’t take away that this location is a great launching point to visit other places in northern Italy though.

After a lazy day yesterday, we drove two hours South today, to Sant’Agata Bolognese and Maranello, to visit two legendary car manufacturers: Lamborghini and Ferrari. While both museums only take around 40 minutes to visit, I definitely was more impressed by the Lamborghini museum. Compared to the Ferrari museum, the Lamborghini museum seemed to exhibit far more (non-racing) eccentric and rare models.

Since both museums failed to entertain us for longer than an hour, we visited Verona on our way back. While we didn’t enter any tourist attractions, we enjoyed strolling through the old town, blending into the mass, only halting on a Roman square to pay way too much for an iced cappuccino which was served lukewarm.

Arrivederci.

Should I unit- or integration test my ASP.NET Web API services?

Sun, 15 Jul 2012 22:37:00 +0200

Over the last two weeks, preparing for a talk, I have been doing some research on ASP.NET Web API. After working my way through the API, and the implementation of certain features, I looked at testing.

Similar to ASP.NET MVC, Web API allows you to create relatively small building blocks, which can replace parts of, or be added to an existing default global setup. This makes it possible for us to test each component in isolation: controllers, dependency resolvers, filters, serialization, type formatters, messagehandlers and routing.

Testing in isolation helps a great deal to keep the magnitude of things to stuff in your head limited, and when you break something, you are able to quickly pinpoint the origin of the error. What unit testing fails to prove however, is the correctness of your code when all the little pieces are put together and configured. And let it be that this is extremely important when you’re exposing an API.

Looking at Web API, I would probably test most infrastructure in isolation - filters, type formatters, messagehandlers and serialization, because these tests will help pinpoint errors in components which will affect a large amount of other code. I wouldn’t test controllers and routing in isolation though.

I would test controllers and routing from a client’s perspective; meaning I’ll send a request to and endpoint on the server, I’ll go through the infrastructure, and I’ll assert the replied response. This would exclude false positives or false negatives which can originate when you unit test controllers and have to fake a bunch of infrastructure just to get it working, while you do include testing the effect the real infrastructure has on your incoming requests or outgoing responses.

An obvious counterargument might be starting and stopping a webserver in your tests, and the associated performance hit. This isn’t something to worry about with Web API though; HttpServer is just another HttpMessageHandler, which makes it possible to consume it using an HttpClient in-memory.

So let me show you some code I wrote trying out these thoughts. The first thing I did was exposing the hosting server’s configuration to my tests. This could be as simple as this.

public class ServerSetup 
{
    public static HttpSelfHostConfiguration GetConfiguration(string baseAdress)
    {
        var config = new HttpSelfHostConfiguration(baseAdress);
        
        var kernel = new StandardKernel();
        kernel.Bind<IResumeStore>().To<ResumeStore>();
        
        config.Routes.MapHttpRoute(
            "DefaultApi", "api/{controller}/{id}",
            new { id = RouteParameter.Optional });
        config.MessageHandlers.Add(new MethodOverrideHandler());
        config.DependencyResolver = new NinjectDependencyResolver(kernel);

        return config;
    }
}

Now in my test I can grab this configuration, and just overwrite the dependencies and the error detail policy. I can initialize an HttpClient by passing in an HttpServer instance which uses the modified configuration.

private HttpClient _client;

[TestInitialize]
public void Setup()
{
    var kernel = new StandardKernel();
    kernel.Bind<IResumeStore>().ToConstant(new Mock<IResumeStore>().Object);

    var config = ServerSetup.GetConfiguration("http://test");
    config.IncludeErrorDetailPolicy = IncludeErrorDetailPolicy.Always;                       
    config.DependencyResolver = new NinjectDependencyResolver(kernel);

    _client = new HttpClient(new HttpServer(config));
}

[TestMethod]
public void Post_Returns_HttpStatus_Code_Created()
{         
    var result = _client.PostAsync<Resume>(
            "http://test/api/resume", 
            new Resume("Jef", "Claes"), 
            new JsonMediaTypeFormatter()).Result;

    result.EnsureSuccessStatusCode();

    Assert.AreEqual(HttpStatusCode.Created, result.StatusCode);
}

Now I’m consuming my API almost exactly as a client would; my request goes through the routing, infrastructure and the controller. Infrastructure is still tested under isolation so finding problems there is easy, but I now have the advantage of testing routing, the effect of my real infrastructure and the logic in my controller actions in one simple integration test. Remember, we are testing a delivery mechanism, not an application; Web API controllers should be skinny as well.

One drawback I stumbled upon is discoverability of controller dependencies, but surprisingly that didn’t bother me much. I can still see an overview of all my dependencies in the controller’s constructor, it’s not a disaster to not have intellisense.

In general, I think this pragmatic approach to testing Web API implementations gets as much value as possible from automated testing with writing as little test code as possible, and without adding too much complexity.

What do you think about this approach to testing Web API solutions?

HtmlHelper to generate a top-level menu for areas

Sun, 08 Jul 2012 19:10:00 +0200

Last week, we had to set up a new ASP.NET MVC web application, using a somewhat customized Twitter Bootstrap build. Because the application has multiple functional contexts, we divided it in multiple parts using areas. Since these areas were a one-to-one mapping with the top-level menu items, we tried abstracting the creation of the menu items, ánd the management of setting the active item, into an HtmlHelper.

Let’s say, for this example, that we have six areas: Images, Maps, Play, Search, Video and Blog, and we want to render a list item for each one of them.

<div class="nav-collapse collapse">
    <ul class="nav">
        // Add list items         
    </ul>
</div>

The first solution we tried, assumed we needed an extreme low-maintenance solution, for which we would write some infrastructure once, and then be able to just create new areas without having to think about updating the top-level menu.

This solution reflected over all the types looking for classes which inherit from the AreaRegistration class. Once you get a list of all the area names, you can iterate over them and create a list item for each one of them, using an instance of UrlHelper to resolve the associated url. You have to impose some routing convention to make the url lookup robust though; in this example, I assume the default route is sufficient. To be able to mark the active area with a css class, you can get the active areaname from the viewcontext, and use that to compare to the iterand value.

public static class TopMenuExtensions
{
    private static IEnumerable<string> _areaNames;

    public static MvcHtmlString RenderTopMenuItems(this HtmlHelper helper)
    {
        var areaNames = GetAreaNames();
        var currentArea = helper.ViewContext.RouteData.DataTokens["area"] as string;

        var html = new StringBuilder();
        foreach (var areaName in areaNames)
        {
            var urlHelper = new UrlHelper(helper.ViewContext.RequestContext);
            var url = urlHelper.Action(string.Empty, string.Empty, new { area = areaName });
            // or similar
            // var url = urlHelper.RouteUrl(areaName + "_default");

            html.AppendLine(areaName.Equals(
                currentArea, StringComparison.OrdinalIgnoreCase) ? 
                "<li class='active'>" : "<li>");
            html.AppendLine(string.Format("<a href='{0}'>{1}</a>", url, areaName));
            html.AppendLine("</li>");
        }

        return new MvcHtmlString(html.ToString());
    }

    private static IEnumerable<string> GetAreaNames()
    {
        if (_areaNames == null)
        {
            _areaNames = Assembly
                .GetExecutingAssembly()
                .GetTypes()
                .Where(t => t.IsClass && typeof(AreaRegistration).IsAssignableFrom(t))
                .Select(a => (AreaRegistration)Activator.CreateInstance(a))
                .Select(r => r.AreaName);
        }

        return _areaNames;
    }
}

Now we can add following line to our _Layout file, and be done with it.

@Html.RenderTopMenuItems()

While this works, we stumbled upon an annoyance pretty quickly: we wanted to change the order of the menu items, but couldn’t. We took a step back, and momentarily considered decorating the arearegistrations with an attribute, but since the added value is so small compared to the extra complexity introduced, we decided just to throw the overengineering out.

public static MvcHtmlString RenderTopMenuItems(
                this HtmlHelper helper, IEnumerable<string> areaNames)
{        
    var currentArea = helper.ViewContext.RouteData.DataTokens["area"] as string;

    var html = new StringBuilder();
    foreach (var areaName in areaNames)
    {
        var urlHelper = new UrlHelper(helper.ViewContext.RequestContext);
        var url = urlHelper.Action(string.Empty, string.Empty, new { area = areaName });
        
        if (url == null)
            throw new NullReferenceException(
                string.Format("Couldn't find an url for the area {0}.", areaName));                
        html.AppendLine(areaName.Equals(
                            currentArea, StringComparison.OrdinalIgnoreCase) ? 
                            "<li class='active'>" : "<li>");
        html.AppendLine(string.Format("<a href='{0}'>{1}</a>", url, areaName));
        html.AppendLine("</li>");
    }

    return new MvcHtmlString(html.ToString());
}

The top-level menu items can now be rendered like this.

@Html.RenderTopMenuItems(new [] { "Search", "Images", "Blog", "Maps", "Play", "Video" } )

And the result looks like this.

While you can go at this problem in a lot of different ways, I think this is one of the most robust and most compact ways I have been able to write this so far. How have you solved this in the past?

On crime and document stores

Sun, 01 Jul 2012 16:04:00 +0200

Having worked with several storage paradigms over these last few months - from flatfiles, to NoSQL, to the big enterprisey relational databases -, I have spent plenty of time trying to make sense of all the options out there. It wasn’t until I watched one of the last episodes of The Wire season 3 that I had an epiphany regarding modeling data in document stores. Yes, I know, I tend to take those things home with me.

Somewhere half way through that episode, you see a detective going through one of those old school, gray and clumsy file cabinets, looking for a dossier on one of the recent murders. Once he finds the dossier, he takes it out of the drawer, scribbles down contact information of an eyeball witness, puts it back in the drawer, and closes the drawer again with a loud stomp.

And that file cabinet actually isn’t very different from a MongoDB collection; it stores and categorizes documents of the same type, and the trade-offs you have to consider when modeling dossiers, or documents, are basically the same.

Let me work the homicide department angle a little further..

A few months later, the murder is still not solved, and one of the detectives, being out of work, starts working the case again. He walks over to that same file cabinet, searches the file and goes through the data one more time. Short on leads, he decides to interrogate the witness again. So he takes the file, steps in his black Buick, and drives over to the address scribbled down next to the name of the witness. When he arrives, after a grueling car ride through morning rush hour, he finds himself standing in front of a vacant house. Son of a… He drives over to city hall, waits in line for 24 minutes, gets the new address, and heads over there. When he gets back to the office, empty handed, and several hours later, he is determined to prevent this from happening again. He suggests the other detectives either check and update their dossiers, or documents, every time somebody moves, or that they just write down a reference to the person in the dossier, and look up the data in the file cabinets at city hall. Since going through all the files every time somebody moves is not feasible, he convinces his chief to enforce the second option. Over time, people get a hang of it, and are content to be relieved of stale data in the dossiers. However, the more they introduce this system in other scenarios, the more they get frustrated doing the manual look-ups. They now first have to fetch the document in the file cabinet, and then go through five more cabinets, just to collect all the bits of the file.

To make matters worse, since it’s often hard to find something in the dossier, chain of command has introduced templates for each of the documents. Now each document has a fixed schema, a list of fields, of which each one is in a fixed position on the document, and some of them are even required. When you want to add a new file, they should be signed off by your superior first. Sigh.

Getting a taste after work, the detectives are discussing the new system. ‘It’s great that we now have a single source of truth, are rid of stale data, and don’t have to manually update everything when something changes. But damn, all the extra paperwork, all the fuss over formats and the going back and forth between file cabinets is getting old quickly.’ ‘However, I’m happy we could at least keep our OT slips and expense notes simple; just one document, which we can fill in and update as we please.’ They collegially gulp down their drinks, and signal the bartender to bring another round.

In this short story, you witnessed the detectives totally ruining their document store. By normalizing their documents and putting constraints on the formats, they no longer reap the benefits of using a document store. Now, they would be far better of with a relational solution.

I hope these analogies made some sense, and maybe made you think about, or even challenge the SQL dogma. What it all comes down to, is having enough knowledge to be able to pick the right tool for the job. Each paradigm has its merits, and as with any other decision in our field, trade-offs have to be considered. The way you can or want to model your data isn’t the only consideration to make though. While for some it is scalability and performance that makes NoSQL the obvious choice, for me it is the simplicity that does it. You don’t have to be an IT pro to install a server instance locally, nor to migrate your application to the cloud (MongoDB, for example, creates its collections on the fly). The way you talk to the database also becomes easier; the mismatch between your code and storage can become a lot smaller, while you also rid yourself of some of the SQL foo. This doesn’t mean that you don’t have to be considerate about how you query your data though; you still need common sense, but there is a lot less black magic you need to master.

NoSQL solutions seem to put the developer first; I see NoSQL, and particularly the document store flavor, not as a silver bullet, but as a great new asset to my toolbox.

Persisting model state when using PRG

Sun, 17 Jun 2012 16:48:00 +0200

I’ve been working on an ASP.NET MVC application in which we frequently apply the Post/Redirect/Get pattern. One of the direct consequences of applying this pattern is that you often want to persist the model state across redirects, so that you don’t lose validation errors, or the values of input fields.

To persist the model state across redirects, we can put TempData to work. The sole purpose of TempData is exactly this; persisting state until the next request.

public ActionResult Index()
{
    ViewData.Model = ...

    if (TempData.ContainsKey("ModelState"))
        ModelState.Merge((ModelStateDictionary)TempData["ModelState"]);

    return View();
}

[HttpPost]        
public ActionResult Update(AddModel inputModel)
{
    if (ModelState.IsValid)
        ...

    TempData["ModelState"] = ModelState;

    return RedirectToAction("Index");
}

So this works, but I found it to be a bit too cumbersome. And so did Davy Brion, he introduced a clean abstraction into the project, smoothing out some of the friction: making use of action filter attributes, we were able to eliminate duplication across controllers, leaving behind an AOP-ish taste.

The SetTempDataModelStateAttribute stores the model state in the TempData dictionary.

public class SetTempDataModelStateAttribute : ActionFilterAttribute
{
    public override void OnActionExecuted(ActionExecutedContext filterContext)
    {
        base.OnActionExecuted(filterContext);         
        filterContext.Controller.TempData["ModelState"] = 
            filterContext.Controller.ViewData.ModelState;
    }
}

While the RestoreModelStateFromTempDataAttribute restores it by pulling it out of TempData again, when it exists.

public class RestoreModelStateFromTempDataAttribute : ActionFilterAttribute
{
    public override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        base.OnActionExecuting(filterContext);
        if (filterContext.Controller.TempData.ContainsKey("ModelState"))
        {
            filterContext.Controller.ViewData.ModelState.Merge(
                (ModelStateDictionary)filterContext.Controller.TempData["ModelState"]);
        }
    }
}

So when we apply these attributes to the example at the beginning of this post, we end up with something like this.

[RestoreModelStateFromTempData]
public ActionResult Index()
{
    ViewData.Model = ...    

    return View();
}

[HttpPost]   
[SetTempDataModelState]     
public ActionResult Update(AddModel inputModel)
{
    if (ModelState.IsValid)
        ...
    
    return RedirectToAction("Index");
}

Very clean. I’m interested to hear how you handle these concerns when using the PRG pattern.

Making my first NancyFx test pass

Mon, 11 Jun 2012 16:47:00 +0200

Like I already said last week, I have been dabbling a bit with NancyFx lately.

This week I took a serious look at testing Nancy modules and Razor views. Due to Nancy’s defaults and conventions, it takes a little while to set up Nancy in a test context. Then again, Nancy’s granularity makes it simple enough to set up a solid test infrastructure by replacing some of its building blocks.

Like always, I had to go through several iterations to get it right.

A first attempt

The first test I wrote looked something like this. This test simply asserts whether a GET request to the root of my application returns a 200 OK status code.

[TestMethod]
public void root_should_return_response_ok()
{        
    var browser = new Browser(new DefaultNancyBootstrapper());

    var response = browser.Get("/");

    Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
}

Assert.AreEqual failed. Expected:<OK>. Actual:<NotFound>.

Telling Nancy which modules to use

I figured out I had to tell Nancy which modules to load. You can do this by using the configurable bootstrapper, which gives you an API to configure parts of Nancy yourself.

var bootstrapper = new ConfigurableBootstrapper(with =>
{
    with.Module<RootModule>();               
});
var browser = new Browser(bootstrapper);

Running this test, I no longer got a NotFound result, but an exception. Making progress.

Test method RootModuleTests.root_should_return_response_ok threw exception: 
System.Exception: ConfigurableBootstrapper Exception ---> 
                    Nancy.RequestExecutionException: Oh noes! ---> 
                    Nancy.ViewEngines.ViewNotFoundException: Unable to locate view 'HomeView'
Currently available view engine extensions: sshtml,html,htm
Locations inspected: HomeView,views/HomeView,views/HomeView,/HomeView,views/Root/HomeView,Root/HomeView

Where is the view engine?

On inspecting the exception, I noticed that the cshtml view engine extension was missing from the list.

To add the view engine, you can just add a reference to the Nancy.ViewEngines.Razor assembly. Nancy will pick up the new engine automatically.

Test method RootModuleTests.root_should_return_response_ok threw exception: 
System.Exception: ConfigurableBootstrapper Exception ---> 
                    Nancy.RequestExecutionException: Oh noes! ---> 
                    Nancy.ViewEngines.ViewNotFoundException: Unable to locate view 'HomeView'
Currently available view engine extensions: sshtml,html,htm,cshtml,vbhtml
Locations inspected: HomeView,views/HomeView,views/HomeView,/HomeView,views/Root/HomeView,Root/HomeView

Locating the views

This time around we get a slightly different exception; the razor view engine extension is available, but Nancy is still having trouble finding the views - which is normal. Nancy is running in a different context now, so the default rootpathprovider will return the wrong path. Implementing another one is children’s play.

public class TestRootPathProvider : IRootPathProvider
{    
    public string GetRootPath()
    {
        return "C:\MyProject\";
    }
}

I discovered that you don’t even need to use the configurable bootstrapper to override the IRoothPathProvider; Nancy seems to pick up the new implementation by herself.

The hardcoded path will work on your machine, but not on the build server. In my second iteration, I made the implementation smarter by traversing my way through parent directories looking for my views folder - starting from my tests out folder all the way up to my web application folder.

public class TestRootPathProvider : IRootPathProvider
{
    private static string _cachedRootPath;

    public string GetRootPath()
    {
        if (!string.IsNullOrEmpty(_cachedRootPath))
            return _cachedRootPath;

        var currentDirectory = new DirectoryInfo(Environment.CurrentDirectory);
        
        bool rootPathFound = false;            
        while (!rootPathFound)
        {
            var directoriesContainingViewFolder = currentDirectory.GetDirectories(
                        "Views", SearchOption.AllDirectories);
            if (directoriesContainingViewFolder.Any())
            {
                _cachedRootPath = directoriesContainingViewFolder.First().FullName;
                rootPathFound = true;
            }

            currentDirectory = currentDirectory.Parent;
        }

        return _cachedRootPath;
    }
}

Result: Passed

I was happy to find out that this technique works for the MSTest runner, NCrunch and Appharbor.

There is one more way you can go at this though, which works and maybe even is the ‘recommended’ technique, but also is a lot more cumbersome: use the DeploymentItem attribute to copy the views folder to your tests out folder.

To recapitulate

I hope this post documented some useful techniques to get started testing Nancy’s modules and views:

Use the configurable bootstrapper to add your modules
Reference the correct view engine
Implement a rootpathprovider telling Nancy where it can find her views

Only a few days ago, @thecodejunkie made a ticket on GitHub addressing some of these issues. I’m pretty confident we can smooth out these rough edges to make Nancy tests also walk the super-duper-happy-path.

Book review: The Art of Agile Development

Sun, 10 Jun 2012 19:42:00 +0200

While I have - obviously - read the Agile Manifesto before, and regularly click through to articles on agile, I had never read an extensive work on it. Browsing for a good book, I was advised by a peer to get The Art of Agile Development.

I wholeheartedly believe in the Agile Manifesto, but somewhere along the way Agile - with a capital A - got somewhat of a bad rep. The authors of this book, James Shore and Shane Warden, already predicted this five years ago. Throughout the book, I never had the feeling they were connected to the Agile mob, but just genuinely care about working software.

I fully expect the big consulting companies to start offering Certified Agile Processes and Certified Agile Consultants - for astronomical fees, of course - any day now. Please don’t get sucked into that mess.

The book consists of three parts: getting started, practicing XP and mastering agility. Notice the title of the second part, this is a book on agile development using the Extreme Programming methodology. This should be mentioned in the title.

The second part is also the dominant chunk of the book - 271 pages to be precise -, in which 37 agile development practices are covered. These practices are bundled into five categories: thinking, collaborating, releasing, planning and developing. Each practice follows the same recipe: breakdown, questions, results, contraindications, alternatives and further reading. These recipes go into detail, making this book suitable as a reference. As a result, it often also reads as reference material; I found it hard at times to stay attentive. The material is a bit tedious, sometimes a bit repetitive, and while some of the real world experiences are very interesting, the made up stories often read as a children’s book. Nonetheless, this doesn’t make the information in this book any less valuable. The authors have lived it through and through, and possess a very healthy and pragmatic vision on agile software development.

Ultimately what matters is success, however you define it. The practices, principles, and values are merely guides along the way. Start by following the practices rigorously. Learn what the the principles mean. Break the rules, experiment, see what works, and learn some more. Share your insights and passion, and learn even more.

I rate this book 4/5. You can get it on Amazon.

The 'everyone should learn to code' dilemma

Sun, 03 Jun 2012 16:40:00 +0200

Back when I was working on software for fire departments, we started thinking about reworking a critical piece of our solution: deployment plans. In a fire department domain, deployment plans help to make a suggestion to the dispatcher about which units should be dispatched to a location when an incident is called in. The suggested composition of units depends on a wide range of variables: availability, response time, ranks, type of incident, required tools, … , even politics. Originally, people high enough in rank could compose these plans using a decision tree-like UI. However, as it turned out, this UI was insufficient; not all variables and conditions were available. Since this was no custom built tool, we had to work around it by composing incomprehensible decision trees or by tricking the underlying services. When talking about how we could do better, we hit a wall pretty soon. We thought about building our own - but more extensive - UI, and damn, even designing a DSL crossed our minds.

Along the process, I heard that Configuration Complexity Clock ticking, and I couldn’t stop myself from thinking that if everyone knew how to code - just some boolean logic and control flow would suffice -, we wouldn’t have gotten into this mess. It seemed impossible to build something that would be intuitive, and still fulfill all the requirements; some things just seem to be best expressed in code.

I empathize with the proponents of teaching everyone how to code. On the other hand, I’ve been watching this movement from a safe distance; there are two sides to the same coin.

Just a few weeks ago, a peer shared one of those horror stories which originate when non-professional developers take matters into their own hands. The IT department of this big company was looking for a piece of software which could standardize the way employees make hardware and support requests. Shopping around, they found the existing products couldn’t satisfy their needs, or they would take a considerable cut out of their yearly budget. Eventually, a system administrator who knew just enough about programming to be dangerous stepped up, and rolled his own tool. For over two years, all was good with the world. Maybe the solution wasn’t very pretty, and a bit on the slow side, but overall - and most importantly - it got the job done. Now, the original developer has decided he can’t be bothered maintaining his brainchild any longer. The peer who told me this story, was also the guy chosen to take over maintenance and feature requests. They assured him that it wouldn’t be too much trouble; just a little feature here and there, and maybe the exceptional bug. What he found however, were the things nightmares are made of; a - classic - ASP.NET web application with just one web page, written in VB.NET, containing over 20k LOC, with updatepanels nested five levels deep.

And this is the perfect example of how we have to be very careful with the ’everyone should learn to code’ meme. Slinging code is easy, but writing good code is hard, and takes a lot of practice. Software might be eating up the world, but maybe it’s for the best we don’t turn it into a monstrous glutton.

The open plan fallacy testimonials

Sun, 27 May 2012 15:36:00 +0200

I wrote an article titled ‘The open plan fallacy’ just two weeks ago. Earlier this week a similar article was published by the New York Times. The content of that article wasn’t particularly extraordinary, but the comments were. I waded through all of them on my daily commute, and it’s really hard to find one in favor of open plan offices - people seem to be enraged.

I handpicked some of the most interesting ones.

Research by Emberson et.al. (Psychological Science, 2010) demonstrate both the impairment in performance for people forced to listen to half a phone conversation and the neurological impossibility of “tuning out” these overheard conversations. The office experience has been described as trying to drive a car full of people all talking on their cell phones.

It’s preferable to have to make the effort to collaborate than the effort to focus on one’s work amidst a distracting open workplace. My profession, architecture, does require a high degree of collaboration, but also a need for intense, individual concentration working on complicated 3D computer models and details.. and long hours being in “the zone”. My firm provides private offices for all of its staff and a variety of collaborative and recreational environments.. particularly the latter, recognizing that just by hanging out together for part of the day, individuals from different project teams and pursuits “cross-pollinate”, come up with imaginative ideas, influence and inspire eachother.

Cubicles were invented as a way to shove more people into a smaller space and save money for corporations. The entire meme about “improved workplace communications” was invented by consultants as a smokescreen. Period. End of story.

“For example, someone whose job requires intense concentration (e.g. computer programmer) needs absolute silence. Programmers who do not get this silence are likely to make mistakes.”
I’ve been around a lot of programmers, among lots of other kinds of workers, and I’d say that 90% of the developers I saw were in a large room with others all around them. It’s actually one of the least likely jobs to provide someone with an office, despite the fact that your diagnosis about what the job requires is absolutely right, in my opinion.
All it takes is a few months in the world of corporations to understand that Dilbert is really a documentary.

I currently work in an open plan office and absolutely hate it. Not having an office with a door that others need to knock on before disturbing me has led to non-stop disturbances all day long. Not to mention having to listen to nonstop chatter of those around me. It is an incredibly inefficient - and I may add, unprofessional - way to work. Any money saved on rent is surely made up for in lost productivity.

I’ve noticed that the person who decides on the open cube layout usually sits in an office.

Having participated in the design of office environments, where we used low cubicle walls or even no walls to support certain kinds of collaborative work, with the full involvement of the employees in that design, it was always distressing to see the ‘open office’ faddishly embraced by management everywhere, regardless of actual practices required for the work. Consultancies, as usual, led the way in yet another blind embrace of ‘innovation’.
Why would individual ’entrepreneurs’ - to take one example from this story - want or need to be able to see and hear each other whilst working? Are the activities in which they are engaged intimately linked, are the tasks often (and necessarily) performed conjointly, is their own working division of labour a concerted one? What do these entrepreneurs themselves think about such matters? (Well, as the story makes plain, they have very clear answers, voting with their headphones!) Anyway, these are the kinds of very practical questions about work activities and worker needs that should drive office design.

In my article, I advocated isolating teams instead of individuals, but most commenters seem to be heavy supporters of private offices - including walls and a door. I never experienced that in a professional environment, so I couldn’t say if that would work for software development. Can you? Collaboration and communication between developers is a necessity. Fostering that just seems hard when everyone is in a separate place - definitely for young teams. While comparing office layouts, I wondered how my proposed solution would scale; how many people can you put in one team before you encounter the same undesired open plan side-effects?

Painless database logging with mongoDB

Sun, 20 May 2012 17:18:00 +0200

While browsing the source code of the ELMAH mongoDB provider, I learned about a special type of collections: capped collections.

From the mongoDB documentation:

Capped collections are fixed sized collections that have a very high performance auto-FIFO age-out feature (age out is based on insertion order). In addition, capped collections automatically, with high performance, maintain insertion order for the documents in the collection; this is very powerful for certain use cases such as logging.

This is such a killer feature. Logging to the database can be extremely useful, but also rather expensive. Using this feature, you can turn on database logging without too many worries.

Insertion into a capped collection is ridiculously fast. To get an idea of how fast it really is, I did some measuring on my own humble machine. I managed to insert 10.000 small documents in less than 3.7 seconds. The headaches of tweaking buffer sizes and rolling asynchronous appenders seem to be miles away.

Something which religiously gets ignored until shit hits the fan, is log table maintenance. With a capped collection there is no need to set up a database job that periodically cleans the logging table. You just set a fixed size, and you’re done. No more middle-of-the-night support calls when the logging table is eating up all the disk space.

Creating a capped collection with the C# driver can look like this.

var server = MongoServer.Create("mongodb://localhost/");
var db = server.GetDatabase("PlayGround");

var options = CollectionOptions
    .SetCapped(true)
    .SetMaxSize(5000)
    .SetMaxDocuments(100);

if (!db.CollectionExists("Log"))
    db.CreateCollection("Log", options);

Now that’s easy sailing.

The open plan fallacy

Sun, 13 May 2012 16:20:00 +0200

I haven’t worked in a whole lot of places, somewhere around four, but every single one of them used an open plan to structure their workplace. From what I hear from others, it’s the standard.

There are a few things to say about the advantages of an open office layout. They should stimulate communication, create more opportunities for observing and learning from others and be more cost-effective. I’m afraid it’s the latter which is the biggest driver though.

In reality open plans really aren’t all that great. The noise alone has proven to reduce productivity by one third. When I look around, I see plenty of signs that people have a hard time getting their job done: programmers buying $500 noise-cancelling headphones in an attempt to keep the environmental noise out, project managers camping in free meeting rooms trying to focus on their number wizardry or even whole teams occupying a meeting room days before a release.

What bothers me the most is overhearing other teams, those distractions amount to nothing at all. What’s the value of me overhearing a discussion on some obscure Sharepoint problem? None. What are the benefits of listening to your team’s Friday after lunch bullshitting? Nothing. Team dynamics differ, that’s normal, but they shouldn’t disturb others.

The extreme alternative would be private offices, but I’m pretty confident that’s far from perfect as well.

What I suggest, is installing each team in their own fully isolated area, free from distractions caused by other teams. This would still do right to all the advantages of an open plan, while taking away some of its biggest bottlenecks. By installing each team in their own cocoon, you create an environment where overhearing conversations on the next feature is an added value, where you can quickly short-circuit discussions on an architectural decision or where you might even concentrate on a problem for 45 minutes straight. The only advantage which is partially gone is cost-effectiveness; there needs to be some flexible infrastructure in place, that makes it easy to swiftly adapt the environment when team compositions change.

Instead of always focusing on that short term budget win, we need to start paying attention to the big picture again: offices should enable teams, not sabotage them.

Is there anybody who genuinely loves open plan offices? If not, what alternative would you prefer?

Why I will always love RSS

Sun, 06 May 2012 16:50:00 +0200

There has been a lot of noise in the tech community earlier this year about how RSS is supposedly having one foot in the grave. If that would be even remotely true, I hope it dies with its boots on. The herald would be browsers and social networking sites killing or hiding support for RSS. While that may be true, their motives shouldn’t rig our opinions.

RSS has never worked out for the regular consumer, not directly anyways. So I get why browsers are dropping support for it, I am not even disappointed. Most popular social networks have enough traction by now so that they can safely start fencing their gardens with the purpose of bringing more money in. Also reasonable.

What startles me is when peers start advertising the death of RSS. The explanations I hear are somewhere along the lines of “Why use RSS? I count on the people I follow on Twitter to share links to good information”, or similarly “I just check Hacker News a few times a day to read up on the latest news”. I strongly believe the contemporary fetish of liking and sharing cheapens the way we consume our information. Don’t get me wrong, I do see value in community driven content, but there’s also a lot of dirt and sensationalism. Some days it feels like I’m reading the front page of a cheap tabloid.

I used to subscribe to a bunch of feeds, but today I’m pickier; I subscribe to authors who share things I care about and who inspire me in some way. By subscribing to their feed, I’m not missing a single update. This forces me to absorb it all, even though most of their content doesn’t go viral. This liberates me from feeling the urge to be connected all the friggin’ time, plus more importantly, there is a gold mine of wisdom to be found in the non-controversial content out there.

I love RSS. It just works, and enables me to learn so much, without having to be connected constantly or having to rely on others to tell me what to consume.

Subscribe to my RSS feed here.

My InfoQ article on HTML5 offline web applications

Wed, 02 May 2012 20:09:00 +0200

After writing a few things on HTML5 offline web applications earlier this year, I got contacted by InfoQ to write an in-detail article on the subject for them. I hesitated at first, because I was afraid that it would feel too much like work, taking the fun out of my writing. Turns out it wasn’t like that at all. The guys at infoQ were really relaxed to work with, asking interesting questions and giving useful feedback, without forcing me into a certain direction. A pleasant experience.

You can find the article here.

Some Servicelocator pattern stinks

Tue, 17 Apr 2012 20:33:00 +0200

I have been working on a somewhat legacy codebase which makes use of the Servicelocator pattern. Although I always thought of Dependecy Injection to be the superior pattern, I was pleased to find some Inversion of Control implementation in there. Working with the codebase, I discovered first hand how easily, when used without caution and discipline, the Servicelocator pattern can introduce code rot.

I will walk you through some of the issues I have with the Servicelocator pattern, mostly looking at it from a test perspective. It’s interesting how you can often quickly discover friction in a codebase by just looking at, or writing, tests.

The first thing that rubs me the wrong way is that by using the Servicelocator pattern, you make each class dependant on the servicelocator, degrading the purity of the models. Although this is a conceptual consideration, it considerably affects development.

What I really like about Dependency Injection is that you can look at the constructor (or properties) from a class and instantly know its dependencies. The Servicelocator pattern makes it too easy to hide them.

To demonstrate some of the beef I have with the Servicelocator pattern, I wrote a FireStation class which has one public method Alert. I chose an example which is not the de facto OrderService or ShoppingBasket implementation, but which still is small enough to grasp easily.

So let’s have a look at this FireStation class. I know it’s a ridiculously naïve implementation, but good enough as an example.

public class FireStation
{
    public void Alert(Incident incident)
    {
        SendPagerMessage();
        SendEmail();

        if (incident.Priority == IncidentPriority.High)            
            ActivateSiren();                        
    }

    private void SendPagerMessage()
    {
        ServiceLocator.Current.GetInstance<IPagingTerminal>().SendMessage();
    }

    private void SendEmail()
    {
        ServiceLocator.Current.GetInstance<IMailServer>().SendMailMessage();
    }

    private void ActivateSiren()
    {
        ServiceLocator.Current.GetInstance<IImmoticaServer>().ActivateSiren();
    }
}

There is one public Alert method, which takes an Incident instance, and uses three alarmation channels to alert firemen: paging, email and a siren.

So, let’s imagine I wanted to test whether the siren is activated when there is a high priority incident. I would just start writing the test, to ultimately find out I have no idea what to assert.

[TestMethod()]        
public void Alert_activates_the_siren_when_the_priority_is_high()
{
    var fireStation = new FireStation();

    fireStation.Alert(new Incident(IncidentPriority.High));

    Assert.Inconclusive("How can I verify the siren is activated?");
}

Looking at the collapsed method definitions, I still don’t know. There is no constructor, so the dependencies aren’t resolved in there. Nothing left to do but inspect the Alert method content, and look at the ActiviateSiren method implementation. That’s where I eventually find out I need to mock the IImmoticaServer interface to assert the interaction.

So I do that, by creating the mock, setting up the container and setting it as the provider for my servicelocator.

[TestMethod()]    
public void Alert_activates_the_siren_when_the_priority_is_high()
{            
    var immoticaServer = new Mock<IImmoticaServer>();

    var kernel = new StandardKernel();
    kernel.Bind<IImmoticaServer>().ToConstant(immoticaServer.Object);
    
    ServiceLocator.SetLocatorProvider(() => new NinjectServiceLocator(kernel));    

    new FireStation().Alert(new Incident(IncidentPriority.High));

    immoticaServer.Verify(i => i.ActivateSiren(), Times.Once());
}

Now I’m polluting my test with responsibilities it shouldn’t really care about. I could probably move that into some infrastructure or the test setup, but still, I’m writing code that could be easily averted.

Anyways, I should now be able to verify the interaction with the IImotticaServer implementation. Boom, red test. You probably figured this one out already, but if I hadn’t expanded the private methods, you would have had no idea that I needed two extra stubs to make the test pass; one for the IPagingTerminal dependency and one for the IMailServer dependency.

kernel.Bind<IPagingTerminal>().ToConstant(new Mock<IPagingTerminal>().Object);
kernel.Bind<IMailServer>().ToConstant(new Mock<IMailServer>().Object);

After binding these two dependencies the test finally turns up green.

This development experience was horrible. The Servicelocator pattern fails at making it easy for me to discover dependencies, leading to an interrupted test flow. Another harmful side-effect is that it also becomes harder for me to read code and understand the high level interactions between objects, without getting knees deep into the implementation details.

By hiding your dependencies, you also promote ignorance to a problematic amount of dependencies. I discovered three in this example, but as this class grows, more and more of them will be buried inside the method implementations.

To make it easier to avoid and spot these smells earlier on, without dropping the Servicelocator pattern as a whole, you could move service resolution to the constructor. This isn’t a completely safe refactoring though. It’s possible that some instantiations are expensive, and should be implemented to be lazy initialized.

There is one more annoyance I would like to spout in this post. The Servicelocator pattern doesn’t encourage good OO. I’m not an OO purist, but seeing dozens of static classes resolving their dependencies through a servicelocator sends shivers down my spine. I would rather pick an Inversion of Control technique which basically makes it impossible to do this.

This post turned out to be longer than I expected it to be, and I even left some annoyances uncovered. I hope I succeeded in explaining my beef with the Servicelocator pattern. I trust that the general consensus will soon dictate that this pattern easily leads to marginal code, and that it actually is an anti-pattern which should be avoided if possible.

Visualizing the offline application cache update progress

Wed, 11 Apr 2012 16:27:00 +0200

I wrote about using the HTML5 application cache earlier, mostly focusing on generating and serving the manifest file using ASP.NET MVC. I also bitched about how not one browser I know of gives an indication of the application cache update progress.

Today I wanted to write something about how you can visualize the application cache update progress yourself.

The applicationCache API has several useful and rather straightforward events we can handle to inform the user of the update progress.

window.applicationCache.onchecking = function (e) {
    updateCacheStatus('Checking for a new version of the application.');
};
window.applicationCache.ondownloading = function (e) {
    updateCacheStatus('Downloading a new offline version of the application');
};
window.applicationCache.oncached = function (e) {
    updateCacheStatus('The application is available offline.');
};
window.applicationCache.onerror = function (e) {
    updateCacheStatus('Something went wrong while updating the offline version 
                        of the application. It will not be available offline.');
};
window.applicationCache.onupdateready = function (e) {
    window.applicationCache.swapCache();
    updateCacheStatus('The application was updated. Refresh for the changes to take place.');
};
window.applicationCache.onnoupdate = function (e) {
    updateCacheStatus('The application is also available offline.');
};
window.applicationCache.onobsolete = function (e) {
    updateCacheStatus('The application can not be updated, no manifest file was found.');
};

One event that is particularly helpful is the progress event. This event fires every time a resource is downloaded and contains three useful attributes we can use to display the download progress: lengthComputable, loaded and total. These attributes should be fairly self-descriptive, but here is the relevant snippet of the W3C specificiations.

For each cache host associated with an application cache in cache group, queue a post-load task to fire an event with the name progress, which does not bubble, which is cancelable, and which uses the ProgressEvent interface, at the ApplicationCache singleton of the cache host. The lengthComputable attribute must be set to true, the total attribute must be set to the number of files in file list, and the loaded attribute must be set to the number of number of files in file list that have been either downloaded or skipped so far. The default action of these events must be, if the user agent shows caching progress, the display of some sort of user interface indicating to the user that a file is being downloaded in preparation for updating the application.

To have a text that updates the percentage of downloaded resources on every download, I came up with this.

window.applicationCache.onprogress = function (e) {               
    var message = 'Downloading offline resources.. ';

    if (e.lengthComputable) {
        updateCacheStatus(message + Math.round(e.loaded / e.total * 100) + '%');
    } else {
        updateCacheStatus(message);
    };
};

These attributes seem to be implemented in WebKit browsers, but not in Firefox. Firefox will fall back to the static message ‘Downloading offline resources..’. Internet Explorer doesn’t support the offline application cache as a whole.

I’m sure more creative souls have it in them to build a really elegant and visually pleasing progress indication using this technique. I’m curious to hear (or see!) how you would represent the update process.

Check for local file browsing with JavaScript

Mon, 02 Apr 2012 15:34:00 +0200

Because I do most of my research while commuting by train, I often pull entire websites offline using httrack. While browsing the [jQuery Mobile documentation](http://jquerymobile.com/demos/1.1.0-rc.1/ locally this morning, I stumbled upon following gem.

I was curious to see how they determine whether a page is browsed locally or not. Looking into the source, I was a bit dissapointed to find nothing but plain common sense. The trick is comparing the protocol of the current location with known local protocols.

if ( location.protocol.substr(0,4)  === 'file' ||
        location.protocol.substr(0,11) === '*-extension' ||
        location.protocol.substr(0,6)  === 'widget' ) {
    // Disable AJAX support etc
}

If you would want to use that check in multiple locations in your codebase, you might want to extend the location object with an isLocal function.

window.location.constructor.prototype.isLocal = function() { 
    return this.protocol.substr(0,4)  === 'file' || 
            this.protocol.substr(0,11) === '*-extension' || 
            this.protocol.substr(0,6)  === 'widget'; 
}

The function could be used like this.

if (window.location.isLocal()) {
    // Disable AJAX support etc
}

Add images to a GitHub readme

Sun, 01 Apr 2012 19:32:00 +0200

Today I wanted to add some screenshots to a GitHub readme for the sake of documenting. While this wasn’t particularly hard, I do had to iterate a few times before I got it right.

Hosting the images

You could simply add the images to your repository and reference them using the raw url’s, but this isn’t very efficient. Using this method, every request needs to go through GitHub’s application layer. It’s far better to make use of GitHub Pages, a feature purely designed to publish web content. I also like how you’re not polluting the repository this way.

Referencing the images

I’m using the - oh so beautiful and simple - markdown format for most of my readme’s. The syntax for embedding images looks like this.

[My image](username.github.com/repository/img/image.jpg)

I hope this post fills in the void I stumbled upon when googling for useful pointers earlier today.

How a web application can download and store over 2GB without you even knowing it

Sun, 25 Mar 2012 13:11:00 +0200

I have been experimenting with the HTML5 offline application cache some more over the last few days, doing boundary tests in an attempt to learn more about browser behaviour in edge cases.

One of these experiments was testing the cache quota.

Two weeks ago, I blogged about generating and serving an offline application manifest using ASP.NET MVC.

I reused that code to add hundreds of 7MB PDF files to the cache.

public ActionResult Manifest()
{     
    var cacheResources = new List<string>();
    var n = 300; // Play with this number

    for (var i = 0; i < n; i++)
        cacheResources.Add("Content/" + Url.Content("book.pdf?" + i));

    var manifestResult = new ManifestResult("1")
    {
        NetworkResources = new string[] { "*" },
        CacheResources = cacheResources
    };

    return manifestResult;
}

I initially tried adding 1000 PDF files to the cache, but this threw an error: Chrome failed to commit the new cache to the storage, because the quota would be exceeded.

After lowering the number of files several times, I hit the sweet spot. I could add 300 PDF files to the cache without breaking it.

Looking into chrome://appcache-internals/, I can see the size of the cache being a whopping 2.2GB now for one single web application.

As a user, I had no idea that the website I’m browsing is downloading a suspicious amount of data in the background. Chrome (17.0.963.83), nor any other desktop browser that I know of, warns me. I would expect the browser to ask for my permission when a website wants to download and store such an excessive amount of data on my machine.

Something else I noticed, is that other sites now fail to commit anything to the application cache due to the browser-wide quota being exceeded. I’m pretty sure this ‘first browsed, first reserved’ approach will be a source of frustration in the future.
To handle this scenario we could use the applicationCache API to listen for quota errors, and inform the user to browse to chrome://appcache-internals/ and remove other caches in favor of the new one. This feels sketchy though; shouldn’t the browser intervene in a more elegant way here?

What are your thoughts? What would you want your browser to do in these scenarios?

Sent from my phone

Sat, 17 Mar 2012 19:46:00 +0100

According to Matt from 37signals the line “Sent from my iPhone” at the bottom of an email means this:

Let’s be honest. “Sent from my iPhone” really means “I’m not going to bother to proofread and correct this because it would take me an extra 30 seconds.”

I agree. I too use this line as an excuse to write a terse message and omit proper salutations.

However, I also think these four simple words greatly helped the viral growth of the mobile phone. Having early adopters brag about how they’re sending emails from their fancy new phone must have been an invaluable form of word-of-mouth advertising.

I think it’s safe to say that those four words first are a marketing technique and then a convenient way of letting your correspondents know that there might be a typo in your message. If not, the default line wouldn’t include the brand name.

I’m turning this thing into a rambling though. What I really wanted to share is a snippet of the much recommended book The Lean Startup, where Eric Ries shares a story on how Hotmail used a similar technique in 1996 to fuel their viral engine of growth.

But everything changed when they made one small tweak to the product. They added to the bottom of every e-mail the message “P.S. Get your free e-mail at Hotmail” along with a clickable link.
Within weeks, that small product change produced massive results. Within six months, Bhatia and Smith had signed up more than 1 million new customers. Five weeks later, they hit the 2 million mark. Eighteen months after launching the service, with 12 million subscribers, they sold the company to Microsoft for $400 million.

I find it incredibly intriguing how a few words can have such an impact. Maybe you do too.

HTML5 Offline Web applications as an afterthought in ASP.NET MVC

Wed, 14 Mar 2012 15:39:00 +0100

Recently I prototyped a mobile web application using ASP.NET MVC, jQuery Mobile and some HTML5 features. One of the key goals was to find out how far you can push a web ‘application’ until the browser starts getting in the way. Working disconnected is one of these things that appear to be a major showstopper at first.

However - to my surprise honestly - the HTML5 Offline Web applications API seems to be widely implemented across modern browsers already. Not of all of them though. Looking into the specifics, the API itself is fairly straightforward. At his core, you will find the manifest file, which dictates which files should be cached by the browser. The API provides other useful events and methods for inspecting the status of the cache and swapping the cache for a newer version, but they are out of scope today. A useful resource to read up on the full API can be found here, and a working example implementation can be found here.

The manifest file

Back to the manifest file. A manifest file could look like this.

The first line in the file should say CACHE MANIFEST. If you want to write comments, you should prefix the lines with a number sign.

In the CACHE section you declare which files should be cached. An important and interesting note is that these files will be served from the cache, even if you’re online.

In the NETWORK section you declare which files the browser should try to download from the server, regardless of whether the user is online or offline.

In the last section, the FALLBACK section, you can define fallback resources to be used when the user is offline.

Serving and generating the manifest file

Now that we got all this theory out of the way, let’s look at generating and serving the manifest file using ASP.NET MVC.

I started by adding a ResourcesController with one action named Manifest.

public class ResourcesController : Controller
{             
    public ActionResult Manifest() { }
}

This action should serve a text file, using a specific cache-manifest MIME type. To accommodate this I created a new action result, which inherits from the FileResult class, and overwrites the content type.

public class ManifestResult : FileResult
{
    public ManifestResult(string version)
        : base("text/cache-manifest") { }    
}

I also made this same class (for the sake of example) responsible for formatting and writing the manifest file to the output stream. That’s why I added a few extra properties to the manifest result, one for each section and one for versioning. Versioning the file comes in handy when you want to expire the cache, because it only expires when the manifest file changes.

public class ManifestResult : FileResult
{
    public ManifestResult(string version)
        : base("text/cache-manifest")
    {
        CacheResources = new List<string>();
        NetworkResources = new List<string>();
        FallbackResources = new Dictionary<string, string>();
        Version = version;
    }

    public string Version { get; set; }
    public IEnumerable<string> CacheResources { get; set; }
    public IEnumerable<string> NetworkResources { get; set; }       
    public Dictionary<string, string> FallbackResources { get; set; }        
}

To write the file to the output stream, I had to override the `WriteFile`
method.  

```csharp
protected override void WriteFile(HttpResponseBase response)
{
    WriteManifestHeader(response);            
    WriteCacheResources(response);
    WriteNetwork(response);
    WriteFallback(response);
}

private void WriteManifestHeader(HttpResponseBase response)
{
    response.Output.WriteLine("CACHE MANIFEST");
    response.Output.WriteLine("#V" + Version ?? string.Empty);            
}

private void WriteCacheResources(HttpResponseBase response)
{
    response.Output.WriteLine("CACHE:");           
    foreach (var cacheResource in CacheResources)
        response.Output.WriteLine(cacheResource);
}

private void WriteNetwork(HttpResponseBase response)
{
    response.Output.WriteLine();
    response.Output.WriteLine("NETWORK:");            
    foreach (var networkResource in NetworkResources)
        response.Output.WriteLine(networkResource);
}

private void WriteFallback(HttpResponseBase response)
{
    response.Output.WriteLine();
    response.Output.WriteLine("FALLBACK:");
    foreach (var fallbackResource in FallbackResources)
        response.Output.WriteLine(fallbackResource.Key + " " + fallbackResource.Value);
}

In the CACHE section I wanted to include all my static resources, meaning the contents of the Scripts and Content folder. To do this in a simple and low-maintenance fashion I introduced the GetRelativePathsToRoot method. This method takes the path of a virtual folder, recursively scans its content and returns a list of relative paths for each file.

private IEnumerable<string> GetRelativePathsToRoot(string virtualPath)
{
    var physicalPath = Server.MapPath(virtualPath);
    var absolutePaths = Directory.GetFiles(physicalPath, "*.*",   SearchOption.AllDirectories);

    return absolutePaths.Select(
        x => Url.Content(virtualPath + x.Replace(physicalPath, ""))
    );
}

For the Content folder, the result could look something like this.

To add pages to the CACHE section, I used the Url.Action method.

For the NETWORK resources, I added an asterisk, which basically means that the cache shouldn’t be used when the user is online. I didn’t specify any fallback resources in this example.

public ActionResult Manifest()
{
    var pages = new List<string>();
    pages.Add(Url.Action("SomeAction", "ControllerName"));    

    var scriptsPaths = GetRelativePathsToRoot("~/Scripts/");
    var contentPaths = GetRelativePathsToRoot("~/Content/");

    var cacheResources = new List<string>();
    cacheResources.AddRange(pages);
    cacheResources.AddRange(contentPaths);
    cacheResources.AddRange(scriptsPaths);
    
    var manifestResult = new ManifestResult("1.0")
    {
        NetworkResources = new string[] { "*" },
        CacheResources = cacheResources
    };            

    return manifestResult;
}

Setting up a route and including the manifest

Now that we are able to generate and serve a manifest file, we should set up a specific route for the manifest file; some browsers aren’t very forgiving and expect it to have a specific name and location: /cache.manifest.

routes.MapRoute("cache.manifest", "cache.manifest", new { controller = "Resources", action = "Manifest" });

The last step I had to take was include a reference to the manifest file in the html element.

<html manifest="@Url.RouteUrl("cache.manifest")")/>

Poor man’s testing

To verify if all of this works, you can look at the console of the Chrome developer tools. You should see something like this.

That console logging has proven to be extremely useful when debugging the manifest file.

You could also just browse to the manifest file to inspect its content. Don’t mind this screenshot too much, obviously there’s plenty of cleaning up to do in my Scripts folder.

Summary

In this post I showed you a technique I came up with to take advantage of ASP.NET MVC to easily generate, maintain and serve an HTML5 Offline Webappliction manifest file:

Create a controller and action that can serve the file
Create a new action result, which returns the correct MIME type and formats the file
Set up a specific route
Include a reference to the manifest in the html tag

Remember, this is a proof of concept, it’s not perfect. I look forward to any feedback you might have!

Learning: the Hacker Way

Sun, 11 Mar 2012 11:08:00 +0100

I have had a fair amount of discussions on continuous learning and knowledge sharing the past few days. It became rather obvious that a lot of us have developed their own techniques, but also that maybe most of us are still in search of more efficient techniques. Having gone through several phases myself, I would like to share my current way of learning: the Hacker Way.

Here are some snippets taken from a recent letter from Mark Zuckerberg addressed to the Facebook shareholders.

Hacking is an inherently hands-on and active discipline. Instead of debating for days whether a new idea is possible or what the best way to build something is, hackers would rather just prototype something and see what works. There’s a hacker mantra that you’ll hear a lot around Facebook offices: Code wins arguments.

I like to believe learning should be a hands-on activity as well. Basically, stop consuming, start producing. Don’t get me wrong, I do think there is value in reading blog posts (I might be slightly biased on this one), reading books and watching videos, but I find that this value is marginal compared to what you gain by actually doing it.

I remember I wanted to step up my JavaScript game two years ago, and ordered a book. A few months after finishing the book, I finally had the chance to implement something in JavaScript, but I couldn’t. I remembered some syntax, and most concepts, but I couldn’t do it. Eventually I really learned JavaScript by doing it on the job, having to constantly fall back on the book and Stackoverflow, making costly mistakes along the way.

It’s not a terrible idea to pick up a new technology on the job, but it very much depends on the environment. If you’re in the consultancy business, there often is no room to pick up a brand new technology on the job. Customers expect you to know what you’re doing.
Imagine you really want to get into mobile development, and you were able to get in the race for a new mobile project. Getting interviewed by the customer, you have to admit that you haven’t built something for mobile yet, but you did read a book and you find mobile development really interesting. It’s really hard to sell that. If you are able to show something you made, and talk about things that work and things that don’t work in the real world, the customer can get a far better feel if you would be a good fit for the project.

The Hacker Way is an approach to building that involves continuous improvement and iteration. Hackers believe that something can always be better, and that nothing is ever complete.

Hacker culture is also extremely open and meritocratic. Hackers believe that the best idea and implementation should always win - not the person who is best at lobbying for an idea or the person who manages the most people.

This is where it becomes interesting and where you can really make the difference as a company.

Depending on various parameters, you often don’t have the room to experiment on the day job, but if you’re building something by yourself or in a small team without these constraints, you can get wild: experiment, innovate and fail often. Not only do you build experience faster, but you also (maybe indirectly) challenge existing practices and build a deeper understanding of the used methodologies and technologies. Also, having the freedom to innovate, aspiring to get closer to the Silver Bullet, can bring yourself, the company and the industry to the next level.

This is just the first step of the cycle though. The next step should be to get these little projects and acquired experiences out there. Share them with your peers and the community, educate those who want to listen and ask for feedback from those who care, hopefully fueling the inspiration of others.

A welcome side-effect might be that the results could be used to extend your personal and your company’s portfolio. This can help you prove that you can do more than just talk the walk, talk is cheap after all.

I want to know from you what the flaws are in my way of thinking. Is the described technique naive, too idealistic or unrealistic? Which technique has proven to work best for you?

ASP.NET MVC4 bundling in ASP.NET MVC3

Sat, 25 Feb 2012 20:26:00 +0100

One of the new wildly evangelized features of ASP.NET MVC4 is the built-in support for bundling and minification of scripts and stylesheets.

I don’t see any reason why this new feature wouldn’t work for ASP.NET MVC3 though. If you open the packages config of an ASP.NET MVC4 beta project, you will find that bundling support lives in the Microsoft.Web.Optimization package.

<package id="Microsoft.Web.Optimization" version="1.0.0-beta" />

So we should just be able to install this package for an ASP.NET MVC3 project. To install the package, run following command. Pay attention to the -Pre switch.

PM> Install-Package Microsoft.Web.Optimization -Pre
Attempting to resolve dependency 'Microsoft.Web.Infrastructure (= 1.0.0)'.
Successfully installed 'Microsoft.Web.Infrastructure 1.0.0.0'.
Successfully installed 'Microsoft.Web.Optimization 1.0.0-beta'.
Successfully added 'Microsoft.Web.Infrastructure 1.0.0.0' to Optimization.
Successfully added 'Microsoft.Web.Optimization 1.0.0-beta' to Optimization.

Adding bundles happens when the application starts, together with registering areas, adding global filters and registering routes.

The quickest way to enable bundling is by enabling the default bundles.

BundleTable.Bundles.EnableDefaultBundles();

This method will add two bundles to the bundle table: one bundle for the stylesheets in the Content folder and one bundle for the scripts in the Scripts folder. The default bundles try to take core scripts into account when ordering the scripts in the bundle. For example, jQuery will be included before any of its plug-ins are included.

To reference these bundles you can add following snippet to your view or layout file.

<link href="@System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("~/Content/css")" 
    rel="stylesheet" type="text/css" />
<script src="@System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("~/Scripts/js")">
</script>

If you start your application now, and inspect the HTML, you will find two versioned links to a minified version of your CSS and JavaScript.

If you’re trying this on a new project, you will probably have no problems. However, if you’re trying this on an existing project, chances are that some things are not included how they should be.

To troubleshoot what’s going wrong, you can inspect the results of the GetRegisteredBundles method.

var registeredBundles = BundleTable.Bundles.GetRegisteredBundles();

If you need more fine-grained control, you can remove the default bundles again and add your own bundles to the bundle table.

For example, this is how you can add a jQuery bundle.

var jQueryBundle = new Bundle("~/Scripts/jquery", new JsMinify());
jQueryBundle.AddDirectory("~/Scripts", "jquery*.js", searchSubdirectories: false, throwIfNotExist: true);

BundleTable.Bundles.Add(jQueryBundle);

<script src="@System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("~/Scripts/jQuery")">
</script>

When you instantiate a new bundle, specify the relative path of the bundle and pick a bundle transformation. You can add a directory to the bundle, filtered by a search pattern. You can also tell the algorithm to search in the subdirectories or to throw an exception when the directory doesn’t exist.

If you don’t want to add a whole directory to the bundle, but just one or more files, you can use the AddFile method.

For example, this is a separate bundle for modernizr.

var modernizrBundle = new Bundle("~/Scripts/modernizr", new JsMinify());
modernizrBundle.AddFile("~/Scripts/modernizr-1.7.js", throwIfNotExist: true);

BundleTable.Bundles.Add(modernizrBundle);

Conclusion

It’s relatively easy to take advantage of bundling in ASP.NET MVC3. Install the NuGet package, set up the bundle table, include the references in your view or layout page and you’re done.

There are some more interesting things you can do using bundling. I just started experimenting with it, so I wouldn’t be surprised if I will be writing a few more things on bundling in the coming weeks.

There's no place for monogamy in technology

Mon, 20 Feb 2012 08:43:00 +0100

In this post I would like to share some of my thoughts on a recent post by James Hague titled ‘Don’t Fall in Love With Your Technology’. If you haven’t read that post yet, please do, it’s so short that me summarizing it here would be silly.

I think there is nothing wrong with falling in love with your technology per se. If you want to have a fair relationship with your technology, you have to invest in her. Don’t have a superficial relationship, take her home with you, spend some cosy Sunday afternoons together. Get to know her inside out. Know when it’s fun to be with her, but also more importantly when it’s not.

The problem with falling in love with your technology is that a lot of us fall blindly in love. Please don’t. Be unfaithful, do - at least - look at other technologies. Take them out on a crazy Friday night date. In the meanwhile you might discover that that new technology does a few things better than the current one and that she’s a lot more fun.

I guarantee that what now seems like the love of your life, will one day be nothing more than a long forgotten memory which might bring a smirk on your face on remembering her, but there will be no regret in leaving her.

Book review: Working with NHibernate 3.0

Wed, 08 Feb 2012 20:21:00 +0100

It’s been a while since I wrote my last book review, mostly because I’m still trying to figure out when it adds value to write one. For this one it was pretty obvious, there are far too little reviews out there.

Being new to NHibernate, and NHibernate being known as having a steep learning curve, I thought it would be a good idea to do some reading. Searching for books on NHibernate 3.0 on Amazon only yielded three results: NHibernate 3 beginner’s guide, NHibernate 3.0 cookbook and working with NHibernate 3.0. None of these books have a decent amount of reviews, so I had to pick judging by the cover and summary. I chose the last one.

The book Working with NHibernate 3.0 by Benjamin Delcamp Perkins contains six chapters, covered over 213 pages.

The first chapter very briefly explains what an ORM is, and then starts looking at configuring NHibernate. This means setting up the session factory and its configuration, creating entities and mapping them to the database (using XML and code), but also configuring log4net (NHibernate’s logging framework), serializing startup and interceptors and events. These last two subjects felt a little misplaced in this chapter though.

In the first chapter the foundation for the Guitar store example application is laid. This example application is written in WPF, which bothered me a bit. Not because I dislike WPF, but because when I’m reading a book on a data access technology I really don’t want it to be littered with fragments of XAML and code-behind. To make matters worse, the author suggests using a console application to test your queries. I think it would have been far more valuable to use real unit tests to prove the queries are correct, more like this. Another complaint about the example application that I read in another review is that there are no scripts available to set up the database, which might be discouraging if you want to follow along.

Chapter two, three and four cover the various NHibernate query API’s: HQL (Hibernate Query Language), ICriteria and LINQ. Every chapter implements the same or at least similar queries. Concepts covered in these chapters are simple queries, complex queries, detached queries, futures and aggregates. I think these chapters succeed in giving a good overview of the ways you can use NHibernate to query data. Also the tips on how to use futures, various fetch modes, the stateless session and aggregates to improve performance will prove useful in the future.

The fifth chapter covers managing state and saving data. In this chapter, the author explains the various ways you can handle database concurrency, listing the advantages and disadvantages of each option. NHibernate caching is also explained, looking at first and second-level caching. Further in this chapter, you can find an example of a custom data type implementation. Finally we arrive at saving data with NHibernate. Next to the standard way of saving data, the author explains the use of Evict, Merge and Persist.

The last chapter, covering only 9 pages, shows how you should set up NHibernate in an MVC3 application.

Conclusion

Although this book doesn’t do a great job showing you how to use NHibernate in the real world, it does do a decent job giving you a basic overview of NHibernate’s capabilities. Reading this book when you’re new to NHibernate will save you from a few costly common NHibernate pitfalls. I don’t think I will be able to use this book for reference, but it should be easier now - knowing the correct terminology - to search in the NHibernate documentation online.

My rating: 3/5. Do you advise other books on NHibernate?

Testing DI bootstrappers

Mon, 06 Feb 2012 08:59:00 +0100

While your Dependency Injection bootstrappers - being responsible for gluing your application together - are a vital part of your application, they are seldom put under test. I don’t see any reason why they shouldn’t be though. The cost of these tests is negligible, definitely if you compare it to the cost of the often catastrophical outcome of bugs in your bootstrappers.

I encourage you to take a look at the commit history of your DI bootstrappers; I bet they change a lot. Wouldn’t it be nice to have a set of tests that proves that the dependency container still behaves like you expect it to at runtime? Next to proving correctness, I think writing these tests also helps you discover various behaviours of your DI container, which is a valuable investment in itself.

Let me show you a few tests that I wrote to put my ASP.NET MVC Ninject bootstrapper under test.

I started by opening up the Ninject bootstrapper, making the CreateKernel method public.

public static IKernel CreateKernel()
{
    var kernel = new StandardKernel();
    
    kernel.Bind<IEntryService>().To<EntryService>();

    return kernel;
}

In the test class, I used the TestInitialize attribute to initialize a new instance of the kernel before every test. I’m not sure this is really necessary, but I want to avoid that my tests experience side-effects of a previous test.

[TestInitialize]
public void Setup()
{         
    _kernel = NinjectMVC3.CreateKernel();
}

Needless to say, the first test should prove that an implementation of the IEntryService interface can be resolved. The behaviour I observed while playing with this case, is that Ninject throws a Ninject.ActivationException when the implementation can’t be resolved.

Ninject.ActivationException: Error activating IEntryService  
   No matching bindings are available, and the type is not
   self-bindable.  
   Activation path:  
       Request for IEntryService  
   
Suggestions:  

1.  Ensure that you have defined a binding for IEntryService. 
2.  If the binding was defined in a module, ensure that the module has
     been loaded into the kernel.
3.  Ensure you have not accidentally created more than one kernel.
4.  If you are using constructor arguments, ensure that the parameter
     name matches the constructors parameter name.
5.  If you are using automatic module loading, ensure the search path
     and filters are correct.

So to test whether an implementation can be resolved, I just make sure no exceptions are thrown on resolving the dependency.

[TestMethod]
public void Test_IEntryService_Can_Be_Resolved()
{
    AssertDoesNotThrowWhenResolved<IEntryService>();
}

The AssertDoesNotThrowWhenResolved method is a helper method which tries to resolve a dependency of T and asserts that no exceptions are thrown while doing so. The assertion is borrowed from Xunit (package available on Nuget).

private void AssertDoesNotThrowWhenResolved<T>() 
{
    Xunit.Assert.DoesNotThrow(() => _kernel.Get<T>());
}

A second useful test is testing the lifetime of the resolved implementation. For most of my dependencies, I expect a new instance every time they are resolved. This test looks like this.

[TestMethod]
public void Test_IEntryService_Is_New_Instance()
{
    AssertNewInstanceIsResolved<IEntryService>();
}

The AssertNewInstanceIsResolved method is another helper method which resolves two instances of T and asserts they are not the same.

private void AssertNewInstanceIsResolved<T>()
{
    var instance = _kernel.Get<T>();
    var secondInstance = _kernel.Get<T>();

    Assert.AreNotSame(instance, secondInstance);
}

That’s it. While these tests are very cheap to write, they do provide great value. I can imagine testing more complex bindings, like contextual bindings, taking a bit more time to set up, but the value of these tests increases proportionally.

Do you put your DI bootstrappers under test? If you don’t, why not?

A solar storm anecdote

Wed, 01 Feb 2012 08:43:00 +0100

Last week, several news channels reported on the strongest solar storm since 2005. This news item reminded me of a peculiar support ticket we received one gray Monday morning a few years ago, when I was still writing software for fire departments.

# Ticket 7238  
Subject: **AVL broken**  
Status: New  
Description  
06:22 Vehicles stay mostly stationary on the map, even when we are  
positive they are en route.

Fire departments that have to cover a large area - and are wealthy enough - often use AVL (Automatic Vehicle Location) to track their vehicles and visualize them on a map. This is extremely valuable, because you always want to dispatch the vehicles with the smallest response time to a high priority intervention. Also being able to advise drivers of possible blockages and toxic gas clouds can save lives. To be able to track a vehicle, an AVL module is installed into each vehicle’s cockpit. This module uses GPS to determine the location and sends the location data over GPRS to a central server.

On our end, we had a third party service listening for those location packets and translating them into a more understandable format. This service, not being mission critical, wasn’t being monitored, so we had to look into the logs to see what was going on. Scrolling through megabytes of debug logs, we couldn’t find anything suspicious.

While I was investigating this, a co-worker had come in and was reading through his mails while sipping on his morning coffee. After reading support ticket 7238, he nonchalantly said he knew what was going on with AVL and he would take over from there.

# Ticket 7238  
Subject: AVL broken  
Status: Pending  
Description  
**08:45 I heard on the radio that there is a solar storm going on at  
the moment, which affects sattelites. The AVL module might have  
a hard time getting a GPS fix. This issue should solve itself over  
time. We will keep an eye on this issue.**  
---------------------------------------------------------------------------------  
06:22 Vehicles stay mostly stationary on the map, even when we are  
positive they are en route.

A few days passed and although the solar storm was over, we were still seeing signifcant packet loss. After spending a few hours working the ticket, in which we restarted the service, monitored network traffic on the machine and conctacted the telephony provider, we were getting a bit desperate. We were discussing other potential paths to investigate, when one of our more seasoned co-workers asked “You guys did try restarting the server, right?”.

Good enough, after restarting the server, we were seeing no more packet loss and the vehicles started moving on the map again.

# Ticket 7238  
Subject: AVL broken  
**Status: Resolved**  
Description  
**15:47 The solar storm must be over. The vehicle locations are  
being updated in a timely fashion again.**  
--------------------------------------------------------------------------------  
08:45 I heard on the radio that there is a solar storm going on at  
the moment, which affects sattelites. The AVL module might have  
a hard time getting a GPS fix. This issue should solve itself over  
time. We will keep an eye on this issue.  
--------------------------------------------------------------------------------  
06:22 Vehicles stay mostly stationary on the map, even when we are  
positive they are en route.

Until today, we never talked about ticket 7238 again.

How Wikipedia uses HTML5 to save bandwidth

Thu, 19 Jan 2012 20:56:00 +0100

Something I hadn’t noticed until recently is that Wikipedia tries to use the browser’s native SVG support to render certain images. For example, if you search for a high resolution image of your country’s flag, you will probably end up viewing an SVG. Wikipedia also offers downloads to the image rendered as a PNG though.

Next to being able to scale to an arbitrary size without suffering data loss, the SVG data format allows images to be far more compact. Basically, SVG is just XML, which also means it can be easily compressed to make its size even smaller. For example, this is the (uncompressed) SVG for the flag of the Kingdom of Belgium.

<svg xmlns="http://www.w3.org/2000/svg" width="450" height="300">
    <rect width="450" height="300"/>
    <rect x="150" width="150" height="300" fill="#FAE042"/>
    <rect x="300" width="150" height="300" fill="#ED2939"/>
</svg>

This svg node only weighs as much as 224 bytes, while the image rendered as a high resolution PNG weighs 13.402 bytes. Stuff like that makes a significant difference when you’re serving millions of page views on a daily basis.

The first time I touched SVG was a few years ago when I was still working on fire department projects. We were working with a third party that used SVG to draw maps in the browser. Being already in the post-Google Maps era, I thought it was terrible I had to download Adobe’s SVG viewer. While the Google Maps technology already works great, there are still things SVG can do better and cleaner, especially for more specialized GIS applications. There is an interesting paper on that subject here, it’s a bit outdated though.

I can only applaud making the browser more capable, and losing yet another plug-in. I’m curious to see how other applications will start taking advantage of the opportunities native SVG support across all modern browsers (even Internet Explorer) presents.

Have you already built things using SVG? Or even considered it?

Autocorrecting unknown actions using the Levenshtein distance

Sun, 15 Jan 2012 16:04:00 +0100

This weekend I prototyped an idea I had earlier this week: autocorrecting unknown actions in ASP.NET MVC.

Handling unknown actions

To give you an example, let’s say I have a Home controller with an action named Kitten on it. If there is an incoming route for the Home controller with Kitty (instead of Kitten) as the action name, the controller will not be able to invoke any action method and instead will call the HandleUnknownAction method.

Here is the snippet from the ASP.NET MVC source.

protected override void ExecuteCore() {
    PossiblyLoadTempData();
    try {
        string actionName = RouteData.GetRequiredString("action");
        if (!ActionInvoker.InvokeAction(ControllerContext, actionName)) {
            HandleUnknownAction(actionName);
        }
    }
    finally {
        PossiblySaveTempData();
    }
}

The HandleUnknownAction is virtual, meaning we can override it in our derived controller. The base implementation of the HandleUnknownAction method does nothing more than throwing a 404 HttpException.

protected virtual void HandleUnknownAction(string actionName) {
    throw new HttpException(404, String.Format(CultureInfo.CurrentCulture,
        MvcResources.Controller_UnknownAction, actionName, GetType().FullName));
}

So let’s override the HandleUnknownAction method and try to autocorrect the unknown action name. To be safe, we will only attempt to autocorrect the action name when it’s a GET HTTP request.

protected override void HandleUnknownAction(string actionName)
{
    if (!HttpContext.Request.HttpMethod.Equals("GET", StringComparison.OrdinalIgnoreCase))
        Throw404HttpException(actionName);
    
    TryToRedirectToAnActionNearby(actionName);           
}

Listing all actions

First we need a list of all available action names. I reflect on the methods of the current controller and select the methods which are public, can be invoked and are an instance method. Also the method should return an ActionResult, not be decorated with the HttpPost attribute and not have a special name. I’m pretty sure I’m missing a few things here, but there seems to be no generic way to extract this metadata from a controller. Places where these type of things are used in the framework seem to be internal or non-public.

private IEnumerable<string> GetAllHttpGetActionNames()
{
    return GetType()
            .GetMethods(BindingFlags.InvokeMethod | 
                        BindingFlags.Public | 
                        BindingFlags.Instance)
            .Where(m => m.ReturnType == typeof(ActionResult) &&
                        !m.IsSpecialName &&
                        !m.GetCustomAttributes(true)
                            .Contains(typeof(HttpPostAttribute)))
            .Select(m => m.Name)
            .Distinct();
}

Once we have all these action names, we want to see how distant they are from the unknown action name we are trying to autocorrect here. To calculate this we can use the Levenshtein distance algorithm.

The Levenshtein distance

The Levenshtein distance is defined by Wikipedia like this.

In information theory and computer science, the Levenshtein distance is a string metric for measuring the amount of difference between two sequences. The Levenshtein distance between two strings is defined as the minimum number of edits needed to transform one string into the other, with the allowable edit operations being insertion, deletion, or substitution of a single character. It is named after Vladimir Levenshtein, who considered this distance in 1965.

An implementation of this algorithm in C# could look like this.

public static int CalculateDistance(string str1, string str2) 
{
    var matrix = new int[str1.Length + 1, str2.Length + 1];

    for (var i = 0; i <= str1.Length; i++)
        matrix[i, 0] = i;
    for (var j = 0; j <= str2.Length; j++)
        matrix[0, j] = j;

    for (var i = 1; i <= str1.Length; i++)
    {
        for (var j = 1; j <= str2.Length; j++)
        {
            var cost = str1[i - 1] == str2[j - 1] ? 0 : 1;

            matrix[i, j] = (new[]
            {
                matrix[i - 1, j] + 1, matrix[i, j - 1] + 1, matrix[i - 1, j - 1] + cost
            }).Min();

            if ((i > 1) && 
                (j > 1) && 
                (str1[i - 1] == str2[j - 2]) &&
                (str1[i - 2] == str2[j - 1]))
            {
                matrix[i, j] = Math.Min(matrix[i, j], matrix[i - 2, j - 2] + cost);
            }
        }
    }

    return matrix[str1.Length, str2.Length];
}

This is a direct port from the pseudocode found on Wikipedia. These tests might, probably a lot more than the implementation, help you understand what the Levenshtein algorithm calculates.

[TestMethod()]
public void Test_CalculateDistance_With_Two_Empty_String()
{           
    Assert.AreEqual(0, Levenshtein.CalculateDistance(string.Empty, string.Empty));
}   

[TestMethod()]
public void Test_CalculateDistance_With_Empty_First_String()
{         
    Assert.AreEqual(6, Levenshtein.CalculateDistance(string.Empty, "kitten"));
}

[TestMethod()]
public void Test_CalculateDistance_With_Empty_Second_String()
{           
    Assert.AreEqual(6, Levenshtein.CalculateDistance("kitten", string.Empty));
}

[TestMethod()]
public void Test_CalculateDistance_With_Missing_Characters()
{           
    Assert.AreEqual(2, Levenshtein.CalculateDistance("kitten", "kitt"));
}

[TestMethod()]
public void Test_CalculateDistance_With_Wrong_Characters()
{           
    Assert.AreEqual(1, Levenshtein.CalculateDistance("kitten", "kittyn"));
}

[TestMethod()]
public void Test_CalculateDistance_With_Too_Much_Characters()
{           
    Assert.AreEqual(5, Levenshtein.CalculateDistance("kitten", "kittenkitty"));
}

[TestMethod()]
public void Test_CalculateDistance_With_Equal_Strings()
{          
    Assert.AreEqual(0, Levenshtein.CalculateDistance("kitten", "kitten"));
}

Now that we have implemented the Levenshtein distance algorithm, we can calculate the distances between the unknown action name and all the available action names.

private Dictionary<string, int> CalculateLevenshteinDistance(IEnumerable<string> actionList, string actionName)
{
    return actionList
            .Select(a => new
            {
                Action = a.ToLower(),
                Distance = Levenshtein.CalculateDistance(a.ToLower(), actionName.ToLower())
            })                    
            .ToDictionary(k => k.Action, v => v.Distance);
}

For the unknown action name ‘Kitty’, when the action names ‘Kitten’, ‘Index’ and ‘Dog’ are available, this method would return a dictionary that looks like this.

'kitten' : 2
'index' : 6
'dog' : 6

Putting it all together

Now we have this dictionary, we want to filter on a certain distance threshold. I picked three, given that when a word is three characters off, the chance of it being a typo is rather small.

If the dictionary still contains some items after filtering, we want to take the action with the shortest distance, this action is the nearest to the unknown action. Only thing left to do is change the action in the RouteData and execute a RedirectResult. The easiest way to generate a url to redirect to, is to use the controller’s UrlHelper to let it generate the url based on the RouteData.

private void TryToRedirectToAnActionNearby(string actionName)
{
    var httpGetActionNames = GetAllHttpGetActionNames();
    if (!httpGetActionNames.Any())
        Throw404HttpException(actionName);

    var actionDistanceMap = CalculateLevenshteinDistance(httpGetActionNames, actionName)
                                .Where(i => i.Value <= 3);
    if (!actionDistanceMap.Any())
        Throw404HttpException(actionName);

    var shortestDistance = actionDistanceMap.Select(v => v.Value).Min();
    var nearestAction = actionDistanceMap.Where(i => i.Value == shortestDistance).First().Key;

    ControllerContext.RouteData.Values["action"] = nearestAction;

    new RedirectResult(Url.RouteUrl(RouteData.Values), permanent: true)
        .ExecuteResult(ControllerContext);
}

The outcome

Now when I, the user, type http://somesite/someController/kitty, I will be redirected to http://somesite/someController/kitten without me even noticing.

Feedback

This implementation definitely is not production ready. It’s a prototype, not even under test. I wonder if this is even something you would want to do. Or is this breaking the Web in one way or the other? Would it bother search engines?

2011 Annual Review

Mon, 26 Dec 2011 21:35:00 +0100

Since the end of the year is approaching, I would like to look back on 2011 and take a peek at 2012. Unlike the years before, I’m hardly setting any goals this year. Interests change so quickly, and new opportunities present themselves so regularly, I feel like it might be ignorant to set these things in stone.

My career

The year started out interesting, working on a brand new product for fire departments at Ferranti Computer Systems. Although my role in this project should have been fairly satisfying, I still felt like my position back then failed to fill a certain void. That’s why I decided to take the leap into the Great Unknown.

In September, I started working for Euricom as a consultant. So far, I haven’t regretted making that decision at all, I ended up in a group of intelligent, progressive and enjoyable people.

Blog

This year I managed to write 69 posts (70 including this one), which is a bit more than the year before.

Plenty of posts have gotten decent attention this year, making the traffic tripple compared to 2010. And no matter how other bloggers claim you shouldn’t care about your blog statistics, this does mean a bunch to me.

The main topics I wrote about this year were mostly web related (HTML5, JavaScript and ASP.NET MVC). I even wrote some opinionated posts this year, which I found harder, but also more rewarding.

Last year, I set a goal to get better at MVC and to at least look at WebMatrix. I spent a large part of the year building things in ASP.NET MVC at the day job, so that goal was easily achieved. I built one small thing in WebMatrix, which was enough to give me a feel of what WebMatrix is all about.

My guess is that I will cover similar topics next year. My faith in the web has only grown stronger. I think the rise and rise of the browser and JavaScript provides us with unseen opportunities, but also with an enormous amount of challenges. I am really eager to start looking more into serious JavaScript development.

Community

I think I did fairly well this year. I attended several local community meetings and got to to know a lot of nice people in the process.

The highlight probably was my own talk on WebSockets at HTML5 Webcamps. I really enjoyed that experience. I do not aspire to be a speaker though, but should I have interesting things to talk about, I probably won’t reject invitations.

I’m not sure if I will be doing a lot of community events in 2012. I often don’t seem to get out of them what I hoped for. Also, Euricom regularly organizes meetups to talk about all things software development. These will already consume plenty of time.

Travelling

This year my girlfriend and I did a three-week roadtrip along the West Coast. This trip was by far the best time of our lives. We are determined to undertake similar projects in the future again.

Next to that, we revisited the Czech Republic, taking plenty of time to visit Prague this time.

We’re not sure what 2012 will bring. Right now, we are looking into something more nearby, Italy.

Sport/Health

Last year I picked up boxing, but I had to give that up due to scheduling conflicts. I have been working a tweaked P90X routine twice a week instead, which is very much recommended.

I reasonably succeeded in achieving my running ambitions. Altogether, I ran somewhere around 800 kilometers (= 500 miles) this year. This summer, I probably was in the best shape of my life, running 15 km pretty (= 9 miles) comfortably. Although the graph below suggests that I’m slacking, it’s only partially true. I have been doing shorter (and higher paced) runs lately, increasing the intensity. Running is probably one of the sports that give me the most energy in return, so I intend to stay at it.

Conclusion

2011 has been an eventful year for me. In 2012, I hope to be able to keep doing the things I love, surrounded by the people I love. Trying to live life as much as I can.

I wish you and your family the best for 2012. That the new year may bring you health, love and plenty of hacking.

2011's most read posts

Sat, 17 Dec 2011 19:33:00 +0100

I compiled a list of most popular posts that were published on this blog in 2011. Unlike last year, and the year before that, this year it’s not a strict top five list. While analyzing the statistics, I found out that a few topics -covered over multiple posts - were popular this year.

Preparing for my HTML5 Webcamps session on WebSockets, I wrote a few posts on the Microsoft WebSockets prototype. I published a step-by-step guide on how to get the prototype working on your machine. A little later I also published two posts covering the internals of the client and server WebSockets prototype. I’m not surprised search drove a lot of traffic to these posts. Microsoft hardly tried to make it easy to play with the prototype: no tutorials, no source, nothing. Now the specification is stable, and WCF 4.5 has WebSockets support out of the box, I suspect MS will make greater evangelizing efforts. Or maybe, they will wait until one of their browsers support it.

In this post I rambled on how liberating it can be to build something small on your own now and then, just to keep your sanity and escape from software in the real world. This post got picked up by Hacker News, making it the second entry in this list.

In one of those sporadic academic moments, I had a look at one of the quirks in anonymous type equality. I also looked at the reason why anonymous types can be compared in the first place, being reference types after all.

To wrap up this link collection, I would like to thank you for reading. I hope to see you contribute to this little outlet of mine for another year.

When should you take performance into consideration?

Sun, 27 Nov 2011 17:55:00 +0100

Before publishing my previous post on rewriting an if, I knew some people would hate it, because the refactored construct is less performant.

Although I think performance is important, relevant performance improvements are, apart from in tight loops, hardly ever to find in language constructs. To put it more bluntly, they are a waste of time. When translating your thoughts into code, you should aim to make your intentions as clear as possible for the person who comes after you. Don’t obfuscate your code for an negligible performance improvement.

I’m not advocating readability and pretty code are an end goal, performance and quality still are. But look for optimizations where it counts: optimizations which, after measuring, prove to be make a significant impact on the overall performance of your system. These are most likely to be found in I/O operations, architectures and algorithms. Not in language constructs.

Rewriting an if

Thu, 24 Nov 2011 21:12:00 +0100

Yesterday I came across an if statement that looked something like this.

if (arg == "a" ||
    arg == "b" ||
    arg == "c" ||
    arg == "d" ||
    arg == "e") 
{
    Console.WriteLine(true);
}

An alternative way of writing this could look like this.

if (new [] { "a", "b", "c", "d", "e" }.Contains(arg))
    Console.WriteLine(true);

I can’t remember in which Github repository I spotted this technique, but I’m sure it was written in something other than C#. I think it works for C# as well though. The language hardly gets in the way, although it would be nice to be able to drop the new.

This is one of these trivial things I tend to geek about. The condition fits on one line now, making the eyes do less work. Also adding a variable is less work; you don’t have to enter and indent accordingly. I think it’s a win in readability, size and maintenance.

But then I stop and wonder: how do you feel about this construct?

Blame no one but yourself

Wed, 23 Nov 2011 21:24:00 +0100

Blame no one but yourself. This is one of the few quotes I remember months after reading this book. Although it’s a harsh statement, there definitely is some truth to it.

Once I started observing my own behavior when faced with failure, I caught myself regularly blaming others for failures to which I am - at least - an accomplice.

Think about it. You might be guilty of this too.

Frustrated with management because you missed their too tight deadline? Did you tell your project manager that taking that ‘one small’ task in between would get you behind on schedule? Dissapointed by your team mate because he messed up the task you asked him to do? Are you sure you gave him all the information and provided enough feedback? Tired of doing the same repetitive task over and over? Why haven’t you automated it yet? Not happy with a certain implementation? Did you speak up and propose alternative solutions?

So next time you blame someone else, ask yourself: Did I try everything within my reach to make this work or did I unconsciously sabotage this myself?

Daydreaming about jQuery Mobile and the WebAPI

Sun, 20 Nov 2011 17:24:00 +0100

I recently blogged about programming for the future of mobile with jQuery Mobile and the WebAPI. You probably heard that jQuery Mobile 1.0 was released earlier this week. Although it will take a while before we will see some actual results from the WebAPI initiative, that shouldn’t keep us from letting our minds play with things that might be possible one day using the WebAPI.

The thoughts in this post were provoked by an interesting comment Kristof Claes left on my previous post.

One thing I don’t like about jQuery Mobile is that it has this iPhony (pun not intended) look to it.

I have a WP7 device and for some reason I don’t want applications to look like I’m using an iPhone. I can imagine the average iPhone user wouldn’t be happy when jQuery Mobile used a Metro-like skin.

And I agree, it does look iPhony. Although there already is a themeroller for jQuery mobile, it’s limited to changing the color scheme. Like Kristof said, the contrast between the Metro UI and the jQuery Mobile UI is enormous.

Platform specific themes might help to close that gap. I really think this makes sense for mobile, because - compared to desktops - most mobile operating systems do leverage a signifantly different feel. Also, the browser is full screen on most mobile devices, saving space, but also losing the native OS context.

I have been thinking a bit about how this could be implemented. So follow along, and let me know if these ramblings seem feasible.

Serving themes based on the operating system should be doable. A naïve implementation, which only supports Metro, could be as simple as this.

return Request.UserAgent.Contains("Windows Phone OS 7.5") ? "jQueryMobile.Metro.css" : "jQueryMobile.Default.css";

On my Windows Phone, I can configure the background- and the accent color for the Metro UI. And this is where it gets interesting. Using the SettingsAPI, defined in the WebAPI standards, we might be able to find out those values in our webapplication. Meaning we could make the mobile UI ’experience’ completely transparant.

The proposed Settings API standard looks like this.

interface SettingsManager
{
    // List of known settings.
    const DOMString FOOBAR = "foobar";
 

    // Setters. SettingsRequest.result is always null.
    SettingsRequest set(DOMString name, DOMString value);
    SettingsRequest set(DOMString name, long value);
    SettingsRequest set(DOMString name, long long value);
    SettingsRequest set(DOMString name, float value);

    // Getters. SettingsRequest.result will be of the requested type if the success event is sent.
    SettingsRequest getString(DOMString name);
    SettingsRequest getInt(DOMString name);
    SettingsRequest getLong(DOMString name);
    SettingsRequest getFloat(DOMString name);
}

So, dynamically modifying the css to comply to the Metro color sheme could be this easy.

var backgroundColorRequest = settingsManager.getString('metro-backgroundColor');

backgroundColorRequest.onSuccess = function() 
    $(".main").css("background-color", this.result);
}

var accentColorRequest = settingsManager.getString('metro-accentColor');

accentColorRequest.onSuccess = function() {
    $(".tile").css("background-color", this.result);
};

Is it just me, or does the WebAPI have some sick potential?

Programming for the future of mobile

Wed, 09 Nov 2011 22:27:00 +0100

I have been working on something small on the side lately. I hardly have anything to show for it though, most of it is still being shaped in my head.

Anyhow, a very important part of the front-end is built using jQuery mobile. Although the framework hasn’t been released - release candidates are available though -, it’s something you should start looking into today. Why? Because the browser is the future of mobile applications. With the Flash and Silverlight bombs that were dropped today, I am even more confident that that future might be nearer than we think.

Built upon the jQuery and jQuery UI foundation, jQuery mobile aims to make mobile web applications seriously cross-platform and cross-device, while optimizing for touchfriendliness. From what I’ve seen so far, the jQuery team has (once again) succeeded at making the web just work.

To start using jQuery mobile, you just have to add one extra script file and one stylesheet on top of the existing jQuery infrastructure. Once that is done, the framework will already do most of the heavy lifting for you. Depending on your needs, you might want to enrich some elements explicitly in a nice semantic fashion.

For example, this is how you would get some collapsible panel action going.

<div data-role="collapsible" data-theme="c" data-content-theme="c" data-collapsed="@dataCollapsed" id="entriesList">
    <h3>@entryGroup.Date.ToString("dd/MM/yyyy")</h3>                
    <ul data-role="listview" data-inset="true" data-theme="d">  
        @foreach (var entry in entryGroup.Entries)
        {           
            <li>
                @entry.CreatedOn.ToString("HH:mm") @entry.Description                     
            </li>         
        }
    </ul>
</div>

To give you an idea, this would result in something that looks like this. Clean, right?

Now that these problems are out of the way soon, the only barrier left for mobile web applications is that the browser has no actual hooks in the device itself. But this last barrier can’t hold forever… Mozilla initiated a very interesting project not so long ago. They started defining standards for a set of APIs that should give you direct access to the device: a camera API, telephony and messaging API, accelerometer API.. Find everything about the WebAPI initiative here. Subscribe to the mailing list, participate and help pushing the web forward.

I’m fairly optimistic about the future of mobile web applications. It’s impossible to put a timeframe on it, but eventually WWW, the Web Will Win.

The gift of legacy

Sun, 30 Oct 2011 13:36:00 +0100

Just after graduating, I hated legacy with the heat of a thousand suns. I felt unfortunate, having to work on old code, built using outdated technologies, while software is all about making new and shiny things. Right, guys? Those naïve expectations of a rookie got crumbled very soon. Legacy is a constant in our industry. You can try to ignore it as long as possible, but it’s impossible to keep that up forever. Over the years, I have come to accept that. I even have been fully embracing it lately.

Legacy is an opportunity to learn from the past. To learn from the costly mistakes our predecessors made, so you can avoid them in the present. Often legacy also helps me fully comprehend the motivation behind more modern architectures and software practices. Finding yourself all alone in thousands of lines of code, where the smallest change can impact an unknown amount of long forgotten scenarios, makes it perfectly clear why we test our code, and why it should be a crime not to.

It’s a shame that there often isn’t a whole lot of glory in maintaining old software. Although, I do think some programmers should be awarded a medal of honor after serving multiple months in the trenches. There is, however, a lot of space to make small refactorings, improving the code quality and in the meanwhile giving you the small wins you need to keep going.

All legacy isn’t bad per se though. Once in a while you discover a ruby in the dust; a creative solution or a small but interesting trick, which you might be able to apply later on.

Legacy is a gift, learn to embrace it.

Book review: The Art of Unit Testing

Thu, 06 Oct 2011 21:08:00 +0200

I think The Art of Unit Testing targets a broad audience. Beginners will find every part of the book useful, where intermediates might be more interested in the final two parts.

Roy Osherove starts this book by laying a solid foundation of the unit testing concept. Why is testing important? What defines a good unit test, and how does a unit test differ from an integration test? In the second part of the book, he demonstrates the use of two core unit testing techniques: stubs and mocks. After showing you how these techniques work, he shows off various isolation frameworks which can help you creating stubs and mocks at runtime (fakes), greatly reducing the effort of writing these objects.

If you have some experience with unit testing, you might not be impressed with the content of the book so far. If you’re somewhat like me, and have been writing tests for some while, but often find yourself wondering if your tests will still be considered solid six months from now, you will find the content of part three very useful. In this part, Roy talks about organizing your tests and how to strive for satisfying the three pillars of good tests: trustworthiness, maintainability and readability.

The final part wasn’t something I expected to find in this book. In this chapter it’s very clear that the author has been an agent of change himself for a long time. Next to sharing successful strategies on how to usher testing into the organization, he answers a bunch of hard questions you will be asked when you’re on your own quest to bring change.

As I said before, there’s something in the book for everyone. While there are a lot of topics discussed, the book counts less than 300 pages. It’s well written at large, sometimes a bit sloppy, but in general a smooth read. All of the concepts are explained using examples written in C#, where the problems are small enough to keep it simple, yet comprehensive enough to make it a plausible scenario.

If you want to start out with unit testing, are looking for confirmation on how you’re doing or want to refine some techniques, this book might be the one to get.

A real developer knows when to pull the plug

Thu, 29 Sep 2011 20:38:00 +0200

My mini-website arealdeveloper.com will no longer be available online after tomorrow.

I slapped it together over a weekend, trying out WebMatrix, which turned out to be the perfect companion for building small things.

A pleasant surprise was the mention of ‘A Real Developer’ by the guys at Channel9. This made the traffic go through the roof for a few days.

A few months later the site was a ghosttown. But I don’t mind, had to happen. It never had the ambition to deliver real value. I was more than happy to deliver five minutes of entertainment.

To have a memento, I uploaded the source to GitHub. Remember, it’s far from clever or impressive. However, I do think it can serve as a nice introduction-by-example to WebMatrix.

I committed the latest database as well, so now you can read all the quotes (even the bad ones), and start your own t-shirt business ;).

Comfortably numb

Wed, 21 Sep 2011 20:51:00 +0200

Something that can bother me tremendously is being surrounded by people who are in a constant state of being comfortably numb. People who don’t welcome change, try to scare away new concepts and are just too much at home in their comfort zone. Some are perfectly happy filling their days keeping up appearances of being busy. They don’t care about self-improvement, but only care about augmenting their paychecks by accumulating as much legacy baggage as possible, with the sole intention of being perceived as an irreplaceable asset to the company, whilst not having to leave their comfort zone. These people also have the tendency to shy away from responsibility and commitment. They are satisfied with doing just enough to not draw any attention to themselves.

It’s really tempting to just go along, and follow. After all, it’s in human nature to strive for comfort: the path of the least resistance. If you are reading this blog, you probably already belong to the group of people who care about their craft, and are aiming for something more than the status quo. I applaud you. For you, this post only serves as a reminder that you should constantly evaluate your current position and be honest with yourself: Am I still moving forward?

Building small things

Sun, 18 Sep 2011 16:35:00 +0200

Due to the nature of things we build in our day to day job, writing software can wear out even the most fit of us.

Most software jobs make you constantly deal with complexity. The amount of things which can lead to a complex software project are immense. A poor first design, and failure to redesign. External dependencies, which seem to behave different every time around. Or just the complexity of the problems itself.
You are almost always working in a team, which can be exhausting as well. If the team doesn’t share your passion and you have a hard time getting your ideas across, you will get frustrated, real soon. Add some coroporate politics to the mix and you’ll be on your way to Paranoia.

It feels like I’m turning this into a rant, but I’ll stop right here, you get where I’m going at. Building software in the real world can be hard. Very hard.

I like to think building something small on your own once in a while can be extremely liberating. It can help you keep your sanity and not lose your passion towards software. You pick what to build. Something small. Something new. Something built with your favorite tools, on your favorite platform. Something that can be shipped. Something which takes only you and your machine.

Having problems finding something meaningful to make? Look around you, the most trivial problem can lead to a satisfying little sideproject. Look for things that bother you, and try making them bother you less. Listen to others, maybe you can help solve their problems. Who cares if it already exists in one way or the other? Look for something that looks fun. And just build it.

Once upon a time in the West

Sat, 17 Sep 2011 11:15:00 +0200

The girlfriend and I returned from our West Coast roadtrip yesterday morning.

We found this trip to be an unforgettable experience, which made us even more hungry for future travels.

This blog will return to business as usual, with mainly technical content and opinions.

Here is an overview of our posts exploring the West Coast:

Once upon a time in the West..

City of fallen angels

Tue, 13 Sep 2011 05:45:00 +0200

After seeing the celebrities at Madame Tussaud’s in Las Vegas, we went for the real deal.

Turns out Hollywood is a lot less glamourous than we expected. The Walk of Fame isn’t a lot different from any other gray sidewalk, paved with long-forgotten stars. The famous Hollywood sign isn’t even lit a night, making the contrast with Beverly Hills at the other side of town even bigger.

Maybe there’s another side to Hollywood that we haven’t seen, since we only stayed for two days, one of which at the Universal Studios. Here we took a sneak peek behind the scenes of movies like The Fast and the Furious, Jaws and Jurassic Park. We also drove through Wisteria Lane and other generic-built shooting sets.

This will probably be our last post from the US. We will spend our last few days driving the coast line back to the Bay Area, to depart there on Thursday.

Fear and loathing in Las Vegas

Sun, 11 Sep 2011 02:49:00 +0200

Driving back into the desert after our breakfast, with Las Vegas in the rear-view mirror, we secretly feel a little relieved to be heading back to the real world. The cocktail of fake pyramids, indoor jungles, Paris landmarks, castles and volcanoes became a little nauseating. With hotels and casinoes resembling labyrinths, designed to get people trapped inside, it feels liberating to drive through the wide open landscape again.

Some of the things we enjoyed the most were the late nights walking the Strip, watching free shows, standing on top of the Stratosphere and posing with the wax celebs at Madame Tussaud’s.

Although we had a great time, Vegas probably only shows its full potential when you’re a highroller.

Edge of eternity

Wed, 07 Sep 2011 06:40:00 +0200

We all know the Grand Canyon as one of the seven natural wonders of the world, today I learned why. As one of the locals described: ‘Pretty amazing for a big hole in the ground’.

Being the closest to the south entrance, we visited the South Rim of the canyon. A trip to the visitor center taught us there were no easy hikes (except for the going down part), so we didn’t attempt at one. A hike down the canyon and back up is a two day endeavor. A shuttle bus took us along the scenic route, dropping us off at three viewpoints. Standing on the edge, stunned by the view, we were accompanied by some California Condors circling above us. However, their view must have been more impressive, being able to stare down right into the 4000 feet (1200m) deep canyon.

After visiting the park, we went to see the National Geographic Grand Canyon movie at the IMAX. This film revealed parts of the inner canyon, following in the footsteps of John Wesley Powell. The show is a cheap alternative to an expensive helicopter flight through the canyon.

Tomorrow morning we will return to the more civilized (?) man-made world: Las Vegas.

From Glen to Grand

Tue, 06 Sep 2011 07:14:00 +0200

Yesterday, we drove from Saint-George to Page, Arizona. The Pipe Springs National Monument happened to be on our way, so we made a stop. Turned out there wasn’t much to see. A 30-minute ranger-led tour made it worthwhile though. During the tour through the Winsor Castle, she told us about the Mormons who used to live there. The Fort manager, also head of their Mormon church, married 58 women, resulting in plenty of children to work the Fort.

Arriving at Page, we hiked the Horseshoe Bend trail, which ended in one of the most amazing views we’ve seen so far. To cool off, we took a swim in Lake Powell, the reservoir above the Glen Canyon Dam.

Today we took a guided tour through Antelope Canyon, which resulted in some interesting pictures.

At the moment we’re in Flagstaff, which will be our launching point to the Grand Canyon tomorrow.

Rusty rocks

Sun, 04 Sep 2011 07:23:00 +0200

We started the day with a short drive to Zion National Park. On our way we came across some colossal rock formations, layered with different shades of red. Since the park is almost entirely car free, we took the shuttle bus along the scenic route. We got off at different stops, to take short hikes to the park’s natural treasures. The Emerald Pools were first up, followed by the Weeping Rock and a riverside walk to the Narrows, which are architected by the Virgin river. Wading in the river gave us some much needed refreshment after a long day of hiking.

Our next destination is Lake Powell.

Sands of Nevada

Sat, 03 Sep 2011 06:29:00 +0200

Yesterday evening we spent the night in a town called Tonopah, which is literally in the middle of nowhere. Today we continued our trip to Zion (Utah), driving hundreds of miles through the Nevada desert, getting sunburned on the way. There was not a soul in sight, except for cattle crossing the road, who seemed to be waiting for us to pass to say hello.

We just arrived at St. George, which is a launching point to Zion National Park.

So, not much to report today. Anyway, here are a few random pictures of our drive through the Nevada desert.

Sands of Nevada

Yosemite skyscrapers

Fri, 02 Sep 2011 07:10:00 +0200

These skyscrapers are, other than the enormous mountains, the redwood sequoia on top of them. Some of these trees are the same height as a thirty story building, which is higher than the Statue Of Liberty. It’s almost impossible to capture these giants on camera.

After a short hike between the sequoia, we drove off to Yosemite Valley to discover the Yosemity Falls. Although the falls must be more impressive in Winter, they were still worth visiting.

Today we drove through Yosemite National Park once again, this time on our way to cross the border to Nevada. On the route, we came across breathtaking swirling roads.

Just before crossing the border to Nevada, we stopped at two Mono Lake vista points.

Tomorrow, we are continuing our trip to Zion.

If you're going to San Francisco

Wed, 31 Aug 2011 09:30:00 +0200

While I’m writing this, the girlfriend and I are on our way from San Francisco to Yosemite National Park. We have spent two full days visiting ‘Fog City’. During these two intensive days we were able to see everything we aimed for, maybe more.

Financial District

This part of town is definitely worth a visit. The multiplicity of skyscrapers merging into the San Francisco fog, is an impressive view. When the fog has gone, the shadows the buildings cast on each other are just as amazing.

Pier 39

While obviously a tourist trap, I found strolling through the shops and restaurants on the pier to be surprisingly amusing.

Alcatraz Island

One of the main attractions of San Francisco, and a must see. If you plan on taking a tour on the island, you must make reservations a few days ahead. We failed to do that, and ended up taking a cruise around Alcatraz Island. I feel like we missed out, but the boat tour was definitely good value for money too.

SS Jeremiah O’Brien

Not so far from Pier 39, you can find a very nicely preserved World War II liberty ship. If you have been on one of those ships before, it might not be worth the 10 bucks.

Lombard Street

Thé most famous street in San Francisco, but kind of overrated. Save yourself the steep hill climb, and try finding a view from a few blocks away.

Chinatown

Always a charming neighbourhood to window shop small boutiques for Chinese kitsch.

Golden Gate Bridge

An amazing piece of architecture. Can you imagine building that in the thirties? We drove there by bike, very recommended. Not for the faint of heart though.

Palace of Fine Arts

On your way to the Golden Gate Bridge, you should drop by the Palace of Fine Arts.

Trivia

If Anonymous isn’t hacking the Sony network, they are protesting in the streets of San Francisco.

I hope this post might have given you some ideas on locations to visit when you’re going to San Francisco.

High Hopes

Mon, 08 Aug 2011 20:30:00 +0200

Last week, I resigned from my job at Ferranti Computer Systems. Three years ago, days after receiving my Graduate Diploma in Applied Computer Science, I had my first real-world working experience at Ferranti Computer Systems.

The first project I was assigned to, was a project for the Antwerp fire department. Along with two other graduates and four seniors, I was thrown in at the deep end. I always had solid grades in school, and was fairly confident that I grasped the material I was taught. Was I misguided (!): it only took me a few days to realize that I knew nothing about building software. It immediately became clear that I had to start learning and probably would need to keep at it forever, if I ever wanted to get remotely good at this.

The Antwerp fire department project turned out to be an immense and complex project, which I have worked on throughout my whole career at Ferranti. Not always that intense though, in the meanwhile I have had a few smaller, more shippable projects and a very similar project for fire department Ghent.

Today, we are a few months into building a promising product for fire departments, which has the potential to be big (in Belgium). Although my role in this project was fun and fairly satisfying, I still felt like it was time for me to move on.

Many might think I’m a fool for throwing away a career where I was actually helping to save lives, building software. There are some frustrations that come along with building software for the public safety sector though. Firstly, how perverted it might sound, this sector has a lot less money to spend on innovation compared to other, commercial sectors. This often leads to solutions which are not perfect, built to meet very tight deadlines. Secondly, making progress is hard. Firefighters want to extinguish fires, save people… Most of them don’t really care about software, making the adaptation and integration process painful. Although, I like to believe that this will change in the future. Our solution has been able to reduce station start-up times by 50%. It’s impossible to ignore those numbers.

This isn’t the real reason I quit my job though. Every sector has its own problems. The real motivation for me to change jobs was that I felt stuck. I have been working with the same team, on the same problem, mostly using the same technology stack, at the same desk, on the same chair for more than three years. This is not what I want, not yet. I want to solve other problems, meet new people, learn from others, try out new technologies and change chairs.

In September, I will start working for Euricom as a consultant, which should give me more opportunities to experience new things.

Will this job be able to give me personal satisfaction? I really don’t know. Will I regret this decision? Probably not, I’d rather make the wrong decision than do nothing.

Anyway, I have high hopes.

Merge sorting in JavaScript

Tue, 26 Jul 2011 19:00:00 +0200

The latest addition to my [data structures and algorithms in JavaScript](https://github.com/JefClaes Data-structures-and-algorithms-in-JavaScript) is the merge sort algorithm.

There are four main steps in the merge sort algorithm (from Wikipedia):

If the list is of length 0 or 1, then it is already sorted.

Otherwise:

Divide the unsorted list into two sublists of about half the size.
Sort each sublist recursively by re-applying the merge sort.
Merge the two sublists back into one sorted list.

I found it a lot easier to understand the algorithm by just looking at this diagram (also from Wikipedia).

I added a public mergeSort function to the sortArray object, which I showed in the first post in these series.

this.mergeSort = function() {                                       
    elements = internalMergeSort(elements, this.onSort);                
    return elements;
};

This function calls the internalMergeSort function passing in the internal elements array and the onSort callback.

var internalMergeSort = function(elements, onSort){            
    if (elements.length < 2){                               
        return elements;  
    }                           

    // Calculate the middle of the elements
    var middle = Math.floor(elements.length / 2);           
    // Divide 
    var leftRange = elements.slice(0, middle);
    var rightRange = elements.slice(middle, elements.length);           
    // Conquer                                                           
    var mergingResult = merge(internalMergeSort(leftRange, onSort), 
                              internalMergeSort(rightRange, onSort));                                   
    onSort(mergingResult);           

    return mergingResult;
};

The function recursively divides the elements in two parts, sorting them and finally merging them back together.

The merge function is implemented like this.


function merge(left, right){                      
    var res = [];           

    while (left.length > 0 && right.length > 0) {                
        if (left[0] <= right[0]) {
            res.push(left.shift());
        } else {
            res.push(right.shift());
        }                                              
    }           
    
    while (left.length > 0) {                
        res.push(left.shift());
    }            

    while (right.length > 0) {            
        res.push(right.shift());
    }

    return res;
};

I peeked at this implementation, which is very similar to mine and which helped me write the first while loop more elegantly.

Testing sorting algorithms is pretty easy. So far, I have only defined one QUnit test for this algorithm.

test("MergeSort sorts.", function() {
    var sortArray = new algorithms.sortArray();

    sortArray.push(15);
    ...
    sortArray.push(130); 

    var actual = sortArray.mergeSort();                               
    var expected = [1, 10, 12, 15, 17, 22, 25, 50, 60, 90, 130];

    deepEqual(actual, expected);
});

A small remark here is that you should use deepEqual instead of equal for array assertions. We want to compare the contents of the array, not their references.

My js data structures and algorithms now on GitHub

Sat, 23 Jul 2011 15:30:00 +0200

If you have been reading my blog lately, you know that I’m implementing some data structures and algorithms in JavaScript. So far, I have blogged about simple sorting algorithms, the stack data structure and the queue data structure. This week I have also implemented a doubly linked list. I started writing a post about that last implementation, but I didn’t like where it was going, so instead of writing about it, I have pushed everything to a public Git repository.

You can find the public repository here.

I structured the repository like this:

Scripts: The JavaScript files which contain the various algorithms and data structures.
Implementations: HTML pages used to demonstrate the usage of some of these algorithms and data structures.
Tests: QUnit tests.

Feel free to watch or fork this project. I will still be writing about most implemenations, but probably not all of them.

Overoptimizing my JavaScript stack implementation for fun

Mon, 18 Jul 2011 21:00:00 +0200

Davy Brion made a comment on my JavaScript stack/queue implementation on Twitter last night: Any reason why you don’t immediately set elements to [] at declaration in your stack/queue example?

var elements;

this.push = function(element) {
    if (typeof(elements) === 'undefined') {
        elements = [];   
    }                            

    elements.push(element);
}

Yes, I made an overoptimization, and a bad one. In this implementation, you save a few bytes in memory if you initialize the stack, but don’t push elements. This might have made some sense 15 years ago, but today a few bytes are very negligible compared to the cost of evaluating the elements reference on every push call.

Anyway, thinking about this driving home, I thought of another optimization, which meets both arguments and is far more fun.

this.push = function(element) {                        
    elements = [];   
    elements.push(element);                            
    this.push = function(element) {
        //Rewriting self. Overoptimization ftw!
        elements.push(element);                                   
    }                                                 
}

The first time the push function is executed, the elements array is initialized. In the same function, the function rewrites itself to no longer initialize the elements array. So the second time the push function is called, it will no longer initalize the array, but only push a new item to the existing elements array.

Dynamilicious!

Stacks and queues in JavaScript

Tue, 12 Jul 2011 21:00:00 +0200

The second assignment in my ‘implementing data structures and algorithms in JavaScript’ quest consists of two popular data structures: the stack and the queue.

The stack

A stack is a last in, first out (LIFO) abstract data type and data structure. A stack can have any abstract data type as an element, but is characterized by only three fundamental operations: push, pop and stack top.

Implementing this turned out to be pretty easy. A native JavaScript array already exposes methods to push and pop elements. In the dataStructures namespace, I defined a stack object. The stack object contains a private array which is initialized as soon as the first element is pushed into the stack. The public push and pop functions expose the corresponding functions of the private array. The stackTop function returns the last element added to the stack, but doesn’t remove it from the internal array.

var dataStructures = {
    stack : function() {                  

        var elements;

        this.push = function(element) {
            if (typeof(elements) === 'undefined') {
                elements = [];   
            }                            
            elements.push(element);
        }

        this.pop = function() {
            return elements.pop();
        }

        this.stackTop = function(element) {
            return elements[elements.length - 1];
        }
    }
}

Although a native array contains most stack operations, the stack object we made is still pretty useful. We end up with a clean encapsulated stack which only exposes the core stack operations.

var someStack = new dataStructures.stack();

someStack.push(1);
someStack.push(2);
someStack.push(3);

var stackTopResult = someStack.stackTop();                         
stackTopResults.html(stackTopResult);

var popResult = "";

popResult += someStack.pop();
popResult += someStack.pop();
popResult += someStack.pop();

The queue

A queue is a particular kind of collection in which the entities in the collection are kept in order and the principal (or only) operations on the collection are the addition of entities to the rear terminal position and removal of entities from the front terminal position. This makes the queue a First-In-First-Out (FIFO) data structure. In a FIFO data structure, the first element added to the queue will be the first one to be removed.

As with the stack, the native array object already contains a few functions which help us implement a queue. The terminology doesn’t completely match though. Enqueue maps with pop and dequeue maps with shift.

I added the queue object to the dataStructures namespace. The queue also holds a private array of elements. The enqueue function pushes a new element on the queue, and the dequeue function removes the first element from the queue. The peek function returns the first element in the array, but does not remove it.

queue : function() {
    var elements;

    this.enqueue = function(element) {
        if (typeof(elements) === 'undefined') {
            elements = [];   
        }
        elements.push(element);                       
    }

    this.dequeue = function() {
        return elements.shift();                                   
    }

    this.peek = function(){
        return elements[0];                  
    }
}

The queue can be used like this.

someQueue.enqueue(1);
someQueue.enqueue(2);
someQueue.enqueue(3);               

var queuePeekResult = someQueue.peek();

queuePeekResults.html(queuePeekResult); 

var dequeueResult = "";                   

dequeueResult += someQueue.dequeue();
dequeueResult += someQueue.dequeue();
dequeueResult += someQueue.dequeue();

Find the final result here. Also check out the previous post on simple sorting in JavaScript.

Simple sorting in JavaScript

Thu, 07 Jul 2011 21:30:00 +0200

About three years ago I graduated and got my degree in Applied Computer Science. Although it says Computer Science, we hardly ever focused on data structures and algorithms. Obviously, I now see that as a shortcoming. So I plan to make up for that by reading up on some of the basics. While at it, I might be blogging on some of the topics.

I am going to start by implementing some of the simple sorting algorithms in JavaScript.

Find the final result here.

Bubble sort

The first sorting algorithm I’m going to implement is probably also the easiest, and slowest in most scenario’s: Bubble sort.

Bubble sort is a simple sorting algorithm that works by repeatedly stepping through the list to be sorted, comparing each pair of adjacent items and swapping them if they are in the wrong order. The pass through the list is repeated until no swaps are needed, which indicates that the list is sorted.

Let’s start by defining a namespace.

var algorithms = { };

We could extend the native array object, but to keep things simple, let’s create our own sortArray object.

var algorithms = {         
    sortArray : function()

    }
};

The sortArray object contains a private array of elements. To add elements to that array, we will expose a push function on our sortArray object.

var algorithms = {        
    sortArray : function() {
        var elements = [];                                                                      
        this.push = function(val){
            elements.push(val);                    
        };                                       
    } 
};

So far, we can do something like this.

var sortArr = new algorithms.sortArray();               

sortArr.push(80);
sortArr.push(20);
sortArr.push(...);

Now we have to add a public sort function. Before we do that we need to define a private swap function, which can swap elements.

var swap = function(one, two) {
    var tmp = elements[one];
    elements[one] = elements[two];
    elements[two] = tmp;                    
};

The bubbleSort function will take a callback argument which will be called every sort iteration.

this.bubbleSort = function(callback) {   
    callback(elements);

    //Loop over all the elements
    for (var out = elements.length - 1; out > 0; out--){                
        for (var inn = 0; inn < out; inn++) {
            //Are they out of order?
            if (elements[inn] > elements[inn+1]){
                swap(inn, inn+1);                                
            } 
        }

        callback(elements);
    }                    
};

That should be it. Now let’s try to push some elements in the array, define a callback and call the bubbleSort function. In the callback function I’m using the Google Charts API to visualize the sorting process.

$(document).ready(function() {             
    ...                                 

    var results = $("#results");

    var drawElements = function(elements) {                            
        var chd = "";                    

        for (var e in elements) {
            chd += elements[e].toString() + ",";
        }

        //Remove that last ","
        chd = chd.substring(0, chd.length - 1);                        

        //Build the Google Charts url
        var chartUrlBase = "http://chart.apis.google.com/chart?chxt=y&chbh=a&chs=300x225&cht=bvg&chco=A2C180,3D7930&chtt=Sorting";
        var chartUrlComplete = chartUrlBase + "&chd=t:" + chd;                                   
        $("#results").append("<img src='" + chartUrlComplete + "'/>").append("<br/>");
    }                              

    sortArr.bubbleSort(drawElements);                               
});

Selection sort

The second algorithm is Selection sort. I found this one the easiest to understand so far: find the mimimum value, swap it with the value in the first position, advance one position and repeat.

Selection sort is a sorting algorithm, specifically an in-place comparison sort. It has O(n2) time complexity, making it inefficient on large lists, and generally performs worse than the similar insertion sort. Selection sort is noted for its simplicity, and also has performance advantages over more complicated algorithms in certain situations, particularly where auxiliary memory is limited.

Let’s start by giving the BubbleSort callback function another place. We will make this a public function on our sortArray. I will name it onSort and set it to an empty function by default.

this.onSort = function() {};

Once that is done we can define our public selectionSort function. This function calls the onSort function every sort iteration.

this.selectionSort = function(){

    this.onSort(elements);

    for (var out = 0; out < elements.length - 1; out++) {
        var min = out;

        for (var inn = out; inn < elements.length; inn++) {
            //If minium greater => new minimum
            if (elements[inn] < elements[min]) {
                min = inn;
            }                            
        }

        swap(out, min);

        this.onSort(elements);
    }
};

Insertion sort

The last simple sorting algorithm I’m going to implement in this post is Insertion sort.

Insertion sort is a simple sorting algorithm: a comparison sort in which the sorted array (or list) is built one entry at a time.

Every repetition of insertion sort removes an element from the input data, inserting it into the correct position in the already-sorted list, until no input elements remain. The choice of which element to remove from the input is arbitrary, and can be made using almost any choice algorithm.

The insertionSort is just another public function on our sortArray object.

this.insertionSort = function(){
    this.onSort(elements);

    for (var out = 1; out < elements.length; out++) {               
        var temp = elements[out];
        var inn = out;            

        //Until one is smaller
        while (inn > 0 && elements[inn-1] >= temp){
            elements[inn] = elements[inn-1];

            inn--;
        }

        elements[inn] = temp;

        this.onSort(elements);
    }
}

You can find the final result here. If you have any remarks on these implementations, please let me know!

Should we get this tool?

Wed, 15 Jun 2011 21:45:00 +0200

This is a decision that often needs to be made by middle management. For managers the most natural way to make this call is by evaluating the return on investment. To calculate the ROI, you need to compare the gain of an investment relative to the amount of investment. And this is exactly where things get hard, if not impossible. Measuring developer productivity is one of the unsolvable problems in our industry.

If a developer wants something which he feels will improve his productivity, not a lot of questions should be asked. The investment is probably ridiculously little, compared to the risk of depressing one of your codemonkeys. Developer happinness is at least as important as developer productivity. Happy developers tend to get passionate about their profession, and therefor ship better software as a result. And that’s what we are all after, right?

I also find that peers saying something along the lines of “What’s wrong with notepad?” are foolish. If you are most productive and happy using notepad, that’s perfectly fine, but don’t go belittling people who enjoy using more rich and intelligent editors.

Book review: More Joel on Software

Tue, 07 Jun 2011 19:00:00 +0200

I was lucky to pick up a copy of More Joel on Software for only 5 euros at a bookfest last week. Turns out the book ships pretty cheap on Amazon as well.

More Joel on Software was written by Joel Spolsky, already a legend for launching The Stack Exchange Network with Jeff Atwood. Although it’s hard to not know him, I admittedly hadn’t read a lot of his material so far. Starting my professional career in late 2008, the Spolsky writing high days were mostly over.

More Joel on Software is the second blook which bundles the best of the digital content of the Joel on Software blog into a dead tree. Although you can find every part of the book on Spolsky’s blog, it was well worth the money. I love how he has, next to amusing anecdotes, a very opinionated view on our industry which really challenges you to broaden your thinking. His writing is also extremely enjoyable, I think he must be one of the best writers in our field.

Here is, for your convenience, the content of the book with links to the original articles online. Might be useful if you are the owner of one of those reading devices.

Part 1 - Managing People

Part 2 - Advice to Potential Programmers

The Perils of JavaSchools
Talk at Yale (1, 2, 3)
Advice for Computer Science College Students

Part 3 - The Impact of Design

Part 4 - Managing Large Projects

Part 5 - Programming Advice

Part 6 - Starting a Software Business

Part 7 - Running a Software Business

Part 8 - Releasing Software

Part 9 - Revising Software

Checking for anonymous types

Sat, 21 May 2011 18:00:00 +0200

Because I blogged about anonymous types last month, I thought following method would also make an interesting post.

private static bool IsAnonymousType(Type type) {
    Debug.Assert(type != null, "Type should not be null");

    // HACK: The only way to detect anonymous types right now.
    return Attribute.IsDefined(type, typeof(CompilerGeneratedAttribute), false)
            && type.IsGenericType 
            && type.Name.Contains("AnonymousType")
            && (type.Name.StartsWith("<>", StringComparison.OrdinalIgnoreCase) ||
                type.Name.StartsWith("VB$", StringComparison.OrdinalIgnoreCase))
            && (type.Attributes & TypeAttributes.NotPublic) == TypeAttributes.NotPublic;
}

For a type to be anonymous:

It should be marked with the CompilerGenerated attribute
It should be a generic type
Its name should contain “AnonymousType”
Its name should start with “<>” or “VB$”
It shouldn’t be publicly accessible

A little fun fact is that the VB and C# compiler generate different type names. The C# compiler makes the type name start with “<>” and the VB compiler uses “VB$”. Both smart safeguards, because the compiler doesn’t allow us to use “<>” or “$” while defining type names. I find the C# way a tad more elegant though.

I stumbled upon this beauty while browsing the [ASP.NET MVC source] (http://aspnet.codeplex.com/) (System.Web.Helpers.ObjectVisitor). Because there is no direct way to detect anonymous types yet, I’m pretty sure this is the best implementation out there.

My thoughts on WebMatrix

Mon, 09 May 2011 21:15:00 +0200

After building arealdeveloper.com, I felt like I had to do a follow-up post sharing my experiences with WebMatrix. While doing some research, I came across this post by Rob Connery. And frankly, I think it’s almost impossible for me to add something to his findings.

In a nutshell

For those who are here for the “tl;dr”, here are my thoughts on WebMatrix..

WebMatrix is the perfect framework to start with ASP.NET development (wish I had). For people who are already familiar with ASP.NET, WebMatrix will hardly have any learning curve. They should also have a look at it though, because it’s a lot of fun ánd it is a great framework to get things done. Every developer has been asked - at least once - to slap together a site for a distant acquaintance, where the site consists of mostly static pages, with a dynamic List<Something> pulled out of a database. I see WebMatrix as a perfect gluing framework to swiftly ship these types of projects. While you could probably also build something more enterprisey, I don’t think you should.

See for yourself

I have uploaded the source of arealdeveloper.com. Download WebMatrix using the WebPlatform Installer and see for yourself.

You might notice that the project is called arealprogrammer. That was the name of the project before I bought the wrong domain name.

arealdeveloper.com mentioned on Channel9

Mon, 02 May 2011 21:30:00 +0200

While nosing around in the arealdeveloper.com analytics, I saw a bunch of traffic coming from Channel9. Clicking through, I found out A Real Developer was mentioned in this video (02:01). That’s a Channel9 front-page video!

That’s so awesome, I think it’s great people find it a fun project.

I have been listening to your feedback, and this Sunday I implemented some of your requests:

Easier linking
Sharing
A feed
Voting
Comments

I try to keep everything as simple as possible, that’s why I implemented most of these features using two popular free plug-ins: AddThis and Disqus.

Oh, and by the way, for Americans, they did pretty well pronouncing my last name ;)

arealdeveloper.com

Tue, 26 Apr 2011 20:00:00 +0200

A Real Developer is a side-project I built over the weekend poking around with WebMatrix.

It got a little out of hand. After publishing, I showed it to a few popular tweeps, and they were kind enough to mention it. 24 hours after the launch it has gotten over 3.5k unique visitors, 39k page views (see what I did there?) and over 240 submissions. Being mentioned by Scott Hanselman and Rey Bango helps.

I kind of felt like a mini - Mark Zuckerberg constantly refreshing the page and reading new quotes hours afer the launch.

I was somewhat surprised by how original and creative our community is. Some of my favorite quotes so far..

A real developer has changed live code in production at least once, begrudgingly. (by @jasonfamularo)
A real developer has code on his t-shirt. (by @stevenmilne)
A real developer knows family is more important than computers. (by @shanselman)
A real developer tries to write as little code as possible. (by @marknijhof)
A real developer works in the dark, but communicates with those in the light. (by @atcrawford)
A real developer understands code is nearly the same as an organic life-form that evolves. (by @stefandidak)

And so much more of them are awesome.

The future

I enjoyed the experience so far. I’m well aware that this buzz will end soon, but I’m going to try to maintain momentum as long as possible.

That’s why I am eager to hear your suggestions and feedback.

Requests received so far:

Make it possible to up- or downvote quotes.
Make a smart algorithm based on those votes.
Make it easier to link to quotes.
Twitter authentication.

I might open source this little project (it’s only 50 lines of code) and I might do a few blog posts on the making of in the next weeks.

Prague impressions

Sun, 24 Apr 2011 15:45:00 +0200

Last year my girlfriend and I already made a trip to the Czech Republic, but we felt like we had spent to little time visiting Prague. We made up for that by making a follow-up trip last week.

Find some of our most memorable impressions captured on picture below..

The Lennon Wall

Love padlocks near the Lennon Wall.

Hiking route in the New Town district.

The Charles bridge from a distance.

The astronomical clock of Prague.

Hand painting Easter eggs like a boss.

More fairylike Easter festivities.

Close-up of a butterfly, shot in the Prague Fata Morgana.

Night at the National Technical Museum.

Prague by night.

Oh, iPads are for taking pictures.

Anonymous type equality follow-up: Equals()

Sun, 24 Apr 2011 12:30:00 +0200

After publishing yesterday’s post on anonymous type equality, I received an interesting comment. The comment stated that even if the sequence of the property assignment were the same, the equality comparison would still return false, because the types generated by the C# compiler are reference types, making their references being tested for equality and not their data.

This is very true, unless the Equals() method is overridden. And this is exactly what the compiler does for us when we define anonymous types.

In the Equals() method the equality of each property is evaluated by using a generic EqualityComparer.

That explains this behaviour.

var a = new { x = 1, y = 2 };
var b = new { y = 2, x = 1 };
var c = new { x = 1, y = 2 };

Console.WriteLine(a.Equals(b)); //False
Console.WriteLine(a.Equals(c)); //True

Anonymous type equality

Sat, 23 Apr 2011 15:15:00 +0200

Let’s say you instantiate two variables (a and b) using anonymous types. They both have the same two properties (x and y) with equal values.

var a = new { x = 1, y = 2 };
var b = new { y = 2, x = 1 };

Do you think they are equal?

Console.WriteLine(a.Equals(b)); //Prints false :O

They are not. Not something I expected!

If we look at the IL the C# compiler produced, it starts making sense though.

There are two different types generated, although the properties we assigned are the same. What differs is the sequence of the property assignment.

This is defined in chapter 7.6.10.6 Anonymous object creation expressions of the C# 4.0 specifications.

Within the same program, two anonymous object initializers that specify a sequence of properties of the same names and compile-time types in the same order will produce instances of the same anonymous type.

Conclusion

When defining anonymous types, the sequence of the property assignment matters. If the sequence of the property assignment differs, different types are defined by the C# compiler.

Using C# keywords as variables

Mon, 11 Apr 2011 19:00:00 +0200

Hold it, don’t shoot me. I know this would be an awful practice, but it is an interesting C# compiler quirk nonetheless.

Keywords are predefined reserved identifiers that have special meanings to the compiler. They cannot be used as identifiers in your program unless they include @ as a prefix. For example, @if is a legal identifier but if is not because it is a keyword.

static void Main(string[] args) {           
    var @if = "oh my..";

    Console.WriteLine(@if);
}

========== Build: 1 succeeded or up-to-date, 0 failed, 0 skipped ==========

It's not how good you are, it's how good you want to be

Sun, 10 Apr 2011 20:30:00 +0200

This Christmas, my girlfriend and I got this little book called It’s not how good you are, it’s how good you want to be.

This book is mainly directed to (advertisement) designers, but since I feel that designers and developers do have a few things in common, we both like to create things, I also read it.

The content of this book is a collection of Paul Ardens observations on business and life presented in a very consumable format. As it turns out, although I am not part of the target audience, a good amount of his thoughts can be applied to software development as well.

Find a few of these thought-provoking quotes below.

If you are involved in something that goes wrong, never blame others. Blame no one but yourself.

Give away everything you know, and more will come back to you.

Don’t look for the next opportunity. The one you have in hand is the opportunity.

The person who doesn’t make mistakes is unlikely to make anything.

If you get stuck, draw with a different pen.

Don’t be afraid to work with the best.

Video, slides and source from my WebSockets talk

Thu, 07 Apr 2011 21:30:00 +0200

Yesterday, I gave a talk on WebSockets at HTML5 WebCamps Belgium.

WebSockets on Fire

While the WebSockets API specification is not stable yet, various browser vendors have already implemented a prototype in their latest browser versions. Microsoft released their WebSockets prototype in December 2010. While there is no native implementation of WebSockets in IE9 just yet, for now they are providing a solution which works cross-browser, relying on a Silverlight client. Server-side a Windows Communication Foundation server is made available, which implements the latest version of the WebSockets protocol.

In this session, Jef covers the WebSockets fundamentals, demonstrates a proof of concept that could be used in various fire department web applications and deep-dives into the code of this proof of concept.

I got a lot of positive feedback on Twitter, which makes me a very happy panda!

You can already find the video of my session on Channel9. The slides can be found on SlideShare.

If you want to play with the demo yourself, you can download the source:

Make sure your firewall allows incoming TCP traffic on port 4502
Deploy the client access policy file to IIS
Also host the Demo.WebSockets.WebClient in IIS

More information on installing the WebSockets prototype here.

Keeping WebSockets alive

Sat, 12 Mar 2011 17:40:00 +0100

The problem with using stateful connections on an imperfect place as the internet is that connections might drop. The server or an intermediary can drop the connection due to an idle timeout. Even a temporary problem at the server or a local network hiccup might kill your connection.

If you aren’t prepared to handle these scenarios, you will not be able to fully rely on WebSockets.

A simple solution

The simplest solution is checking every few seconds whether the WebSocket is still opened. This might suffice in a good amount of scenarios, but a lot of other scenarios require more stable connectivity.

$(document).ready(function () {        
    setInterval(openWebSocket, 5000);
}

function openWebSocket(){
    if (websocket.readyState === undefined || websocket.readyState > 1) {
        ...
    }
}

Keepalives

As mentioned before, the server or an intermediate might drop the connection when the connection becomes idle. To prevent this, you could make your server send keepalive messages at predefined intervals.

I implemented this in the HTML5 Labs WCF Server.

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession, 
                 ConcurrencyMode = ConcurrencyMode.Multiple)]

public class CadSubscriptionService : WebSocketsService {
    private static KeepAliveBroadcaster _keepAliveBroadcaster;

    public CadSubscriptionService() {           
        if (_keepAliveBroadcaster == null) {
            _keepAliveBroadcaster = new KeepAliveBroadcaster();
        }
    }
 
    public override void OnOpen() {
        _keepAliveBroadcaster.AddService(this);

        base.OnOpen();
    }

    protected override void OnClose(object sender, EventArgs e) {
        _keepAliveBroadcaster.RemoveService(this);

        base.OnClose(sender, e);
    }     
}

The KeepAliveBroadcaster class maintains a list of connected sessions. Using a Timer it sends a message to each client every 15 seconds. This should stop the connection from dropping due to an idle timeout.

public class KeepAliveBroadcaster {

    private List<WebSocketsService> _services;
    private Timer _timer;

    public KeepAliveBroadcaster() {
        _timer = new Timer((o) => {
            if (_services == null) {
                return;
            }

            lock (_services) {
                if (_services.Count > 0) {
                    foreach (var service in _services) {
                        service.Send("staying alive!");
                    }
                }
            }
        }, null, 100, 15000);              
    }

    public void AddService(WebSocketsService service) {
        if (_services == null) {
            _services = new List<WebSocketsService>();
        }

        lock (_services) {               
            _services.Add(service);
        }
    }

    public void RemoveService(WebSocketsService service) {
        lock (_services) {
            _services.Remove(service);
        }
    }
}

Reopening ASAP

Something you could also implement is reopening the connection as soon as it closes.

Retry opening the connection when the onclose event fires. Think about limiting the number of retries, or you might end up with an infinite loop.

    websocket.onclose = function (event) {            
        $('#status').html('Socket closed');    
        
        retryOpeningWebSocket();
    };


    function retryOpeningWebSocket(){
        if (retries < 2) {            
            setTimeout(openWebSocket, 1000);            
            retries++;
        }
    }

Some afterthought

Depending on what type of connectivity you require, these solutions might not suffice. If you can’t afford to lose a single message, you might want to think about implementing queues at the client and server to overcome a gap of connection loss. Maybe you even want to implement some sort of acknowledge messaging. Something for a future post maybe!

Why I still buy real books

Tue, 01 Feb 2011 08:30:00 +0100

As a technology geek, it should be hard to keep ignoring the latest generation of eReaders (read: the Kindle). With over three million devices sold, the Kindle has proven that it adds value to people’s lives, and that it’s a lot more than just a gadget.

Some of the most obvious benefits… You can carry a whole library in your backpack. The price of an eBook often is substantially less than the hardcover version. I am far from a Green, but I can’t neglect the fact that no trees need to be chopped down to print an eBook. Reading a 1000+ page book in bed or next to the pool is the opposite of convenient.

Sufficient arguments to get a Kindle, but still, I just can’t. Like serial killers collect trophies of their victims, I want to collect every book I have ever read. Unlike serial killers, I want to exhibit every trophy. I catch myself regularly halting for a few minutes when I pass by my bookcase. I love to have a look at the bookspines and associate the title with the time and place I read the book. Part of my love for paper books might also be caused by the global perception that books equal wisdom.

Book review: The Art of Non-Conformity

Sat, 22 Jan 2011 14:00:00 +0100

The book The Art of Non-Conformity: Set Your Own Rules, Live the Life You Want, and Change the World was written by Chris Guillebeau.

I learned about this book through Chris’ blog. He hosts a very popular blog where he writes on lifehacking, entrepreneurship and his goal to travel to each country in the world.

In this book Chris tries to show the reader that you don’t have the live your life the way other people expect you to. As long as you are open to new ideas, are dissatisfied with the status quo and are willing to work hard, you should be able to live the life of your dreams.

The first part of this book was an eye-opener for me. Once you find something that you really enjoy doing, your passion, make a commitment and be dedicated to use it to make a positive change. For yourself and for the world. Don’t be afraid to be somehow different, a lot of great things have been accomplished outside the status quo. If living the life you want, takes it for you to be unconventional and nonconformist, overcome your fear, and pursue it.

In the second and last part of the book, Chris lost most of my interest. In the second part he shares his personal experience with education, building an army of followers and financing a noncomformist lifestyle. The last part of this book is on building a legacy.

Throughout the book Chris shares his personal stories and aneckdotes. There are also dozens of stories about other people changing their lifes for the better. I especially enjoyed the latter. In his personal stories some of his achievements are repeated throughout the book, which sometimes gives the impression he wrote the book to brag and to promote the AONC brand. Another letdown of this book is that some of the advice isn’t useful for most of us. Frequent flyer miles anyone?

I enjoyed the first part of the book a lot. Too bad the other parts mostly failed in keeping me interested.

My rating: 3/5.

Book review: Got Fight?

Sat, 15 Jan 2011 18:00:00 +0100

I rarely review books with non-technical content here, but I just felt like I had to with this one.

The book Got Fight? was written by Forrest Griffin, assisted by Erich Krauss.

Forrest Griffin is one of the toughest light-heavyweight mixed martial artists competing in the UFC. If you are unfamiliar with MMA and UFC, I encourage you to watch one of the Forrest Griffin tribute videos on Youtube.

This book touches several subjects. Forrest talks about the path he followed to become a professional fighter and what made him who he is today. Forrest also shares a bunch of advice on fighting and non-fighting topics. The last part of the book contains a series of how-to MMA techniques. All these topics are talked together in the form of hilarious, and sometimes offending (to some of you), ramblings. After all, the main goal to read this book shouldn’t be to learn, but to be entertained, and maybe learn something along the way!

Below you can find some quotes taken from the book, which will probably tell you a lot more than my small summary above.

Forrest on..

.. sphincter control

Back in the day, when I first met a chick, there was no way I would let one rip. I would wait until at least the second or third date before I introduced her to my Love Potion 109.

.. advice for the fat and tremendously out-of-shape fighter

Steroids, steroids, steroids. This magic elixir can replace all forms of training and will add to your intimidating appearance.

.. cheating

When asked about the tactic after the fight, Tito said, ‘If you ain’t cheating, you ain’t trying’. I would have to agree with him. I would rather be known as a cheater than a loser. Seriously, I have this deep hatred of losing.

.. fear

A black eye will dissapear and a broken nose can be mended. Fear is a good thing because it keeps you alive, but it becomes so great that it hinders you from doing what you want, you need to confront it head-on.

.. eating challenges

I kept mumbling over and over, ‘Oh God, I made a mistake’, and vociferously promised sweet baby Jesus I would never accept an eating challenge again.

.. being humble

Although it can be depressing to admit to yourself that you will never be the best, it is liberating at the same time. Instead of trying to be better than everyone else, which is existentially arrogant, you can focus on being the best that you can be.

My rating: 4/5. This is probably the funniest book I have ever read.

Photo credit: ESPN

Book review: Pro HTML5 Programming

Thu, 06 Jan 2011 14:30:00 +0100

The Apress book Pro HTML5 Programming was written by three authors: Peter Lubbers, Brian Albers and Frank Salim. In this book the authors introduce the reader to the most popular and useful features of HTML5.

The book contains 11 concise chapters: An overview of HTML5, the Canvas API, audio and video, the Geolocation API, the Communication API, the WebSocket API, Forms, the Web Workers API, the Web Storage API, Offline Web Applications and the future of HTML5.

This book succeeds in breaking the myth that HTML5 will not be applicable until 2022. The authors demonstrate this by accompanying all theory with a ton of examples based on realistic scenarios. The examples are not to complex, and most of the time, cut in well sized pieces, which makes them easy to digest. The authors show you how to detect whether a certain feature is supported by the hosting browser. They do not provide information on fallback mechanismes though, which I think is a good thing, because it would only break the tight scoping of the book.

Developers who are new to HTML and JavaScript should pick up other books first, some experience with client-side webdevelopement is adviced.

I think this book is a good read and can serve as an excellent introduction to HTML5. You should get it before it’s obsolete ;).

My rating: 4/5.

Reviewing and renewing resolutions

Wed, 29 Dec 2010 20:17:00 +0100

My career

I’m still working on the same projects as I was last year: Firestation Antwerp and Ghent. This year we did not build a lot of brand new things, but we have added a decent amount of new functionalities. Next to that there were a bunch of bugs to fix and technical debt to pay off. I learned a lot from the mistakes we made and refactoring makes you really think about what good and clean code should look like.

In 2011 I will still be spending time on Firestation Antwerp and Ghent. There is a hazy bright light on the horizon though, chances are there is a new interesting challenge ahead.

This year I had some fun side-challenges. I gave internal technical sessions on ASP.NET 4 & ASP.NET AJAX, something I enjoyed doing more than I thought. I also passed the MCTS 70-536 exam, which I didn’t enjoy studying for that much. In 2011 I want to pass at least one MCTS ASP.NET exam, and give more technical sessions.

Blog

I have written 62 posts this year. Not as much as I would have liked to. There are several reasons I didn’t produce more content. I have had a technology hiatus this summer, because I was just too busy with other life stuff. Next to that, I have been working on a side-project and have been studying for the MCTS 70-356 exam. A lot of excuses, but I plan to make up for it in 2011.

I am pretty satisfied with the qualtiy I produced this year, although there is a lot of room for improvement. I like to think a good metric for quality is the number of visitors and retweets. I got double the traffic compared to last year, which makes me feel a little warm inside.

This year I mainly blogged about Webforms (refactoring), ASP.NET AJAX, the Bing API, TechEd, IE9 and HTML5. In 2011 I plan to continue to focus on Microsoft and the web. I’m still an MVC and WebMatrix novice, which needs to change next year.

Community

Last year I planned on being more active on StackOverflow, but I failed miserably. I don’t think I will be making this a goal this year. I’m just going to continue to go there for questions only.

I also planned on going to a big event in 2010, which turned out to be TechEd Europe. Seriously, I could hang around there an entire month.

In being part of the Belgian .NET community I was less successful. I have been talking a lot with other Belgian developers through blogs and Twitter, but I should attend more real-life events. With the DrinksWithDevs community Davy Brion is starting, I think I might be doing better this year.

Travelling

One of the goals I’m totally satisfied about.

Next to citytripping in Belgium, I was lucky enough to travel abroad 5 times this year:

Czech Republic
Holland
Denmark
Spain
Germany (TechEd)

There are no plans for next year (yet), but I don’t think I will be able to do much better.

Sport/Health

This is a new section in this year’s report. Sport has become more and more important to me over the years. Next to going to the gym and running, I picked up boxing this September.

In the next year I want to focus more on conditioning and less on strength. It’s hard to set a specific goal right now, because I haven’t measured the speed and distance of my runs this year. For Christmas I got the Nike+ iPod gear, so that should help me to collect some data the following months.

Conclusion

This year had its ups and downs. Not a lot of exciting things happened at work, but 2011 should change that. I am overall content with the progress of my blog, although I think I should do better next year. I should be very satisfied with the travelling I did this year, I have spent more than a month abroad! I think I improved my health noticeably this year, I hope I stay injury-free and can put enough effort in to keep on improving my conditioning.

All by all 2010 was a good year and I hope 2011 will be at least as good. And to be honest, I don’t mind not being able to accomplish all goals, because the love of my family and friends compensates a lot. That was cheesy, right?

HTML5: Exception handling with the Geolocation API

Sun, 26 Dec 2010 20:30:00 +0100

In my previous post on the Geolocation API I passed in a PositionErrorCallback to the geolocation.getCurrentPosition() method. When I received this callback I displayed a generic message informing the user something went wrong. In real-world scenarios you probably want the message to be more specific. You might also want to call a specific fallback method depending on what went wrong.

This is where the PositionError argument of the PositionErrorCallback comes in handy. This object has two properties: code and message.

The code property can return three codes:

PERMISSION_DENIED
POSITION_UNAVAILABLE
TIMEOUT

The message property returns a string describing what went wrong. Be careful, this property is primarily intended for debugging!

Example

The codesnippet below only shows the part where I am handling the PositionErrorCallback.

function onError(error){
    var content = document.getElementById("content");        
    var message = "";
    
    switch (error.code) {
        case 0:
            message = "Something went wrong: " + error.message;
            break;
        case 1:
            message = "You denied permission to this page to retrieve a location.";
            break;
        case 2:
            message = "The browser was unable to determine a location: " + error.message;
            break;
        case 3:
            message = "The browser timed out before retrieving the location.";
            break;
    }

    content.innerHTML = message;
}

HTML5: The Geolocation API is scary (good)

Sun, 19 Dec 2010 17:30:00 +0100

I read about the HTML5 Geolocation API in the Pro HTML5 Programming book a while ago, and decided to play with it on this lazy Sunday.

Using the Geolocation API to make a one-shot position request is very straight-forward. Get a reference to the navigator.geolocation object and call the getCurrentPosition() method, passing in at least a PositionCallback. In this example I’m also passing in a PositionErrorCallback. In the PositionCallback you can examine the properties of the position object. Here I am only using the latitude and longitude properties.

function showLocation() {                       
    if (navigator.geolocation) {
        navigator.geolocation.getCurrentPosition(onSuccess, onError);
    } else {
        var content = document.getElementById("content");        
        content.innerHTML = "Geolocation is not supported by your browser!";
    }                
}
    
function onSuccess(position){
    var content = document.getElementById("content");        
    content.innerHTML = position.coords.latitude + ", " + position.coords.longitude;
}

function onError(error){
    var content = document.getElementById("content");        
    content.innerHTML = "Something went wrong..";
}

That’s really easy, right?

The scary part

The privacy part isn’t that scary, because the specs state that browsers must acquire permissions through a user interface. The scary part is, that it really works and is very accurate even though I’m on a desktop with no known hotspots nearby! After getting the results, I pasted them in Google Maps and they were only a few meters off. Why is it that while most geolocation services have been failing over the years, it suddenly works this good?

There is some documentation out there, but this documentation is very simplistic and doesn’t touch the internals.

The Pro HTML5 Programming book lists which sources can be used.

A device can use any of the following sources:

IP address

Coordinate triangulation:

Global Positioning System (GPS)

Wi-Fi with MAC addresses from RFID, Wi-Fi and Bluetooth

GSM or CDMA cell phone IDs

User defined

FireFox lets the Google Location Services assist them.

If you consent, Firefox gathers information about nearby wireless access points and your computer’s IP address. Then Firefox sends this information to the default geolocation service provider, Google Location Services, to get an estimate of your location. That location estimate is then shared with the requesting website.

HTML5 selectors and jQuery

Sun, 12 Dec 2010 13:09:00 +0100

In my first post on the HTML5 javascript Selector API I wondered how the new methods querySelector() and querySelectorAll() would influence jQuery.

At the time, I couldn’t find any information on the subject, but yesterday I found out that jQuery has been taking advantage of these new methods since version 1.4.3.

From the release notes..

The performance of nearly all the major traversal methods has been drastically improved. .closest(), .filter() (and as a result, .is()), and .find() have all been greatly improved.

These improvements were largely the result of making greater use of the browsers querySelectorAll and matchesSelector methods (should they exist). The jQuery project petitioned the browsers to add the new matchesSelector method (writing up a test suite, talking with vendors, and filing bugs) and the whole community gets to reap the excellent performance benefits now.

The above performance results specifically look at three very common cases in jQuery code: Using .closest() on a single DOM node, using .filter() (or .is()) on a single DOM node, and using .find() rooted on a DOM element (e.g. $(“#test”).find(“something”)).

Note that the the browsers shown are those that actually support querySelectorAll or matchesSelector – existing browsers that don’t support those methods continue to have the same performance characteristics.

Looks like it’s not a bad idea to keep an eye on the release notes..

HTML5: Drawing images to the canvas gotcha

Sun, 05 Dec 2010 19:55:00 +0100

While I was playing with the Canvas API I came across a weird issue: I was trying to draw an image to the canvas, but the image failed to render very often.

Have a look at the source. Do you spot the problem?

<!DOCTYPE html>
<html>
    <head>
        <title>HTML5: Canvas</title>
        <script type="text/javascript">        
            window.addEventListener("load", draw, true);
            
            function draw(){                            
                var canvas = document.getElementById('canvas');
                var context = canvas.getContext('2d');    
 
                var img = new Image();
                img.src = "logo.png";                

                context.drawImage(img, 0, 0);                
            }            
        </script>        
    </head>

    <body>
        <canvas id="canvas" height="500" width="500">
            Looks like canvas isn't supported in your browser! 
        </canvas>
    </body>
</html>

It wasn’t until I opened Firebug and saw an unhandled exception in the console that I discovered what was going on.

uncaught exception: \[Exception... "Component returned failure code:
0x80040111 (NS\_ERROR\_NOT\_AVAILABLE)
\[nsIDOMCanvasRenderingContext2D.drawImage\]" nsresult: "0x80040111
(NS\_ERROR\_NOT\_AVAILABLE)" location: "JS frame :: file:///....html
:: draw :: line 15" data: no\]

Browsers load images asynchronously while scripts are already being interpreted and executed. If the image isn’t fully loaded the canvas fails to render it. Turns out the weird issue, is pretty logical.

Luckily this isn’t hard to resolve. We just have to wait to start drawing until we receive a callback from the image, notifying loading has completed.

window.addEventListener("load", draw, true);

function draw(){                                    
    var img = new Image();
    img.src = "logo.png";                

    img.onload = function(){
        var canvas = document.getElementById('canvas');
        var context = canvas.getContext('2d');    

        context.drawImage(img, 0, 0);        
    };            
}

Et voila!

HTML5: More on selectors

Sun, 05 Dec 2010 13:15:00 +0100

Last weekend I blogged on new addittions to the javascript Selector API: querySelector() and querySelectorAll(). These two new methods enable you to find elements by matching against a group of selectors. I only scratched the surface in the previous post, that’s why you can find a few more examples in this post. These examples should demonstrate the power and ease of use of the new Selector API features. It’s impossible to show you all of the selectors usages in just one post, that’s why I strongly encourage you to have a look at the W3C Selectors specifications.

I experimented on the asp.net homepage using the IE9 developer tools.

Attribute selectors

Select all elements which have an attribute named title.

document.querySelectorAll('[title]');
[object] {
    length : 5,
    0 : http://www.asp.net/,
    1 : [object HTMLImageElement],
    2 : http://www.asp.net/get-started,
    3 : http://www.asp.net/downloads,
    4 : http://www.asp.net/rss/spotlight
}

Select all elements where the title attribute has the value set to Rss.

document.querySelectorAll('[title=Rss]');
[object] {
    length : 1,
    0 : http://www.asp.net/rss/spotlight
}

Select all elements where the href attribute is set to http://umbraco.org/ and where the target attribute is set to _blank.

document.querySelectorAll('[href="http://umbraco.org/"][target="_blank"]');
[object] {
    length : 1,
    0 : http://umbraco.org/
}

Select all elements where the href attribute contains umbraco.

document.querySelectorAll('[href*="umbraco"]');
[object] {
    length : 3,
    0 : [object HTMLLinkElement],
    1 : [object HTMLLinkElement],
    2 : http://umbraco.org/
}

Class selectors

Find the first element where the class is set to .search_box.

document.querySelector('.search_box')

[object HTMLInputElement] {
    jQuery1291484082884 : 1,
    align : "",
    border : "",
    hspace : 0,
    vspace : 0,
    accept : "",
    alt : "",
    checked : false,
    defaultChecked : false,
    defaultValue : "Search"
    ...
}

Id selectors

Find the first element with the id #WLSearchBoxInput.

document.querySelector('#WLSearchBoxInput')

[object HTMLInputElement] {
    jQuery1291484082884 : 1,
    align : "",
    border : "",
    hspace : 0,
    vspace : 0,
    accept : "",
    alt : "",
    checked : false,
    defaultChecked : false,
    defaultValue : "Search"
    ...
}

Pseudo classes

Select the first hyperlink that is being hovered over.

document.querySelector('a:hover')

http://umbraco.org/ {
    charset : "",
    coords : "",
    hash : "",
    host : "umbraco.org:80",
    hostname : "umbraco.org",
    href : "http://umbraco.org/",
    hreflang : "",
    name : "",
    pathname : "",
    port : "80"
    ...
}

Select the third child element from the first element with the class .welcom_nav.

document.querySelector('.welcome_nav:nth-child(3)')
    
    type : "",
    compact : false,
    currentStyle : [object MSCurrentStyleCSSProperties],
    runtimeStyle : [object MSStyleCSSProperties],
    accessKey : "",
    className : "welcome_nav",
    contentEditable : "inherit",
    dir : "",
    disabled : false,
    id : ""
    ...
}

As you can see searching for elements with native javascript has become a lot easier. I tried to show some examples which can be applied to real life scenarios. Examples based on more complex scenarios can be found in the W3C Selectors specifications.

HTML5: New in the javascript Selector API

Mon, 29 Nov 2010 19:30:00 +0100

Because I finally got the MCTS 70-536 certification out of the way, I can start experimenting with some fun stuff again. One of the things on the top of my list is HTML5. I started reading the book Pro HTML5 Programming, so expect more posts on HTML5 in the near future.

In this post I will show you two new methods in the javascript Selector API which are extremely useful to find elements.

Specificiations

Methods we can now use to find elements are getElementById(), getElementsByName() and getElementsByTagName(). In HTML5 there are two new methods in the Selector API: querySelector() and querySelectorAll(). These new methods find elements by matching against a group of selectors.

Their description as in the W3C specifications is:

The querySelector() method on the NodeSelector interface must, when invoked, return the first matching Element node within the node’s subtrees. If there is no such node, the method must return null.

The querySelectorAll() method on the NodeSelector interface must, when invoked, return a NodeList containing all of the matching Element nodes within the node’s subtrees, in document order. If there are no such nodes, the method must return an empty NodeList.

Throwing out some feelers

To play with these new methods I made a simple page which displays an unordered list of methods in the Selector API. Existing methods have the class exists and new methods have the class new.

<!DOCTYPE html>
<html>
    <head>
        <title>HTML5: Selector API</title>        
    </head>
    <body>
        <div id="content">
            <ul id="methodslist">
                <li class="exists">getElementById</li>
                <li class="exists">getElementsByName</li>
                <li class="exists">getElementsByTagName</li>
                <li class="new">[NEW] querySelector</li>
                <li class="new">[NEW] querySelectorAll</li>   
            </ul>            
        </div>
    </body>
</html>

To select the first new method we can use the querySelector() method with the li.new Selector. The first matching element is returned.

function selectFirstNewMethod(){
    var firstNewMethod = document.querySelector('li.new');
    firstNewMethod.style.color = 'Red';
}

To select all new methods we can use the querySelectorAll() method with the li.new Selector. All matching elements are returned.

function selectAllNewMethods(){
    var allNewMethods = document.querySelectorAll('li.new');
    for (var i = 0; i < allNewMethods.length; i++){
        allNewMethods[i].style.color = 'Blue';
    }
}

To select all methods (existing and new) we can use the querySelectorAll() method with the li.exists, li.new Selector.

function selectExistingAndNewMethods(){
    var existingAndNewMethods = document.querySelectorAll('li.exists, li.new');
    for (var i = 0; i < existingAndNewMethods.length; i++){
        existingAndNewMethods[i].style.color = 'Yellow';
    }
}

These examples just demonstrate the tip of the iceberg. Check out the Selectors specifications to see how they can be used to greatly improve the way of finding elements in the DOM.

The HTML5 Selector API and jQuery?

I am far from a jQuery or javascript expert, so I wonder: how will these methods and selectors influence jQuery? Will they be encapsulated into jQuery, because I can imagine that these natively implemented methods will be a few times faster than jQuery Selectors?

Edit: querySelectorAll is being used by jQuery since version 1.4.3. (Source).

Trip Report Denmark (Part 2)

Thu, 14 Oct 2010 15:30:00 +0200

If you missed the first part of this trip report, you can find it here.

Funen

After three days of Copenhagen we stayed at Funen for two days. This is the smallest of the three big islands of Denmark. Our motel was in a small town called Faaborg.

During these two days in Funen the most impressive attraction we went to see was the Egeskov Castle. This castle is Europe’s best preserved Renaissance water castle. You can also walk around in the big park surrounding the castle, or visit the five museums on the Egeskov domain. The vintage automobile and motorcycle museums are awesome!

Jutland

The last three days of our trip we stayed at the biggest island (peninsula) of Denmark, Jutland.

Here we visited the oldest city of Denmark, went hiking and strolled the cleanest beaches in Europe. A perfect place to relax and find new energy.

Conclusion

This trip was amazing. From Copenhagen to Jutland, this country has surprised me hundreds of times. We got the best out of Denmark’s culture and history in Copenhagen, and enjoyed the beauty of Denmark’s nature in Funen and Jutland. Summarized Denmark has it all!

One small drawback is that Denmark isn’t cheap. Everything is twice as expensive compared to Belgium. Although with some good looking around you can save a lot.

Trip Report Denmark (Part 1)

Wed, 13 Oct 2010 20:30:00 +0200

Last month me and my girlfriend went on a holiday to Denmark for a week. We had a great time, visited a lot of places and took hundreds of pictures. In this post you can find a trip report containing the best pictures and tips on places to see.

Copenhagen

The first three days of this holiday we visited Copenhagen. These three days were the most interesting, but also the most intensive days of the holiday. There are so many places to visit in Copenhagen!

If you are in Copenhagen to do some sightseeing I advice you to get yourself a Copenhagen card. This card is valid for 72 hours and includes entrance fees for almost all attractions and public transport. And it only costs 63€.

Ripley’s believe it or not museum

Next to the TV show hosted by Superman, there are Ripley’s believe it or not museums spread over the world. Although this museum isn’t a must-see, the bizarre items featured are definitely entertaining.

Dansk Design Center

Although this is one of the museums my girlfriend had to drag me to, it turned out to be not that bad. The UX people out there will find this very interesting.

Christianborg

This palace used to be the home of the Danish monarchy. Today Christianborg is the seat of the parliament, the Prime Minister’s Office and the Supreme Court.

The palace is probably most known for it’s turbulent past. It had to be rebuilt two times due to serious fires.

A small part of the palace is open to the public. The inside of the palace is even more impressive than the outside!

Tivoli

Tivoli is the second oldest amusement park in the world. It was built over 150 years ago. Although you can’t compare it to Disney World, this is a must-see. If you go there, I advice you to go by night.

Zoological Museum

In this museum there is a permanent exhibition called From Pole To Pole. There is also a semi-permanent Darwin exhibition.

Copenhagen Zoo

The Copenhagen Zoo is definitely worth a visit. The main reason we wanted to go there is because we wanted to see a Tasmanian Devil. Copenhagen Zoo is the only zoo outside of Australia that has Tasmanian Devils in captivity. It turned out it didn’t totally live up to our expectation.

The Tasmanian Devil guarding its territory borders, is not that scary, right?

Christiania

Christiania is a special part of Copenhagen. This part of Copenhagen is a self-proclaimed autonomous neighbourhood. In the seventies a group of hippies founded this freetown. Although the government tried to turn the situation around at first, in the late seventies Christiania was tolerated and considered a social experiment.

This probably was the weirdest experience of this trip. Soft drugs are tolerated and publicly sold in Christiania, giving this freetown a very special atmosphere. People from all around Copenhagen get together here to smoke, socialize and chill.

Too bad taking pictures at Pusher Street isn’t allowed.

Rosenborg castle

This castle used to be a country summerhouse for the monarchy. Today it is a museum exhibiting the Royal Collection and Crown Jewels.

Nyhavn

Nyhaven, meaning new harbor, is very popular among tourists. It is best known for its colorful buildings and cozy restaurants.

Architecture

For people with an architectural fetish there are tons of buildings worth visiting. The ones that impressed me the most are the Black Diamond and the new Opera House.

Things good to know about SQL State Server

Thu, 23 Sep 2010 19:31:00 +0200

While installing a SQL State Server last week, I came across a few things worth sharing about the installation and use of SQL State Server.

Finding a good tutorial

There are lots of tutorials out there on how to install SQL State Server but most of them are not great. To do a basic installation you only need this MSDN documentation on how to run the Aspnet_regsql.exe tool and edit your web.config.

All the objects in Session need to be serializable

If you try to store an object in Session which isn’t marked as serializible an HttpException will get thrown with following message.

Unable to serialize the session state. In 'StateServer' and
'SQLServer' mode, ASP.NET will serialize the session state objects,
and as a result non-serializable objects or MarshalByRef objects are
not permitted. The same restriction applies if similar serialization
is done by the custom session state store in 'Custom' mode.

Marking your classes with the Serializible attribute shouldn’t be a problem. Be careful when storing WebControls in Session though, most of these aren’t marked as serializible!

When you are using UpdatePanels this exception doesn’t fully propagate to the front-end. The javascript error shown will have following message.

Error: Sys.WebForms.PageRequestManagerServerErrorException: An unknown
error occurred while processing the request on the server. The status
code returned from the server was: 500

Use the same machinekey on multiple servers

To make the SQL State Server work across servers hosting the same application, you need to make sure the machineKey element in the machine.config is identical.

Redundancy options are limited

Looking at the MSDN documentation, it looks like only SQL Server clustering is supported. You can’t specify a Failover Partner, so SQL Server mirroring isn’t supported.

In SQLServer mode, you can configure several computers running SQL Server to work as a failover cluster, which is two or more identical computers running SQL Server that store data for a single database. If one computer running SQL Server fails, another server in the cluster can take over and serve requests without session-data loss.

Tripping in Holland. Honey, who shrunk Holland?

Wed, 21 Jul 2010 17:30:00 +0200

People who follow me on Twitter might have seen my tweets about going on a weekend to Holland with some friends. Next to excessive eating and drinking, we took the time to do some sightseeing in The Hague.

The sight which gave us the most interesting pictures is Madurodam. Madurodam is a miniature city featuring the most famous Dutch buildings and landmarks. Hope you enjoy these pictures!

Switching with non-constant cases in C#

Wed, 14 Jul 2010 22:25:00 +0200

Last week I came across a scenario where I wanted to switch over non-constants (aka variables), but while I was compiling I got Compiler Error CS0150 (A constant value is expected). This is one of those things I always forget. You can’t use variables in your case statements because the C# compiler doesn’t allow you to. It’s very logical though, the compiler forces you to use constants because otherwise there is no way of knowing there are equal case statements.

The scenario

Let’s say I have an ASP.NET page where the user can input and submit a value. On the server-side I want to match this value with a value from a local resource file. Depending on the match I want to execute other code.

Remember I can’t use a switch because the values in the local resource file are variable.

Option one: Using conditional statements

As shown in the code snippet below you can use else-if statements to search for a match.

if (input == GetLocalResourceObject("CaseOne").ToString()) {
      this.ltResult.Text = "Case one matched.";
} else if (input == GetLocalResourceObject("CaseTwo").ToString()) {
      this.ltResult.Text = "Case two matched.";
} else if (input == GetLocalResourceObject("CaseThree").ToString()) {
      this.ltResult.Text = "Case three matched.";
} else {
      this.ltResult.Text = "No matching case found.";
}

This option has at least two disadvantages:

It allows equal conditions (cases) which might have horrible consequences.
It’s ugly.

Option two: Using a dictionary

You can also use a generic dictionary where the pairs have a string as the key and a delegate as the value. Because the code I’m executing when a match is found is so compact I’m using the simplest delegate of them all: an Action delegate. An Action delegate takes no parameters and does not return a value.

Dictionary<string, Action> mappings = new Dictionary<string, Action>() {
      { GetLocalResourceObject("CaseOne").ToString(), () => this.ltResult.Text = "Case one matched."},
      { GetLocalResourceObject("CaseTwo").ToString(), () => this.ltResult.Text = "Case two matched."},
      { GetLocalResourceObject("CaseThree").ToString(), () => this.ltResult.Text = "Case three matched."}
};

if (mappings.ContainsKey(input)) {
      mappings[input]();
} else {
      this.ltResult.Text = "No matching case found.";
}

I think this option is a lot better than the previous one:

This code is elegant, no spaghetti here.
And better yet, having equal conditions (cases) is impossible. You can’t add duplicate keys to a dictionary. If you do, an ArgumentException gets thrown at runtime.

Handling the AggregateException

Sun, 23 May 2010 13:10:00 +0200

Last week I showed you how you can use the AggregateException to apply consistent exception handling in batch operations. You can find that post here.

Bart De Smet read that post and pointed out that I should check out the Handle method of the AggregateException.

The Handle method

As found in the MSDN documentation.

Description

Invokes a handler on each Exception contained by this AggregateException.

Parameters

System.Func<Exception, Boolean> predicate  

The predicate to execute for each exception. The predicate accepts as an argument the Exception to be processed and returns a Boolean to indicate whether the exception was handled.

Remarks

Each invocation of the predicate returns true or false to indicate whether the Exception was handled. After all invocations, if any exceptions went unhandled, all unhandled exceptions will be put into a new AggregateException which will be thrown. Otherwise, the Handle method simply returns. If any invocations of the predicate throws an exception, it will halt the processing of any more exceptions and immediately propagate the thrown exception as-is.

In practice

I refactored the example in my previous post to make use of the Handle method.

static void Main(string[] args)
{
      try
      {
            ExecuteBatch();
      }
      catch (AggregateException aggEx)
      {
            aggEx.Handle(HandleBatchExceptions);
      }

      Console.ReadLine();
}

I’m passing a Func<T, TResult> delegate to the Handle method. In this delegate I decide whether I’m handling the exception or not. If I handle the exception, I return true, else I return false.

private static bool HandleBatchExceptions(Exception exceptionToHandle)
{
      if (exceptionToHandle is ArgumentNullException)
      {
            // I'm handling the ArgumentNullException.
            Console.WriteLine("Handling the ArgumentNullException.");
            // I handled this Exception
            return true;
      }
      else
      {   
            // I'm only handling ArgumentNullExceptions.
            Console.WriteLine(string.Formaxt("I'm not handling the {0}.", exceptionToHandle.GetType()));
            //I didn't handle this Exception, return false.
            return false;
      }          
}

When we run this example a new AggregateException is thrown with the exceptions I didn’t handle.

Conclusion

Make use of the Handle method to run over each InnerException and decide which exception you want to handle or not. The exceptions you didn’t handle are automatically wrapped in a new AggregateException which gets rethrown.

Exception handling in batch operations with the AggregateException

Sat, 15 May 2010 18:34:00 +0200

Doing batch operations and elegantly handling exceptions is a problem which every developer has faced before. In .NET 3.5 or older there is no out-of-the-box solution to handle exceptions in these types of scenarios, without being inconsistent to the normal flow of exception handling. .NET 4 introduces the AggregateException; an exception representing multiple exceptions. The AggregateException was introduced in the first place to be used with the parallel framework, but it can be used in other scenarios as well, such as batch operations.

Take a look at the following example..

I have a single action which I want to peform in batch. This single action might throw some exceptions.

private static void ExecuteSingleAction(int i)
{
      if (i == 5)            
            throw new ArgumentNullException("You forgot an argument.");       
      if (i == 18)
            throw new ArgumentException("This argument doesn't make sense.");
}

Most of the exceptions the single action might throw shouldn’t break my whole batch operation. While I’m executing the single actions I catch the exceptions which shouldn’t break the batch operation and hold them in a list of exceptions. The AggregateException has a public constructor taking an IEnumerable of exceptions. If I catch some exceptions while executing the single action I throw an AggregateException passing in that list of exceptions to its constructor.

private static void ExecuteBatch()
{
      List<Exception> exceptions = new List<Exception>();
      
      for (int i = 0; i < 100; i++)
      {
            try
            {
                  ExecuteSingleAction(i);
            }
            catch (ArgumentNullException nullRefEx)
            {
                  exceptions.Add(nullRefEx);
            }
            catch (ArgumentException argumentEx)
            {
                  exceptions.Add(argumentEx);
            }             
      }

      if (exceptions.Count > 0)           
            throw new AggregateException(exceptions);            
}

In the front-end I can catch the AggregateException, run over its InnerExceptions and act based on the type of exception.

static void Main(string[] args)
{
      try
      {
            ExecuteBatch();
      }
      catch (AggregateException aggEx)
      {
            foreach(Exception ex in aggEx.InnerExceptions)
            {                  
                  Console.WriteLine(ex.Message);

                  if (ex is ArgumentNullException)
                  {
                        //Do something
                  }

                  if (ex is ArgumentException)
                  {
                        //Do something else
                  }
            }
      }

      Console.ReadLine();
}

The result looks like this.

Can you imagine other scenarios where the AggregateException might be able to add some value?

Related post: Handling the AggregateException

Trip Report Czech Republic

Mon, 12 Apr 2010 20:16:00 +0200

Last week my girlfriend and I went on a short holiday to the city of Plzeň in the Czech Republic. In this post you can find a small trip report.

Roadtrippin'

Minutes after leaving home we discovered an unpleasant surprise; my Tom Tom has no map of the Czech Republic. Good thing that people used to print this stuff on paper. After consulting a paper map of Europe we did have an idea which direction to follow.

The route was easy enough though; straight ahead for 800km. Driving through Germany was a blast. For those not from Europe; a big part of the German autobahn has no speedlimit.

Prague

The highlight of our holiday was Prague. A lot of the historic architecture is still intact, making Prague an impressive city and deservingly one of the most-visited cities in Europe.

Some of the sights we visited were..

The Charles Bridge. This bridge is one of the most popular sights in Prague and has a very rich history. It’s over 600 (!) years old.

The Petřínská rozhledna on top of the Petřín hill is a miniature version of the Eiffel Tower. On top of the Petřínská rozhledna you have a great view of the Prague skyline.

The Prague Castle is one of the biggest castles in the world. Today it still houses the Czech government.

The Saint Vitus’s Cathedral next to the Prague Castle is the biggest and most important church of the Czech Republic.

Plzeň

Plzeň is known worldwide for Pilsener beer and its Skoda factories. Although industry made Plzeň one of the biggest cities of the Chzech Republic, the number of tourist attractions is rather limited..

We had some good laughs at the Dinopark.

We also visited the famous Pilsner Urquell museum and got a bonus show! That evening the soccer match Viktoria Plzen vs Sparta Prague took place. While we were standing in front of the Pilsner Urquell museum 80 fully equipped cops stormed a restaurant on the other side of the street to arrest some hooligans. Sweet action!

Final thoughts

Although the Czech Republic does not have a good reputation, I think all the problems this country used to have belong to the past. The people were nice, the culture was rich and everything was pretty cheap compared to most European countries. Definitely worth visiting!

Cannot Start Service from the command line or debugger. A Windows Service must first be installed (using installutil.exe)..

Sat, 13 Mar 2010 16:36:00 +0100

When you create a new Windows Service project and try to debug it, Visual Studio will show you a Windows Service Start Failure with the message Cannot Start Service from the command line or debugger. A Windows Service must first be installed (using installutil.exe) and then started with the Server Explorer, Windows Services Administrative tool or the NET START command..

The trick my team and I use to workaround this problem, makes use of the service Debug flag. If the Debug flag is on, we just start the service by using our own public Start method. When the OnStart event is fired in the service itself, we call the same public Start method.

This goes in your service.

protected override void OnStart(string[] args)
{
    Start();
}

public void Start()
{
    //Start!
}

And this goes in Program.cs.

namespace UdpListener
{
    static class Program
    {
        static void Main()
        {
            #if (!DEBUG)
                ServiceBase[] ServicesToRun;
                ServicesToRun = new ServiceBase[] 
                { 
                    new UdpListener() 
                };

                ServiceBase.Run(ServicesToRun);
            #else
                UdpListener listener = new UdpListener();

                listener.Start();
            #endif           
        }
    }
}

You can set the Debug flag of your service in your service properties.

The only problem with this solution is that you can’t debug your OnStop event but this hasn’t been an issue for us so far.

Book review: Object-Oriented JavaScript

Mon, 22 Feb 2010 19:47:00 +0100

Although I’ve made my feet wet with various JavaScript frameworks, I am not a JavaScript hero at all. This might be one of the disadvantages of being a WebForms developer and having great toolkits at hand like the AJAX Control Toolkit and Telerik RADControls.

Because you really can’t call yourself a webdeveloper without having some understanding of JavaScript I decided to order Object-Oriented Javascript by Stoyan Stefanov.

This book really handles the language itself. The first chapters handle the basics: Types, loops, functions and objects. Although these concepts and the syntax resemble C#, JavaScript is a total different beast. After covering the basics, Stoyan goes deeper into OO JavaScript with Prototype and Inheritance. Then he shows the Browser Environment and he finishes the book with Coding and Design patterns.

All these concepts are accompanied by a ton of easy-to-read examples. After every chapter he also gives you some homework, because you learn coding by coding. After finishing this book you can also use the excellent Appendix for your reference.

I thought this was a great and very complete book. Definitely one of the best software books I’ve read in a while.

Get the book here (Amazon).

Webforms lessons learned the hard way (Part 2)

Mon, 15 Feb 2010 17:43:00 +0100

If you missed part 1, you can find it here.

Use the built-in goodies

ASP.NET Webforms has a lot of good stuff built into it. Do your homework before you start building the next big Webforms thing! A perfect example of this is ASP.NET Membership. ASP.NET provides an out-of-box membership solution. I’ve seen people who were to lazy to do some research or thought they could do better and ended up with a solution which put its doors wide open to people with bad intentions.

When a feature doesn’t exactly match your needs, try extending it. The ASP.NET team are artists as it comes to writing solid extensible frameworks. I’d like to point to ASP.NET Membership again. You can write your own provider which integrates with ASP.NET seamlessly.

Move as much as possible away from your page

Separating the domain and persistence away from your presentation logic positively affects the quality of your page and code-behind in so many ways.

One of the big drawbacks of Webforms is that testing your pages isn’t easy. That’s why it’s important to keep the quantity of code in the code-behind as low as possible. Code outside your presentation logic can easily be covered using unittests.

Dividing your web application in multiple layers also makes the code in the page and page-behind easier to grasp (encapsulation etc..).

Let others do it for you

Because most of the ASP.NET Server controls are easy to extend, there a lot of third party control vendors out there.

I’ve only been using the Telerik RadControls for a few months now, and using this control suite, made my life so much easier.1000$ might look like a big investment, but you can’t believe how many time it has saved me. Especially the RadGrid control is a great time-saver. Binding a list of objects to a RadGrid, and finding out sorting, paging,.. just works is awesome.

What are some of your Webforms lessons learned the hard way?

Webforms lessons learned the hard way (Part 1)

Sun, 14 Feb 2010 16:22:00 +0100

I’ve been spending a lot of my days in Webforms the last two years. In this post I want to share some best practices I’ve learned the hard way over these years. A lot of MVC developers might think this post comes a bit late (who still cares about Webforms?!), I do think (in the real world) a lot of the ASP.NET developers are still using Webforms. This post is directly targeting them.

Keep your pages light

There are a few things where you should pay attention to.

Most of the times the Viewstate is the number one performance killer. Although most developers know storing a lot of ViewState on the page has issues, and that there are numerous alternatives. Most developers don’t take the time to think about the Viewstate. I don’t blame them. Persisting the ViewState to another medium is not as easy as it should be. The Viewstate is enabled by default, disabling the Viewstate on each control is a hassle.. In ASP.NET 4 the ViewStateMode will make it much easier though. Once the users start complaining about performance, you are too late. I really encourage you to start thinking about the Viewstate as early as possible. Only use it when you need to persist changes across Postbacks. You will be surprised how little you really needs to persist.

A Masterpage is great for applying a consistent style to your pages. But be careful with what you put in the head tag of your Masterpage. Often you only need a certain stylesheet or javascript library in a few pages. Avoid including references to these stylesheets and javascript libraries in your Masterpage. Better is to put a ContentPlaceHolder in the head tag of your Masterpage and use that for your conditional resources.

Updatepanels are great controls, but you should apply them wisely. Don’t blindly wrap your whole form in an Updatepanel. Think about which part of your page really needs to be refreshed. The less HTML and ViewState (!) coming back from the server the better.

Use javascript. I think to many Webforms developers are uncomfortable leaving their serverside habitat. Avoid a postback whenever you can.

Stay away from the designer

Creating pages using the designer doesn’t work on so many levels.

The Design view doesn’t always give a correct representation of how the page will look like when it’s rendered by the browser.

Using drag-and-drop, databinding wizards.. does not end well. I have a big problem with defining datasources in the ASPX markup. In my opinion the ASPX markup should be as clean as possible. It should serve as a view, and as a view only. Another downside of defining your datasources in your ASPX markup is that your application tends to be more fragile.

Let me prove this by this example..

In this example I’m binding a List of FireStationEvents to a GridView. Notice the TypeName and SelectMethods are strings.

<form id="form1" runat="server">
      <div>           
            <asp:GridView ID="gvEvents" runat="server" DataSourceID="odsEvents">
            </asp:GridView>        
      </div>
      <asp:ObjectDataSource ID="odsEvents" runat="server" 
            SelectMethod="Load" 
            TypeName="WFDemo.BusinessLogic.FireStationEvents">
      </asp:ObjectDataSource>
</form>

A few weeks later I decide to change the methodname Load to LoadAll. I rebuild my solution and all is good. I publish my site to production, and a few hours later I get a critical bug report assigned to me. After looking into it, I discover I forgot to change the SelectMethod property of the DataSource. Loading the page throws an unhandled Exception at runtime: ObjectDataSource 'odsEvents' could not find a non-generic method 'Load' that has no parameters. One small refactoring made the application fail big time and I didn’t even have a clue.

Stay posted for Part 2 tomorrow! What are some of your Webforms lessons learned the hard way?

Log4Net: log.Debug(String.Format()) versus log.DebugFormat()

Wed, 20 Jan 2010 20:30:00 +0100

Log4net is one of the most popular opensource logging frameworks available in the .NET world. I’ve been using this framework for over a year now, and today I discovered something new.

I often use string.Format() to format my log messages. Earlier this morning I made a typo formatting my message and an Exception was thrown in the beginning of my method which caused the application flow to break. You can avoid this by using the DebugFormat() method. If you mistype here, no exception will be thrown, but a WARN message will be logged.

Example using `String.Format`

log.Debug(string.Format("{0}{1}", "ABC_1"));

As expected this throws a System.FormatException: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.

Example using `log.DebugFormat`

log.DebugFormat("{0}{1}", "ABC_1");

Using DebugFormat() the logger logs a WARN message, but it doesn’t break the application flow.

WARN StringFormat: Exception while rendering format [{0} {1}.]

So to avoid logging breaking your application, you should use the DebugFormat(), WarningFormat(), InfoFormat(), ErrorFormat() methods instead of String.Format() to format your log message.

Looking back, moving forward (New years post)

Mon, 21 Dec 2009 18:57:00 +0100

In this post I’m going to reflect on the year that’s almost finished and the year to come.

I came up with a few topics that are worth discussing in this post.

My career

I don’t think I ever mentioned where I work. The first of January I will be working 18 months for Ferranti. The only company I’ve worked for since I graduated.

The first project I’ve ever worked on was a rather big one. Firestation Antwerp. Last year we heard that sales scored another firestation for us (Firestation Ghent). So I’ve been working on these two projects since I started.

Do these projects get boring? Well, not boring, exhausting maybe. These projects are very complex (Dispatching, telecommunication, web applications, GIS, ..) and the requirements are very high. Although these projects are exhausting from time to time, I don’t think there are other projects where I would’ve learned as much in such a small period of time. As a developer I have touched and worked on lots of stuff: ASP.NET, WindowsServices, WCF, WebServices, GIS, API’s, Telecommunication.. You name it.

Blog

I’ve been blogging since October 2008. When I just started out the quality of the content was rather poor. Although I’m not totally satisfied with the content I produced this year, I hope the quality has improved noticeable. I saw the traffic increase significantly this year, so I hope that’s a good sign.

There are a few posts that scored pretty good out there. You can find them here.

By blogging I’ve made lots of new friends. I enjoy reading through all of your Tweets. Bart De Smet in particular has been a very good mentor.

Community

I managed to go to two events this year:

TechDays Belgium
Scott Guthrie in Belgium

I hope I will get the opportunity to go to some big event next year. I also plan on going to more VISUG sessions.

I’ve been neglecting the ASP.NET forums lately. In general I think the quality of the questions is rather poor and there are plenty of good answerers. I’ve been a long time StackOverflow lurker. Next year I’ll try to contribute to that community.

Travelling

I managed to travel four times this year:

France (Albi)
Spain (Madrid)
Belgium (Ardennes)
France (Normandie)

In 2010 I’ll try to go abroad at least three times.

Wrap up

In general, my year was pretty fun. I hope 2010 will be at least as good. Excited about hearing how you did this year and what you hope to achieve in 2010!

Response.RedirectPermanent in .NET 3.5 and older

Sat, 05 Dec 2009 17:50:00 +0100

One of the new features in ASP.NET 4.0 is permanently redirecting to a page using Response.RedirectPermanent.

It is common practice in Web applications to move pages and other content around over time, which can lead to an accumulation of stale links in search engines. In ASP.NET, developers have traditionally handled requests to old URLs by using by using the Response.Redirect method to forward a request to the new URL. However, the Redirect method issues an HTTP 302 Found (temporary redirect) response, which results in an extra HTTP round trip when users attempt to access the old URLs.

Source

You can achieve this functionality in ASP.NET 3.5 and older by writing a 301 Moved Permanently Status and a ``Location Headerto theResponse` stream. This can be found in the HTTP specifications.

Here is an example.

Response.Clear();
Response.Status = "301 Moved Permanently";
Response.AddHeader("Location", "PageOne.aspx");
Response.End();

You can verify the result by using Fiddler.

Book review: Professional Refactoring in C# & ASP.NET

Tue, 17 Nov 2009 19:37:00 +0100

Three months ago I was looking up to refactoring a legacy ASP.NET Webforms application. Although I was familiar with refactoring, I ordered “Professional Refactoring in C# & ASP.NET” by Daniejel Arsenovski to get deeper into refactoring.

The title

Although the title states “Professional”, I think this book should find its main audience in coding novices. Another problem with the title is the word ASP.NET in it. The book has one chapter covering ASP.NET, with only 35 poor-quality pages in it.

Now the title has some issues, but how was the content?

The content

Danijel identifies codes mells and shows refactoring techniques to solve them. Too bad some of these examples are spreaded over multiple pages, which makes them a tad sloppy and hard to read. Danijel stresses how important Object Oriented design is. He proves this by transforming procedural code into Object Oriented code, greatly improving the readability. This book should help developers who are new to or come from a non-OO background to apply the basic Object Oriented concepts. Next to refactoring techniques, he also touches refactoring tools, unit testing and design patterns.

I think the beginning .NET developer might be able to significally start improving his codebase by reading this book. For more experienced .NET developers most of the topics handled in the book should be part of their day-to-day skillset.

Conclusion

All by all I was a bit dissapointed by the book, it was an easy read, had a few interesting tricks, but I expected it would go more into depth.

Something valuable I realised though.. is that refactoring isn’t something you learn by reading a book, but you learn it by obeying the constant hunger for improving your code.

If you feel confident that you dig the basic concepts of writing quality code, I would instead of buying this book, invest in a good book on Object Oriented concepts or design patterns.

My trip to Madrid in ten pictures

Mon, 09 Nov 2009 20:30:00 +0100

Last week I went to Madrid with my family-in-law to visit my girlfriend’s brother and cousin.

Although our stay there was rather short (only four days), we visited some of the most impressive places in Madrid:

- Stadium Real Madrid

The Royal Palace

Prado Museum

Queen Sofia Museum

Buen Retiro Park

Plaza Mayor

…

Stadium Real Madrid

Pay attention to the stitching Jesus in first place.

Royal Palace

Ruïnes of an Egyptian temple

Buen Retiro Park

Older lady feeding the birds. I wonder what her trick is.. Four of us tried letting them eat out of our hands, but we all failed.

Why code reviews should be a team thing

Sun, 23 Aug 2009 19:22:00 +0200

Often code reviews are done by one person, the technical lead of your team.

In my opinion it’s better to make each developer of your team do code reviews.

My arguments

Having all developers do code reviews leads to discussions about what coding standards should be used..
Leading to well-documented standards.
Each developer is forced to understand the workflow of an application which isn’t his, which expands his domain knowledge.
Each developer learns from others and is given an opportunity to teach and discuss why someone chose a certain path.. There is always something new to learn, even for seasoned developers. You might just pick up that latest trick doing whizbang Silverlight 3 stuff from that freshman that just started working this year.
Multiple perspectives on a problem can only benefit the solution.

Stored Procedures: Make them or break them

Sun, 15 Feb 2009 18:58:00 +0100

I’ve been thinking about the use of stored procedures lately.

Some questions that popped up in my head: What are the advantages/disadvantages? When and how should they be used? Will they extinct with LINQ to SQL becoming more and more popular? Why should you store your queries in your database, instead of in your code..

I asked the opinion of a few ASP.NET/.NET professionals. Hope you find this interesting!

Kris Van Der Mast (Blog)

Since I’m a consultant I have to go with the specifications of my clients. Most still use stored procedures though I see a shift occurring. My current client allows us to use Linq To Sql for select statements (generated by L2S) but for update, delete and insert statements they require us to make sprocs since they put security on it on a database level. My former client allowed us to use NHibernate (winforms project) but for 3-4 special occasions we used sprocs.

SyntaxC4 (Profile)

I do use Stored Procedures on a regular basis. I develop mostly in DotNetNuke, which has an extensive use of stored procs. I have read blogs, and forum posts that suggest that stored procedures shouldn’t be used as they can be a performance bottleneck, but I believe they are an excellent way of promoting SQL Abstraction.

I do however try to preserve the number of stored procedures I write. I normally create one stored procedure to handle both insert and updates to the a particular table, one for deleting, and one to select (sometimes i do multiple select statements within one stored proc if it makes sense).

To work away from bottlenecks, i try to use subqueries in my stored procedures instead of using a view to query against. unless of course a view is more efficient.

TATWORTH (Profile)

It is not much a case of would I use stored procedures as when would I not use them! Occasionally for searching, dynamic TSQL needs to be generated and for this I would use parameterized TSQL. Otherwise I use stored procedures all the time. I do have stored procedures for generating from the table definition stored procedures plus wrapper code. This allows very rapid development of the data layer.

The use of stored procedures (or the occasional parameterized TSQL) is in my opinion one of a series of essential steps to produce robust applications.

Tom Peeters (a co-worker)

I prefer to (read always) use stored procedures when I’m developing front end applications.

The ones who find creating stored procedures is intolerable overhead, I want to give some advantages..

They are modular, I’d prefer to troubleshoot a stored procedure than an embedded query buried within many lines of GUI code.

They are tunable. Changes can be made to the stored procedures –in terms of join methods, differing tables, etc.– that are transparent to the front-end interface.

Stored procedures abstract or separate server-side functions from the client-side. It is much easier to code a GUI application to call a procedure than to build a query through the GUI code.

Stored procedures are usually written by database developers/administrators. Persons holding these roles are usually more experienced in writing efficient queries and SQL statements. This frees the GUI application developers to utilize their skills on the functional and graphical presentation pieces of the application. If you have your people performing the tasks to which they are best suited, then you will ultimately produce a better overall application.

In short, queries are best handled via stored procedures. While the initial development overhead is greater, you will more than make up for the investment down the line.

Looks like a lot of people are still very fond of stored procedures. The arguments make a lot of sense!

Mon, 01 Jan 0001 00:00:00 +0000

Hello world

Jef Claes

GES scavenging and the hidden cost of link events

Amazon Redshift - Fundamentals

Consumed in 2017

Books

The Zen and Art of Motor Cycle Maintenance

The Soul of a New Machine

Winter is Coming: Why Vladimir Putin and the Enemies of the Free World Must Be Stopped

Shows

Westworld

Stranger Things 2

Podcasts

The Joe Rogan Experience

Conversations with Tyler

Dan Carlin’s History X

Passing the AWS Certified Solutions Architect exam

Exam Blueprint

A Cloud Guru

FAQs and Whitepapers

AWS Open Guide

Whizzlabs

Exam Guru

Test exam

Test day

Fast projections

The stream

First attempt

Less checkpointing

Batching

Batching with transformation

Conclusion

From human decisions, to suggestions to automated decisions

A use case specific view

Making suggestions

Automating decisions

From human decisions, to suggestions, to automated decisions

How to organize a meetup

Finding a speaker

Contacting a speaker

Gathering speaker requirements

Selecting a location sponsor

Contacting a location sponsor

Meetup.com

Speaker gift

Recording

Day of the meetup

Give thanks

Printing money - hacking loyalty points

Everyone has to serve somebody, but software has to serve more than one

Consumed in 2016

Books

Movies

Shows

Podcasts

Commands and events with JustSaying and AWS

My InfoQ interview on DDD, events and legacy

Pieter Hintjens

Using a batch layer for fast(er) aggregations

Notifications from an event log

Functional one-liner for running totals in C#

Consumed in 2015

Visualizing event streams

Slides from my talk "Evil by Design" at Build Stuff

Defining big wins

Bulk SQL projections with F# and type providers

Aspect ratio calculation

Basic casino math

Consumed: Queries and projections (F#)

Consumed: Handling commands (F#)

Validation

Producing events

Dependencies

An eventstore

Handlers

Side effects

Putting it all together

Finding unused code (F#)

Consumed: Parsing command line arguments (F#)

Checked errors in F#

Scaling promotion codes