Part 4: Constant evaluation and improvement: Finding sources for feedback.

2010-10-24 08:13
In recent years demand for shorter feedback cycles especially in software development has increased. Agile development, lean management and even Scrum are all for short feedback cycles: Coming from the dark ages when software projects would last for months or even years before any results could be delivered to customers we are transforming development into a process that integrates the customer in the design and evolution of his own product. Developers have learned that planning ahead for years does not work: It's not only customers changing their mind so fast, it's requirements changing quickly. The only achievement from months-long cycles is getting input on your work later, being able to hide deficiencies longer.

However not only for planning and management does it make sense to install fast feedback loops. A few days ago I finished reading the book "Apprenticeship patterns". A book that gives an overview of various patterns that help improve software development skills.

One major theme was about getting fast feedback constantly. On the (agile) development side, automated tests (unit and integration) and continuous integration systems are technical aids that can help. Pair programming and code review take the idea of fast feedback one step further by having humans give you feedback on what cannot possibly be evaluated automatically.

There is just one minor glitch: With really short feedback loops any mistake you make gets revealed instantly. This is not particularly special to agile development. Another area with these kinds of fast feedback loops are projects developing free software - see also the last paragraph in Bertrand's blog on This is how we work at Apache.

There are developers who have a hard time living with exposing their work to a wider audience quickly. However it has the huge benefit of revealing any errors as early as possible - ideally at a point in time where fixing them is still cheap. Examples for options spotting mistakes early in various environments are listed below:


  • Open source development: Mistakes are revealed during code review of patches (or checkins).
  • Scrum: Speeding up error spotting can be implemented by closely integrating the product owner during each sprint. As soon as a feature is done - according to the developer team - it gets reviewed by the product owner this way reducing risk of features getting rejected during the sprint review.
  • In the team: Get each change set either mailed to all developers allowing for fast review of incoming code.


These are all good ways for feedback, however what about non-coding areas? Are there ways to build fast feedback into tasks that do not involve coding? I'll pick just one example to highlight some ways that facilitate fast feedback in a non-coding environment.

From time to time even hard-core coders have to meet face-to-face to discuss new designs, learn more about customers' requirements. This may involve going to conferences, giving talks, organising internal workshops, public meetups or even conferences.

Usually people doing the organisation are too busy to "watch what happens": They already drown in tasks to do and have no time (and are in no good position) to objectively evaluate the conference's quality.

Still there are ways to build feedback loops even into this kind of setup. Most of them have to do with communication:

  • Ask people to give you feedback in an official feedback form: Don't expect more than a fraction of the actual attendees to answer that form. Still it can be a source for honest feedback when done correctly. Include a few open questions, don't ask people to rate each and every task - they'll never find the time to do that anyway. Read the free-form comments - usually they provide way more insight than any rating question anyway.
  • Talk to people, ask for proposals on what you should do differently next time.
  • Watch what people are telling on the net - however keep in mind that those statements usually are a bit biased showing only the extreme ends of the spectrum.


The same applies to people giving presentations: Talk to people from the audience after your presentation is over. If your talk was video-taped, you are in a really great situation, as now you can judge for yourself what should be improved and where there are glitches in your arguments.

According to my experience people very rarely give explicit feedback - except when being really disappointed or extremely positively surprised. However when asked for precise feedback on specific aspects people are usually more than willing to share their experience, tell you where to improve and what to change. Usually it turns out to be a good idea to actively seek out people for evaluation of your projects to get better at what you do, to encourage peers to tell you what you do wrong or even where you could get slightly better.

Video: Sebastian Schelter on Recommendation w/ Apache Mahout

2010-10-21 13:55
A few weeks ago we had the autumn edition of the Apache Hadoop Get Together in newthinking store in Berlin. I am glad to announce the first video online:

Mahout Sebastian Schelter from Isabel Drost on Vimeo.



Thanks to JTeam for sponsoring video taping, thanks to newthinking for providing the location and thanks to Martin Schmidt from newthinking for producing the video.

Stay tuned for the second video to be published next week.

Apache Mahout at Apache Con NA

2010-10-15 20:39
The upcoming Apache Con NA to take place in Atlanta will feature several tracks relevant to users of Apache Mahout, Lucene and Hadoop: There will be a full track on Hadoop as well as one on NoSQL on Wednesday featuring talks on the framework itself, Pig and Hive as well as presentations from users on special use cases and on their way of getting the system to production.

The track on Mahout, Lucene and friends starts on Thursday afternoon, followed by another series of Lucene presentations on Friday.



Also don't miss the track on the community and business tracks for a glimpse behind the scenes. Of course there will also be tracks on well-known Apache Tomcat, httpd, OSGi and many, many more.

Looking forward to meeting you in Atlanta!

Salsa steps

2010-10-14 20:50
If you always wanted to start learning Salsa and never really had the time to do so: Today a new beginners' course starts at Salsa Con Corazon in Berlin Schöneberg. The course is still open for registration teaching the very basics in four sessions - one per week.

Slides available

2010-10-08 09:19
Yesterday evening the Autumn Hadoop Get Together took place in Berlin. The meetup this time focussed mainly on latest developments at Apache Mahout. The meetup was kindly sponsored by JTeam, providing video taping of the presentations as well as for free drinks. Thanks a lot for that.

After the meetup the group went over to Cafe Aufsturz for drinks, food and lots of interesting discussions - I left them there as I still have to get rid of a persistent cold. Hope you guys had fun!

The two speakers were so kind to provide the slides:



Videos of the talks will be posted very soon - so stay tuned.

Reminder: Apache Hadoop Get Together Berlin Today

2010-10-07 09:52
Just a brief reminder: The Apache Hadoop Get Together Berlin is supposed to take place today in newthinking store, Tucholskystr. 48 at 5p.m.

The meeting features two talks on Apache Mahout: Committer Sebastian Schelter will explain how to scale recommender systems with Mahout. Contributor Max Heimel is going to give an introduction to the sequence labeling facilities available in Mahout.

As usual the group will move over to Cafe Aufsturz after the meetup is over.

A big Thanks goes to JTeam for sponsoring video taping as well as to newthinking store for providing the venue for free.

A Get Together Checklist

2010-10-06 19:38
Still on the list of potentially interesting books: The Checklist Manifesto - explaining why checklists can still be valuable - especially for complex problems and tasks.

Though not very complex, I chose to come up with a checklist for running a Hadoop Get Together in Berlin as an exercise. I'm trying to stick with advise provided by the Checklist for Checklists.

Parties involved


  • Find two to three speakers two months in advance.
  • Find a sponsor for the videos.

Gathering information


  • Double check time and date with all speakers and newthinking store.
  • Get name, title, abstract from the speakers.
  • Get logo and exact conditions from sponsor.

Spreading the word


  • Put together an announcement text including thanks to video and venue sponsors.
  • Publish the event on Upcoming.
  • Publish the event on Xing.
  • Augment the announcement text by the Xing event and Upcoming links.
  • Send a newsletter to the Meetup Xing group.
  • Send the text to the Get Together mailing list, and if appropriate to the Hadoop, HBase, katta, Lucene, Solr and Mahout mailing lists.
  • On event day send a reminder to the Get Together mailing list
  • Create meetup intro slides including thanks for the sponsors, schedule, announcements of future events.

During the meetup


  • Mention newthinking bar during introduction.
  • Self-introduction of all participants.
  • Get mail addresses of future mailing list subscribers.
  • Keep presentations at 30 to 40 minutes.
  • Get speakers' slides.

After the event


  • Publish talks' slides.
  • Publish links to videos.


The more meetups you have run the larger the chance of the main organiser getting sick the day the meetup takes place. To avoid having to re-schedule the event make sure there are people that are capable and willing to take over moderation.

Apache Dinner October Today in Potsdam

2010-10-04 08:16
The Apache Dinner Berlin will take place today. As always, everybody is invited even if you didn't participate in the poll (http://doodle.com/8bi2456enwe6z2g6). This time around the dinner was organised by Daniel Naber. Thanks!

When: Monday, 2010-10-04, 19:30
Where: Lindencafe, Rudolf-Breitscheid-Str. 47/48, 14482 Potsdam-Babelsberg, This is directly next to the S-Bahn (S7, Babelsberg).

Looking forward to meeting you there!

Are devs contributing to OSS happier?

2010-09-24 20:18
When talking to fellow developers or meeting with students it happens from time to time that I get the question of why on earth I spent my freetime working on an open source project? Why do I spend weekends at developers' conferences like FOSDEM? Why do spent afternoons organising meetups? Why is it that I am reviewing and writing code after work for free?

Usually I point people to a post by Shalin explaining some of his reasons to contribute to open source. The post quite nicely summarises most reasons that match well with why I contribute back.

On the Apache Community mailing list Grant Ingersoll asked the question about whether devs who work on or use open source are happier in their employment.

In his response Mike posted a link to a video on what motivates people that adds another piece of information to the question of why work on open source software can be perceived as very rewarding though no money is involved: With people doing cognitively challenging tasks, motivation via payment can get you only so far. There are other motivational factors that might play an equal if not larger role in getting people to perform well on their day-to-day work:


  • Autonomy: If people are supposed to be engaged with their project they need time and freedom to chose how to solve their tasks. Many large engineering driven companies like Google or Atlassian have gone even further by introducing the concept of giving people a day a week to work on what they want how they want provided they share their results. These so-called 20% projects have shown to have high potential of turning into new, creative project ideas but also even into bugs or problems getting fixed.
  • Mastery: Great developers strive to get better at what they do - simply because realizing that you actually learn something and get better at what you do can be very satisfying. One way of achieving that goal is to work together with peers on common projects. The larger the pool of peers to draw from, the higher the probability of you finding mentors to help you out and to point out mistakes you make.

    There is one more factor why working on open source increases your coding level that should not be underestimated. Grant Ingersoll nicely described it in the thread mentioned above: "I was just talking with a friend yesterday, and fellow committer, who said he is a much better programmer since contributing. Of course, it makes sense. If your underwear is on display for all to see, you sure better make sure it is clean!"
  • Purpose: People like to work on projects for a purpose. Be it to make all information accessible to the world or to turn earth into a better place by making cheap calls available to everyone. As a counter example deploying some software only for the purpose of selling a license and not make life of your client better by recommending the best solution to help solve his problem may not be half as satisfying.


There is quite some documentation out there on what drives people who contribute to open source projects. The video shared by Mike nicely summarizes some of the motivations of people that are independent of open source work but are closely related to it.

Apprenticeship patterns (O'Reilly)

2010-09-23 08:17
A few days ago I finished reading the book "Apprenticeship Patterns" - Guidance for the Aspiring Software Craftsman, by
Dave Hoover, Adewale Oshineye. The book is addressed to readers who have the goal of becoming great software devleopers.

One naive question one could ask is why there is a need for such a book at all? Students are trained in computer science at university, then enter some IT departement and simply learn from their peers. So how is software development any different than other professions? Turns out there are a few problems with that approach: At university students usually don't get the slightest idea of what professional software development looks like. After four years of study they still have a long way to go before writing great software. When entering your average IT shop these juniors usually are put on some sort of customer project with tight deadlines. However learning implies making mistakes, it implies having time to try different routes to find the best one. Lucky are those very few who join a team that has a way for integrating and training junior developers. Last but not least at least in Germany tech carrier paths are still rare: As soon as developers excel they are offered a promotion - which usually leads straight into management before they even had a chance to become masters in their profession.

So what can people do who love writing software and want to become masters in their profession? The book provides various patterns, grouped by task:

  • Emptying the cup deals with setting up an attitude that enables learning: To be able to learn new skills the trainee first has to face his ignorance and realise that what he knows already is just a tiny little fraction of what differenciates the master from the junior.
  • In the second chapter "Walking the long road" the book deals with the problem of deciding whether to stick with software development or to go into management. Both paths provide their own hurdles and rewards - in the end the developer himself has to decide which one to go. Deciding for a technical carrier however might involve identifying new kinds of rewards: Instead of being promoted to senior super duper manager, this may involve benefits like getting a 20% project, setting up a company internal user group, getting support for presenting ones projects at conferences. The chapter also deals with motivational side of software development: Let's face it, professional development usually is way different from what we'd do if we had unlimited time. It may involve deadlines that cannot be met, it may invovle customers that are hard to communicate with. One might even have to deal with unmovtivated colleagues who have lower quality standards and no intention to learn more than what is needed to accomplish the task at hand. So there is the problem of staying motivated even if times get rough. Getting in touch with other developers - external and internal - here can be a great help: Attending user groups (or organising one), being part of an open source project, meeting regularly with other developers in one's general geografical area all may help to remember the fun things about developing software.
  • The third group of patterns has been put under the headline "Accurate self-assessment" - as people get better and better it get ever harder to remember that there are techniques out there one does not yet know. Being the best in a team means that there is not more room to learn in that environment. It's time to find another group to get in touch with others again: To be the worst in a team means there is a lot of room for learning, finding mentors helps with getting more information on which areas to explore next. Especially helpful is working on a common project with others - doing pair programming can help even with picking up just minor optimisations in their work environment.
  • The fourth chapter "Perpetual learning" deals with finding opportunities to learn new technologies - either in a toy project that in contrast to professional work is allowed to break and can be used to try and test new techniques and learn new languages. Other sources for learning are the source code itself, tech publications on magazines, books (both new and classic), blogs and mailing lists. Reflecting on what you learned helps remember it later - on option to reflect may involve writing up little summaries of what you read and keeping them in a place where you can easily retrieve them (for me this blog has turned into such a resource - yeah, I guess writing this book summary is part of the exercise, even was a proposal in the book itself). Last but not least one of the best resources for reflection and continued learning is to share knowledge - though you may feel there are others out there way better then you are, you are the one who just went though all the initial loops that no master remembers anymore. You can explain concepts in easy to understand words. Sharing and teaching means quickly finding gaps in your own knowledge and fixing them as you go forward. Last but not least it is important to create feedback loops: It does not help to learn after three years of coding that what you did does not match a customers expectations. As an apprentice you need faster feedback: On a technical level this may involve automated tests, code analysis and continuous integration. On a personal level it involves finding people to review your code. It means discussing your ideas with peers.
  • The last chapter on "Constructing your curriculum" finally dealt with the task of finding a way to remain up to date, e.g. by following re-known developers' blogs. But also studying the classic literature - there are various books in computer science and software development that have been written back in the 60s and 70s but are still highly relevant.


The book does not give you a recipe to turn from junior to master in the shortest possible time. However it successfully identifies situations many a software developer has encountered in his professional life that made him quesion his current path. It provides ideas on what to do to improve one's skills even if the current IT industry may not be best equipped with tools for training people.

My conclusion from the book was that most important is getting in touch with other developers, exchanging ideas and working on common projects. Open source get several mentions in the book, but also for me has turned out to be a great source for getting feedback, help and input from the best developers I've met so far.

In addition meeting people who are working on similar projects face-to-face provides a lot of important feedback as well as new ideas to try out. Talking with someone over a cup of coffee for two hours sometimes can be more productive than discussing for days over e-mail. Hacking on a common project, maybe even in the same location, usually is the most productive way not only to solve problems but also to pick up new skills.