September Apache Hadoop Get Together @ Berlin

2009-08-23 20:48
The upcoming Apache Hadoop Get Together Berlin is to take place on September 29th in newthinking store. Details are up on the web page at upcoming and will be sent out to the mailing list soon.

Flying back home from Cologne

2009-08-23 20:40
Last weekend FrOSCon took place in Sankt Augustin, near Cologne. FrOSCon is organized on a yearly basis at the university of applied sciences in Sankt Augustin. It is a volunteer driven event with the goal of bringing developers and users of free software projects together. This year, the conference featured 5 tracks, two examples being cloud computing and the Java track.

Unfortunately this year the conference started with a little surprise for me and my boyfriend: Being both speakers, we had booked a room in Hotel Regina via the conference committee. Yet on Friday evening we had to learn that the reservation never actually reached the hotel... So, after several minutes talking to the receptionist, calling the organizers we ended up in a room that was booked for Friday night by someone who was known to arrive no earlier than Saturday. Fortunately for us we have a few friends close by in Düsseldorf: Fnord was so very kind to let us have his guest couch for the following night.

Checkin time next morning: On the right hand side the regular registration booth. On the left hand side the entrance for VIPs only. The FSFE quickly realized it's opportunity: They soon started distributing flyers and stickers among the waiting exhibitors and speakers.






Set aside the organizational issues, most of the talks were very interesting and well presented. The Java track featured two talks by Apache Tomcat committer Peter Roßbach, the first one on the new Servlet 3.0 API, the second one on Tomcat 7. Too sad, my talk was in parallel to his Tomcat talk, so I couldn't attend that. I appreciate several of the ideas on cloud computing highlighted in the keynote: Cloud computing as such is not really new or innovative, it is several good ideas so far known for instance as utility computing that are now being improved and refined to make computation a commodity. At the very moment however cloud computing providers tend to sell their offers as new, innovative products. There is no standard API for cloud computing services. That makes switching from one provider to another extremely hard and leads to vendor-lockin for its users.

The afternoon was filled by my talk. This time I tried something, that so far I only have done in user groups of up to 20 people: I first gave a short introduction into who I am and than asked the audience to describe themselves in one sentence. There were about 50 people, after 10 minutes everyone had given is self-introduction. It was a nice way of getting detailed information of what knowledge to expect from people, and it was interesting to hear people from IBM and Microsoft being in the room.

After that I attended the RestMS talk by Thilo Fromm and Peter Hintjens. They showed a novel, community driven way to standards creation. RestMS is a messaging standard that is based on a restful way for communication. So far the standard itself is still in it's very early stages, still there are some very “alpha, alpha, alpha” implementations out there that can be used for playing around. According to Peter there are actually people who already use these implementations for production servers and send back bug reports.

Sunday started with an overview of the DaVinci VM by Dalibor Topic, the author of the OpenJDK article series in the German Java Magazin. Second talk of the day was an introduction to Scala. I already know a few details of the language, but the presentation made it easy to learn more: It was organised as an open question and answer session with live coding leading through the talk.

After lunch and some rest, the last two topics of interest were on details on the campaigns of FFII against software patents and an overview of the upcoming changes in gnome3.0.

This year's FrOSCon did have some organizational quirks but the quality of most of the talks was really good with at least one interesting topic in one of the sessions at nearly every time slot - though I must admit that that was easy in my case with Java and cloud computing being of interest to me.

Update: Videos are up online.

September 2009 Hadoop Get Together Berlin

2009-08-17 09:11
The newthinking store Berlin is hosting the Hadoop Get Together user group meeting. It features talks on Hadoop, Lucene, Solr, UIMA, katta, Mahout and various other projects that deal with making large amounts of data accessible and processable. The event brings together leaders from the developer and user communities. The speakers present projects that build on top of Hadoop, case studies of applications being built and deployed on Hadoop. After the talks there is plenty of time for discussion, some beer and food.

There is also a related Xing Group on the topic of building scalable information retrieval systems. Feel free to join and meet other developers dealing with the topic of building scalable solutions.


Agenda:

Please see upcoming page for updates.


  • Thilo Götz: JAQL
  • Uwe Schindler: Lucene 2.9
  • nugg.ad: Ad Recommendation with Hadoop
  • T. Schuett: Solving puzzles with Hadoop.


If you yourself would like to give a presentation: There are additional slots of 20 minutes each available. There is a beamer provided. Just bring your slides. To include your topic on this web site as well as the upcoming.org entry, please send your proposal to Isabel.

After the talks there will be time for an open discussion. We are going into a nearby restaurant after the event so there will be plenty of time for talking, discussing and new ideas.

Location

The Apache Hadoop Get Together takes place at the newthinking store Berlin:



newthinking store GmbH

Tucholskystr. 48

10117 Berlin



View Larger Map

Accomodation

  • Homeli - not exactly in walking distance, but only a few S-Bahn stations away. Very nice Bed and Breakfast hotel. (The offer is only valid if you stay for at least three nights.)

  • Circus Berlin is a combination of hostel and hotel close by.

  • Zimmer in Berlin is yet another Bed and Breakfast hotel.

  • House boat near Friedrichshain



Announcements

If you would like to be notified on news please subscribe to our mailinglist. The meetings usually are also announced on the project mailing lists as well as on the newthinking store website.


Contact

In case you have any trouble reaching the location or finding accomodation feel free to contact the organiser Isabel.

Past events

Solr at AOL

2009-07-02 13:06
Grant Ingersoll has posted a very interesting interview with Ian Holsman on Solr at Relegance, now AOL. It describes the business side of the decission to switch to an open source solution, provides some inside on the size of the installation and details which technological reasons have driven the decission to switch from a proprietary implementation to Solr:

http://www.lucidimagination.com/Community/Hear-from-the-Experts/Podcasts-and-Videos/Interview-Ian-Holsman-Relegence

Large Scalability - Papers and implementations

2009-06-23 12:08
In recent years the Googles and Amazons on this world have released papers on how to scale computing and processing to terrabytes of data. These publications have led to the implementation of various open source projects that benefit from that knowledge. However mapping the various open source projects to the original papers and assigning tasks that these projects solve is not always easy.

With no guarantee of completeness this lists provides a short mapping from open source project to publication.

There are further overviews available online as well as a set of slides from the NOSQL debrief.















Map Reduce Hadoop Core Map Reduce Distributed programming on rails, 5 Hadoop questions, 10 Map Reduce Tips
GFS HDFS (Hadoop File System) Distributed file system for unstructured data
Bigtable HBase, Hypertable Distributed storage for structured data, When to use HBase.
Chubby Zookeeper Distributed lock- and naming service
Sawzall PIG, Cascading, JAQL, Hive Higher level langage for writing map reduce jobs
Protocol Buffers Protocol Buffers, Thrift, Avro, more traditional: Hessian, Java serialization Data serialization, early benchmarks
Some NoSQL storage solutions CouchDB, MongoDB CouchDB: document database
Dynamo Dynomite, Voldemort, Cassandra Distributed key-value stores
Index Lucene Search index
Index distribution katta, Solr, nutch Distributed Lucene indexes
Crawling nutch, Heritrix, droids, Grub, Aperture Crawling linked pages



Tomcat Tuesday talk

2009-05-21 09:07
Since several months at neofonie we have a talk given by external or internal developers on various subjects each Tuesday. Usually these presentations are a nice way to get an overview of new emerging technologies, to get an overview of current conference topics or to gain insight into interesting internal projects.

This week we had Apache Tomcat Committer and PMC Peter Rossbach here at neofonie to talk about the Tomcat architecture and Tomcat clustering solutions. He gave two pretty in-depth presentations on the Tomcat internals, Tomcat optimization and extension points.

Some points that were especially interesting to me: The project started out in the late nineties, initiated by a bunch of developers who just wanted to see what it takes to write a web application container and that fullfills the spec. The goal basically was a reference implementation. Soon enough however users defined the resulting code as production ready and used it.

There are a few caveats from this history that are still visible. One is the lack of tests in the codebase. Sure, each release is tested agains the Sun TCK - but these tests cannot be opened to the general public. So if you as a developer make extensions or modifications to the code base there is no easy way of knowing whether you broke something or not.

For me as a developer it was interesting to see really how complex it quickly gets to cluster tomcat deployments and make them failure resistant. Some tools mentioned that help automatic with easier deployment are Puppet and FAI. One issue however that is still on the developer's agenda is Tomcat monitoring.

To summarize: The conference room was packed with developers expecting two very interesting talks. Thanks to Peter Rossbach for coming to neofonie and explaining more on the internals of the Tomcat software, the project and the community behind.

Back from Zürich

2009-05-05 16:58
I spend the last five days in Zurich. I wanted to visit the city again - and still owed one of my friends there a visit. I am really happy the weather was quite nice over the weekend. That way I could spend quite some time in town (got another one of those puzzles) and go for a hike on the Ütli mountain: I took the steep way up that had quite a lot of stairs. Interestingly though, despite being quite tired when I finally arrived on top, my legs did not have sore muscles the next day. Seems going to work and back again by bike does indeed help a bit, even if we have no hills in Berlin.

Yesterday I was allowed to present the Apache project Mahout in a Google tech talk. Usually I am talking to people well familiar with the various Apache projects. Giving my talk I asked people who was familiar with Lucene, with Hadoop. To me it was pretty unusual that very few engineers were aware of these. It almost seemed like it is unusual to have a look at what is going outside the company? Or was it just the selection of people that were interested in my talk?

I tried to cover most of the basics, put Mahout into the context of the Lucene umbrella project. I tried to show some of the applications that can be built with Mahout and detailed some of the things that are on our agenda.

Some of the questions I received were on the scalability of Hadoop, on the general distribution of people being paid to work on Free Software projects vs. those working on them in their freetime. Another question was whether the project is targeted to text only applications (which of course it is not, as feature extraction so far has been left to the user). Last but not least the relation to UIMA was brought up by a former IBM-UIMA engineer.

To summarize: For me it was a pretty interesting experience to give this tech talk. I hope it did help me to do away with some of my "Apache bias". It is always valuable to look into what is going outside one's community.

Feedback from the Hadoop User Group UK

2009-04-29 08:54
A few weeks after the Hadoop User Group UK is over, there are quite a few postings on the event online. I will try to keep this page updated if there are any further reviews. The one I found so far:

http://huguk.org/2009/04/huguk-2-wrap-up.html - the wrap-up of the event itself.

http://blog.oskarsson.nu/2009_04_01_archive.html - a short summary by the organiser - Thanks again for a great event.

http://www.cloudera.com/blog/2009/04/21/hadoop-uk-user-group-meeting/ - a short summary on the Cloudera blog.

http://people.kmi.open.ac.uk/adam/?p=26 - a quick overview with a Mahout focus by Adam Rae.

June 2009 Apache Hadoop Get Together @ Berlin

2009-04-23 19:30
Title: Apache Hadoop Get Together @ Berlin
Location: newthinking store Berlin Mitte
Link out: Click here
Description: I just announced the fifth Apache Hadoop Get Together in Berlin at the newthinking store. Torsten Curdt offered to give a talk on data serialization with Thrift and Protocol Buffers.

If you have a topic you would like to talk about: Feel free to just bring your slides - there will be a beamer and lots of people interested in scalable information retrieval.
Start Time: 17:00
Date: 2009-06-25

Mahout on EC2

2009-04-21 21:00
Amazon released Elastic Map Reduce only a few weeks ago. EMR is based on a hosted Hadoop environment and offers machines to run map reduce jobs against data in S3 on demand.

Last week Stephen Green has spent quite some effort to get Mahout running on EMR. Thanks to his work Mahout is running on EMR since last Thursday night. Read the weblog of Tim Bass for further information.