Hadoop and HBase for OSGi

2008-08-27

Here’s a tip if you need to access HBase from an OSGi environment. I tried packaging them into two separate Hadoop and HBase bundles, but I ran into an unexpected, and quite unseemly, issue. To make a short story sweet (or so): You will need to package them together in a single bundle, and write a manifest that exports the HBase and the Hadoop packages.

The issue, shockingly, is that HBase takes the liberty of declaring a class in a Hadoop package! Specifically, the org.apache.hadoop.ipc.HBaseClient class extends Hadoop’s Client class, defined in that same package. This is the kind of necessary evil that pops up from time to time between two projects, especially in early days – after all, they are in 0.17.1 and 0.2.0, respectively. It does break an important OSGi rule: A package can only be imported from a single bundle, it cannot contain classes contributed from multiple bundles. Which means that other bundles can only ever depend on one of them, never both.

I haven’t thought of any import/export acrobatics that could resolve this, and I’m not sure if anyone should. The HBase bundle needs to see that package exported from the Hadoop bundle, or it can’t extend its Client class. Given that, the HBase bundle then needs to export that same package. I can’t imagine any way that would work, except by accidents in the implementation. Even if it somehow is an edge case within the spec, it certainly isn’t something I would like to spend too much think-time concerning myself about, let alone maintain. The quick fix is to package them together. Do that, and your stuff works.

Feel free to ask for the manifest, if you’re interested. I didn’t have it at hand while writing this post.

Advertisements

3 Responses to “Hadoop and HBase for OSGi”

  1. Kjetil V. Says:

    For anyone still lured in by three buzzwords at once – I think I heard they’d fixed it. Move on.

  2. Leen Toelen Says:

    I created a ‘patch’ (more a howto) to make hadoop common OSGI compatible natively. Hopefully this helps a bit in resolving these issues earlier.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: