Question

I have a lot of data in an Amazon Simple DB Domain. I want to start Hive on Elastic Map Reduce (on top of hadoop) and somehow, either import data from simpledb or, connect to simpledb and run hiveql queries on it. I have having issues importing the data. Any pointers?

Answer 1

As input to a streaming hadoop job you could have a sequence of select statements for simpleDB.

for example, your input could contain (in a less verbose form):

collectionA between dates 123 and 234
collectionA between dates 235 and 559
collectionA between dates 560 and 3000
...

Then you would implement a mapper script that performed the following transformation: input_select_statement => execute_select_statement => output_results

This would be super easy using streaming because you could use any library for any language you like and not have to worry about implementing any of the complicated Hadoop java stuff.

Hope this helps.

(the hacky way to do it would be to have a single script that you run locally that does the same as above, but loads the results into s3. I run a script like that nightly for a lot of our database data)

友情链接