Question

I m using hadoop to update some records in a mysql db... The issue that I m seeing is that in certain cases, multiple reducers are launched for the same key set. I ve seen up to 2 reducers running on different slaves for the same key. This leads to the issue of both reducers updating the same record in the db.

I was thinking of turning off the autocommit mode to alleviate this issue.... but and doing the commit as part of the "cleanup" operation in the reducer, but was wondering what to do with the reducer(s) that lag behind...would the cleanup operation still be called for that...if so....is there a way to tell if the reducer finished normally or not, since I d like to call "rollback" on the reducer(s) that didn t finish processing the data entirely?

Answer 1

• 减少就业财产:

mapred.map.tasks.speculative.execution

具有虚假价值。这将取代投机性处决。

Answer 2

两点:

I really doubt that two (EQUAL) keys inside a reduce get partitioned to different slaves. Since HashPartitioner is used. You should override hashCode on your key class.
You have the option to set the number of reduce tasks. It can be done with an API call to Job.setNumReduceTasks(X). Obviously you can set this to 1.

Answer 3

一般来说(在不了解您的使用情况的情况下),通常最好避免与Hadoop发生“效果”。这基本上依靠Hadoop之外的第3个政党制度,因为它可能阻碍你的业绩,并可能因翻新而使系统瘫痪。我建议你调查从云拉角起的Sqoop,在完成绘制地图的工作后,完成批量。作为大负荷,我取得了良好的成功。

Sqoop Documentation

如果你仍然希望从Hadoop直接指数化。你们可以利用公平程序来评定能够随时运行的地图仪或减标器的数量。开始这项工作。页: 1 你们正在寻找最大/最大参数。

rel=“nofollow”> 公平时间安排文件

友情链接