I am trying to transfer lots of files from local to hadoop hdfs. In my java code, I just have one connection to hadoop. But I call
fileSystem.transferFromLocal
同时,在50条深线。
我认为,这样做可能不是好的,因为它确实很缓慢。
请允许我就此提出一些建议吗? 非常感谢。
I am trying to transfer lots of files from local to hadoop hdfs. In my java code, I just have one connection to hadoop. But I call
fileSystem.transferFromLocal
同时,在50条深线。
我认为,这样做可能不是好的,因为它确实很缓慢。
请允许我就此提出一些建议吗? 非常感谢。
你们需要看到导致转让缓慢的瓶颈,这是可能的。 刚刚增加校对人数,科索沃民盟的书本也相应增加。 如果不了解你的Hadoop集群的细节,就很难诊断出问题。
这里需要考虑的一些事项。
检查当地机器和Hadoop集群之间的网络带宽。
当地磁盘(i/o)也可能是瓶颈。
Try increasing the number of data nodes. Note that the data is directly streamed from the client to the first data node in the pipe line. The first forwards to the second, which forwards to the next data node.
检查任何组合参数,以调整科索沃警察部队。
Hi have this j2ee web application developed using spring framework. I have a problem with rendering mnessages in nihongo characters from the properties file. I tried converting the file to ascii using ...
Check this, List<String> list = new ArrayList<String>(); for (int i = 0; i < 10000; i++) { String value = (""+UUID.randomUUID().getLeastSignificantBits()).substring(3, ...
I am in the middle of solving a problem where I think it s best suited for a decorator and a state pattern. The high level setting is something like a sandwich maker and dispenser, where I have a set ...
I have been trying to execute a MS SQL Server stored procedure via JDBC today and have been unsuccessful thus far. The stored procedure has 1 input and 1 output parameter. With every combination I ...
I have a system which contains multiple applications connected together using JMS and Spring Integration. Messages get sent along a chain of applications. [App A] -> [App B] -> [App C] We set a ...
If I m given two Java Libraries in Jar format, 1 having no bells and whistles, and the other having lots of them that will mostly go unused.... my question is: How will the larger, mostly unused ...
I have a Class variable that holds a certain type and I need to get a variable that holds the corresponding array class. The best I could come up with is this: Class arrayOfFooClass = java.lang....
I m working on a Java desktop application that reads and writes from/to different files. I think a better solution would be to replace the file system by a SQLite database. How hard is it to migrate ...