English 中文(简体)
2. 当地人使用许多深层的 had子向 had子转移
原标题:TransferFromLocal to hadoop hdfs using many threads

I am trying to transfer lots of files from local to hadoop hdfs. In my java code, I just have one connection to hadoop. But I call

 fileSystem.transferFromLocal 

同时,在50条深线。

我认为,这样做可能不是好的,因为它确实很缓慢。

请允许我就此提出一些建议吗? 非常感谢。

问题回答

你们需要看到导致转让缓慢的瓶颈,这是可能的。 刚刚增加校对人数,科索沃民盟的书本也相应增加。 如果不了解你的Hadoop集群的细节,就很难诊断出问题。

这里需要考虑的一些事项。

  1. 检查当地机器和Hadoop集群之间的网络带宽。

  2. 当地磁盘(i/o)也可能是瓶颈。

  3. Try increasing the number of data nodes. Note that the data is directly streamed from the client to the first data node in the pipe line. The first forwards to the second, which forwards to the next data node.

  4. 检查任何组合参数,以调整科索沃警察部队。

查阅





相关问题
Spring Properties File

Hi have this j2ee web application developed using spring framework. I have a problem with rendering mnessages in nihongo characters from the properties file. I tried converting the file to ascii using ...

Logging a global ID in multiple components

I have a system which contains multiple applications connected together using JMS and Spring Integration. Messages get sent along a chain of applications. [App A] -> [App B] -> [App C] We set a ...

Java Library Size

If I m given two Java Libraries in Jar format, 1 having no bells and whistles, and the other having lots of them that will mostly go unused.... my question is: How will the larger, mostly unused ...

How to get the Array Class for a given Class in Java?

I have a Class variable that holds a certain type and I need to get a variable that holds the corresponding array class. The best I could come up with is this: Class arrayOfFooClass = java.lang....

SQLite , Derby vs file system

I m working on a Java desktop application that reads and writes from/to different files. I think a better solution would be to replace the file system by a SQLite database. How hard is it to migrate ...

热门标签