Hadoop has configuration parameter hadoop.tmp.dir which, as per documentation, is `"A base for other temporary directories." I presume, this path refers to local file system.
I set this value to /mnt/hadoop-tmp/hadoop-${user.name}. After formatting the namenode and starting all services, I see exactly same path created on HDFS.
Does this mean, hadoop.tmp.dir refers to temporary location on HDFS?
It s confusing, but hadoop.tmp.dir is used as the base for temporary directories locally, and also in HDFS. The document isn t great, but mapred.system.dir is set by default to "${hadoop.tmp.dir}/mapred/system", and this defines the Path on the HDFS where where the Map/Reduce framework stores system files.
If you want these to not be tied together, you can edit your mapred-site.xml such that the definition of mapred.system.dir is something that s not tied to ${hadoop.tmp.dir}
Let me add a bit more to kkrugler s answer:
There re three HDFS properties which contain hadoop.tmp.dir in their values
dfs.name.dir: directory where namenode stores its metadata, with default value ${hadoop.tmp.dir}/dfs/name.dfs.data.dir: directory where HDFS data blocks are stored, with default value ${hadoop.tmp.dir}/dfs/data.fs.checkpoint.dir: directory where secondary namenode store its checkpoints, default value is ${hadoop.tmp.dir}/dfs/namesecondary.This is why you saw the /mnt/hadoop-tmp/hadoop-${user.name} in your HDFS after formatting namenode.
Had a look around for information on this one. Only thing I could come up with was this post on the Amazon Elastic MapReduce Dev Guide:
In hadoop-site.xml, we set hadoop.tmp.dir to /mnt/var/lib/hadoop/tmp. /mnt is where we mount the “extra” EC2 volumes, which can contain a lot more data than the default volume. (The exact amount depends on instance type.) Hadoop s RunJar.java (the module that unpacks the input JARs) interprets hadoop.tmp.dir as a Hadoop file system path rather than a local path, so it writes to the path in HDFS instead of a local path. HDFS is mounted under /mnt (specifically /mnt/var/lib/hadoop/dfs/. So, you can write lots of data to it.
I want to redirect the sub-domain webmail to /roundcube for that domain. This have to work for all virtual hosts in apache. Example: webmail.example.com must point to [www.]example.com/roundcube ...
What is the fastest way to store config data in PHP so that it is easily changeable (via PHP)? First I thought about having config.php file, but I can t edit it on fly with PHP, at least not very ...
I m trying to use siFR 3. I need set to the flash element to use max width for a text replaced. The width of flash text can use 200px MAX. The problem of the following code is that Here s is my ...
Often in example code for software, I see a connection string eg. mysql://user@localhost in the configuration file for the software to get access to the database. This seems fine for a test/...
Is there a helper method/object/methodology for getting a reference to a Zend_Application s config resource? I know I can do something like $config = new Zend_Config_Ini($file, $environment); but ...
I am currently running a Hudson instance on a VM slice. As I don t have a need to run more than a couple small applications from it, I m minimizing how much space I devote to it. The sizes I ve ...
I m attempting to make use of configuration transformations in a continuous integration environment. I need a way to tell the TFS build agent to perform the transformations. I was kind of hoping it ...
hi i m trying make a setup msi for my application with wix v3 I have a problem about this task; I need a user input that will be stored in my application s config file (for example i need a dialog for ...