Question

我发现了这个问题 Where HDFS stores files locally by default?

我的HDFS将数据存储在/ tmp /文件夹中，由系统删除。

我想更改 HDFS在本地存储文件的位置。

我正在查看hdfs-default.xml但找不到dfs.data.dir

运行bin/hadoop version

Hadoop 2.8.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 66c47f2a01ad9637879e95f80c41f798373828fb
Compiled by jdu on 2017-10-19T20:39Z
Compiled with protoc 2.5.0
From source with checksum dce55e5afe30c210816b39b631a53b1d
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.8.2.jar

修改
我想知道细节：
哪个文件以及如何编辑以在本地更改HDFS存储文件？

Answer 1

寻找dfs.datanode.data.dir。这里的文档 - http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Answer 2

感谢@ultimoTG提示。

所以，我的解决方案是在我的hadoop目录中找到文件名hdfs-default.xml（此文件仅供参考，在此处更改配置 NOT WORK ）。

$HADOOP_HOME/share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

然后，我在修改后的值之前将我想要从hdfs-default.xml更改为$HADOOP_HOME/etc/hadoop/hdfs-site.xml的行。

这是我的$HADOOP_HOME/etc/hadoop/hdfs-site.xml，它将HDFS将文件本地目录更改为下载文件夹。

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>


<property>
  <name>dfs.namenode.name.dir</name>
  <value>/home/my_name/Downloads/hadoop_data/dfs/name</value>
  <description>Determines where on the local filesystem the DFS name node
      should store the name table(fsimage).  If this is a comma-delimited list
      of directories then the name table is replicated in all of the
      directories, for redundancy. </description>
</property>

<property>
  <name>dfs.datanode.data.dir</name>
  <value>/home/my_name/Downloads/hadoop_data/dfs/data</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices. The directories should be tagged
  with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
  storage policies. The default storage type will be DISK if the directory does
  not have a storage type tagged explicitly. Directories that do not exist will
  be created if local filesystem permission allows.
  </description>
</property>

</configuration>

Answer 3

当您第一次提取hadoop时，hdfs-site.xml存在于$ HADOOP_HOME / etc / hadoop中，默认为空。您可以将以下配置添加到hdfs-site.xml以更改本地存储位置：

<property> 
      <name>dfs.data.dir</name> 
      <value>path_to_dir</value> 
</property> 

<property> 
      <name>dfs.name.dir</name> 
      <value>path_to_dir</value> 
</property>

如何更改Hadoop HDFS在本地存储文件

3 个答案: