HDFS的shell操作
基本语法
hadoop fs 具体命令hdfs dfs 具体命令
命令大全
[root@hadoop03 bin]# hadoop fsUsage: hadoop fs [generic options][-appendToFile <localsrc> ... <dst>][-cat [-ignoreCrc] <src> ...][-checksum <src> ...][-chgrp [-R] GROUP PATH...][-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...][-chown [-R] [OWNER][:[GROUP]] PATH...][-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>][-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>][-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] [-e] <path> ...][-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>][-createSnapshot <snapshotDir> [<snapshotName>]][-deleteSnapshot <snapshotDir> <snapshotName>][-df [-h] [<path> ...]][-du [-s] [-h] [-v] [-x] <path> ...][-expunge][-find <path> ... <expression> ...][-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>][-getfacl [-R] <path>][-getfattr [-R] {-n name | -d} [-e en] <path>][-getmerge [-nl] [-skip-empty-file] <src> <localdst>][-head <file>][-help [cmd ...]][-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...]][-mkdir [-p] <path> ...][-moveFromLocal <localsrc> ... <dst>][-moveToLocal <src> <localdst>][-mv <src> ... <dst>][-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>][-renameSnapshot <snapshotDir> <oldName> <newName>][-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...][-rmdir [--ignore-fail-on-non-empty] <dir> ...][-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]][-setfattr {-n name [-v value] | -x name} <path>][-setrep [-R] [-w] <rep> <path> ...][-stat [format] <path> ...][-tail [-f] [-s <sleep interval>] <file>][-test -[defsz] <path>][-text [-ignoreCrc] <src> ...][-touch [-a] [-m] [-t TIMESTAMP ] [-c] <path> ...][-touchz <path> ...][-truncate [-w] <length> <path> ...][-usage [cmd ...]]
命令实操
1、hadoop fs -help rm
结果：
[root@hadoop03 hadoop-3.1.3]# hadoop fs -help rm
-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ... :
Delete all files that match the specified file pattern. Equivalent to the Unix
command "rm <src>"
-f If the file does not exist, do not display a diagnostic message or
modify the exit status to reflect an error.
-[rR] Recursively deletes directories.
-skipTrash option bypasses trash, if enabled, and immediately deletes <src>.
-safely option requires safety confirmation, if enabled, requires
confirmation before deleting large directory with more than
<hadoop.shell.delete.limit.num.files> files. Delay is expected when
walking over large directory recursively to count the number of
files to be deleted before the confirmation.
2、上传（从本地上传到HDFS）
2.1 moveFromLocal （剪切）
hadoop fs -mkdir /wkmac
vim wkmac.txt(随便写几个字)
hadoop fs -moveFromLocal ./wkmac.txt /wkmac
2.2 copyFromLocal （拷贝）
vim zlsmac.txt
hadoop fs -copyFromLocal ./zlsmac.txt /wkmac
2.3 put 等同于 copyFromLocal
vim ztmac.txt
hadoop fs -put ./ztmac.txt /wkmac
OR hdfs dfs -put ./ztmac/txt /wkmac
2.4 appendToFile 追加一个文件到已经存在的文件末尾
3、下载（从HDFS下载到本地）
3.1、copyToLocal
hadoop fs -copyToLocal /wkmac/wkmac.txt ./
3.2、get 等同于copyToLocal
hadoop fs -get /input ./
4、显示目录信息
hadoop fs -ls /wkmac
5、查看文件内容
hadoop fs -cat /wkmac/wkmac.txt
6、修改文件权限
hadoop fs -chmod 777 /wkmac/wkmac.txt
hadoop fs -chown labour:labour /wkmac/wkmac.txt
7、创建路径
hadoop fs -mkdir /test
8、拷贝（HDFS目录之间的拷贝）
hadoop fs -cp /wkmac/wkmac.txt /test
9、移动
hadoop fs -mv /wkmac/wkmac.txt /test
10、显示一个文件末尾1kb的数据
hadoop fs -tail /test/wkmac.txt
11、删除
hadoop fs -rm /wkmac
hadoop fs -rm -rf /wkmac (递归删除)
12、统计文件夹大小
hadoop fs -du -s -h /wkmac
输出：
42 126 /wkmac （42 文件大小 126 表示文件大小*副本数）
hadoop fs -du -h /wkmac
输出：
21 63 /wkmac/wkmac.txt
11 33 /wkmac/zlsmac.txt
10 30 /wkmac/ztmac.txt
13、设置HDFS副本数量
hadoop fs -setrep 4 /wkmac
给文件夹设置副本数时，文件夹下所有的文件都会被设置
Replication 4 set: /wkmac/wkmac.txt
Replication 4 set: /wkmac/zlsmac.txt
Replication 4 set: /wkmac/ztmac.txt
idea中操作HDFS
导入依赖包
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>3.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>3.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>3.1.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>3.1.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-yarn-api -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-api</artifactId>
<version>3.1.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-yarn-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
<version>3.1.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-yarn-client -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<version>3.1.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-yarn-server-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-common</artifactId>
<version>3.1.3</version>
</dependency>
编写操作代码
使用客户端操作HDFS时，是有一个用户身份的。默认情况下，HDFS客户端API会采用Windows/MacOS的默认用户去访问HDFS，会报权限异常错误，所以在访问HDFS时，需要配置用户
/**
* 测试创建文件
*/
public void testMkdirs() throws URISyntaxException, IOException, InterruptedException{
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop03:8020"), configuration,"root");
// 2 创建目录
fs.mkdirs(new Path("/ideatest/cjml/"));
// 3 关闭资源
fs.close();
}
/**
* 测试上传文件
*/
public static void testCopyFromLocalFile() throws IOException, URISyntaxException, InterruptedException {
Configuration configuration = new Configuration();
configuration.set("dfs.replication", "2");
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop03:8020"), configuration, "root");
// 2 上传文件
fs.copyFromLocalFile(new Path("/Users/wukai/Downloads/iShot2021-12-08 23.47.13.png"), new Path("/ideatest/cjml/"));
// 3 关闭资源
fs.close();
}
/**
*测试文件下载
*/
public static void testCopyToLocalFile() throws URISyntaxException, IOException, InterruptedException {
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop03:8020"), configuration, "root");
// 2 执行下载操作
// boolean delSrc 指是否将原文件删除
// Path src 指要下载的文件路径
// Path dst 指将文件下载到的路径
// boolean useRawLocalFileSystem 是否开启文件校验
fs.copyToLocalFile(false, new Path("/ideatest/cjml/iShot2021-12-08 23.47.13.png"), new Path("/Users/wukai/Downloads/iShot2021-12-08 23.47.13.png"), true);
// 3 关闭资源
fs.close();
}
/**
*测试文件改名
*/
public static void testRename() throws IOException, URISyntaxException, InterruptedException {
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop03:8020"), configuration, "root");
fs.rename(new Path("/ideatest/cjml/iShot2021-12-08 23.47.13.png"), new Path("/ideatest/cjml/beauty.png"));
fs.close();
}
/**
*测试删除文件
*/
public static void testDelete() throws IOException, URISyntaxException, InterruptedException {
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop03:8020"), configuration, "root");
fs.delete(new Path("/ideatest/cjml/beauty.png"),true);
fs.close();
}
/**
*获取文件详情（查看文件名称、权限、长度、块信息）
*/
public static void testListFiles() throws IOException, URISyntaxException, InterruptedException {
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop03:8020"), configuration, "root");
// 2 获取文件详情
RemoteIterator<LocatedFileStatus> listFiles = fs.listFiles(new Path("/"), true);
while (listFiles.hasNext()) {
LocatedFileStatus fileStatus = listFiles.next();
System.out.println("========" + fileStatus.getPath() + "=========");
System.out.println(fileStatus.getPermission());
System.out.println(fileStatus.getOwner());
System.out.println(fileStatus.getGroup());
System.out.println(fileStatus.getLen());
System.out.println(fileStatus.getModificationTime());
System.out.println(fileStatus.getReplication());
System.out.println(fileStatus.getBlockSize());
System.out.println(fileStatus.getPath().getName());
// 获取块信息
BlockLocation[] blockLocations = fileStatus.getBlockLocations();
System.out.println(Arrays.toString(blockLocations));
}
// 3 关闭资源
fs.close();
}
HDFS写数据流程
1、客户端通过Distributed FileSystem模块向NameNode请求上传文件，NameNode检查目标文件是否已存在，父目录是否存在
2、NameNode返回是否可以上传
3、客户端请求第一个 Block上传到哪几个DataNode服务器上
4、NameNode返回3个DataNode节点，分别为dn1、dn2、dn3
5、客户端通过FSDataOutputStream模块请求dn1上传数据，dn1收到请求会继续调用dn2，然后dn2调用dn3，将这个通信管道建立完成
6、dn1、dn2、dn3逐级应答客户端
7、客户端开始往dn1上传第一个Block（先从磁盘读取数据放到一个本地内存缓存），以Packet为单位，dn1收到一个Packet就会传给dn2，dn2传给dn3；dn1每传一个packet会放入一个应答队列等待应答
8、当一个Block传输完成之后，客户端再次请求NameNode上传第二个Block的服务器。（重复执行3-7步）
在HDFS写数据的过程中，NameNode会选择距离待上传数据最近距离的DataNode接收数据
HDFS读数据流程
1、客户端通过DistributedFileSystem向NameNode请求下载文件，NameNode通过查询元数据，找到文件块所在的DataNode地址
2、挑选一台DataNode（就近原则，然后随机）服务器，请求读取数据
3、DataNode开始传输数据给客户端（从磁盘里面读取数据输入流，以Packet为单位来做校验）
4、客户端以Packet为单位接收，先在本地缓存，然后写入目标文件
若有收获，就点个赞吧
0 人点赞
