Bashscript将文件上传到hdfs

时间:2017-03-24 17:56:10

标签: bash hadoop hdfs

我正在尝试创建一个bashscript来将文件从本地边缘节点文件系统上传到hdfs。我想知道在文件中添加时间戳的好方法。让时间戳工作有一些问题。

Mar 24 14:40:04 ip-172-31-8-83 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  323]     0   323     8840      356      18        0             0 systemd-journal
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  357]     0   357    11645      536      22        0         -1000 systemd-udevd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  392]     0   392    13854      112      26        0         -1000 auditd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  451]    81   451     8176      138      17        0          -900 dbus-daemon
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  455]     0   455   127319      539      63        0             0 NetworkManager
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  456]     0   456     6048       73      15        0             0 systemd-logind
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  458]   998   458   131878     1609      54        0             0 polkitd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  460]   997   460    28962      103      26        0             0 chronyd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  467]     0   467    31556      156      21        0             0 crond
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  474]     0   474    27509       32      13        0             0 agetty
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  482]     0   482    27509       30      10        0             0 agetty
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  499]     0   499    28219     3124      54        0             0 dhclient
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  733]     0   733   138288     2664      88        0             0 tuned
Mar 24 14:40:04 ip-172-31-8-83 kernel: [  736]     0   736    56068      482      37        0             0 rsyslogd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 1670]     0  1670    22472      281      42        0             0 master
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 1699]    89  1699    22515      276      45        0             0 qmgr
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2077]     0  2077    20631      217      39        0         -1000 sshd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2130]    27  2130   280031    53066     167        0             0 mysqld
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2333]     0  2333    26971       24       7        0             0 rhnsd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2397]     0  2397    35749      320      72        0             0 sshd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2400]  1000  2400    35787      348      70        0             0 sshd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2401]  1000  2401    13225      190      29        0             0 sftp-server
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2410]     0  2410    35749      319      72        0             0 sshd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2413]  1000  2413    35770      331      69        0             0 sshd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2414]  1000  2414    28846       94      14        0             0 bash
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2433]     0  2433    48346      173      49        0             0 sudo
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2434]     0  2434    46918      152      48        0             0 su
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2435]     0  2435    28878      123      14        0             0 bash
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2533]     0  2533    94123     1575     180        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2534]    48  2534    98410     5599     184        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2535]    48  2535    97125     4817     181        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2538]    48  2538    97641     4819     182        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2888]  1001  2888    28281       52      14        0             0 standalone.sh
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 2952]  1001  2952   665416   106289     289        0             0 java
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3088]    48  3088    97260     5185     182        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3115]    48  3115    98612     5715     185        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3169]    89  3169    22498      273      43        0             0 pickup
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3230]    48  3230    98419     5602     185        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3231]    48  3231    95656     3494     179        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3232]    48  3232    97320     5048     182        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3233]    48  3233    94517     2289     176        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3235]    48  3235    96425     4303     181        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3246]    48  3246    94157     1567     175        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3248]     0  3248    94123     1584     170        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3249]     0  3249    94123     1594     170        0             0 httpd
Mar 24 14:40:04 ip-172-31-8-83 kernel: [ 3250]     0  3250    94123     1596     170        0             0 httpd

1 个答案:

答案 0 :(得分:2)

无法使用date +%T,因为命令结果中包含:个字符,如11:12:45,并且在HDFS中无法创建具有:字符的文件名。见Hadoop-3275

在脚本中尝试此命令,

hdfs dfs -put /home/myname/folder1/* /user/myname/example_1_`date +%H%M%S`.txt

这将创建类似/user/myname/example_1_111245.txt的文件名。