如何在HDFS中删除x天以前的数据

时间:2015-12-01 14:40:03

标签: hadoop hdfs

如何删除超过x天的数据。在linux hdfs非常简单,但hdfs dfs -ls /<path>找不到。

此外,即使Sub SomeSub() Dim StartWord As String, EndWord As String Dim Find1stRange As Range, FindEndRange As Range Dim DelRange As Range, DelStartRange As Range, DelEndRange As Range 'Setting up the Ranges Set Find1stRange = ActiveDocument.Range Set FindEndRange = ActiveDocument.Range Set DelRange = ActiveDocument.Range 'Set your Start and End Find words here to cleanup the script StartWord = "From: Research.TA@traditionanalytics.com" EndWord = "This message has been scanned " 'Starting the Find First Word With Find1stRange.Find .Text = StartWord .Replacement.Text = "" .Forward = True .Wrap = wdFindAsk .Format = False .MatchCase = False .MatchWholeWord = False .MatchWildcards = False .MatchSoundsLike = False .MatchAllWordForms = False 'Execute the Find Do While .Execute 'If Found then do extra script If .Found = True Then 'Setting the Found range to the DelStartRange Set DelStartRange = Find1stRange 'Having these Selections during testing is benificial to test your script DelStartRange.Select 'Setting the FindEndRange up for the remainder of the document form the end of the StartWord FindEndRange.Start = DelStartRange.End FindEndRange.End = ActiveDocument.Content.End 'Having these Selections during testing is benificial to test your script FindEndRange.Select 'Setting the Find to look for the End Word With FindEndRange.Find .Text = EndWord .Execute 'If Found then do extra script If .Found = True Then 'Setting the Found range to the DelEndRange Set DelEndRange = FindEndRange 'Having these Selections during testing is benificial to test your script DelEndRange.Select End If End With 'Selecting the delete range DelRange.Start = DelStartRange.Start DelRange.End = DelEndRange.End 'Having these Selections during testing is benificial to test your script DelRange.Select 'Remove comment to actually delete 'DelRange.Delete End If 'Ending the If Find1stRange .Found = True Loop 'Ending the Do While .Execute Loop End With 'Ending the Find1stRange.Find With Statement End Sub 也会提供未排序的有序数据。

1 个答案:

答案 0 :(得分:0)

以下是执行此操作的代码段:

days_diff=10
now=$(date +%s)
hdfs dfs  -ls /path/todir/ | while read f; do
  dir_date=`echo $f | awk '{print $6}'`
  filename=`echo $f | awk '{print $8}'`
  difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) ))
  if [ $difference -gt $days_diff ]; then
    echo "Deleting file:" $filename ;
    hdfs dfs  -rm $filename;
  fi
done