Question

Text的结构是这样的;

Tag001
 0.1, 0.2, 0.3, 0.4
 0.5, 0.6, 0.7, 0.8
 ...
Tag002
 1.1, 1.2, 1.3, 1.4
 1.5, 1.6, 1.7, 1.8
 ...

文件可以包含任意数量的TagXXX，每个标签可以包含任意数量的CSV值。

==== PPPS。（抱歉这些东西： - ）

更多改进;现在我的原子笔记本电脑上的31842行数据需要1秒左右，这比原始代码快7倍。但是，C版本比这快20倍。

(defn add-parsed-code [accu code]
  (if (empty? code)
    accu
    (conj accu code)))

(defn add-values [code comps]
  (let [values comps
        old-values (:values code)
        new-values (if old-values
                     (conj old-values values)
                     [values])]
    (assoc code :values new-values)))

(defn read-line-components [file]
  (map (fn [line] (clojure.string/split line #","))
       (with-open [rdr (clojure.java.io/reader file)]
         (doall (line-seq rdr)))))

(defn parse-file [file]
  (let [line-comps (read-line-components file)]
    (loop [line-comps line-comps
           accu []
           curr {}]
      (if line-comps
        (let [comps (first line-comps)]
          (if (= (count comps) 1) ;; code line?
            (recur (next line-comps)
                   (add-parsed-code accu curr)
                   {:code (first comps)})
            (recur (next line-comps)
                   accu
                   (add-values curr comps))))
        (add-parsed-code accu curr)))))

==== PPS。

虽然我无法弄清楚为什么第一个比第二个慢10倍，而不是 slurp，map和with-open确实让阅读更快;整个阅读/处理时间不会减少（从7秒到6秒）

(time
 (let [lines (map (fn [line] line)
                  (with-open [rdr (clojure.java.io/reader
                                   "DATA.txt")]
                    (doall (line-seq rdr))))]
   (println (last lines))))

(time (let [lines
            (clojure.string/split-lines
             (slurp "DATA.txt"))]
        (println (last lines))))

==== PS。 Skuro的解决方案确实有效。但是解析速度不是那么快，所以我必须使用基于C的解析器（在1~3秒内读取400个文件，而clojure对于单个文件需要1~4秒;是文件大小相当大）用于读取和仅为统计分析构建DB和clojure。

Answer 1

以下解析上述文件，保持任何值分隔行。如果那不是您想要的，您可以更改add-values功能。解析状态保存在curr变量中，而accu保存先前解析的标记（即在找到“TagXXX”之前出现的所有行）。它允许没有标记的值：

更新：副作用现已封装在专用的load-file函数中

(defn tag? [line]
  (re-matches #"Tag[0-9]*" line))

; potentially unsafe, you might want to change this:
(defn parse-values [line]
  (read-string (str "[" line "]")))

(defn add-parsed-tag [accu tag]
  (if (empty? tag)
      accu
      (conj accu tag)))

(defn add-values [tag line]
  (let [values (parse-values line)
        old-values (:values tag)
        new-values (if old-values
                       (conj old-values values)
                       [values])]
    (assoc tag :values new-values)))

(defn load-file [path]
  (slurp path))

(defn parse-file [file]
  (let [lines (clojure.string/split-lines file)]
    (loop [lines lines ; remaining lines 
           accu []     ; already parsed tags
           curr {}]    ; current tag being parsed
          (if lines
              (let [line (first lines)]
                (if (tag? line)
                    ; we recur after starting a new tag
                    ; if curr is empty we don't add it to the accu (e.g. first iteration)
                    (recur (next lines)
                           (add-parsed-tag accu curr)
                           {:tag line})
                    ; we're parsing values for a currentl tag
                    (recur (next lines)
                           accu
                           (add-values curr line))))
              ; if we were parsing a tag, we need to add it to the final result
              (add-parsed-tag accu curr)))))

我对上面的代码并不十分兴奋，但它确实起了作用。给出如下文件：

Tag001
 0.1, 0.2, 0.3, 0.4
 0.5, 0.6, 0.7, 0.8
Tag002
 1.1, 1.2, 1.3, 1.4
 1.5, 1.6, 1.7, 1.8
Tag003
 1.1, 1.2, 1.3, 1.4
 1.1, 1.2, 1.3, 1.4
 1.5, 1.6, 1.7, 1.8
 1.5, 1.6, 1.7, 1.8

它产生以下结果：

user=> (clojure.pprint/print-table [:tag :values] (parse-file (load-file "tags.txt")))
================================================================
:tag   | :values
================================================================
Tag001 | [[0.1 0.2 0.3 0.4] [0.5 0.6 0.7 0.8]]
Tag002 | [[1.1 1.2 1.3 1.4] [1.5 1.6 1.7 1.8]]
Tag003 | [[1.1 1.2 1.3 1.4] [1.1 1.2 1.3 1.4] [1.5 1.6 1.7 1.8] [1.5 1.6 1.7 1.8]]
================================================================

Answer 2

这可以使用分区依据功能完成。阅读可能有点神秘，但可读性很容易提高。这个功能在我的mini-mac上执行大约500毫秒。

首先，我使用以下函数创建了测试数据。

(defn write-data[fname]
   (with-open [wrtr (clojure.java.io/writer fname) ]
     (dorun 
        (for [ x (take 7500 (range)) ]
          (do
             (.write wrtr (format "Tag%010d" x))
             (.write wrtr "
                            1.1, 1.2, 1.3, 1.4
                            1.1, 1.2, 1.3, 1.4
                            1.5, 1.6, 1.7, 1.8
                            1.5, 1.6, 1.7, 1.8
                           " ))))))

(write-data "my-data.txt")

; "a b c d " will be converted to [ a b c d ]
(defn to-vec[st]
   (load-string (str "[" st "]")))


(defn my-transform[fname]
   (let [tag (atom {:tag nil})]
      (with-open [rdr (clojure.java.io/reader fname)]
         (doall 
           (into {} 
               (map 
                  (fn[xs] {(first xs) (map to-vec (rest xs))}) 
                     ( partition-by 
                          (fn[y] 
                             (if(.startsWith 
                                  (str y) "Tag") 
                                  (swap! tag assoc :tag y) @tag)) 
                       (line-seq rdr))))))))


(time (count (my-transform "my-data.txt")))
;Elapsed time: 517.23 msecs

如何使用Clojure读取/解析以下文本？

2 个答案: