Ep 033: Cake or Ice Cream? Yes!

Functional Design in Clojure

Nate needs to parse two different errors and takes some time to compose himself.

• Previously, we were able to parse out errors and give the parsing function the ability to search as far into the future as necessary.
• We did this by having the function take a sequence and return a sequence, managed by lazy-seq.
• (01:30) New Problem: We need to correlate two different kinds of errors.
• The developers looked at our list of sprinkle errors and they think that they're caused by the 357 errors.
• They have requested that we look at the entire log and generate a report of 357 and sprinkle errors, so we can tell if they're correlated.
• "When someone says, do I want cake or ice cream, the right answer is: yes, I want both!"
• Before, we were only parsing out a single type of error and summarizing it, but now we need to parse out both types of errors.
• If we try to parse both kinds of errors with the same function, we will quickly get ourselves into nested ifs or maybe an infinite cond. Perhaps a complex state machine with backtracking?
• (05:55) Realization: Each error stands alone. Once you detect the beginning of a sprinkle error, you won't need to look for a 357 error.
• You can take each one in turn.
• (06:30) Solution step 1: What if we had two functions, one for each type of error.
• Each of these functions would take the entire sequence and tell us if there was an error at the beginning.
• Previously, our function both recognized errors and handled the sequence generation. If we pull those apart, we can add parsing for more errors easily.
• Each error parsing function would return nil if no error was found at the head of the sequence.
• (08:46) Solution step 2: Create a function that uses the two detectors to find out what error is at the head of the sequence.
• It takes the sequence, and wraps consecutive calls in an or block.
• The or block will try each one in turn until one matches and then that is the result.
• Each error's parsing is in its own function, and the combining function serves as an inventory.
• (11:35) Solution step 3: Create a lazy sequence that wraps calls to the combined detector function.
• Last week's code has parsing and lazy in one function.
• Now that we've pulled the parsing out, we can use the remaining structure to create our lazy sequence.
• The combined detector function is parse-next, and the function that manages the lazy sequence is parse-all.
• "Now we've fulfilled our obligation to have bike-shedding on naming. Next up, cache consistency. And finally, off-by-one errors."
• The top of parse-all has a call to lazy-seq.
• It will use the result of calling parse-next on the sequence.
  • If it gets something, it will use cons to add that value to the beginning of a recursive call to itself.
  • If it gets nil, it will recursively call itself with the rest of the sequence, thus advancing the parsing forward one step.
• It's not a ton of boilerplate, but it is nice to put all the mechanics of the sequence creation into a function by itself.
• Now we have a heterogeneous sequence of errors, and we can transform it into any report that is useful.
• Each parsing function doesn't need to worry about advancing down the sequence, that is handled by the higher parse-all function.
• Since we have a new lazy sequence, we can take it and make recognizers that take it and generate an even higher level of sequence.
• We ruminate more on higher level data in Episode 020.

Related episodes:

• 020: Data Dessert
• 028: Fail Donut
• 029: Problem Unknown: Log Lines
• 030: Lazy Does It
• 031: Eager Abstraction
• 032: Call Me Lazy

Clojure in this episode:

• seq, cons, rest
• lazy-seq
• or, cond

Code sample from this episode:

(ns devops.week-05  (:require  [devops.week-01 :refer [parse-line]]  [devops.week-02 :refer [process-log]]  [devops.week-03 :refer [sprinkle-errors-by-type]]  ))  (defn parse-sprinkle  [lines]  (let [[first-line second-line] lines  [_whole donut-id] (some->> first-line :log/message (re-matches #"failed to add sprinkle to donut (\d+)"))  [_whole error] (some->> second-line :log/message (re-matches #"sprinkle fail reason: (.*)"))]  (when (and donut-id error)  (merge first-line  {:kind :sprinkle  :sprinkle/donut-id donut-id  :sprinkle/error error}))))  (defn parse-357-error  [lines]  (let [[first-line] lines  [_whole user] (some->> first-line :log/message (re-matches #"transaction failed while updating user ([^:]+): code 357"))]  (when user  (merge first-line  {:kind :code-357  :code-357/user user}))))  (defn parse-next  [lines]  (or (parse-357-error lines)  (parse-sprinkle lines)))  (defn parse-all  [lines]  (lazy-seq  (when (seq lines)  (if-some [found (parse-next lines)]  (cons found (parse-all (rest lines)))  (parse-all (rest lines))))))  (defn kind?  ([kind]  #(kind? % kind))  ([line kind]  (= kind (:kind line))))   (comment  (process-log "sample.log" #(->> % (map parse-line) parse-all (map :kind) doall))  (process-log "sample.log" #(->> % (map parse-line) parse-all (filter (kind? :sprinkle)) sprinkle-errors-by-type))  )

Log file sample:

2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 23948 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: should never happen 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 94238 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: timeout exceeded threshold 2019-05-14 16:48:56 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user sally: code 357 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 24839 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: too many requests 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 19238 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: should never happen 2019-05-14 16:48:57 | process-Poster | INFO | com.donutgram.poster | transaction failed while updating user joe: code 357 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | failed to add sprinkle to donut 50493 2019-05-14 16:48:55 | process-Poster | INFO | com.donutgram.poster | sprinkle fail reason: unknown state