Snippets

orbitzN sedlex tokenizing

Created by orbitzN
View revision

File lexer.ml Added

  • Ignore whitespace
  • Hide word diff 
+let rec token buf =
+  let open Token in
+  match%sedlex buf with
+    | "@@" -> Escaped_at::token buf
+    | "@" -> At::token buf
+    | Star (Sub (any, "@")) ->
+      Printf.printf "length = %d\n" (Sedlexing.lexeme_length buf);
+      Printf.printf "array length = %d\n" (Array.length (Sedlexing.lexeme buf));
+      String (Sedlexing.Utf8.lexeme buf)::token buf
+    | eof -> []
+    | _ -> failwith "nyi"
HTTPS SSH

You can clone a snippet to your computer for local editing. Learn more.