seanmcl avatar seanmcl committed 30ab372 Merge

Comments (0)

Files changed (5)

 
 *.old.*
 
+.DS_Store
 *.annot
 *.cmi
 *.cmx
+# -*- mode: org -*-
+#+STARTUP: indent
+
+* Feature requests
+
+* Open
+
+** Omake.next-error is slow on large builds
+*** sweeks
+I am doing a build of our main repo, with about 75k targets
+and 50 errors. Running next-error takes 5-10 seconds to run. I think
+the performance of this used to be OK. I haven't done a large build in
+a while, so I guess this problem was introduced in the last month.
+
+*** sweeks
+I should add that there was no obvious CPU use of emacs or
+omake_server while I was waiting. I never saw their CPU use move about
+2% during the 5-10%.
+
+
+** omake-mode degrading emacs keystroke performance
+*** sweeks
+I have two emacsen running on the same machine, with the only
+difference being that one is connected to omake-server doing a build
+and the other isn't connected to omake server. The visible response of
+keypresses (especially with hold to repeat) is much better in the
+emacs that isn't talking to omake server. If I stop the build, the
+emacs talking to omake server gets better, but it's still worse than a
+clean emacs. If I stop omake server, then the emacs reverts to good
+keypress resposiveness.  I think there's room for improvement in
+interfering less with emacs. I find that the poor responsiveness with
+a build running negatively affects my interaction with emacs.
+
+*** sweeks
+I just compared the keystroke responsiveness of tip to that of
+the most recent roll to test here, 705108ac39fe. Things have gotten
+much worse in the last month. I think this issue needs to be
+addressed. The responsiveness is bad enough that I'm switching back to
+705108ac39fe.
+
+** omake-server spinning at 100%+ CPU
+*** sweeks
+Ron hit a case today in which his omake_server_test.exe was spinning
+using 110% CPU. There was one other omake_server_test.exe process
+running, with arguments "server running", and it was hung not using
+CPU, I believe since the time the server started spinning.  The
+mode-log for the relevant emacs was last updated at the time the
+server started spinning. Here's the tail of the log.
+
+Async   : server ping -counter 511 -pid 29122 -version 13
+Read    : (progn (Omake.Server.logf "Server  : (Omake.Ping.ack 511)") (Omake.Ping.ack 511))\n
+Eval    : (progn (Omake.Server.logf "Server  : (Omake.Ping.ack 511)") (Omake.Ping.ack 511))
+Server  : (Omake.Ping.ack 511)
+Read    : (progn (Omake.Server.logf "Server  : (load \"/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el\" t t t)") (load "/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el" t t t))\n
+Eval    : (progn (Omake.Server.logf "Server  : (load \"/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el\" t t t)") (load "/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el" t t t))
+Server  : (load "/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el" t t t)
+Read    : (progn (Omake.Server.logf "Server  : (load \"/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el\" t t t)") (load "/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el" t t t))\n
+Eval    : (progn (Omake.Server.logf "Server  : (load \"/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el\" t t t)") (load "/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el" t t t))
+Server  : (load "/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el" t t t)
+Here's the tail of the server log.
+
+[2012-07-25 09:48:20.597985] omake server received: (Ping 29122((version 13)(uid 511)))
+[2012-07-25 09:48:20.598030] Writing: (progn (Omake.Server.logf "Server  : (Omake.Ping.ack 511)") (Omake.Ping.ack 511))
+[2012-07-25 09:48:20.714021] Writing: (progn (Omake.Server.logf "Server  : (load \"/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el\" t t t)") (load "/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el" t t t))
+[2012-07-25 09:48:21.717247] Writing: (progn (Omake.Server.logf "Server  : (load \"/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el\" t t t)") (load "/tmp/omake-server/yminsky/mnt/local/sda1/yminsky/jane/live-prod/elisp.el" t t t))
+I did an strace on the spinning omake server, and didn't see much. I did notice that there weren't any calls to select, other than from the OCaml ticker thread. So async cycles weren't running. That seems to point to an infinite loop within an async job as opposed to an infinite loop continually running new jobs.
+
+*** sweeks
+Ron adds that his entire emacs was hung when the problem happened.
+
+*** seanmcl
+It would be cool if there were a way to send the server a
+message saying 'print a stack trace on stdout'.  Where can I learn
+more about the ocaml ticker thread? For example, I don't know how you
+inferred your comment about the select calls.
+
+*** seanmcl
+Interestingly, there are only two 'let rec' calls in the
+entire program. One 'Error.dedup' is obviously terminating by
+induction on the input. The second 'Files.omakeroot_dir' walks up the
+file name for omakeroot_dir. It is less obvious that it terminates
+since it's dealing with a string, and while the string usually gets
+smaller, Filename.dirname "/" == "/". But I check for that case, so I
+don't see any obvious looping.
+
+*** seanmcl
+I'm putting this on hold, since I haven't heard anything in over a month.
+
+*** sweeks
+Sam E caused omake-server to spin using 100% CPU several times today. Interestingly, on the last one, while we were looking into it, omake-server stopped spinning, and appeared to continue functioning normally. It now seems likely to me there is a quadratic (or worse) performance issue in omake server's processing of omake output. Regexps are always a good candidate.
+Here is the tail of the project-log.
+
+[2012-09-06 11:11:44.481373] New output
+[2012-09-06 11:11:44.534868] File_tail warning: /tmp/omake-server/sehrlichman/mnt/local/sda1/sehrlichman/trees/hedger-tools.108.05.00.00/omake reached EOF
+[2012-09-06 11:11:45.034537] File_tail warning: /tmp/omake-server/sehrlichman/mnt/local/sda1/sehrlichman/trees/hedger-tools.108.05.00.00/omake reached EOF
+[2012-09-06 11:11:45.481933] New output
+[2012-09-06 11:11:46.536472] New output
+[2012-09-06 11:11:51.713523] File_tail warning: /tmp/omake-server/sehrlichman/mnt/local/sda1/sehrlichman/trees/hedger-tools.108.05.00.00/omake did not reach EOF for 6.1s
+[2012-09-06 11:11:55.892051] New output
+[2012-09-06 11:11:55.901067] File_tail warning: /tmp/omake-server/sehrlichman/mnt/local/sda1/sehrlichman/trees/hedger-tools.108.05.00.00/omake reached EOF
+[2012-09-06 11:12:14.752257] File_tail warning: /tmp/omake-server/sehrlichman/mnt/local/sda1/sehrlichman/trees/hedger-tools.108.05.00.00/omake did not reach EOF for 7.3s
+[2012-09-06 11:12:32.609426] File_tail warning: /tmp/omake-server/sehrlichman/mnt/local/sda1/sehrlichman/trees/hedger-tools.108.05.00.00/omake did not reach EOF for 25s
+[2012-09-06 11:12:43.124088] File_tail warning: /tmp/omake-server/sehrlichman/mnt/local/sda1/sehrlichman/trees/hedger-tools.108.05.00.00/omake did not reach EOF for 35s
+[2012-09-06 11:12:43.126133] File_tail warning: /tmp/omake-server/sehrlichman/mnt/local/sda1/sehrlichman/trees/hedger-tools.108.05.00.00/omake reached EOF
+[2012-09-06 11:13:07.277727] New output
+[2012-09-06 11:13:08.288586] New output
+The "omake did not reach EOF for" warnings fit with omake server spinning on something computational, preventing the async file-tail job from getting a chance to run.
+
+The raw buffer is small, 137,361 bytes. However, it has some very long lines (10k characters), further pointing at something superlinear in line length like regexp handling. Here are the line lengths:
+
+41
+89
+92
+89
+29
+77
+16
+43
+26
+48
+28
+28
+28
+31
+36
+25
+56
+89
+24
+38
+41
+7594
+115
+114
+110
+188
+115
+114
+110
+109
+115
+114
+110
+123
+32
+43
+47
+43
+9714
+17
+61
+89
+60
+215
+32
+56
+47
+56
+10048
+17
+61
+89
+73
+202
+32
+43
+47
+43
+9996
+17
+61
+89
+60
+208
+32
+49
+47
+49
+10020
+17
+61
+89
+66
+210
+32
+51
+47
+51
+10028
+17
+61
+89
+68
+210
+32
+51
+47
+51
+10028
+17
+61
+89
+68
+268
+45
+82
+213
+46
+57
+38
+57
+53
+57
+10034
+23
+67
+95
+74
+44
+55
+38
+55
+53
+55
+10026
+23
+67
+95
+72
+46
+57
+38
+57
+53
+57
+10034
+23
+67
+95
+74
+38
+49
+38
+49
+53
+49
+10002
+23
+67
+95
+66
+51
+62
+38
+62
+53
+62
+10054
+23
+67
+95
+79
+38
+49
+38
+49
+53
+49
+9720
+23
+67
+95
+66
+41
+All the long lines were of the form:
+
+      + ../../../bin/link-quietly ocamlopt.opt ...
+where the "..." is lots of -I switches and libraries.
+
+I don't see that omake-server even needs to understand such lines, so
+perhaps we could put a simple hack in to just not process them.
+
+*** seanmcl
+Yeah, any line may go through 13 regexp calls, some of which
+have a number of alternations. I'd rather not tinker with Pcre or the
+regular expressions themselves. I like your idea of filtering
+problematic lines. I ran the server on a file with very long lines,
+but it terminated quickly. There must have been some property of Sam's
+particular lines.
+
+** Stopping omake server leaves around omake processes
+*** sweeks
+If I run "controller stop" from the command line, omake server leaves
+around the omake processes that it was managing.
+
+*** seanmcl
+Ug. Right. Maybe service_command could have an argument that runs some
+cleanup code before stopping the daemon. I'll have to dig into the
+commands themselves to get the behavior I want.
+
+*** seanmcl
+Now we use service_command's start, but the old stop which kills the
+children processes. On hold whether we should fix service_command to
+install a signal handler for sigint.
+
+*** sweeks
+I chatted with Nathan about this. He agrees that [Service_command]
+should handle SIGTERM and call [exit] so that [at_exit] handlers can
+run.  He also pointed out that [Service_command] may be going away
+from [Core_extended], being replaced by something that subsumes it,
+but is also more tied to being inside Jane Street.
+
+To make progress now, how about you fix [Service_command] in your
+clone of Core to do this?
+
+** After closing dedicate omake status frame, C-c C-c dies
+*** mml
+with "Wrong type argument: frame-live-p, #<dead frame Emacs: 0x1d869540>
+
+*** seanmcl
+I run into this sometimes in other packages. Emacs gets
+confused somehow by referring to a dead frame. In emergencies you can
+do: run this if you can't kill a buffer due to a dead-frame error
+(setq menu-updating-frame nil) but I'll work on finding why it's
+happening to omake-mode.
+
+** Remove 'watching' a project from user consideration.
+
+*** seanmcl
+Now the only way to kill a project will be from the projects window or
+command line. It will significantly simplify the user interface.
+
+** omake-mode just hangs when omake runs out of memory
+*** seanmcl
+I'm running on a low-memory virtual OS, and while building core I ran
+out of memory. Omake-mode just hung, even though there was a readable
+message in the raw output:
+
+{{{
+:* omake: blocked (25 min 04.50 sec, 10/777 scans, 1387/2133 rules, 1907/7179 digests)
+[===========================================================   ] 03857 / 03954
+                                                                               
+:* omake: targets were not rebuilt because of errors:
+   core_extended-108.07.00/lib/patience_diff.cmx
+      depends on: core_extended-108.07.00/lib/patience_diff.ml
+      - build core_extended-108.07.00/lib patience_diff.cmx
+      + <compute 1 value dependencies>
+      - build core_extended-108.07.00/lib patience_diff.cmx
+      + ocamlopt.opt -I +camlp4 -thread -w @a-4-7-29-28-9 -strict-sequence -annot -inline 20 -nodynlink -g -pp 'camlp4o   ../../lib/pa_type_conv/pa_type_conv.cmo ../../lib/pa_sexp_conv/pa_sexp_conv.cmo ../../lib/pa_bin_prot/pa_bin_prot.cmo [[../../lib/pa_fields_conv/pa_fields_conv.cmo ../../lib/pa_variants_conv/pa_variants_conv.cmo ../../lib/pa_compare/pa_compare.cmo ../../lib/pa_pipebang/pa_pipebang.cmo ../../lib/pa_ounit/pa_ounit.cmo pa_macro.cmo -pa-ounit-ident core_extended-108.07.00/lib/patience_diff.ml' -I ../../lib/bin_prot -I ../../lib/fieldslib -I ../../lib/sexplib -I ../../lib/variantslib -I ../../lib/core -I ../../lib/pcre -I ../../lib/oUnit -I ../../lib/sexplib -for-pack Core_extended -c patience_diff.ml
+      Fatal error: exception Out_of_memory
+      Called from file "arg.ml", line 216, characters 4-32
+      - exit core_extended-108.07.00/lib patience_diff.cmx, 55.11 sec, code 2
+   core_extended-108.07.00/lib/patience_diff.o
+      depends on: core_extended-108.07.00/lib/patience_diff.ml
+   core_extended-108.07.00/lib/procfs.cmx
+      depends on: core_extended-108.07.00/lib/procfs.ml
+      - build core_extended-108.07.00/lib procfs.cmx
+      + <compute 1 value dependencies>
+      - build core_extended-108.07.00/lib procfs.cmx
+      + ocamlopt.opt -I +camlp4 -thread -w @a-4-7-29-28-9 -strict-sequence -annot -inline 20 -nodynlink -g -pp 'camlp4o   ../../lib/pa_type_conv/pa_type_conv.cmo ../../lib/pa_sexp_conv/pa_sexp_conv.cmo ../../lib/pa_bin_prot/pa_bin_prot.cmo ../../lib/pa_fields_conv/pa_fields_conv.cmo ../../lib/pa_variants_conv/pa_variants_conv.cmo ../../lib/pa_compare/pa_compare.cmo ../../lib/pa_pipebang/pa_pipebang.cmo ../../lib/pa_ounit/pa_ounit.cmo pa_macro.cmo -pa-ounit-ident core_extended-108.07.00/lib/procfs.ml' -I ../../lib/bin_prot -I ../../lib/fieldslib -I ../../lib/sexplib -I ../../lib/variantslib -I ../../lib/core -I ../../lib/pcre -I ../../lib/oUnit -I ../../lib/sexplib -for-pack Core_extended -c procfs.ml
+      - exit core_extended-108.07.00/lib procfs.cmx, 2 min 16.04 sec, code 127
+   core_extended-108.07.00/lib/procfs.o
+      depends on: core_extended-108.07.00/lib/procfs.ml
+   core_extended-108.07.00/lib/readline.cmx
+      depends on: core_extended-108.07.00/lib/readline.ml
+   core_extended-108.07.00/lib/readline.o
+      depends on: core_extended-108.07.00/lib/readline.ml
+      - build core_extended-108.07.00/lib readline.o
+      + <compute 1 value dependencies>
+      - build core_extended-108.07.00/lib readline.o
+      + ocamlopt.opt -I +camlp4 -thread -w @a-4-7-29-28-9 -strict-sequence -annot -inline 20 -nodynlink -g -I ../../lib/bin_prot -I ../../lib/fieldslib -I ../../lib/sexplib -I ../../lib/variantslib -I ../../lib/core -I ../../lib/pcre -for-pack Core_extended -c readline.ml
+      - exit core_extended-108.07.00/lib readline.o, 39.92 sec, code 127
+:* omake: polling for filesystem changes
+}}}
+
+* Hold 
+
+* Closed 
+

ocaml/omake/OMakefile

   core_extended
   pcre
   ocaml_inotify
+  ounit
 
 EXES[] =
   omake_server

ocaml/omake/config.ml

   let key = String.lowercase key in
   let value = String.lowercase value in
   T.replace t ~key ~data:value
+

ocaml/omake/regex.ml

 let matches rex s =
   try Pcre.exec_all ~rex s |! Array.to_list with
   | Not_found -> []
+
+TEST_MODULE = struct
+
+  TEST =
+    let rex = of_string "abc" in
+    rewrite rex ~template:"def" "abc" = "def"
+  ;;
+
+end
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.