DjangoCon 2011 Notes / docs / pdx_python_meetup_sept_11.rst

PDX Python Meetup

Delicious data with mmstats

Problem:

  • You have an app.
  • It has state.
  • What is my app doing?
  • How do you inspect for that state?
  • Simple in-memory stats get hard to expose in multi-process environments.

Solution #1: Logging


Pros

  • Universally supported
  • Easily enhanced
  • Persistent

Cons

  • Libraries suck
  • Operational burden (rotating, shipping, routing, etc)
  • Records events more than inspects state
  • Difficult to predict where needed
  • Performance impact if too verbose

Solution #2: Graphite


Pros

  • Fast, sexy, "enterprise"
  • Python

Cons

  • Do you want a graph or a graph?
  • Have fun installing it.
  • Still not great for introspection.

Solution #3: socketconsole


Pros:

  • Pure Python.

  • Very useful for deadlocks, blocking code, threaded app.

  • Simple to integrate:

    import socketconsole
    socketconsole.launch()
    

Cons:

  • CPython only.
  • Doesn't work with gevent or eventlet monkeypatching.
  • Doesn't work with greenthreads.
  • Limited functionality.
  • All the fun of Python threads.

Solution #4: REPL Backdoors


Pros:

  • Pure Python!
  • Changing code at runtime is for winners.
  • Inspect all the things!

Cons:

  • With great power comes great responsibility.
  • Requires threads or event loop.
  • Still can't reach all state.

Solution #5: GDB


Pros:

  • Well, you wanted introspection.
  • Has some Python helpers: pygdb, gdb-heap

Cons:

  • Seriously? Definitily only a a last resort.

Solution #6: JMX


Pros:

  • Universally supported.
  • Powerful, extensible, 2-way.
  • Helpful tools.

Cons:

  • Where "universal" means "runs on the JVM".
  • Not as easy to monitor as you'd think.

Note

Python needs this.

mmstats Goals


  • Simple API to expose state.
  • Separate publishing from reading, aggregating, etc.
  • Language, platform, framework agnostic.
  • Minimal & predictable performance impact.
  • Optional persistence (e.g. post-mortems)
  • 1-way (for now?)

What is mmstats?


  • mmap alows sharing memory between processes.

  • Language independent data structure:

    • Series of fields (structs)
    • Fields have label, type, and values.
  • Exposed in Python app as a model class.

Performance Implementation


Single writer, multi-reader

  • No locks.
  • No syscalls (write, read, send, recv, etc).
  • All in userspace for readers & writers.
  • Reading has no impact on writers.
  • Fixed field sizes.
Q: How are reads consistent without locks?
A: Double buffering.
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.