HTTPS SSH

sbt-perf License

sbt-perf is a plugin for the simple-build-tool sbt to measure and compare the memory and runtime (experimental) usage of code units. It is also possible to run assertions on the results of the measurements.

It has a similar interface to unit tests to define measurements. The runtime and memory usage of the measurements are saved in a file called .perf. On subsequent measurements, the runtime is compared to the previous one saved in this file. If they exceed a certain threshold, warnings are shown to raise awareness about probable performance regressions.

Requirements

  • sbt 1.1.5 or newer
  • scala 2.12.6 for sbt plugin
  • scala 2.12.6 for measurements

Installation

Add to your project/plugins.sbt file:

    resolvers += Resolver.bintrayIvyRepo("delors", "sbt-plugins")
    addSbtPlugin("de.opal-project" % "sbt-perf" % "0.2.0-SNAPSHOT")

Note: The plugin injects the library dependency "de.opal-project" % "perf-api_2.12" % sbtPerfVersion to the projects to be able to compile the measurement classes.

Usage

Measurements can be easily defined by the measure() function:

class SomePerformanceMeasurements extends PerfSpec {
    measure("Some measurement") {
        someFunction()
        someOtherFunction() // the memory required by the returned value / object graph is measured
    }
}

Run sbt perf to execute the defined measurements to measure the memory consumption of the code unit. If a certain theshold exceeds, a warning or error will occur and the measurement will fail.

The results are saved in the .perf file in the project directory. Note: This file should be included in the .gitignore since the measurements depends on the machine and JVM (e.g. 32 bit vs. 64 bit) they are running on.

On subsequent runs, the new runtimes and memory consumption is not saved. This is to prevent creeping performance decreases, e.g. only 2% per change. To update the measurement, set updateMeasurments settings key to true or clear the measurements.

// in sbt console
set updateMeasurements in Perf := true

// or in build.sbt
updateMeasurements in Perf := true

Note: If set to true and used with perfOnly com.example.Perf, all measurement but com .example.Perf will be removed!

If the function does not return a Unit, the memory usage of the returned object (graph) is measured as well.

More examples are provided in the examples folder or in the unit tests.

Measure runtime (experimental)

sbt-perf uses a simple time diff mechanism to measure the runtime of a code unit. This is highly unstable and can differ on a large scale depending on the tasks running on the machine. Therefore, runtime measurements are disabled by default. You can enable them for a certain test:

    measure("measure 100ms", measureRuntime = true) {
        Thread.sleep(100)
    }

When activating runtime measurements it behaves the following: If a measurements runtime is 20% slower than the previous run, a warning message is shown. If the runtime difference exceeds 50% the measurement will fail and throw an exception.

Assert measurements

After all measurements for a class are done, it is possible to get the results and perform assertions on them. Adding the AssertResults trait to the measurement class adds a assertResults(measurements: Map[String, MeasurementResult]) callback. Using assert it is possible to perform additional analysis on those results. For example:

class AssertionPerfTrue extends PerfSpec with AssertResults {
    measure("1MB array", measureRuntime = false) {
        new Array[Byte](1024*1024)
    }

    measure("2MB array", measureRuntime = false) {
        new Array[Byte](2*1024*1024)
    }

    override def assertResults(measurements: Map[String, MeasurementResult]) = {
        // 1MB + small overhead for array object
        assert(measurements("1MB array").memoryUsage < 1024*1024 + 100)

        // 1MB array should always be smaller than the 2MB array
        assert(measurements("1MB array").memoryUsage < measurements("2MB array").memoryUsage)
    }
}

Specific measurements can be obtained with measurements("name of measurement").

Tasks

  • perf: Run all measurements defined in the project or subprojects.
  • perfOnly <className>: Run the measurements for the given class only.
  • cleanMeasurements: Delete all measurements. This removes the .perf file.
  • cleanMeasurement <className>: Remove the runtime for all measurements in the given class.

Options

You can configure the measurement thresholds with the keys:

  • runtimeThresholdWarning: Percentage when a warning message is shown. Default: 0.2
  • runtimeThresholdError: Percentage when the measurement fails. Default: 0.5
  • memoryThresholdWarning: Percentage when a warning message is shown. Default: 0.2
  • memoryThresholdError: Percentage when the measurement fails. Default: 0.5
  • runRuntimeMeasurements: Measure the runtime of the measurements. Default: false
  • runMemoryMeasurements: Measure the memory consumption of the measurements. Default: true
  • updateMeasurements: Update the measurements if a measurement file (.perf) exists. Note: If set to true and used with perfOnly, all measurements but the current one will be removed! Default: false
  • forkedJvmOptions: A Seq containing JVM options which will be passed to the JVM that executes the measurements, e.g. Seq("-Xmx3072M"). Default: Seq.empty[String]

Most of this settings you can set per measurement. Just pass the relevant option as a parameter in the measure method, for example:

measure("Sleep 100ms", runtimeWarningThreshold = Some(RuntimeWarningThreshold(0.7))) {
    Thread.sleep(100)
}

How it works

sbt-perf works by injecting a library dependency into the project which provides the PerfSpec trait and the measurement executor code. This enables sbt to compile the classes derived from PerfSpec. To run the measurements, sbt-perf forks a new JVM using scala 2.12 and calls the main method inside the spec sub-project. This is needed because the code inside a sbt plugin the JVM needs a javaagent for jamm to measure the memory.

Build sbt-perf

Follow the steps to compile sbt-perf:

  • Clone repository:
$ git clone https://bitbucket.org/OPAL-Project/sbt-perf.git
  • Build sbt-perf:
$ sbt publishLocal
  • Run tests with:
$ sbt coverage test coverageReport

License

sbt-perf is released under the BSD 2-Clause license.