I want to make some performance measures (mainly runtime) for my Java code, a single-threaded, local, complex algorithm. (So I do not want a macro-benchmark to measure a JVM implementation.)
我想为我的Java代码制定一些性能度量(主要是运行时)，一个单线程的、本地的、复杂的算法。(因此，我不希望使用宏基准来度量JVM实现。)
With the tool, I would like to
有了这个工具，我想
- analyse the complexity, i.e. see how my code scales for a parameter n (the search depth). (I already have a junit test parameterized in n.)
- 分析复杂性，例如，看看我的代码如何缩放参数n(搜索深度)。(我已经在n中使用了一个junit测试参数。)
- do some trend analysis to get warned if some change to the code base makes the code slower.
- 如果对代码库的一些更改导致代码速度变慢，请进行一些趋势分析以得到警告。
For this, I would like to use a tool or framework that
为此，我想使用一个工具或框架
- does the statistics, optimally computing the mean value, standard deviation and confidence intervals. This is very important.
- 统计数据，最优计算平均值，标准差和置信区间。这是非常重要的。
- can be parameterized (see parameter n above). This is also very important.
- 可以参数化(参见上面的参数n)。这也非常重要。
- is able to produce a fancy plot would be nice, but is not required
- 能够产生一个漂亮的情节是不错的，但不是必需的吗?
- can be used in an automated (junit-)test to warn me if my program slows done, but this is also not required, just a plus.
- 可以在自动化(junit-)测试中使用，以警告我程序是否变慢，但这也不是必需的，只是一个附加条件。
What tools/frameworks fulfill these requirements? Which one would be well suited for complexity and trend analysis, and why?
哪些工具/框架满足这些需求?哪一个更适合于复杂性和趋势分析，为什么呢?
3 个解决方案
#1
65
Below is an alphabetical list of all the tools I found. The aspects mentioned are:
下面是我找到的所有工具的字母列表。提到的方面有:
- is it easily parameterizable
- 它是容易可参数化的
- is it a Java library or at least easily integratable into your Java program
- 它是Java库还是至少可以轻松地集成到Java程序中
- can it handle JVM micro benchmarking, e.g. use a warmup phase
- 它能否处理JVM微基准测试，例如使用热身阶段
- can it plot the results visually
- 它能直观地描绘出结果吗?
- can it store the measured values persistently
- 它能持久存储测量值吗?
- can it do trend analysis to warn that a new commit caused a slow down
- 它能做趋势分析来警告一个新的承诺会导致一个慢下来吗
- does it provide and use statistics (at least max, min, average and standard deviation).
- 是否提供和使用统计数据(至少是最大值、最小值、平均值和标准差)。
自动舵
parameterizable; Perl library; no JVM micro benchmarking; plotting; persistence; trend analysis!?; good statistics (run a given test until results stabilize; highlight outliers).
可参数化的;Perl库;没有JVM微基准测试;策划;持久性;趋势分析! ?;良好的统计(运行给定的测试直到结果稳定;突出异常值)。
基准测试框架
not parameterizable; Java library; JVM micro benchmarking; no plotting; no persistence; no trend analysis; statistics.
不是可参数化;Java库;JVM微基准测试;没有策划;没有坚持;没有趋势分析;统计数据。
Does the statistics extremely well: besides average, max, min and standard deviation, it also computes the 95% confidence interval (via bootstrapping) and serial correlation (e.g. to warn about oscillating execution times, which can occur if your program behaves nondeterministically, e.g. because you use HashSets). It decides how often the program has to be iterated to get accurate measurements and interprets these for reporting and warnings (e.g. about outliers and serial correlation).
统计数据做得非常好:除了平均值、最大值、最小值和标准差之外，它还计算95%置信区间(通过自举)和序列相关性(例如，警告执行时间的振荡，如果程序的行为不确定，可能会出现这种情况，例如，因为您使用了HashSets)。它决定需要迭代程序多久才能得到准确的度量，并将它们解释为报告和警告(例如关于异常值和序列相关性)。
Also does the micro-benchmarking extremely well (see Create quick/reliable benchmark with java? for details).
微基准测试也做得非常好(参见使用java创建快速/可靠的基准测试)吗?详情)。
Unfortunately, the framework comes in a util-package bundled together with a lot of other helper-classes. The benchmark classes depend on JSci (A science API for Java) and Mersenne Twister (http://www.cs.gmu.edu/~sean/research/). If the author, Brent Boyer, finds time, he will boil the library down and add a simpler grapher so that the user can visually inspect the measurements, e.g. for correlations and outliers.
不幸的是，这个框架是一个附带了许多其他helper类的util-package。基准类依赖于JSci (Java的科学API)和Mersenne Twister (http://www.cs.gmu.edu/~sean/research/)。如果作者布伦特·博耶(Brent Boyer)找到了时间，他将简化库，增加一个更简单的编写器，以便用户能够直观地检查度量，例如相关性和离群值。
卡尺
parameterizable; Java library; JVM micro benchmarking; plotting; persistence; no trend analysis; statistics.
可参数化的;Java库;JVM微基准测试;策划;持久性;没有趋势分析;统计数据。
Relatively new project, tailored towards Android apps. Looks young but promising. Depends on Google Guava :(
相对较新的项目，适合Android应用。看上去很年轻但有前途。取决于谷歌番石榴:(
共享监控
not parameterizable!?; Java library; no JVM micro benchmarking!?; plotting; persistence through a servlet; no trend analysis!?; no statistics!?.
不是可参数化的! ?Java库;没有JVM微基准测试! ?;策划;通过一个servlet的持久性;没有趋势分析! ?没有统计! ?。
Supports AOP instrumentation.
支持AOP工具。
安达卢
not parameterizable; Java library; no JVM micro benchmarking; plotting, persistence and trend analysis with additional tools (Jarep or JMX); statistics.
不是可参数化;Java库;没有JVM微基准测试;使用其他工具(Jarep或JMX)进行绘图、持久性和趋势分析;统计数据。
Good monitoring, intertwined with log4j, data can also be programmatically accessed or queried and your program can take actions on the results.
良好的监控，与log4j交织在一起，数据也可以通过编程访问或查询，您的程序可以对结果采取行动。
Java西蒙
not parameterizable!?; Java library; no JVM micro benchmarking; plotting only with Jarep; persistence only with JMX; no trend analysis; no statistics!?.
不是可参数化的! ?Java库;没有JVM微基准测试;策划只有Jarep;坚持只与JMX;没有趋势分析;没有统计! ?。
Competitor of Jamon, supports a hierarchy of monitors.
Jamon的竞争对手，支持一个监视器等级。
JETM
not parameterizable; Java library; JVM micro benchmarking; plotting; persistence; no trend analysis; no statistics.
不是可参数化;Java库;JVM微基准测试;策划;持久性;没有趋势分析;没有统计。
Nice lightweight monitoring tool, no dependencies :) Does not offer sufficient statistics (no standard deviation), and extending the plugIn correspondingly looks quite difficult (Aggregators and Aggregates only have fixed getters for min, max and average).
很好的轻量级监控工具，没有依赖项:)没有提供足够的统计数据(没有标准偏差)，相应地扩展插件看起来相当困难(聚合器和聚合器只有固定的最小值、最大值和平均值的getter)。
jmeter
parameterizable!?; java library; no JVM micro benchmarking!?; plotting; persistence; trend analysis!?; statistics!?.
可参数化的! ?。java库;没有JVM微基准测试! ?;策划;持久性;趋势分析! ?;统计! ?。
Good monitoring library that is tailored towards load testing web applications.
为负载测试web应用程序定制的良好监控库。
Java Microbenchmark Harness (jmh)
Java微基准测试工具(jmh)
parametrizable (custom invokers via Java API); Java library; JVM microbenchmarking; no plots; no persistence; no trend analysis; statistics.
参数化(通过Java API自定义调用程序);Java库;JVM微基准测试;没有情节;没有坚持;没有趋势分析;统计数据。
The benchmarking harness built by Oracle's HotSpot experts, thus very suitable for microbenchmarking on HotSpot, used in OpenJDK performance work. Extreme measures are taken to provide the reliable benchmarking environment. Besides human-readable output, jmh provides a Java API to process the results, e.g. for 3rd party plotters and persistence providers.
由Oracle的HotSpot专家构建的基准测试工具，因此非常适合在OpenJDK性能工作中使用的HotSpot的微基准测试。为了提供可靠的基准测试环境，采取了极端的措施。除了人类可读的输出之外，jmh还提供了一个Java API来处理结果，例如针对第三方绘图仪和持久性提供程序。
junit-Benchmarks
parameterizable; Java library; JVM micro benchmarking; plotting; persistence (using CONSOLE, XML or database H2); graphical trend analysis; statistics (max, min, average, standard deviation; but not easily extensible for further statistics).
可参数化的;Java库;JVM微基准测试;策划;持久性(使用控制台、XML或数据库H2);图形化趋势分析;统计量(max, min, average, standard deviation)但是对于进一步的统计数据不容易扩展)。
Simply add a junit-4-rule to your junit tests :)
只需在junit测试中添加junit-4规则:)
junit-Benchmarks is open source, under the Apache 2 licence.
junit- benchmark是开源的，使用Apache 2许可证。
Update: project moved to jmh
更新:项目转移到jmh
junitperf
Mainly for doing trend analysis for performance (with the JUnit test decorator TimedTest) and scalability (with the JUnit test decorator LoadTest).
主要用于对性能进行趋势分析(使用JUnit测试decorator TimedTest)和可伸缩性(使用JUnit测试decorator LoadTest)。
parameterizable; Java library; no JVM micro benchmarking; no plotting; no persistence; no statistics.
可参数化的;Java库;没有JVM微基准测试;没有策划;没有坚持;没有统计。
perf4j
not parameterizable; Java library; no JVM micro benchmarking; plotting; persistence via JMX; trend analysis via a log4j appender; statistics.
不是可参数化;Java库;没有JVM微基准测试;策划;通过JMX持久性;通过log4j appender进行趋势分析;统计数据。
Builds upon a logging framework, can use AOP.
构建在日志框架上，可以使用AOP。
项目百老汇
Very general concept: monitors observe predefined conditions and specify how to react when they are met.
非常一般化的概念:监视器观察预定义的条件，并指定如何在遇到它们时进行响应。
speedy-mcbenchmark
Main focus is on parameterizability: check whether your algorithm scales, i.e. check if it's O(n), O(n log(n)), O(n²)...
主要重点是parameterizability:检查是否你的算法尺度,即检查如果是O(n),O(n log(n)),O(n²)……
java library; JVM micro benchmarking; no plotting; persistence; trend analysis; no statistics.
java库;JVM微基准测试;没有策划;持久性;趋势分析;没有统计。
Grinder
parameterizable; Jython library; no JVM micro benchmarking; plotting; persistence; no trend analysis; no good statistics, but easily extensible.
可参数化的;Jython库;没有JVM微基准测试;策划;持久性;没有趋势分析;没有好的统计数据，但是很容易扩展。
Depends on Jython, HTTPClient, JEditSyntax, ApacheXMLBeans, PicoContainer.
取决于Jython、HTTPClient、JEditSyntax、apachexmlbean、PicoContainer。
TPTP
parameterizable!?; Java tool platform; no JVM micro benchmarking!?; plotting; persistence; graphical trend analysis; no statistics!?
可参数化的! ?。Java工具平台;没有JVM微基准测试! ?;策划;持久性;图形化趋势分析;没有统计!
The Test & Performance Tools Platform is a huge generic and extensible tool platform (based on Eclipse and four EMF models). Hence it is powerful but quite complex, can slow Eclipse down, and extending it for your own needs (e.g. with statistics so that they influence the number of iterations) seems to be very difficult.
测试和性能工具平台是一个巨大的通用和可扩展的工具平台(基于Eclipse和四个EMF模型)。因此，它功能强大但相当复杂，可以降低Eclipse的运行速度，并且根据您自己的需要扩展它(例如，使用统计数据来影响迭代的数量)似乎非常困难。
Usemon
parameterizable!?; Java library; no JVM micro benchmarking; plotting; persistence; trend analysis!?; statistics!?.
可参数化的! ?。Java库;没有JVM微基准测试;策划;持久性;趋势分析! ?;统计! ?。
Tool is tailored towards monitoring in large clusters.
工具是针对大型集群中的监控而定制的。
#2
9
Another alternative is caliper from google. It allows parameterized testing.
另一种选择是谷歌中的caliper。它允许参数化测试。
#3
7
Try using http://labs.carrotsearch.com/junit-benchmarks.html. This is an extention to JUni4, features:
尝试使用http://labs.carrotsearch.com/junit-benchmarks.html。这是JUni4的延伸，特性:
Records execution time average and standard deviation.
Garbage collector activity recording.
Per-benchmark JVM warm-up phase.
Per-run and historical chart generation.
Optional results persistence in the H2 SQL database (advanced querying, historical analysis).记录执行时间平均值和标准偏差。垃圾收集器活动记录。Per-benchmark JVM热身阶段。每次运行和历史图表生成。H2 SQL数据库中的可选结果持久性(高级查询、历史分析)。
#1
65
Below is an alphabetical list of all the tools I found. The aspects mentioned are:
下面是我找到的所有工具的字母列表。提到的方面有:
- is it easily parameterizable
- 它是容易可参数化的
- is it a Java library or at least easily integratable into your Java program
- 它是Java库还是至少可以轻松地集成到Java程序中
- can it handle JVM micro benchmarking, e.g. use a warmup phase
- 它能否处理JVM微基准测试，例如使用热身阶段
- can it plot the results visually
- 它能直观地描绘出结果吗?
- can it store the measured values persistently
- 它能持久存储测量值吗?
- can it do trend analysis to warn that a new commit caused a slow down
- 它能做趋势分析来警告一个新的承诺会导致一个慢下来吗
- does it provide and use statistics (at least max, min, average and standard deviation).
- 是否提供和使用统计数据(至少是最大值、最小值、平均值和标准差)。
自动舵
parameterizable; Perl library; no JVM micro benchmarking; plotting; persistence; trend analysis!?; good statistics (run a given test until results stabilize; highlight outliers).
可参数化的;Perl库;没有JVM微基准测试;策划;持久性;趋势分析! ?;良好的统计(运行给定的测试直到结果稳定;突出异常值)。
基准测试框架
not parameterizable; Java library; JVM micro benchmarking; no plotting; no persistence; no trend analysis; statistics.
不是可参数化;Java库;JVM微基准测试;没有策划;没有坚持;没有趋势分析;统计数据。
Does the statistics extremely well: besides average, max, min and standard deviation, it also computes the 95% confidence interval (via bootstrapping) and serial correlation (e.g. to warn about oscillating execution times, which can occur if your program behaves nondeterministically, e.g. because you use HashSets). It decides how often the program has to be iterated to get accurate measurements and interprets these for reporting and warnings (e.g. about outliers and serial correlation).
统计数据做得非常好:除了平均值、最大值、最小值和标准差之外，它还计算95%置信区间(通过自举)和序列相关性(例如，警告执行时间的振荡，如果程序的行为不确定，可能会出现这种情况，例如，因为您使用了HashSets)。它决定需要迭代程序多久才能得到准确的度量，并将它们解释为报告和警告(例如关于异常值和序列相关性)。
Also does the micro-benchmarking extremely well (see Create quick/reliable benchmark with java? for details).
微基准测试也做得非常好(参见使用java创建快速/可靠的基准测试)吗?详情)。
Unfortunately, the framework comes in a util-package bundled together with a lot of other helper-classes. The benchmark classes depend on JSci (A science API for Java) and Mersenne Twister (http://www.cs.gmu.edu/~sean/research/). If the author, Brent Boyer, finds time, he will boil the library down and add a simpler grapher so that the user can visually inspect the measurements, e.g. for correlations and outliers.
不幸的是，这个框架是一个附带了许多其他helper类的util-package。基准类依赖于JSci (Java的科学API)和Mersenne Twister (http://www.cs.gmu.edu/~sean/research/)。如果作者布伦特·博耶(Brent Boyer)找到了时间，他将简化库，增加一个更简单的编写器，以便用户能够直观地检查度量，例如相关性和离群值。
卡尺
parameterizable; Java library; JVM micro benchmarking; plotting; persistence; no trend analysis; statistics.
可参数化的;Java库;JVM微基准测试;策划;持久性;没有趋势分析;统计数据。
Relatively new project, tailored towards Android apps. Looks young but promising. Depends on Google Guava :(
相对较新的项目，适合Android应用。看上去很年轻但有前途。取决于谷歌番石榴:(
共享监控
not parameterizable!?; Java library; no JVM micro benchmarking!?; plotting; persistence through a servlet; no trend analysis!?; no statistics!?.
不是可参数化的! ?Java库;没有JVM微基准测试! ?;策划;通过一个servlet的持久性;没有趋势分析! ?没有统计! ?。
Supports AOP instrumentation.
支持AOP工具。
安达卢
not parameterizable; Java library; no JVM micro benchmarking; plotting, persistence and trend analysis with additional tools (Jarep or JMX); statistics.
不是可参数化;Java库;没有JVM微基准测试;使用其他工具(Jarep或JMX)进行绘图、持久性和趋势分析;统计数据。
Good monitoring, intertwined with log4j, data can also be programmatically accessed or queried and your program can take actions on the results.
良好的监控，与log4j交织在一起，数据也可以通过编程访问或查询，您的程序可以对结果采取行动。
Java西蒙
not parameterizable!?; Java library; no JVM micro benchmarking; plotting only with Jarep; persistence only with JMX; no trend analysis; no statistics!?.
不是可参数化的! ?Java库;没有JVM微基准测试;策划只有Jarep;坚持只与JMX;没有趋势分析;没有统计! ?。
Competitor of Jamon, supports a hierarchy of monitors.
Jamon的竞争对手，支持一个监视器等级。
JETM
not parameterizable; Java library; JVM micro benchmarking; plotting; persistence; no trend analysis; no statistics.
不是可参数化;Java库;JVM微基准测试;策划;持久性;没有趋势分析;没有统计。
Nice lightweight monitoring tool, no dependencies :) Does not offer sufficient statistics (no standard deviation), and extending the plugIn correspondingly looks quite difficult (Aggregators and Aggregates only have fixed getters for min, max and average).
很好的轻量级监控工具，没有依赖项:)没有提供足够的统计数据(没有标准偏差)，相应地扩展插件看起来相当困难(聚合器和聚合器只有固定的最小值、最大值和平均值的getter)。
jmeter
parameterizable!?; java library; no JVM micro benchmarking!?; plotting; persistence; trend analysis!?; statistics!?.
可参数化的! ?。java库;没有JVM微基准测试! ?;策划;持久性;趋势分析! ?;统计! ?。
Good monitoring library that is tailored towards load testing web applications.
为负载测试web应用程序定制的良好监控库。
Java Microbenchmark Harness (jmh)
Java微基准测试工具(jmh)
parametrizable (custom invokers via Java API); Java library; JVM microbenchmarking; no plots; no persistence; no trend analysis; statistics.
参数化(通过Java API自定义调用程序);Java库;JVM微基准测试;没有情节;没有坚持;没有趋势分析;统计数据。
The benchmarking harness built by Oracle's HotSpot experts, thus very suitable for microbenchmarking on HotSpot, used in OpenJDK performance work. Extreme measures are taken to provide the reliable benchmarking environment. Besides human-readable output, jmh provides a Java API to process the results, e.g. for 3rd party plotters and persistence providers.
由Oracle的HotSpot专家构建的基准测试工具，因此非常适合在OpenJDK性能工作中使用的HotSpot的微基准测试。为了提供可靠的基准测试环境，采取了极端的措施。除了人类可读的输出之外，jmh还提供了一个Java API来处理结果，例如针对第三方绘图仪和持久性提供程序。
junit-Benchmarks
parameterizable; Java library; JVM micro benchmarking; plotting; persistence (using CONSOLE, XML or database H2); graphical trend analysis; statistics (max, min, average, standard deviation; but not easily extensible for further statistics).
可参数化的;Java库;JVM微基准测试;策划;持久性(使用控制台、XML或数据库H2);图形化趋势分析;统计量(max, min, average, standard deviation)但是对于进一步的统计数据不容易扩展)。
Simply add a junit-4-rule to your junit tests :)
只需在junit测试中添加junit-4规则:)
junit-Benchmarks is open source, under the Apache 2 licence.
junit- benchmark是开源的，使用Apache 2许可证。
Update: project moved to jmh
更新:项目转移到jmh
junitperf
Mainly for doing trend analysis for performance (with the JUnit test decorator TimedTest) and scalability (with the JUnit test decorator LoadTest).
主要用于对性能进行趋势分析(使用JUnit测试decorator TimedTest)和可伸缩性(使用JUnit测试decorator LoadTest)。
parameterizable; Java library; no JVM micro benchmarking; no plotting; no persistence; no statistics.
可参数化的;Java库;没有JVM微基准测试;没有策划;没有坚持;没有统计。
perf4j
not parameterizable; Java library; no JVM micro benchmarking; plotting; persistence via JMX; trend analysis via a log4j appender; statistics.
不是可参数化;Java库;没有JVM微基准测试;策划;通过JMX持久性;通过log4j appender进行趋势分析;统计数据。
Builds upon a logging framework, can use AOP.
构建在日志框架上，可以使用AOP。
项目百老汇
Very general concept: monitors observe predefined conditions and specify how to react when they are met.
非常一般化的概念:监视器观察预定义的条件，并指定如何在遇到它们时进行响应。
speedy-mcbenchmark
Main focus is on parameterizability: check whether your algorithm scales, i.e. check if it's O(n), O(n log(n)), O(n²)...
主要重点是parameterizability:检查是否你的算法尺度,即检查如果是O(n),O(n log(n)),O(n²)……
java library; JVM micro benchmarking; no plotting; persistence; trend analysis; no statistics.
java库;JVM微基准测试;没有策划;持久性;趋势分析;没有统计。
Grinder
parameterizable; Jython library; no JVM micro benchmarking; plotting; persistence; no trend analysis; no good statistics, but easily extensible.
可参数化的;Jython库;没有JVM微基准测试;策划;持久性;没有趋势分析;没有好的统计数据，但是很容易扩展。
Depends on Jython, HTTPClient, JEditSyntax, ApacheXMLBeans, PicoContainer.
取决于Jython、HTTPClient、JEditSyntax、apachexmlbean、PicoContainer。
TPTP
parameterizable!?; Java tool platform; no JVM micro benchmarking!?; plotting; persistence; graphical trend analysis; no statistics!?
可参数化的! ?。Java工具平台;没有JVM微基准测试! ?;策划;持久性;图形化趋势分析;没有统计!
The Test & Performance Tools Platform is a huge generic and extensible tool platform (based on Eclipse and four EMF models). Hence it is powerful but quite complex, can slow Eclipse down, and extending it for your own needs (e.g. with statistics so that they influence the number of iterations) seems to be very difficult.
测试和性能工具平台是一个巨大的通用和可扩展的工具平台(基于Eclipse和四个EMF模型)。因此，它功能强大但相当复杂，可以降低Eclipse的运行速度，并且根据您自己的需要扩展它(例如，使用统计数据来影响迭代的数量)似乎非常困难。
Usemon
parameterizable!?; Java library; no JVM micro benchmarking; plotting; persistence; trend analysis!?; statistics!?.
可参数化的! ?。Java库;没有JVM微基准测试;策划;持久性;趋势分析! ?;统计! ?。
Tool is tailored towards monitoring in large clusters.
工具是针对大型集群中的监控而定制的。
#2
9
Another alternative is caliper from google. It allows parameterized testing.
另一种选择是谷歌中的caliper。它允许参数化测试。
#3
7
Try using http://labs.carrotsearch.com/junit-benchmarks.html. This is an extention to JUni4, features:
尝试使用http://labs.carrotsearch.com/junit-benchmarks.html。这是JUni4的延伸，特性:
Records execution time average and standard deviation.
Garbage collector activity recording.
Per-benchmark JVM warm-up phase.
Per-run and historical chart generation.
Optional results persistence in the H2 SQL database (advanced querying, historical analysis).记录执行时间平均值和标准偏差。垃圾收集器活动记录。Per-benchmark JVM热身阶段。每次运行和历史图表生成。H2 SQL数据库中的可选结果持久性(高级查询、历史分析)。