当测试失败时，TravisCI并没有让我的构建失败

I have tests written in JavaScript and I use TravisCI for tests.

我有用JavaScript编写的测试，我使用TravisCI进行测试。

Setup

My package.json is something like:

我的package.json是这样的：

"scripts": {
  "test": "node testsRunner.js"
}

And my .travis.yml is:

我的.travis.yml是：

language: node_js
node_js:
- '0.12.7'

'testsRunner.js' is:

'testsRunner.js'是：

var nodeunit = require('nodeunit');
var path = require('path');

nodeunit.reporters.default.run([
  path.join(__dirname, 'suite1/test.js')
]);

And suite1/test.js finally is:

而suite1 / test.js最终是：

module.exports = {
  setUp: function(callback) {
    // Initialization code...
    callback();
  },

  tearDown: function(callback) {
    // Cleanup...
    callback();
  }, 

  test1: function(test) {
    test.expect(10); // This test expects 10 assertions to be run
    // Doing stuff...
    test.done();
  },

  test2: function(test) {
    test.expect(10); // This test expects 20 assertions to be run
    // Doing stuff...
    test.done();
  }
};

Log from Travis

Here is the build log and test execution from Travis:

以下是Travis的构建日志和测试执行：

Using worker: worker-linux-docker-19fc8ef0.prod.travis-ci.org:travis-linux-6
system_info
Build system information
Build language: node_js
Build image provisioning date and time
Thu Feb  5 15:09:33 UTC 2015
Operating System Details
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.5 LTS
Release:    12.04
Codename:   precise
Linux Version
3.13.0-29-generic
Cookbooks Version
a68419e https://github.com/travis-ci/travis-cookbooks/tree/a68419e
GCC version
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
LLVM version
clang version 3.4 (tags/RELEASE_34/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Pre-installed Ruby versions
ruby-1.9.3-p551
Pre-installed Node.js versions
v0.10.36
Pre-installed Go versions
1.4.1
Redis version
redis-server 2.8.19
riak version
2.0.2
MongoDB version
MongoDB 2.4.12
CouchDB version
couchdb 1.6.1
Neo4j version
1.9.4
RabbitMQ Version
3.4.3
ElasticSearch version
1.4.0
Installed Sphinx versions
2.0.10
2.1.9
2.2.6
Default Sphinx version
2.2.6
Installed Firefox version
firefox 31.0esr
PhantomJS version
1.9.8
ant -version
Apache Ant(TM) version 1.8.2 compiled on December 3 2011
mvn -version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00)
Maven home: /usr/local/maven
Java version: 1.7.0_76, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-oracle/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "3.13.0-29-generic", arch: "amd64", family: "unix"
git.checkout
0.61s$ git clone --depth=50 --branch=master git://github.com/myuser/myproj.git myuser/myproj
Cloning into 'myuser/myproj'...
remote: Counting objects: 1473, done.
remote: Compressing objects: 100% (1053/1053), done.
remote: Total 1473 (delta 300), reused 1401 (delta 250), pack-reused 0
Receiving objects: 100% (1473/1473), 8.93 MiB | 0 bytes/s, done.
Resolving deltas: 100% (300/300), done.
Checking connectivity... done.
$ cd myuser/myproj
$ git checkout -qf d603836a30ea1bc213f7b97682df507c91af8404
This job is running on container-based infrastructure, which does not allow use of 'sudo', setuid and setguid executables.
If you require sudo, add 'sudo: required' to your .travis.yml
See http://docs.travis-ci.com/user/workers/container-based-infrastructure/ for details.
2.99s$ nvm install 0.12.7
######################################################################## 100.0%
Now using node v0.12.7
$ node --version
v0.12.7
$ npm --version
2.11.3
$ nvm --version
0.23.3
install
0.84s$ npm install 
npm WARN package.json myproj@0.1.0 No license field.
0.64s$ npm test
> myproj@0.1.0 test /home/travis/build/myuser/myproj
> node testsRunner.js
test.js
✔ test1
✖ test2
TypeError: Cannot read property...
    at extract (...)
    at Object.at (...)
    at Object.module.exports...
    ... <stacktrace continues>

FAILURES: 1/45 assertions failed (19ms)
The command "npm test" exited with 0.
Done. Your build exited with 0.

The problem

Problem is that test1 succeeds and test2 does not pass. Before test2 threw exceptions and this made Travis report my build as a failure.

问题是test1成功，test2没有通过。在test2抛出异常之前，这使Travis报告我的构建失败。

However now i fixed the test, so test2 does not throw exceptions, but its asserts fail. However Travis is reporting that build as a pass, while it should report it as a failed build.

但是现在我修复了测试，所以test2不会抛出异常，但是它的断言失败了。然而，Travis报告将构建作为传递，而它应该将其报告为失败的构建。

How to handle this?

怎么办呢？

Edit

I have changes my runner like this:

我改变了我的跑步者：

var nodeunit = require('nodeunit');
var path = require('path');

nodeunit.reporters.default.run([
  path.join(__dirname, 'suite1/test.js')
]);
console.log('Done!'); // ADDED THIS LINE

Well, in the log I can see that Done! is displayed before everithing, after that I can see each single test log and stacktraces. Is it possible that these tests are executing async and this is causing the main process to exit with 0 status?

好吧，在日志中我可以看到完成！在进行之前显示，之后我可以看到每个单独的测试日志和堆栈跟踪。是否有可能这些测试正在执行异步，这导致主进程以0状态退出？

Fixed

Following Chris Beck's suggestion I changed the runner like this:

按照Chris Beck的建议，我改变了这样的跑步者：

var nodeunit = require('nodeunit');
var path = require('path');

nodeunit.reporters.default.run([
  path.join(__dirname, 'suite1/test.js')
], null function(data) {
  // `data` has a value if errors occurred
  if (data) { throw 'Error1'; } 
});

2 个解决方案

#1

I have used travis-ci for unit tests.

我用travis-ci进行单元测试。

If you want the test failures to represent a build failure, the command-line utility that invokes the tests needs to report a nonzero error code result (as would be visible in the bash value $? after running the tests.)

如果您希望测试失败表示构建失败，则调用测试的命令行实用程序需要报告非零错误代码结果（在运行测试后可以在bash值$？中看到）。

If your js environment does not cause assert failures to do this, the most simple thing most likely is to write your own assert function which throws an uncaught exception, or, which sets a global variable to indicate failure, and before exist your test runner script checks the global variable and throws an uncaught exception if it is set.

如果你的js环境没有导致断言失败，最简单的事情就是编写你自己的断言函数抛出一个未捕获的异常，或者设置一个全局变量来指示失败，并且在你存在测试运行脚本之前检查全局变量并在设置时抛出未捕获的异常。

(The latter has the advantage that you run all the tests even if some asserts fail, but the disadvantage that if the tests loop infinitely then your build will stall for a while even after an earlier assertion failure.)

（后者的优点是，即使某些断言失败，您也可以运行所有测试，但缺点是如果测试无限循环，那么即使在先前的断言失败之后，您的构建也会暂停一段时间。）

#2

There are usually plugins for unit test integration through JUnit xml. Is this default in travis ci? Are you reporting your tests usin Junit?

通常有通过JUnit xml进行单元测试集成的插件。这是默认的travis ci吗？您是否在Junit上报告您的测试？

nodeunit has a built in junit reporter:

nodeunit有一个内置的junit记者：

junit - Creates jUnit compatible XML reports, which can be used with continuous integration tools such as Hudson.

junit - 创建与jUnit兼容的XML报告，可以与Hudson等持续集成工具一起使用。

#1