Builds with Appium tests pass even if Jasmine tests fail

[janpio]

During development of #178 I noticed, that the tests were passing although there were Jasmine (main) test failures:

https://travis-ci.org/apache/cordova-paramedic/jobs/552245365

Seems it is enough if the Appium tests pass for the whole run to be counted as successful. Not good.

[janpio]

#178 is blocked by this, because it makes the failing tests invisible for runs with Appium.

[janpio]

Does anyone know where exactly the success/failure of a run is defined?

Because I added some output of what I though it was, but that indicates I didn't understand correctly:
https://travis-ci.org/apache/cordova-paramedic/jobs/552280079

cordova-paramedic results (SauceLabs): isTestPassed = false, isAppiumTestPassed = true

https://github.com/apache/cordova-paramedic/pull/178/files#diff-ff902472818c9b031ca73ec778f68526
I thought that last return would bubble up to Travis, but as this would be false here that can't be the case.

[janpio]

ping @alsorokin

[alsorokin]

isTestPassed used to be an indicator of successful test run. Maybe this .catch or .timeout gets in the way?
https://github.com/apache/cordova-paramedic/blob/master/lib/paramedic.js#L95-L100

[janpio]

I don't think so - that all looks perfectly fine.
The problem is the other way around: one of the two TestPassed vars are false, but paramedic still reports success to Travis.

[alsorokin]

Ah, I see.
But also I think it wasn't false but undefined. So there are two problems here, it seems:

• isTestPassed var doesn't catch the result of jasmine test
• whatever the result was, it doesn't get accounted

[janpio]

Why do you think undefined?

I added some output, see the earlier comment: #179 (comment)

whatever the result was, it doesn't get accounted

Yep, that I agree with. I don't understand the code where this gets returned actually. I just can't find out what is executed as the last thing and defines that exit status.

[alsorokin]

Because of the line Completed tests at 23:26:58 with result: undefined

[janpio]

Oh, I didn't notice that. Looking into it.

[janpio]

Strange, this doesn't seem to influence the result:
https://travis-ci.org/apache/cordova-paramedic/jobs/552280069 vs. https://travis-ci.org/apache/cordova-paramedic/jobs/552280077

failing build:

cordova-paramedic results (SauceLabs): isTestPassed = false, isAppiumTestPassed = true
Completed tests at 23:21:35 with result: undefined

(wrongly) succeeding build:

cordova-paramedic results (SauceLabs): isTestPassed = false, isAppiumTestPassed = true
Completed tests at 23:26:49 with result: undefined

[janpio]

I notice that the wrongly succeeding ones mess up on "Getting saucelabs jobs details..." - there is nothing output for some reason. That generally happens when the Appium tests are actually run.

[alsorokin]

It seems paramedic could use some more refactoring ;)

[janpio]

I don't disagree - more or less what we are doing right now.
Problem is understanding what is actually going on in the current code.

Like this:

cordova-paramedic results (local): isTestPassed = true, isAppiumTestPassed = true: result = true
---------------------------------------------------------
6. Collect data and clean up
---------------------------------------------------------
Completed tests at 8:48:44 AM with result: undefined

So that fin is definitely not getting the return value (

cordova-paramedic/lib/paramedic.js

Lines 101 to 104 in 084dd2b

	.fin((result) => {
	isTestPassed = result;
	logger.normal('Completed tests at ' + (new Date()).toLocaleTimeString() + ' with result: ' + isTestPassed);

)

---

But: Then this also can't be where the actual return status code of the whole paramedic run is set anyway.