wait for kube controllers to be ready before starting informers

[deads2k]

Fixes the various controller panic problems.

/assign derekwaynecarr
/assign gnufied

[gnufied]

So As I understand it - the reason we had to make this change is because - in case of openshift (where StartInformers is not nil), we do not want to close the ctx.InformersStarted channel immediately after calling StartInformers because we want control over that in openshift code.

Do we want to backport this to upstream at all?

[deads2k]

So As I understand it - the reason we had to make this change is because - in case of openshift (where StartInformers is not nil), we do not want to close the ctx.InformersStarted channel immediately after calling StartInformers because we want control over that in openshift code.

Correct

Do we want to backport this to upstream at all?

No. In 3.9, we plan to start executing this separately.

[deads2k]

/retest

[deads2k]

/retest

[derekwaynecarr]

/lgtm
/approve no-issue

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, derekwaynecarr

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• pkg/cmd/OWNERS [deads2k,derekwaynecarr]
• vendor/k8s.io/kubernetes/cmd/kube-controller-manager/OWNERS [deads2k,derekwaynecarr]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[deads2k]

/retest

[deads2k]

/retest

[deads2k]

/retest

[deads2k]

@stevekuznetsov are the e2e tests borked?

[deads2k]

/retest

[stevekuznetsov]

/retest

No idea -- router tests seem 100% broken in that last run

[stevekuznetsov]

We're not seeing that in other runs of the job. Could be local to this.

[deads2k]

We're not seeing that in other runs of the job. Could be local to this.

@stevekuznetsov It's happening on every pull to older branches. See #17896 and #17807 and and #17849

[stevekuznetsov]

@deads2k OK -- can you help root-cause it or determine which merge into the branch broke it? Or ask @knobunc to nominate someone from his team to dig?

[deads2k]

@deads2k OK -- can you help root-cause it or determine which merge into the branch broke it? Or ask @knobunc to nominate someone from his team to dig?

It wasn't us. Nothing merged since the last successful merge here (#17690) and that passed while running those tests. We merged zero commits and now it breaks every time. It's something in the infrastructure.

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17690/test_pull_request_origin_end_to_end/6930/

Passed

github.com/openshift/origin/test/end-to-end TestIngressConfiguredRouter
github.com/openshift/origin/test/end-to-end TestRouter
github.com/openshift/origin/test/end-to-end TestRouterBindsPortsAfterSync
github.com/openshift/origin/test/end-to-end TestRouterDuplications
github.com/openshift/origin/test/end-to-end TestRouterHealthzEndpoint
github.com/openshift/origin/test/end-to-end TestRouterPathSpecificity
github.com/openshift/origin/test/end-to-end TestRouterReloadCoalesce
github.com/openshift/origin/test/end-to-end TestRouterServiceUnavailable
github.com/openshift/origin/test/end-to-end TestRouterStatsPort

[deads2k]

@deads2k OK -- can you help root-cause it or determine which merge into the branch broke it? Or ask @knobunc to nominate someone from his team to dig?
It wasn't us. Nothing merged since the last successful merge here (#17690) and that passed while running those tests. We merged zero commits and now it breaks every time. It's something in the infrastructure.

@stevekuznetsov aka "I didn't ask at random"

[stevekuznetsov]

This is probably related to the Go version on the AMI. We have never really had an approach to versioning the AMIs and approaches that would maintain N AMIs for each version we needed were shot down in the distant past. Right now I think we should just ignore these tests for the older branches. In the future we need to think critically about whether or not we can either containerize the test as-is or remove it's dependency on Docker so that it can run wherever.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[soltysh]

/retest

[mfojtik]

/retest

@deads2k this now should work I hope

[deads2k]

/retest

[openshift-merge-robot]

/test all [submit-queue is verifying that this PR is safe to merge]

[deads2k]

/retest

[stevekuznetsov]

There is no point in re-testing this PR. As explained above we do not have support in the Origin tests for backports to a version of Origin that does not use the same Go version as master. Green-button this or try to get the OSE CI to cooperate, as it has that support.

[deads2k]

There is no point in re-testing this PR. As explained above we do not have support in the Origin tests for backports to a version of Origin that does not use the same Go version as master. Green-button this or try to get the OSE CI to cooperate, as it has that support.

We updated the tests. Note the green e2e run.

/retest

[mfojtik]

/retest

[openshift-ci-robot]

@deads2k: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/extended_conformance_install	5f41e97	link	/test extended_conformance_install
ci/openshift-jenkins/end_to_end	5f41e97	link	/test end_to_end

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[deads2k]

This had all tests green. CI seems globally broken now, but it clears a blocker bug. The two other changes that went in are unrelated. Merging.