Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WFLY-17986] Support external client access to EAP deployments on OpenShift oer http #523

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
234 changes: 234 additions & 0 deletions ejb/WFLY-17986.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
= Support external EJB client access to EAP deployments on OpenShift over http
:author: Richard Achmatowicz
:email: [email protected]
:toc: left
:icons: font
:idprefix:
:idseparator: -

== Overview
Existing EAP customers use the EJB client library to access EJB-based deployments (SFSBs, SLSBs, MDBs) on EAP
on bare metal and such access needs to be provided whether the EAP deployment is on bare metal or on OpenShift.

This analysis document aims to highlight the issues and concerns in providing EJB client access to EJB-based
deployments on EAP in OpenShift.

=== Client location and protocol
Generally, an EJB client application may be internal or external to the OpenShift cluster, and may use any one of
three transport protocols supported by the EJB client: remote, remote+http, or http (or their encrypted variants).

This issue specifically focuses on the case where the EJB client application is:

* external to the OpenShift cluster
* the transport protocol is http (or its encrypted variant https)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Tomasso
The first link describes the two options available for looking up beans in a scenario where one OpenShift EAP deployment (represented by a cluster) needs to communicate with another OpenShift EAP deployment (represented by another cluster), and so both clusters are deployed on OpenShift. This is really an issue for EAP7-2063.
So I would say the type of client envisaged for EAP7-2062 was a standalone client only and not a client in an EAP deployment on a cluster external to the OpenShift cluster.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, that client should be able to lookup the proxy for the EJB bean in the OpenShift EAP deployment either by using JNDI/HTTP or by creating the proxy programmatically.

For the other location / protocol combinations, please see the related issues
(https://issues.redhat.com/browse/EAP7-2063[EAP7-2063], https://issues.redhat.com/browse/EAP7-2064[EAP7-2064],
https://issues.redhat.com/browse/EAP7-2065[EAP7-2065])

=== Access to cluster
One issue of relevance here is how an external EJB client application gains entry to the OpenShift cluster.

NOTE: We sometimes use the word `cluster` to refer to the OpenShift cluster in which the EAP application is
deployed; at other times, to refer to the set of Pods which make up the EAP deployment when the
HA (high availability) server profile is used. We will try to refer either to the OpenShift cluster or
the EAP cluster to avoid ambiguity.

OpenShift provides a limited set of entry points for external applications to gain access to the cluster. For binary
protocols, NodePorts are used to gain access. For protocols based on HTTP, routes or ingress controllers
are available. A route is a per-deployment load balancer which is accessible external to the cluster.
An ingress controller is a load balancer external to the cluster which can be shared by multiple deployments.

Copy link

@tommaso-borgato tommaso-borgato Jul 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/7.4/html/getting_started_with_jboss_eap_for_openshift_container_platform/eap-operator-for-automating-application-deployment-on-openshift_default#jakarta-enterprise-beans-remoting-on-openshift_default we can find details about "headless service" created by the EAP Operator which, is not exposed outside of the cluster and, hence, not relevant for EAP7-2062;

On the other hand, the EAP Operator creates a Route like e.g.:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  annotations:
    openshift.io/host.generated: 'true'
    wildfly.org/wildfly-server-generation: '1'
  resourceVersion: '27583543'
  name: eap-server-route
  uid: 4613b2f8-7ad6-4db7-8338-1109de1c1630
  creationTimestamp: '2023-07-13T10:24:49Z'
  managedFields:
    - manager: openshift-router
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2023-07-13T10:24:49Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:ingress': {}
      subresource: status
    - manager: wildfly-operator
      operation: Update
      apiVersion: route.openshift.io/v1
      time: '2023-07-13T10:24:49Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:wildfly.org/wildfly-server-generation': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:app.openshift.io/runtime': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"1d9d5b27-aa2b-4c9d-b267-cfdfc0c1ebf5"}': {}
        'f:spec':
          'f:port':
            .: {}
            'f:targetPort': {}
          'f:to':
            'f:kind': {}
            'f:name': {}
            'f:weight': {}
          'f:wildcardPolicy': {}
  namespace: <OPENSHIFT_NAMESPACE>
  ownerReferences:
    - apiVersion: wildfly.org/v1alpha1
      kind: WildFlyServer
      name: eap-server
      uid: 1d9d5b27-aa2b-4c9d-b267-cfdfc0c1ebf5
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/managed-by: eap-operator
    app.kubernetes.io/name: eap-server
    app.openshift.io/runtime: eap
spec:
  host: eap-server-route-<OPENSHIFT_NAMESPACE>.apps.<OPENSHIFT_CLUSTER>
  to:
    kind: Service
    name: eap-server-loadbalancer
    weight: 100
  port:
    targetPort: http
  wildcardPolicy: None
status:
  ingress:
    - host: eap-server-route-<OPENSHIFT_NAMESPACE>.apps.<OPENSHIFT_CLUSTER>
      routerName: default
      conditions:
        - type: Admitted
          status: 'True'
          lastTransitionTime: '2023-07-13T10:24:49Z'
      wildcardPolicy: None
      routerCanonicalHostname: router-default.apps.<OPENSHIFT_CLUSTER>

This Route is the only entry point to the EAP Cluster for clients outside the OpenShift cluster;

For this reason, unless we plan to ask for modifications to the EAP Operator, we should proceed relying on this route for EAP7-2062;

This Route, in turn, forwards request to the following service:

kind: Service
apiVersion: v1
metadata:
  annotations:
    wildfly.org/wildfly-server-generation: '1'
  resourceVersion: '27583531'
  name: eap-server-loadbalancer
  uid: d37c718f-45a0-4743-a0b5-511492800e63
  creationTimestamp: '2023-07-13T10:24:49Z'
  managedFields:
    - manager: wildfly-operator
      operation: Update
      apiVersion: v1
      time: '2023-07-13T10:24:49Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            .: {}
            'f:wildfly.org/wildfly-server-generation': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:app.openshift.io/runtime': {}
            'f:wildfly.org/operated-by-headless': {}
            'f:wildfly.org/operated-by-loadbalancer': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"1d9d5b27-aa2b-4c9d-b267-cfdfc0c1ebf5"}': {}
        'f:spec':
          'f:internalTrafficPolicy': {}
          'f:ports':
            .: {}
            'k:{"port":8080,"protocol":"TCP"}':
              .: {}
              'f:name': {}
              'f:port': {}
              'f:protocol': {}
              'f:targetPort': {}
          'f:selector': {}
          'f:sessionAffinity': {}
          'f:type': {}
  namespace: <OPENSHIFT_NAMESPACE>
  ownerReferences:
    - apiVersion: wildfly.org/v1alpha1
      kind: WildFlyServer
      name: eap-server
      uid: 1d9d5b27-aa2b-4c9d-b267-cfdfc0c1ebf5
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/managed-by: eap-operator
    app.kubernetes.io/name: eap-server
    app.openshift.io/runtime: eap
    wildfly.org/operated-by-headless: active
    wildfly.org/operated-by-loadbalancer: active
spec:
  clusterIP: 172.122.69.0
  ipFamilies:
    - IPv4
  ports:
    - name: http
      protocol: TCP
      port: 8080
      targetPort: 8080
  internalTrafficPolicy: Cluster
  clusterIPs:
    - 172.122.69.0
  type: ClusterIP
  ipFamilyPolicy: SingleStack
  sessionAffinity: None
  selector:
    app.kubernetes.io/managed-by: eap-operator
    app.kubernetes.io/name: eap-server
    app.openshift.io/runtime: eap
    wildfly.org/operated-by-headless: active
    wildfly.org/operated-by-loadbalancer: active
status:
  loadBalancer: {}

This Service, is backed by an endpoint like the following, supposing we have a 6 nodes cluster:

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2023-07-13T10:25:29Z"
  creationTimestamp: "2023-07-13T10:24:49Z"
  labels:
    app.kubernetes.io/managed-by: eap-operator
    app.kubernetes.io/name: eap-server
    app.openshift.io/runtime: eap
    wildfly.org/operated-by-headless: active
    wildfly.org/operated-by-loadbalancer: active
  name: eap-server-loadbalancer
  namespace: <OPENSHIFT_NAMESPACE>
  resourceVersion: "27583968"
  uid: 50196306-a098-44a6-8492-518b46c3d7b9
subsets:
- addresses:
  - ip: 10.128.3.95
    nodeName: tborgato-yhvd-wsjm5-worker-0-xkd9q
    targetRef:
      kind: Pod
      name: eap-server-2
      namespace: <OPENSHIFT_NAMESPACE>
      uid: c90369a8-d65a-4d22-9a50-e2cd2d5c0dfe
  - ip: 10.128.3.96
    nodeName: tborgato-yhvd-wsjm5-worker-0-xkd9q
    targetRef:
      kind: Pod
      name: eap-server-5
      namespace: <OPENSHIFT_NAMESPACE>
      uid: 09bfa6b6-46ad-4253-8ec5-b7898900030a
  - ip: 10.129.3.109
    nodeName: tborgato-yhvd-wsjm5-worker-0-jg72t
    targetRef:
      kind: Pod
      name: eap-server-1
      namespace: <OPENSHIFT_NAMESPACE>
      uid: e30ebbb5-472c-459f-ad42-87da30c4f45d
  - ip: 10.129.3.110
    nodeName: tborgato-yhvd-wsjm5-worker-0-jg72t
    targetRef:
      kind: Pod
      name: eap-server-4
      namespace: <OPENSHIFT_NAMESPACE>
      uid: cefa4e48-75d1-451f-a468-ccbad745c275
  - ip: 10.131.1.149
    nodeName: tborgato-yhvd-wsjm5-worker-0-jqklq
    targetRef:
      kind: Pod
      name: eap-server-0
      namespace: <OPENSHIFT_NAMESPACE>
      uid: 324c1425-dbe1-407e-a421-0d986ba3bf27
  - ip: 10.131.1.150
    nodeName: tborgato-yhvd-wsjm5-worker-0-jqklq
    targetRef:
      kind: Pod
      name: eap-server-3
      namespace: <OPENSHIFT_NAMESPACE>
      uid: c3cf24c4-877c-47db-921a-801511c77c97
  ports:
  - name: http
    port: 8080
    protocol: TCP

If you think this is correct, could you please incorporate this info into this AD?

Next step could be giving instructions on how-to setup the EJB call, e.g. how-to set the java.naming.provider.url using that Route (e.g. http://eap-server-route-<OPENSHIFT_NAMESPACE>.apps.<OPENSHIFT_CLUSTER>/)

Copy link

@tommaso-borgato tommaso-borgato Jul 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I light of our latest call, and the changes @rhusar is making to the Operator, we might want elaborate a little on this topic: IIUC, @rhusar is going to modify the load balancer used and will replace the Route with an Ingress

NOTE: External access points for a deployment may be set up automatically (as in the case of a `route`,
when the application is deployed using the WildFly operator) or may need to be set up manually.

This implies that, for the case of an external EJB client using http, client access to the OpenShift cluster must be
via a route or ingress controller configured for the EAP deployment in some way. This fits well with the fact that
in the case of the http protocol, the EJB client application delegates load balancing and fail-over to a load balancer.

=== Singleton or clustered deployment
Another issue concerns whether the nature of the EAP deployment itself.
A EAP deployment in OpenShift may be deployed using a non-HA server profile, or using an HA server profile. In the
Copy link
Contributor Author

@rachmatowicz rachmatowicz May 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarification required: can a non-HA WildFly deployment nstalled by the WilfFly operator be scaled to multiple(non-clustered) replicas? Or will it always be deployed as clustered?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @rachmatowicz , yes, the operator will scale up/down your application image. If your application image is not configured as a HA, for example, you have trimmed it and it doesn't configure JGroups, your deployment will be scaled up and you will get a bunch of nodes that will not form a cluster.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yersan thanks for confirming!

former case, the deployment will run on one Pod in the OpenShift cluster. In the latter case, the deployment will
run on multiple Pods in the OpenShift cluster. In both cases, the external entrypoint to the cluster will be a route
or an ingress controller endpoint.

=== Features of the EJB client
The main purpose of the EJB client library is to allow a client application to discover and interact with EJBs
contained within an EJB-based application deployment on EAP: the client obtains a proxy for the specific bean
in the deployment it wants to interact with, and uses the proxy to make invocations on the bean.
However, over and above this basic invocation mechanism, the EJB client provides a number of important features,
depending on the type of deployment.

The key features for both HA and non-HA deployments include:

* lookup via JNDI or programmatic creation of proxies for EJBs in a deployment
* client-specific security and discovery configuration
* transactional scoping of invocations on EJBs in a deployment

Additional features for HA deployments:

* load balancing - requests to SLSBs can be load balanced across the Pods in the EAP deployment's cluster
* affinity management - requests to SFSBs will be routed to the specific Pod in the EAP cluster on which the session
was initially created; in other words, the client proxy for the SFSB will have an affinity to a particular backend Pod
and that affinity will be maintained
* failover - requests to SFSBs will be retried on a different Pod in the EAP cluster if the Pod processing a request
fails, and the affinity will be updated

A major part of this issue is to ensure that these key features work in OpenShift as they do in bare metal.

=== The WildFly operator
The OpenShift environment differs from a bare metal environment in many key ways; for example, in OpenShift, Pod IP
addresses and storage volume references are transient, whereas on bare metal they are persistent; in OpenShift,
Pods which fail can be restarted automatically, whereas on bare metal, they are under the control of a system
administrator; and in OpenShift, the number of instances of Pods backing a service can be scaled up or scaled down
automatically. In other words, the OpenShift environment is more dynamic than a corresponding bare metal environment.

Unless otherwise constrained, it would be very difficult for an EJB client application to interact with an EAP
deployment in OpenShift if stricter guarantees were not imposed. For example, discovery of the members of a
EAP cluster and affinity management of SFSBs would be next to impossible without an assumption persistent Pod
IP addresses.

The `WildFly operator` can be used to deploy an EAP application in OpenShift and provide the following guarantees:

* persistent IP addresses for Pods
Copy link

@tommaso-borgato tommaso-borgato Jun 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think here the correct statement is "persistent name for Pods"

basically what you get on a Pod named wildfly-build-from-server-0 with:

sh-4.4$ hostname
wildfly-build-from-server-0

If you try and scale up and down deployments, you can see Pods IPs always change whenever a Pod is destroyed and then re-created (this was observed on an OpenShift 4.13 cluster using EAP Operator 2.4.0 - but that would most probably be the same with other cluster/operator versions):

scale to 3:

wildfly-build-from-server-0 Pod IP: 10.129.3.31 Host IP: 172.215.0.255
wildfly-build-from-server-1 Pod IP: 10.131.0.9 Host IP: 172.215.1.38
wildfly-build-from-server-2 Pod IP: 10.129.3.32 Host IP: 172.215.0.255

scale to 1 and then back to 3:

wildfly-build-from-server-0 Pod IP: 10.129.3.31 Host IP: 172.215.0.255 <-- This POD wasn't stopped and IP stays the same
wildfly-build-from-server-1 Pod IP: 10.131.0.10 Host IP: 172.215.1.38 <-- This POD was destroyed and recreated: IP changes
wildfly-build-from-server-2 Pod IP: 10.131.0.11 Host IP: 172.215.1.38 <-- This POD was destroyed and recreated: IP changes

scale to 0 and then to 4:

wildfly-build-from-server-0 Pod IP: 10.129.3.34 Host IP: 172.215.0.255 <-- This POD was destroyed and recreated: IP changes
wildfly-build-from-server-1 Pod IP: 10.131.0.12 Host IP: 172.215.1.38 <-- This POD was destroyed and recreated: IP changes
wildfly-build-from-server-2 Pod IP: 10.129.3.35 Host IP: 172.215.0.255 <-- This POD was destroyed and recreated: IP changes
wildfly-build-from-server-3 Pod IP: 10.131.0.13 Host IP: 172.215.1.38

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good point. Thanks for providing proof. The wording needs to be changed. I'm wondering how this feature/ guarantee is described in the documentation for stateful sets.

* persistent storage references for storage used by EAP
* scaling up or scaling down of services which are transaction-aware

In what follows, we assume that deployments are made using the WildFly operator and that the EJB client may depend
on these additional guarantees. For more information on the WildFly operator, see
https://operatorhub.io/operator/WildFly[WildFly Operator on operatorhub.io]

=== Scope
As mentioned earlier, the scope of this issue is limited to `external` EJB clients using the `http` protocol to

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rachmatowicz where are we going to tests HELM based deployments? in some different RFE?

access `WildFly-operator-based` EAP deployments on OpenShift. Within this limited use case scenario, the aim is to
provide the same set of features to the application programmer on OpenShift as are available on bare metal.

== Issue Metadata

=== Issue

* https://issues.redhat.com/browse/WFLY-17986[WFLY-17986]

=== Related Issues

* https://issues.redhat.com/browse/EAP7-2062[EAP7-2062]

=== Dev Contacts

* mailto:{email}[{author}]

=== QE Contacts

=== Testing By
* [ ] Engineering

* [x] QE

=== Affected Projects or Components

* https://github.com/WildFly/WildFly[WildFly]
* https://github.com/WildFly/WildFly-http-client[WildFly HTTP Client]

=== Other Interested Projects

* https://github.com/WildFly-security/WildFly-elytron[WildFly Elytron]

=== Relevant Installation Types
// Remove the x next to the relevant field if the feature in question is not relevant
// to that kind of WildFly installation
* [x] Traditional standalone server (unzipped or provisioned by Galleon)

* [ ] Managed domain

* [x] OpenShift s2i

* [x] Bootable jar

== Requirements
The requirements for this issue center around the key features of the EJB client mentioned previously.
We assume that the deployments are deployed using the WildFly operator.

=== Hard Requirements

* the key features of the EJB client should work on OpenShift as they do on bare metal, those features being:
** lookup via JNDI or programmatic creation of proxies for EJBs in a deployment (non-HA and HA)
** client-specific security and discovery configuration (non-HA and HA)
** transactional scoping of invocations on EJBs in a deployment (non-HA and HA)
** load balancing (HA only)
** affinity management (HA only)
** failover (HA only)

* the affinity management feature should work with all load balancers supported by OpenShift; this includes:
Copy link

@tommaso-borgato tommaso-borgato Jun 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the mechanism that is going to provide such a feature? does it already exist? is it cookie base affinity?

Copy link
Contributor Author

@rachmatowicz rachmatowicz Jun 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using the EJB client with protocol http, the wildfly-http-client library is used to provide the transport of invocations from the client to the server. This transport has a client side part and a server side part and they work together to make sure that HTTP requests have their affinity managed correctly. The current implementation is broken and the issue is addressed by WEJBHTTP-81.

** httpd-based load balancers, which append affinity metadata onto the JSESSIONID cookie value (JSESSIONID + route)
** non-httpd-based load balancers (like HAProxy and IS) which use a separate cookie to store affinity metadata

=== Nice-to-Have Requirements
NONE

=== Non-Requirements
NONE

== Backwards Compatibility
Backward compatibility with previous versions of WildFly is not a valid consideration for two reasons:

* the behaviour of EJB client over HTTP with a load balancer on bare metal is known to be incorrect
(see https://issues.redhat.com/browse/WEJBHTTP-81[WEJBHTTP-81])
* the behaviour of EJB client over HTTP to access WildFly deployments on OpenShift has never been tested

=== Default Configuration
The configuration of the EJB client is provided by WildFly-config.xml on the client application classpath.
This client configuration file allows configuring various aspects of the EJB client behaviour, such as
authentication, discovery, initial targets for the http client, XNIO-related properties, etc.
It may be required to adjust this configuration file to allow specifying affinity management properties to
the http-client configuration.

=== Importing Existing Configuration
N/A

=== Deployments

Deployments must make use of the WildFly operator.

=== Interoperability
N/A

== Implementation Plan
There are a number of issues with the WildFly HTTP client which need to be fixed before this issue can be resolved.
The issues are:

* https://issues.redhat.com/browse/WEJBHTTP-81[WEJBHTTP-81]
* https://issues.redhat.com/browse/WEJBHTTP-103[WEJBHTTP-103]
* https://issues.redhat.com/browse/WEJBHTTP-110[WEJBHTTP-110]
* https://issues.redhat.com/browse/WFLY-17577[WFLY-17577]

== Security Considerations
There are two aspects to consider here:

* security of access between EJB client and EAP server (or EAP cluster)
* security of access to the OpenShift cluster

The EJB client can be configured for authentication and encryption by way of the WildFly-client.xml configuration
file, thus providing configurable security in the communication pathways between client and server.

Security of access to the OpenShift cluster would be the responsibility of the OpenShift administrator, and it is
unclear if this touches on the configuration of the client.

== Test Plan
TODO

== Community Documentation
This issue will require community documentation, at a minimum to explain how to use the EJB client to access
WildFly-operator-based deployments on OpenShift.

== Release Note Content
TODO
////
Draft verbiage for up to a few sentences on the feature for inclusion in the
Release Note blog article for the release that first includes this feature.
Example article: http://WildFly.org/news/2018/08/30/WildFly14-Final-Released/.
This content will be edited, so there is no need to make it perfect or discuss
what release it appears in. "See Overview" is acceptable if the overview is
suitable. For simple features best covered as an item in a bullet-point list
of features containing a few words on each, use "Bullet point: <The few words>"
////