Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WFLY-19221] Incorporate channel metadata in the download zips #17936

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

jmesnil
Copy link
Member

@jmesnil jmesnil commented Jun 6, 2024

  • WildFly full distribution is built using the wildfly-maven-plugin based on the org.wildfly.channels:wildfly channel.
  • WildFly ee distribution is built using the wildfly-maven-plugin based on the org.wildfly.channels:wildfly-ee channel.
  • WildFly Preview distribution is built based on the org.wildfly.channels:wildfly-preview channel

preview/dist and dist poms depends on the channels artifacts to ensure that they are build first so that the distributions can use them during provisioning.

This is required for wildfly/wildfly-proposals#577

JIRA: https://issues.redhat.com/browse/WFLY-19221

@jmesnil
Copy link
Member Author

jmesnil commented Jun 6, 2024

I manually verified that the new distributions content is identical to the existing ones with additional metadata in .installation:

$ diff -r ~/tmp/wildfly-33.0.0.Beta1-SNAPSHOT-galleon-maven-plugin/ ~/tmp/wildfly-33.0.0.Beta1-SNAPSHOT-wildfly-maven-plugin/
Only in /Users/jmesnil/tmp/wildfly-33.0.0.Beta1-SNAPSHOT-wildfly-maven-plugin/.installation: README.txt
Only in /Users/jmesnil/tmp/wildfly-33.0.0.Beta1-SNAPSHOT-wildfly-maven-plugin/.installation: installer-channels.yaml
Only in /Users/jmesnil/tmp/wildfly-33.0.0.Beta1-SNAPSHOT-wildfly-maven-plugin/.installation: manifest.yaml
Only in /Users/jmesnil/tmp/wildfly-33.0.0.Beta1-SNAPSHOT-wildfly-maven-plugin/.installation: manifest_version.yaml
Only in /Users/jmesnil/tmp/wildfly-33.0.0.Beta1-SNAPSHOT-wildfly-maven-plugin/.installation: provisioning_record.xml

@github-actions github-actions bot added the deps-ok Dependencies have been checked, and there are no significant changes label Jun 6, 2024
@jmesnil
Copy link
Member Author

jmesnil commented Jun 6, 2024

I also updated the docs module to provision with the WildFly Maven plugin for consistency sake.

@bstansberry I did not update the ee-dist distributions that we are not publishing but for consistency, I can do it if you want.

@jmesnil jmesnil requested a review from bstansberry June 6, 2024 12:22
Copy link
Contributor

@bstansberry bstansberry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmesnil Thanks! Please have a look at a3b0ccd -- there's some stuff in there that should be in here:

  1. Consideration for the external.channels.profile. That profile isn't meant to impact the 'dist' modules, as it's just for provisioning used by the testsuite. TS doesn't test the 'dists'.

  2. It uses wildfly-maven-plugin's overwrite-provisioned-server setting, and gets rid of some maven-clean-plugin cruft that's not needed when that's used.

Also, please do deal with ee-dist as well. We do distribute those, just not via the wildfly.org/downloads or the Github release page:

https://repository.jboss.org/org/wildfly/wildfly-ee-dist/32.0.1.Final/

This code will also become downstream code so we should do things consistently so we can consistently adapt to downstream use.

dist/pom.xml Outdated Show resolved Hide resolved
galleon-pack/channel/pom.xml Outdated Show resolved Hide resolved
@jmesnil
Copy link
Member Author

jmesnil commented Jun 10, 2024

@bstansberry Thanks for the review, I incorporated your changes in a3b0ccd (including ee-dist and the manual mode test)

@jmesnil jmesnil requested a review from bstansberry June 10, 2024 12:52
@jmesnil
Copy link
Member Author

jmesnil commented Jun 10, 2024

@spyrkob Something I'm not sure about... The current distribution is provisioned with the universe's feature pack locations for our feature packs:

cat ./.installation/provisioning_record.xml
<?xml version="1.0" ?>

<installation xmlns="urn:jboss:galleon:provisioning:3.0">
    <transitive>
        <feature-pack location="wildfly-ee@maven(org.jboss.universe:community-universe):current#33.0.0.Beta1-SNAPSHOT">
            <packages>
                <include name="docs.examples.configs"/>
            </packages>
        </feature-pack>
    </transitive>
    <feature-pack location="wildfly@maven(org.jboss.universe:community-universe):current#33.0.0.Beta1-SNAPSHOT"/>
    <options>
        <option name="jboss-fork-embedded" value="true"/>
    </options>
</installation>

(this file is identical to .galleon/provisioning.xml)

When I try to update it with Prospero, it fails with the error:

$ ./prospero update perform -vv --debug  --dir=/Users/jmesnil/Developer/wildfly/dist/target/wildfly-33.0.0.Beta1-SNAPSHOT --use-default-local-cache

Updating server: /Users/jmesnil/Developer/wildfly/dist/target/wildfly-33.0.0.Beta1-SNAPSHOT

18:06:06,122 INFO  [prospero] PRSP000012: Checking available updates
18:06:06,122 INFO  [prospero] PRSP000012: Checking available updates
Jun 10, 2024 6:06:06 PM java.lang.System$LoggerFinder lambda$accessProvider$0
WARNING: Failed to instantiate LoggerFinder provider; Using default.


ERROR: org.jboss.galleon.universe.maven.MavenUniverseException: Failed to resolve Maven universe org.jboss.universe:community-universe

org.jboss.galleon.ProvisioningException: org.jboss.galleon.universe.maven.MavenUniverseException: Failed to resolve Maven universe org.jboss.universe:community-universe
        at org.jboss.galleon.impl.ProvisioningUtil.getCoreVersion(ProvisioningUtil.java:113)
        at org.jboss.galleon.impl.ProvisioningUtil.getCoreVersion(ProvisioningUtil.java:89)
        at org.jboss.galleon.api.GalleonBuilder.newProvisioningBuilder(GalleonBuilder.java:101)
        at org.wildfly.prospero.galleon.GalleonUtils.newProvisioning(GalleonUtils.java:197)
        at org.wildfly.prospero.galleon.GalleonEnvironment.<init>(GalleonEnvironment.java:140)
        at org.wildfly.prospero.galleon.GalleonEnvironment$Builder.build(GalleonEnvironment.java:328)
        at org.wildfly.prospero.actions.UpdateAction.getGalleonEnv(UpdateAction.java:157)
        at org.wildfly.prospero.actions.UpdateAction.findUpdates(UpdateAction.java:143)
        at org.wildfly.prospero.cli.commands.UpdateCommand.buildUpdate(UpdateCommand.java:434)
        at org.wildfly.prospero.cli.commands.UpdateCommand$PerformCommand.performUpdate(UpdateCommand.java:136)
        at org.wildfly.prospero.cli.commands.UpdateCommand$PerformCommand.call(UpdateCommand.java:122)
        at org.wildfly.prospero.cli.commands.UpdateCommand$PerformCommand.call(UpdateCommand.java:83)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2045)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
        at picocli.CommandLine.execute(CommandLine.java:2174)
        at org.wildfly.prospero.cli.CliMain.execute(CliMain.java:132)
        at org.wildfly.prospero.cli.CliMain.main(CliMain.java:60)
Caused by: org.jboss.galleon.universe.maven.MavenUniverseException: Failed to resolve Maven universe org.jboss.universe:community-universe
        at org.jboss.galleon.universe.maven.MavenUniverseFactory.getUniverse(MavenUniverseFactory.java:65)
        at org.jboss.galleon.universe.UniverseFactoryLoader.getUniverse(UniverseFactoryLoader.java:110)
        at org.jboss.galleon.universe.UniverseResolver.getUniverse(UniverseResolver.java:87)
        at org.jboss.galleon.universe.UniverseResolver.getUniverse(UniverseResolver.java:81)
        at org.jboss.galleon.universe.UniverseResolver.resolve(UniverseResolver.java:127)
        at org.jboss.galleon.impl.ProvisioningUtil.getCoreVersion(ProvisioningUtil.java:101)
        ... 21 more
Caused by: org.jboss.galleon.universe.maven.MavenUniverseException: Can not resolve latest Maven artifact (no stream found) : org.jboss.universe:community-universe:jar:
        at org.wildfly.prospero.galleon.ChannelMavenArtifactRepositoryManager.resolve(ChannelMavenArtifactRepositoryManager.java:79)
        at org.wildfly.prospero.galleon.ChannelMavenArtifactRepositoryManager.resolveLatestVersion(ChannelMavenArtifactRepositoryManager.java:306)
        at org.jboss.galleon.universe.maven.repo.MavenRepoManager.resolveLatestVersion(MavenRepoManager.java:68)
        at org.jboss.galleon.universe.maven.MavenUniverse.resolveUniverseArtifact(MavenUniverse.java:64)
        at org.jboss.galleon.universe.maven.MavenUniverse.<init>(MavenUniverse.java:101)
        at org.jboss.galleon.universe.maven.MavenUniverseFactory.getUniverse(MavenUniverseFactory.java:63)
        ... 26 more
Caused by: org.wildfly.channel.NoStreamFoundException: Can not resolve latest Maven artifact (no stream found) : org.jboss.universe:community-universe:jar:
        at org.wildfly.channel.ChannelSession.lambda$findChannelWithLatestVersion$5(ChannelSession.java:282)
        at java.base/java.util.Optional.orElseThrow(Optional.java:408)
        at org.wildfly.channel.ChannelSession.findChannelWithLatestVersion(ChannelSession.java:274)
        at org.wildfly.channel.ChannelSession.resolveMavenArtifact(ChannelSession.java:118)
        at org.wildfly.prospero.galleon.ChannelMavenArtifactRepositoryManager.resolve(ChannelMavenArtifactRepositoryManager.java:73)
        ... 31 more

Prospero is looking for a steam inside the channel to get the version of org.jboss.universe:community-universe:jar but this GAV is not part of the server and will not be in the channel.

I tried to generate the distributions by using Maven GAVs for the feature packs (eg org.wildfly:widfly-galleon-pack:33.0.0.Beta1-SNAPSHOT instead of their universe feature pack locations (wildfly@maven(org.jboss.universe:community-universe):current#33.0.0.Beta1-SNAPSHOT) and Prospero was able to succeeds:

$ cat .installation/provisioning_record.xml
<?xml version="1.0" ?>

<installation xmlns="urn:jboss:galleon:provisioning:3.0">
    <transitive>
        <feature-pack location="org.wildfly:wildfly-ee-galleon-pack:33.0.0.Beta1-SNAPSHOT">
            <packages>
                <include name="docs.examples.configs"/>
            </packages>
        </feature-pack>
    </transitive>
    <feature-pack location="org.wildfly:wildfly-galleon-pack:33.0.0.Beta1-SNAPSHOT"/>
    <options>
        <option name="jboss-fork-embedded" value="true"/>
    </options>
</installation>

./prospero update perform -vv --debug  --dir=/Users/jmesnil/Developer/wildfly/dist/target/wildfly-33.0.0.Beta1-SNAPSHOT --use-default-local-cache
Updating server: /Users/jmesnil/Developer/wildfly/dist/target/wildfly-33.0.0.Beta1-SNAPSHOT

18:15:40,430 INFO  [prospero] PRSP000012: Checking available updates
18:15:40,430 INFO  [prospero] PRSP000012: Checking available updates
Jun 10, 2024 6:15:40 PM java.lang.System$LoggerFinder lambda$accessProvider$0
WARNING: Failed to instantiate LoggerFinder provider; Using default.
18:15:43,842 INFO  [prospero] PRSP000013: Found 0 updates
18:15:43,842 INFO  [prospero] PRSP000013: Found 0 updates
No updates found.
Operation completed in 3.64 seconds.

@spyrkob Should Prospero be able to work with universe FPL? In my case, I only produced snapshots of WildFly in my local repo cache so I'm not sure if that's a correct setup.

@spyrkob
Copy link
Contributor

spyrkob commented Jun 11, 2024

@jmesnil we should probably find a way to support the FPL in Prospero. The easiest approach would be to add the community-universe and wildfly-producers to the WildFly channel, but like you said they're not part of the server and don't really belong there. Maybe we can create a universe channel that the WildFly channel would depend on?

@jmesnil
Copy link
Member Author

jmesnil commented Jun 12, 2024

I see 3 possible solutions:

  1. Do not use Galleon FPL when we store the provisioning info and record instead the Maven artefact GAV directly. There is a galleon options for this (store-input-provisioning-config)
  2. Add a stream in WildFly channels for the the org.jboss.universe:community-universe artifact (the latest release is 1.2.0.Final and was done 3 years ago so that seems a low maintenance solution)
  3. Make Prospero "understand" the maven(org.jboss.universe:community-universe) FPL out of the box (like the Galleon CLI tool is doing)

@jfdenise As you filed GAL-341, what do you think we should use to store the provisioning metadata in WildFly.
We need to have a solution that would not break any usage of galleon.sh.

@jmesnil jmesnil force-pushed the WFLY-19221_channel_metadata_in_distributions branch from 16b9af0 to 5a50e06 Compare June 13, 2024 09:06
@jfdenise
Copy link
Contributor

@jmesnil , you should not have the transitive feature-pack wildfly-ee, the package 'docs.examples.configs' should be directly resolved from wildfly feature-pack. In addition you should remove the feature-pack versions from the plugin configuration. This versionshould not be set in a channel context.

galleon.sh update expects the usage of its own channels: current, 32.0, ... in the form of wildfly@maven(org.jboss.universe:community-universe):current and doesn't work well with GAV

If we want an installation to be updated from the 2 different tools, we should have the community universe in the channel. As you said that is not a big deal. The channel contains feature-pack, galleon-plugins that are not part of the server either, just used at provisioning time.

@jmesnil
Copy link
Member Author

jmesnil commented Jun 13, 2024

you should not have the transitive feature-pack wildfly-ee, the package 'docs.examples.configs' should be directly resolved from wildfly feature-pack. In addition you should remove the feature-pack versions from the plugin configuration.

The .galleon/provisioning.xml is identical to the one generated by the galleon-maven-plugin. The changes you suggest should be done in our existing setup, right?

@jfdenise
Copy link
Contributor

@jmesnil yes.

@jmesnil jmesnil force-pushed the WFLY-19221_channel_metadata_in_distributions branch 2 times, most recently from 49c04a0 to f0d9c3b Compare August 30, 2024 14:45
@jmesnil jmesnil force-pushed the WFLY-19221_channel_metadata_in_distributions branch 2 times, most recently from 9bb4026 to bc966a6 Compare September 3, 2024 12:32
@wildfly-bot wildfly-bot bot added the rebase-this PR has a merge conflict. label Sep 6, 2024
@jmesnil jmesnil force-pushed the WFLY-19221_channel_metadata_in_distributions branch from 7ec4680 to 4c20f1f Compare September 10, 2024 11:32
* WildFly full distribution is built using the wildfly-maven-plugin based
  on the org.wildfly.channels:wildfly channel.
* WildFly ee distribution is built using the wildfly-maven-plugin based
  on the org.wildfly.channels:wildfly channel-ee.
* WildFly Preview distribution is built based on the
  org.wildfly.channels:wildfly-preview channel
* Generate docs from a server provisioned with wildfly-maven-plugin

preview/dist, ee-dist and dist poms depends on the channels artifacts to ensure that they
are build first so that the distributions can use them during provisioning.

JIRA: https://issues.redhat.com/browse/WFLY-19221

Signed-off-by: Jeff Mesnil <[email protected]>
so that they are in the generated manifest for wildfly, wildfly-ee & wildfly-preview channels.

Signed-off-by: Jeff Mesnil <[email protected]>
Add a <name> parameter to the wildfly-maven-plugin configuration
so that the channels that are stored in the metadata installation are
named.

Signed-off-by: Jeff Mesnil <[email protected]>
@jmesnil jmesnil force-pushed the WFLY-19221_channel_metadata_in_distributions branch from 4c20f1f to df0aa83 Compare September 10, 2024 13:28
@wildfly-bot wildfly-bot bot removed the rebase-this PR has a merge conflict. label Sep 10, 2024
<exists>true</exists>
</file>
<file>
<location>target/${server.output.dir.prefix}-${server.output.dir.version}/.installation/manifest_version.yaml</location>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note, manifest_version.yaml and provisioning_record.xml are both optional files, not sure if they need to be verified.

@@ -545,6 +545,8 @@
<version.org.jboss.spec.jakarta.el.jboss-el-api_5.0_spec>4.0.1.Final</version.org.jboss.spec.jakarta.el.jboss-el-api_5.0_spec>
<version.org.jboss.spec.jakarta.xml.soap.saaj-api_3.0_spec>1.0.0.Final</version.org.jboss.spec.jakarta.xml.soap.saaj-api_3.0_spec>
<version.org.jboss.spec.jakarta.xml.ws.api_4.0_spec>1.0.0.Final</version.org.jboss.spec.jakarta.xml.ws.api_4.0_spec>
<version.org.jboss.universe.community>1.2.0.Final</version.org.jboss.universe.community>
<version.org.jboss.universe.producer.wildfly>1.3.11.Final</version.org.jboss.universe.producer.wildfly>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if that matters, but are the producer artifacts updated before or after WildFly release?

For example before WildFly 33.0.0.Final is released, do we need to release the producer artifact including the Galleon channel 33.0 so that we can include it in the wildfly channel?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deps-ok Dependencies have been checked, and there are no significant changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants