Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for authenticated package sources #24

Closed
tintoy opened this issue Feb 9, 2021 · 21 comments
Closed

Add support for authenticated package sources #24

tintoy opened this issue Feb 9, 2021 · 21 comments

Comments

@tintoy
Copy link
Owner

tintoy commented Feb 9, 2021

Needs latest (v5) NuGet.Client libraries and configuration for NuGet's ICredentialService.

Relates to tintoy/msbuild-project-tools-vscode#39 and #11.

tintoy added a commit that referenced this issue Feb 23, 2021
(while updating NuGet package references for a project)

#24
tintoy added a commit to tintoy/msbuild-project-tools-vscode that referenced this issue Feb 23, 2021
@TripleEmcoder
Copy link

@tintoy thank all for all of your work on this useful extension. Do you have any idea when you might find the time to finish and merge this feature?

@tintoy
Copy link
Owner Author

tintoy commented Sep 19, 2021

Hi :)

Sorry, I'd completely forgotten about this issue - I'll take a look tomorrow and see what I can do.

@tintoy
Copy link
Owner Author

tintoy commented Sep 20, 2021

Hi. I've built a preview version of the extension with this functionality; can you try installing it from the VSIX package below and see if it works for you? You'll need to uninstall the existing extension first if you installed it from the gallery.

msbuild-project-tools-0.4.0-preview4.zip

@TripleEmcoder
Copy link

Thank you. I installed the preview build and now there aren't any exceptions related to authentication, however I do get this one:

NuGet.Protocol.Core.Types.FatalProtocolException: The source does not have a Search service!
   at async Task<IEnumerable<string>> NuGet.Protocol.AutoCompleteResourceV3.IdStartsWith(string packageIdPrefix, bool includePrerelease, ILogger log, CancellationToken token)
   at async Task<SortedSet<string>> MSBuildProjectTools.LanguageServer.Utilities.NuGetHelper.SuggestPackageIds(IEnumerable<AutoCompleteResource> autoCompleteResources, string packageIdPrefix, bool includePrerelease, ILogger logger, CancellationToken cancellationToken) in C:\Development\github\tintoy\msbuild-project-tools-vscode\lib\server\src\LanguageServer.Common\Utilities\NuGetHelper.cs:line 139
...

My sources are (redacted DevOps project name):

Both trigger the same exception when isolated in NuGet.config. So it seems there is something different about Azure DevOps Artifacts repositories. One suggestion would be to catch the exception and skip the offending source, as this also breaks nuget.org completion. However a true fix would be ideal, if you have any ideas what the reason here could be.

@tintoy
Copy link
Owner Author

tintoy commented Sep 24, 2021

It sounds like Azure DevOps package repositories don’t fully implement the v3 protocol (e.g. the search facility). I imagine that their main use case could be installing packages with known Ids not finding package Ids by partial match…?

I think we can have it skip Azure DevOps sources (or have a setting representing a list of package sources never to offer completion from).

Will see what I can do 🙂

@tintoy
Copy link
Owner Author

tintoy commented Sep 24, 2021

(part of the v3 protocol is discovery of endpoints within the API and their functionality - if the client library is throwing that exception it means that the search endpoint is not present)

@tintoy
Copy link
Owner Author

tintoy commented Sep 24, 2021

@TripleEmcoder
Copy link

TripleEmcoder commented Sep 24, 2021

It sounds like Azure DevOps package repositories don’t fully implement the v3 protocol (e.g. the search facility). I imagine that their main use case could be installing packages with known Ids not finding package Ids by partial match…?

Not really - those source work normally in Visual Studio's Package Manager, with search and all. Maybe check how the requests differ between VS and NuGet client libraries with Fiddler? Although I would think they share code.

@tintoy
Copy link
Owner Author

tintoy commented Sep 24, 2021

Interesting - they don’t seem to play nice with the offical NuGet client libraries though. Well, the version I’m using anyway. I’ll try with the latest version later this morning - I wouldn’t be surprised if they have special-case code in there somewhere.

@tintoy
Copy link
Owner Author

tintoy commented Sep 24, 2021

Can you try creating a .net core console project with this code and running it?

https://gist.github.com/tintoy/3d9467d59b2547ad227ba5d1560ee838

I figure that may save us a bunch of time figuring out what the exact error you're seeing really is.

@tintoy
Copy link
Owner Author

tintoy commented Sep 24, 2021

Never mind - it turns out that was a complete waste of time:

https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Protocol/Resources/AutoCompleteResourceV3.cs#L55

😂😂😂

@tintoy
Copy link
Owner Author

tintoy commented Sep 24, 2021

@TripleEmcoder
Copy link

TripleEmcoder commented Sep 25, 2021

Not much more output, am I missing some logging configuration?

[Program/INF] Found 2 configured package sources.
[NuGet/INF]   CACHE https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json
[NuGet/INF]   CACHE https://api.nuget.org/v3/index.json
[Program/INF] Requesting suggestions for package prefix Microsoft.AspNetCore from package source Azure SDK ("https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json")...
[Program/ERR] Unexpected error.
NuGet.Protocol.Core.Types.FatalProtocolException: The source does not have a Search service!
   at async Task<IEnumerable<string>> NuGet.Protocol.AutoCompleteResourceV3.IdStartsWith(string packageIdPrefix, bool includePrerelease, ILogger log, 
CancellationToken token)
   at async Task NuGetFeedLister.Program.Main() in E:\Marcin\Repos\3d9467d59b2547ad227ba5d1560ee838-826a4a2916f9e2b90fb8fabd8ef968c5a700bf24\Program.cs:line 62

That's from dotnet run after downloading the entire gist as zip and copying NuGet.config over.

@tintoy
Copy link
Owner Author

tintoy commented Sep 25, 2021

Hmm - that cache business is interesting.

I think, for now, I’ll add a package source ignore list to the extension settings so at least you won’t be blocked entirely and then do some experimentation around upgrading the NuGet client libraries when I get some time (experience has taught me that these upgrades don’t always go smoothly).

@tintoy
Copy link
Owner Author

tintoy commented Sep 26, 2021

Ok, can you try updating Program.cs from the gist linked above?

I've modified it to dump out the service index for each configured package source (this will tell us what capabilities the package source has; the service we need is SearchAutocompleteService). Here's example output from nuget.org:

[Program/INF] Service index for package source nuget ("https://api.nuget.org/v3/index.json"):
[Program/INF]   SearchQueryService (v0.0.0) => "https://azuresearch-usnc.nuget.org/query"
[Program/INF]   SearchQueryService (v0.0.0) => "https://azuresearch-ussc.nuget.org/query"
[Program/INF]   SearchAutocompleteService (v0.0.0) => "https://azuresearch-usnc.nuget.org/autocomplete"
[Program/INF]   SearchAutocompleteService (v0.0.0) => "https://azuresearch-ussc.nuget.org/autocomplete"
[Program/INF]   SearchGalleryQueryService/3.0.0-rc (v0.0.0) => "https://azuresearch-usnc.nuget.org/"
[Program/INF]   SearchGalleryQueryService/3.0.0-rc (v0.0.0) => "https://azuresearch-ussc.nuget.org/"
[Program/INF]   RegistrationsBaseUrl (v0.0.0) => "https://api.nuget.org/v3/registration5-semver1/"
[Program/INF]   PackageBaseAddress/3.0.0 (v0.0.0) => "https://api.nuget.org/v3-flatcontainer/"
[Program/INF]   LegacyGallery (v0.0.0) => "https://www.nuget.org/api/v2"
[Program/INF]   LegacyGallery/2.0.0 (v0.0.0) => "https://www.nuget.org/api/v2"
[Program/INF]   PackagePublish/2.0.0 (v0.0.0) => "https://www.nuget.org/api/v2/package"
[Program/INF]   SymbolPackagePublish/4.9.0 (v0.0.0) => "https://www.nuget.org/api/v2/symbolpackage"
[Program/INF]   SearchQueryService/3.0.0-rc (v0.0.0) => "https://azuresearch-usnc.nuget.org/query"
[Program/INF]   SearchQueryService/3.0.0-rc (v0.0.0) => "https://azuresearch-ussc.nuget.org/query"
[Program/INF]   SearchQueryService/3.5.0 (v0.0.0) => "https://azuresearch-usnc.nuget.org/query"
[Program/INF]   SearchQueryService/3.5.0 (v0.0.0) => "https://azuresearch-ussc.nuget.org/query"
[Program/INF]   SearchAutocompleteService/3.0.0-rc (v0.0.0) => "https://azuresearch-usnc.nuget.org/autocomplete"
[Program/INF]   SearchAutocompleteService/3.0.0-rc (v0.0.0) => "https://azuresearch-ussc.nuget.org/autocomplete"
[Program/INF]   SearchAutocompleteService/3.5.0 (v0.0.0) => "https://azuresearch-usnc.nuget.org/autocomplete"
[Program/INF]   SearchAutocompleteService/3.5.0 (v0.0.0) => "https://azuresearch-ussc.nuget.org/autocomplete"
[Program/INF]   RegistrationsBaseUrl/3.0.0-rc (v0.0.0) => "https://api.nuget.org/v3/registration5-semver1/"
[Program/INF]   ReportAbuseUriTemplate/3.0.0-rc (v0.0.0) => "https://www.nuget.org/packages/{id}/{version}/ReportAbuse"
[Program/INF]   PackageDisplayMetadataUriTemplate/3.0.0-rc (v0.0.0) => "https://api.nuget.org/v3/registration5-semver1/{id-lower}/index.json"
[Program/INF]   PackageVersionDisplayMetadataUriTemplate/3.0.0-rc (v0.0.0) => "https://api.nuget.org/v3/registration5-semver1/{id-lower}/{version-lower}.json"
[Program/INF]   SearchQueryService/3.0.0-beta (v0.0.0) => "https://azuresearch-usnc.nuget.org/query"
[Program/INF]   SearchQueryService/3.0.0-beta (v0.0.0) => "https://azuresearch-ussc.nuget.org/query"
[Program/INF]   SearchAutocompleteService/3.0.0-beta (v0.0.0) => "https://azuresearch-usnc.nuget.org/autocomplete"
[Program/INF]   SearchAutocompleteService/3.0.0-beta (v0.0.0) => "https://azuresearch-ussc.nuget.org/autocomplete"
[Program/INF]   RegistrationsBaseUrl/3.0.0-beta (v0.0.0) => "https://api.nuget.org/v3/registration5-semver1/"
[Program/INF]   ReportAbuseUriTemplate/3.0.0-beta (v0.0.0) => "https://www.nuget.org/packages/{id}/{version}/ReportAbuse"
[Program/INF]   PackageDetailsUriTemplate/5.1.0 (v0.0.0) => "https://www.nuget.org/packages/{id}/{version}?_src=template"
[Program/INF]   RegistrationsBaseUrl/3.4.0 (v0.0.0) => "https://api.nuget.org/v3/registration5-gz-semver1/"
[Program/INF]   RegistrationsBaseUrl/3.6.0 (v0.0.0) => "https://api.nuget.org/v3/registration5-gz-semver2/"
[Program/INF]   RegistrationsBaseUrl/Versioned (v4.3.0-alpha) => "https://api.nuget.org/v3/registration5-gz-semver2/"
[Program/INF]   RepositorySignatures/4.7.0 (v0.0.0) => "https://api.nuget.org/v3-index/repository-signatures/4.7.0/index.json"
[Program/INF]   RepositorySignatures/5.0.0 (v0.0.0) => "https://api.nuget.org/v3-index/repository-signatures/5.0.0/index.json"
[Program/INF]   Catalog/3.0.0 (v0.0.0) => "https://api.nuget.org/v3/catalog0/index.json"

tintoy added a commit that referenced this issue Sep 26, 2021
…urces that don't support the NuGet completion API

#24
tintoy added a commit to tintoy/msbuild-project-tools-vscode that referenced this issue Sep 26, 2021
tintoy added a commit that referenced this issue Sep 26, 2021
…urces that don't support the NuGet completion API

#24
tintoy added a commit to tintoy/msbuild-project-tools-vscode that referenced this issue Sep 26, 2021
@tintoy
Copy link
Owner Author

tintoy commented Sep 26, 2021

I've also created a preview version of the extension supports both manually and automatically ignoring configured package sources (if you have time, could you try it out and let me know if it works for you?).

To manually ignore a package source, use the extension's new msbuildProjectTools.nuget.ignorePackageSources setting (you can ignore package sources by name or URI).
Package sources are automatically ignored if the NuGet client library indicates that they don't support the AutoComplete API.

@TripleEmcoder
Copy link

Here's the output of the updated gist, hope it helps:

> dotnet run
[Program/INF] Found 2 configured package sources.
[NuGet/INF]   CACHE https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json
[NuGet/INF]   CACHE https://api.nuget.org/v3/index.json
[Program/INF] Service index for packagesource Azure SDK ("https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json"):
[Program/INF]   PackagePublish/2.0.0 (v0.0.0) => "https://pkgs.dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_packaging/fa8c16a3-dbe0-4de2-a297-03065ec1ba3f/nuget/v2/"
[Program/INF]   LegacyGallery/2.0.0 (v0.0.0) => "https://pkgs.dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_packaging/fa8c16a3-dbe0-4de2-a297-03065ec1ba3f/nuget/v2/"
[Program/INF]   RegistrationsBaseUrl/3.0.0-beta (v0.0.0) => "https://pkgs.dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_packaging/fa8c16a3-dbe0-4de2-a297-03065ec1ba3f/nuget/v3/registrations2/"
[Program/INF]   RegistrationsBaseUrl/3.6.0 (v0.0.0) => "https://pkgs.dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_packaging/fa8c16a3-dbe0-4de2-a297-03065ec1ba3f/nuget/v3/registrations2-semver2/"
[Program/INF]   RegistrationsBaseUrl/Versioned (v0.0.0) => "https://pkgs.dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_packaging/fa8c16a3-dbe0-4de2-a297-03065ec1ba3f/nuget/v3/registrations2-semver2/"
[Program/INF]   SearchQueryService/3.0.0-beta (v0.0.0) => "https://pkgs.dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_packaging/fa8c16a3-dbe0-4de2-a297-03065ec1ba3f/nuget/v3/query2/"
[Program/INF]   PackageBaseAddress/3.0.0 (v0.0.0) => "https://pkgs.dev.azure.com/azure-sdk/29ec6040-b234-4e31-b139-33dc4287b756/_packaging/fa8c16a3-dbe0-4de2-a297-03065ec1ba3f/nuget/v3/flat2/"
[Program/INF]   VssBaseUrl (v0.0.0) => "https://pkgs.dev.azure.com/azure-sdk/"
[Program/INF]   VssFeedId (v0.0.0) => "urn:uuid:fa8c16a3-dbe0-4de2-a297-03065ec1ba3f"
[Program/INF]   VssQualifiedFeedViewId (v0.0.0) => "com.visualstudio.feeds.feedview:fa8c16a3-dbe0-4de2-a297-03065ec1ba3f"
[Program/INF]   AzureDevOpsProjectId (v0.0.0) => "urn:uuid:29ec6040-b234-4e31-b139-33dc4287b756"
[Program/INF] Requesting suggestions for package prefix Microsoft.AspNetCore from package source Azure SDK ("https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json")...
[Program/ERR] Unexpected error.
NuGet.Protocol.Core.Types.FatalProtocolException: The source does not have a Search service!
   at async Task<IEnumerable<string>> NuGet.Protocol.AutoCompleteResourceV3.IdStartsWith(string packageIdPrefix, bool includePrerelease, ILogger log, 
CancellationToken token)
   at async Task NuGetFeedLister.Program.Main() in E:\Marcin\Repos\3d9467d59b2547ad227ba5d1560ee838-826a4a2916f9e2b90fb8fabd8ef968c5a700bf24\Program.cs:line 85

By the way, this feed is public, so you can test on your end too. I'm happy to help, don't get me wrong, but it might speed things up for you :)

With the new preview I get no suggestions and no errors/messages at all... Output only shows:

Starting MSBuild language service...
MSBuild language service is running.

@tintoy
Copy link
Owner Author

tintoy commented Sep 26, 2021

Thanks for trying that - it looks like they don’t support the auto-complete API (the NuGet client library has a facility for looking up API end-points and out of the ones listed the SearchCompletion service is notably absent).

I didn’t realise that feed was public, I’ll try some more tests with it later today and see what I can find.

BTW, if you set the extension’s logging level to Verbose you should see log messages about validating and configuring package sources.

@tintoy
Copy link
Owner Author

tintoy commented Sep 26, 2021

Oh, and there seem to be 2 output window panes called MSBuild Project Tools (I think that's due to a change in the behaviour of the VS Code Extension SDK). The second one will have log entries from the language server.

Here's what mine says now that I also have the Azure SDK feed in nuget.config:

[Info  - 9:29:15 am] 2 package sources configured for project "c:\Development\test-projects\NuGetFeedLister\NuGetFeedLister.csproj".
[Info  - 9:29:15 am]   Locally-configured package source "nuget" (v2) => https://api.nuget.org/v3/index.json
[Info  - 9:29:15 am]   Locally-configured package source "Azure SDK" (v2) => https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json
[Warn  - 9:29:15 am] Ignoring configured package source "Azure SDK" ("https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json") because it does not appear to support a compatible version of the NuGet auto-complete API.

Unfortunately Azure DevOps package feeds don't seem to support any version of the SearchAutoComplete API which is why that exception was being thrown by the NuGet client library. I'll have a think about whether there is any other way to safely tackle this issue.

@tintoy
Copy link
Owner Author

tintoy commented Sep 27, 2021

Ok, have figured out why there were no suggestions. Will at least publish a version of the extension in the meanwhile with support for authenticated package feeds (but still won't work for Azure DevOps feeds at this stage because they don't currently implement the part of the NuGet protocol that we need for this).

@tintoy
Copy link
Owner Author

tintoy commented Sep 27, 2021

Implemented in v0.4.0.

(sorry, still doesn't work with Azure DevOps package feeds but that's a separate issue; I'll keep thinking about that one)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants