No matching version found for dependencies@undefined

[Xquick]

Issue description:

After converting yarn.lock to package-lock.json I run npm install.
output is

npm i
npm ERR! code ETARGET
npm ERR! notarget No matching version found for dependencies@undefined
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.

Disclaimer
I reinstalled node_modules fresh using yarn install, then I used following command to convert yarn.lock synp --source-file yarn.lock and it ran correctly and created package-lock.json

Attaching files

yarn.lock https://pastebin.com/FZWC7mUS
package-lock.json https://ufile.io/3rsmr
package.json https://pastebin.com/9fMwYfGt

[imsnif]

Hey, thanks for reporting this and my apologies for the late reply!

Could you walk me through the reproduction please? From downloading the files to a specific folder until you get this error? Because I'm getting different results and I want to make sure it's not just a specific repo particularity (eg. I needed to change the version in package.json because npm wouldn't accept 1.xxxx).

[DavidReinberger]

Hi @imsnif I actually work with @Xquick and I've decided to investigate my self and found out that there is a dependencies key a direct child to dependencies object like this, once I delete the object, npm install does not throw an error.

"dependencies": {
    ...
    "dependencies": {
          "minimist": {
          "version": "1.2.0",
          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
          "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ="
        }
    }
}

[imsnif]

Hey @DavidReinberger, thanks! I appreciate you looking into this.

Ouch, that does not look good!... When I convert the file, this does not happen to me. Which is more than a little odd. If I may ask... if you convert again now on a different box, does the field appear there? How about if you remove node_modules, reinstall them (with yarn) and then convert?

[DavidReinberger]

@imsnif So I've tried to copy out my package.json and yarn.lock to a new folder and doing a fresh install, which yielded the same result... We are using node:9 on macOS, which setup are you using?

[DavidReinberger]

Just a heads up, my colleague ran this on windows and it worked fine. So the problem is probably with macOS? 🤷‍♂️

[imsnif]

Hey @DavidReinberger - I think this has to do with some optional dependencies that are only installed on macOS (I'm using linux).

I'm guessing this has to do with the way we traverse the node_modules folder. synp's node_modules traversing logic is outsourced to a different library which might make this easier to debug (if you'd be willing to help).

Could you run this in the folder after a clean npm install? npx nmtree . and send me the results, along with a normal tree command output? That way I can try and check if there's a discrepancy between the modules nmtree finds and what's actually there. Thanks!

[DavidReinberger]

are you sure about the tree cmd? Or npm list?

[imsnif]

Sorry, for some reason I thought it came out of the box on macOS. :) I think you can install tree with brew install tree?

[imsnif]

ping @DavidReinberger ?

[djvs]

Issue also exists on up to date Arch Linux

[djvs]

So I went into the package-lock.json with vim. You have __"dependencies (two spaces) at the very beginning, ______"dependencies (six spaces) for every dep, and then at the veeery end of the file, 14 lines back, there's a single ____"dependencies (four spaces), with two packages under it. Both are subdependencies of the bcrypt package and also primary (top-level) dependencies - yallist and safe-buffer. Let me know if you need any other info.

[imsnif]

Thanks for the details, @djvs! I'm on arch too, so that's good news - will be easier to debug. :)
My problem with this issue up until now was that I was unable to reproduce it.

Can you consistently reproduce it on your box (the creation of the package-lock.json file)? Do you have a package.json + yarn.lock combination that reproduces this? Could you please post them so that I could take a look?

Much appreciated!

[sragu]

Worked on Windows for me as well.

[oliverlockwood]

We have a few projects which were failing due to this issue, on both MacOS and Linux.

For us, the workaround is as follows:

1. Run cat package-lock.json | jq -e ".dependencies.dependencies?"
2. If the above returns a zero return code, run the following block of shell script:

   cat package-lock.json | jq "if .dependencies.dependencies then del(.dependencies.dependencies) else . end" > package-lock.tmp
   mv package-lock.tmp package-lock.json
   

Hope this excerpt might be of use to others while this bug persists.

[imsnif]

Hey @oliverlockwood - thanks for this workaround.

I think most of the cases talked about in this thread involve private repositories (which makes it hard to reproduce and thus address). I hope this is not a lot to ask, but I wonder if you could help me out with some brief debugging? If you try to reproduce this not through some CI or script, but simply by cloning the repository, installing with yarn and then running synp... does this happen?

[oliverlockwood]

@imsnif - I believe so; yes.

My examples are also in private repos, but I will experiment with trimming down to something I can make public as a mini-repo to reproduce this issue. Watch this space...

[oliverlockwood]

@imsnif a number of interesting observations. All of this was executing manually on my machine (a Mac OS X). Note that the original issue (on an internal project) also reproduced on Linux; so with luck the stripped-down variant should also do so if necessary.

1. Initially, yarn install and synp followed by cat package-lock.json | jq '.dependencies.dependencies' gave this:

   {
     "sax": {
       "version": "1.2.4",
       "resolved": "https://nexus.ooflex.net/repository/npm/sax/-/sax-1.2.4.tgz",
       "integrity": "sha1-KBYjTiN4vdxOU1T6tcqold9xANk="
     }
   }
   

2. When I stripped out the internal dependencies from one example project, ran yarn install and synp again, the issue went away 😞
3. When I re-added those same dependencies, ran yarn install and synp again, the issue was still not present 🤔

Now for some comparisons.

• The diff between yarn.lock from (1) and (3) is here: https://gist.github.com/oliverlockwood/5916ecc2d5d7d9ff165c5a27112300fb
• In each case I ran find . -name "sax"
  From (1):

  ./node_modules/sax
  ./node_modules/gc-stats/node_modules/sax
  

  From (2):

  ./node_modules/sax
  ./node_modules/gc-stats/node_modules/needle/node_modules/sax
  

• Concentrating, then, on the gc-stats dependency we note the following difference in the lockfiles:

     gc-stats@^1.2.1:
-  version "1.2.1"
-  resolved "https://nexus-host-redacted/repository/npm/gc-stats/-/gc-stats-1.2.1.tgz#f3e1bf7e28a385780db22a81681b064932e358da"
-  integrity sha512-CPQfMBQPGkqG4upxCn4zHxYZo20woPClSeqnC/WK8pFqlfAtz6zpxbOfnmxOIDYiC26H/pYlWQfdoPVGoqxFUA==
+  version "1.4.0"
+  resolved "https://nexus-host-redacted/repository/npm/gc-stats/-/gc-stats-1.4.0.tgz#66cd194c5a8eae1138407300bc6cb42c2f6f3cd6"
+  integrity sha512-4FcCj9e8j8rCjvLkqRpGZBLgTC/xr9XEf5By3x77cDucWWB3pJK6FEwXZCTCbb4z8xdaOoi4owBNrvn3ciDdxA==
   dependencies:
-    nan "^2.10.0"
-    node-pre-gyp "^0.11.0"
+    nan "^2.13.2"
+    node-pre-gyp "^0.13.0"

From all of this, I was able to construct a very simple example repo, just including gc-stats with a yarn.lock which has already resolved ^1.2.1 to exactly version 1.2.1. You can find this repo at https://github.com/oliverlockwood/synp-test.

Note that we do have other projects which end up generating quite a large block in the dependencies.dependencies object - I just wanted to start with the simplest one.

Let me know if you need anything else in order to progress this!

[imsnif]

Wow, @oliverlockwood, great work!!

We finally found the issue!!! :D :D
I fixed it and published as 1.7.0 - could you take a look and see if it solves this for you in the private repos as well?

[oliverlockwood]

@imsnif thanks for this. Unfortunately, although version 1.7.0 does fix the issue in the stripped-down mini-repo, it does not fix the issue in the private repos (even the one from which i derived the mini-repo still causes problems.)

If you would like to add some debugging and either publish beta packages or just put the code on a branch, I'm happy to run such things against our private repos and send you the output, to help dig into this further. Alternatively please let me know how else I can help out.