-
Notifications
You must be signed in to change notification settings - Fork 613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: symlinks in node_modules #257
Conversation
Thanks for the PR! ❤️ I'm not super familiar with the module resolution logic, so I will need some time to be able to review it properly. @jeanlauliac: you may have a better knowledge of it, can you take a look? |
The react native bundler, 'metro', doesn't follow symlinks. This meant that in the example app we couldn't do import Foo from "react-native-feed-media-audio-player" when the library was installed via 'npm install ../package', because that npm a symlink in 'node_modules/react-native-feed-media-audio-player' and metro wouldn't follow it. Maddening. Hopefully this will be fixed with: facebook/metro#257 This patch inverts things, so the top level 'package' directory is a symlink, and the node_modules/react... is the actual source. Now we can run the example app while changing code in the library.
Waiting on the Jest team, but wouldn't mind getting this reviewed soon. 😎 |
Nice to see some of the this work finally coming together. Really hoping this can land incrementally while adding more features later on. One of the bigger features that is missing right now (correct me if wrong) is the ability to recursively follow symlinks. Did some initial work on this almost two years ago. See "Recursive detection of symlinked modules" of facebook/react-native#12400 description. Just an attempt to get symlinks in |
I don't know if I'd characterize this work as "coming together". It's been almost 7 weeks since this PR was initially pushed up and 2 weeks since I checked in to see the status of this PR. (to which there was no response) @jeanlauliac @rafeca how can we help get traction on this? It's preventing us from adopting ReactNative 0.57 -- until then we are stuck on 0.56.1 |
We are also stuck on 0.56, can we do something to help? Same for this issue : jestjs/jest#6993 |
Will look this week. Haven't worked on that code for a while but that should be workable! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeanlauliac has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
@jeanlauliac have you had a chance to take a look? This problem makes a lot of different build environments impossible. |
Yes, I've been looking today, thank you for the reminder. Unfortunately I'm working on other projects these days so I struggle to take the time to work on Metro. Anyway, I think that looks good overall! Even though there are more changes than seem strictly necessary for the feature itself. I was trying to understand where's the "follow" function in HasteFS, and I realised jestjs/jest#6993 isn't in yet. So I guess we have to figure it out first. Also, I notice the |
In jestjs/jest#6993, the Jest team has mentioned they want to make symlinks transparent eventually. This PR is only focused on symlinks in The main issue with transparent symlinks is the negative effect on performance for developers not utilizing symlinks. For this reason, transparent symlinks should be opt-in. On the other hand, symlinks in We should fast-track this PR (and the Jest PR), since many developers have had issues with symlinks in And if you find time, please add test stubs to this PR (to fill out once the Jest team merges jestjs/jest#6993). |
Any update here? This has been blocking for us for 3 months now... |
I think that's blocked by jestjs/jest#6993 as we need to get that one right first, I'll comment there. |
Now that jestjs/jest#6993 is supplanted by jestjs/jest#7549, all symlinks are cached by As far as emulating |
if (result.type === 'resolved') { | ||
return result.resolution; | ||
/** Generate the potential module paths */ | ||
function* genModulePaths( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fancy
(scopeName ? extraNodeModules[scopeName] : void 0); | ||
|
||
if (parent) { | ||
yield path.join(follow(path.join(parent, packageName)), ...bits); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't this need to be path.join(parent, 'node_modules', packageName)
? If not, why not?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The previous code dealing with extraNodeModules
didn't do that, so I didn't either.
The idea is that a package name can be mapped to a node_modules
directory it may exist in.
extraNodeModules: {
react: '/a/b/c/node_modules'
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, nevermind. I think I misread the old code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome. Here are the next steps:
- Get the Jest PR merged.
- We'll publish a new release of jest-haste-map and update Metro to use it (this may take a while if we also have to update Jest for RN at FB).
- Merge this PR.
- Make a Metro release.
- Update the react-native-cli to use the new version of Metro.
205ed77
to
b46da65
Compare
- resolve symlinks inside "node_modules" directories - add `follow` function to `ResolutionContext` type - move "node_modules" logic into the `genModulePaths` method - feat: scope-only keys in `extraNodeModules` map - fix: always preserve scope when checking `extraNodeModules` map - fix: throw an error if `resolveRequest` fails to resolve a direct import - [BREAKING] refactor the `FailedToResolveNameError` class This commit adds support for symlinks in any `node_modules` directory. This includes scoped packages. Also, PNPM users rejoice! The `genModulePaths` method avoids extra work by lazily generating the potential module paths for indirect imports. Also, the resolver now skips over module paths that contain `/node_modules/node_modules/`. The `extraNodeModules` map now has support for keys like `@babel` to act as a fallback for all modules starting with `@babel/`. Previously, if the `resolveRequest` function failed to resolve a direct import, we would continue onto the "node_modules" search logic. The correct behavior is to throw an error.
We cannot assume that `filePath` is _not_ a directory.
Rebased onto v0.56 |
Just another dev and metro user who was hoping to use Lerna hopping on here to say that this is a blow to my plans for wanting a component library monorepo for RN, that it isn't merged in yet. Would love to see this feature looked at by the metro team and liaise with jest to get jest/9351 in too. Until then, sadness :'( |
@James-Reed-cognito In the meantime, you can try haul |
@jsamr Ooh interesting, though looks like it doesnt have hot reload/fast refresh? |
@jsamr I actually found a way to make this work by tweaking the metro config. Should I detail this here or perhaps add a PR to add a section in the docs for it, do you reckon? |
@James-Reed-cognito definitely post here. This issue is the top search result for others experiencing the issue, if you have a good solution that's easy and straightforward you'll probably help lots of people. |
Alright then, changing my metro.config.js to this seemed to make everything work:
All the credit actually goes to a friend of mine who gave me this solution. the |
Hi everyone. Any updates on this? Have been trying to make a working monorepo with yarn workspaces for several days, no success... |
@vlad-dewitt I recently set up a RN monorepo and found this blog post very useful https://medium.com/@ratebseirawan/react-native-0-63-monorepo-walkthrough-36ea27d95e26 |
I was able to get it to work using this guide: https://medium.com/@dushyant_db/how-to-use-lerna-with-react-native-1eaa79b5d8ec const path = require('path');
const watchFolders = [
// YOUR relative path to the packages directory
path.resolve(__dirname + '/../packages'),
];
module.exports = {
transformer: {
// ......
},
watchFolders,
}; |
Any update for this PR? |
I'd recommend @callback/repack to anyone who wants symlinks to work with react-native since the metro maintainers do not work actively on this thread. |
In case someone is looking for a workaround, pnpm has now a |
Symlinks were the very first issue filed on this repo over 5 and a half years ago. I think it's safe to say they're never going to happen from the metro maintainers. There is an alternative now however! https://microsoft.github.io/rnx-kit/docs/tools/metro-resolver-symlinks works perfectly to make metro follow symlinks. Got my app up and running on a pnpm workspace build by EAS with just a little bit of effort. Edit: and by "little bit of effort" I mean add |
I know it's been a while but believe it or not, we're actually working on this now. That was part of the motivation for decoupling from |
I'm not asking this to insult anybody, I'm genuinely curious -- Why has it taken 5+ years to add support for symlinks despite a community PR and every major JS package manager using them? Is there anything that can be done to help move Metro forward into the modern JS ecosystem? As a user of metro it's getting more and more difficult to use as it's pretty far behind the capabilities and best practices of all modern tooling. There are pain points with bundle sizes, bundling speed, ESM support, monorepos, etc all of which are long solved problems in modern JS tooling. I think there's a sincere desire from the community to collaborate with the metro team on bringing metro into the modern JS tooling world. How can we make that happen? |
That's a fair question - I'm not sure this is the best place for a proper answer, but broadly - personnel changes, priority shifts, challenges posed by Metro's deep integration into Meta infrastructure in ways not visible from GitHub, and sometimes - as in this case with Watchman and then Jest - technical dependencies outside this project. What I can say is that the team behind Metro at Meta is larger, growing and more engaged in Metro than it has been in several years. We know we have a lot of catching up to do, and we can do a better job of communicating what we're doing, but we're investing in and optimistic about Metro's future. As far as collaboration is concerned - it's very much appreciated, we're open to it - I'd just advise opening an issue to discuss anything non-trivial before spending time on it, because this repo is currently the foundation of a Jenga-like tower of infrastructure at Meta and some changes may be more complicated than they appear - which is actually one of the things we're looking to sort out. So, apologies, but please bear with us :) |
Thanks for this @aleclarson - I know it’s been several years, but this PR and all of the discussion is where we started when designing the general symlinks approach now in main. I’m closing this because it’s superseded by changes landed, but thanks for the groundwork. 🙏 |
@robhogan Congrats on your efforts :) |
follow
function toResolutionContext
typeyieldPotentialPaths
functionextraNodeModules
mapextraNodeModules
mapresolveRequest
fails to resolve a direct importFailedToResolveNameError
classSummary
This commit adds support for symlinks in any
node_modules
directory. This includes scoped packages. Also, PNPM users rejoice!The
yieldPotentialPaths
function avoids extra work by lazily generating the potential module paths for indirect imports. Also, the resolver now skips over module paths that contain/node_modules/node_modules/
.The
extraNodeModules
map now has support for keys like@babel
to act as a fallback for all modules starting with@babel/
.Previously, if the
resolveRequest
function failed to resolve a direct import, we would continue onto the "node_modules" search logic. The correct behavior is to throw an error.Partially fixes: #1
Depends on: jestjs/jest#7549
Test plan
None yet. Not familiar with Metro's test suite, and I assume you'll want some changes, or deny this altogether in favor of a more informed direction.
I've tested this manually with arbitrary symlinks and PNPM installations.