-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC Discussion] Legion ECS Evolution #22
Comments
Looks good! Maybe you could add some benchmarks to back up the "Significant performance gains" point? |
Benchmark visualization is here: https://github.com/jaynus/legion/raw/master/bench.png To discuss more candid, discussion-style explanation of why: Specs specifically utilizes sparse component storage. That means whenever you are joining 2 or more components, you are jumping between 2 arbitrary locations in memory repeatedly, for every single entity. This is very hard on CPU caching, as you are repeatedly loading and processing N completely separate regions of memory. Legion allocations like-entities in contiguous memory, and iterates based on allocated "chunks" for iteration. This means that whenever performing a query against N-components, the memory is almost always guaranteed to be local and adjacent because its archetype is the same for that query. This is a high level explanation, and leaving out some edge cases, but that is the gist of why we see significant performance benefits with legion. The more complex of these benefits can't really be realized in any current use of amethyst, so those are basically theoretical. |
For full disclosure, that image only shows creation and iteration and so forth, it does not demonstrate the timings of when components are added and/or removed as is common in many ECS engine styles. The adding/removing pattern is very slow compared to specs because it forced a copy of every component an entity has into a new 'chunk'. A combined pattern where components can be marked as what 'chunks' (if any) they should appear with in regards to other components would fix the issue as it would keep the bulk components together and the rapidly changing ones would then remain out of band. |
I've started adding benchmarks to my fork jaynus/amethyst-ecs-benchmarks The reports can be seen Criterion Reports Here They are a WIP as I add more optimized cases. |
Awesome @jaynus, looking forward. My minigames tend to do a significant amount of component swapping of a set amount of component types with rare swapping of another set of component types and little to no swapping of the remaining amount of component types, so optimizing for these is very doable with legion with some more design. |
Really excited for the new transforms. It could also potentially let us easily create alternative coordinate systems (most notably spherical coordinates) to very easily create 3rd person cameras for example. |
There's been a lot of positive feedback to this change. Do we have any issues that would require delaying confirming this as our course of action? Would we want to publicize this discussion more before doing so? |
It seem to me like it would be good to make this as public as possible before confirming it, being that it is probably one of the single largest possible changes that we could make. It seems unlikely at this point that we would change our direction, but I think it makes sense to give people as much an opportunity to state their thoughts against it before we go ahead with it. |
Khionu and I have posted it around now, thanks. |
As a non-amethyst-user who is hopeful about legion's future, I'm concerned that amethyst adopting it wholesale in the near future might make it difficult for legion to make significant changes to its public API. Legion's a very new crate and issues like amethyst/legion#27 illustrate that significant refinements are still occurring, and that locking in a stable API is therefore premature. Conversely, adoption by amethyst could provide a substantial amount of the experimental use required to drive future design revisions, if those revisions aren't prevented by stability requirements, so the key question is how much churn amethyst is willing to tolerate in the name of improvements to upstream libraries. |
Great work all around! This definitely looks promising. A word of caution, I've seen a world like concept crop up in different game engines and it nearly always turned out to have a lot of the same problems as global mutable state, especially the more complex a project got. Also with something this fundamental for the engine, I'd highly advise also running benchmarks in more complex use cases like a simple 3d game, as microbenchmarking tells an incomplete story. |
Are the changes to the transform component tied to the legion change, or can they be done independently? I really like the transform suggestion, but I do have some comments:
|
As long as it is enabled only as a feature first, it's fine by me. |
There are also lots of other cool transforms that can be added to TL;DR: Yes, it's high customizable and you only pay the cost of the transformations you actually use.
Agreed. Need lots more documentation. Nalgebra Transform API IssueOne big issue I have with my own library: the nalgebra API for computer graphics is... to be polite let's just say "difficult to use", and their documentation is hard to navigate. Because |
These answer my first question well. Great to hear this whole system is being thought out well. |
I agree that nalgebra's API is pretty difficult. It isn't like it is totally bad, but the whole dimensions abstraction really leverages Rust's type system in a way that, while it actually works fine, is very non-intuitive and comparatively difficult to think about. It also doesn't seem to be the fastest solution out there, but that is probably another discussion. I absolutely love what you are saying about the transforms being pay for what you use and highly customizable. 👍 |
Small personal pet peeve around terminology. :)
Should say "Sparse storage has poor cache locality." Cache coherency is the (generally in-hardware) process by which memory writes from one CPU are propagated so the caches on other CPUs in a shared-memory system don't have stale data. Cache locality is the likelihood of subsequent memory operations being in cache and is the root of why we prefer contiguous and sequential memory access pattern. |
It would be great to see a few more benchmarks. As just the iteration time of a basic component is only marginally faster and all of the other benchmarks are slower ( I realize they are a WIP ). One case that isn't really discussed is memory usage? Does one have more overhead? Also, these benchmarks seem to focus on cases where all entities have all components instead of the more likely scenario where things like render targets and transforms are used alot and other things like Uibuttons are used more sparingly. If we could get a benchmark where we can test both cases (and one in between maybe) that could be revealing. |
Reminder that I linked above https://github.com/jaynus/amethyst-ecs-benchmarks, which is the WIP benchmark comparisons I showed the results for here. The current benchmark results show specs ahead of legion because these are worst case scenario benchmarks for legion, and best case scenario benchmarks for specs. E.g. the only benches I implemented are basically to show legion in the worst possible light, and it still holds up fairly well. Anyways, the whole point of this is that I could really use help writing more benchmarks more reflective of real world use. I only have so much time, so any help will be greatly appreciated. I'll take all the PR's in the world to that project to start visualizing some true usage. |
Also, just to reiterate and not to negate the need for benchmarks, but performance isn't the only reason for using Legion, though it is among the biggest reasons. Legion is also more flexible and easier to setup an FFI with, which is needed for scripting. |
@jaynus I tried cloning that repo bit ran into alot of issues with getting it to build, I think because you are using custom crates. So decided to make my own. I can publish the source later if there is interest ,but I ran two tests: -Second I varied the frequency that entities had velocity components. I tested (90%,50%, and 10%). Legion had significant speed increases as that frequency decreased as you would expect due to less components to join. Specs was relatively the same regardless of the frequency which was concerning. I am on board that the change should lead to significant improvements in amethyst. |
I'm for legion being chosen as well, however it still sorely needs grouping support like other 'batched' ECS's as well. Sometimes components change rapidly so they should be in a different group. Sometimes components need to be tightly packed together with only themselves to allow for OpenCL or whatever work, those should be in a null group or a group by themselves, etc... Regardless, grouping needs to be supported. |
@OvermindDL1 We've been thinking about it here: amethyst/legion#16 This issue is the "correct" way to provide that kind of functionality, so its not like its not on our minds or something that is not going to be implemented; its just not implemented yet. I know you've brought this up many times in discord and on the community post, so I just wanted to inform you on the plan. @mrhatman Oh sorry, I don't think I've updated it for the latest API safety changes. I'll get it updated shortly for you to PR into. |
I like the current API for creating systems as it makes sense in my mind. Is there a need/reason the change the developer facing API? I'd love to hear any arguments for or against moving to the new API. |
The shred trait-based API /w a You can see the current thoughts on that here: amethyst/legion/issues/31 |
I'm onboard with choosing Legion, primarily for performance gains, secondarily for the scripting API (Which will be great to finally have moving forward again). |
I like with the idea of having a scripting API. |
There's discussion of scripting in other threads, including the Scripting RFC and on the forums. It'd be best if discussion of features that are not exactly in scope are held where they are. |
I haven't directly done any work on it and @jaynus might be able to speak better to this, but as far as I know there hasn't been any further development of the port of Amethyst to Legion, but there has been miscellaneous development work on Legion itself. I've seen a bit of general activity, PRs, issues, etc. on the Legion repo. |
Ah, it looks like I was inadvertently on a non-master branch when I was looking at commit dates. There is definitely some more recent activity. |
Still, I think folks would all love to hear how things are coming along and if there are places that help is needed! |
I've been asked a bit for an update, and I've taken a small break for amethyst related work, hence my fork stalled for a bit. I have pushed the latest legion work @ https://github.com/amethyst/amethyst/tree/legion This includes another rewrite of the dispatcher and some other minor changes. This branch DOES NOT CURRENTLY BUILD due to being in the middle of updating everything to the latest legion master code. TLDR Status Here is a task list of items that are complete:
Following are items that STILL NEED TO BE COMPLETED from a implementation standpoint:
Supporting work that needs to be completed:
|
Hm, perhaps it's useful to do a release of amethyst with its current changes, and have the version after that purely the legion switchover (2-smaller-step upgrades, instead of one big step). One benefit is people get the non- I'll likely take on |
I can give a shot at amethyst network, I've been working on network code for legion where I used amethyst network as an reference. |
@jaynus what do you mean by "(This should probably just get gutted and removed)" ? |
I mean, exactly what I said. The time of porting isn't really worth it considering it's purely CPU inefficient animation. We could spend the time integrating a GPU skeletal system, which imo would be a better use of time. It's just opinion, really |
ok I get that for animation, kinda. I was mostly concerned about gutting the ui system. |
There's been a desire to rewrite the UI system, but I'm on the fence on whether it would be better to put in the effort to do a minimal port for now or include it as another large, breaking change in the same release... |
I personally lean towards porting it as it is, because a lot of people use the ui system as is and I don't want to change to much in a single release. |
One breaking change that I would like is putting the UI coordinate origin
in the top left, same as rendering.
21.2.2020 5.25 ip. "mrhatman" <[email protected]> kirjoitti:
… I personally lean towards porting it as it is, because a lot of people use
the ui system as is and I don't want to change to much in a single release.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#22?email_source=notifications&email_token=AC66YFGG3WQ4ECETVGIPAXDRD7W5FA5CNFSM4JEVY24KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTB2JY#issuecomment-589700391>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AC66YFD7NYAUINVMCBDGHM3RD7W5HANCNFSM4JEVY24A>
.
|
So how about this: we won’t let the porting of amethyst_ui block a release, but if someone does do the work and port it over, it will of course be included. |
Many of you have probably caught it but in case not, adding relevant blog post that has been published recently: https://csherratt.github.io/blog/posts/specs-and-legion/ |
I want to ask why chose |
@Joe23232 Shipyard looks very young, and this move to Legion far predates it. Also I see no evidence of your performance claim. |
Yes the move towards Legion was made before shipyard was even made I think. Also shipyard is rather immature and changing quite rapidly, which would make it more difficult to integrate with Amethyst because of Amethyst now having to keep up with the changing API of shipyard. There was some apparently controversial discussion about the different ECS models on the forum, and I the gist that I gleaned from it was that:
I think for Amethyst the Legion model of being more beginner friendly might be better for it either way. |
|
Yeah, I understand that. And, personally, I'm actually am using It seems like it would be good to have a "vision" or "zen of Amethyst" or some other form of guidance for decisions like this that will drive Amethyst as an engine and show its own philosophy. Any project will have to make trade-offs, but as it stands we don't have any guide that shows us, as users, what general concepts guide Amethyst's decisions. |
This would be a great thing to discuss, need a coherent vision to move towards! |
Given how big of a breaking change switching ECS engines is, and how much this one move is blocking us, it's safe to say that it would need to be a massive reason to switch off of Legion, and certainly not something we can afford doing for at least a year. That said, Amethyst can certainly keep an eye on shipyard, for inspiration if nothing else. |
I've actually started a discussion about this on the forum some months ago, but the topic wasn't public yet. We just moved the topic to the public section of the forum so we can open it up to community discussion: ECS Design and the Amethyst Vision. |
What is the current status of transition Amethyst from Specs to Legion? |
We are making good progress recently on the Legion port, but we could always use more help from the community. Please consider reviewing the recent PRs or sharing your thoughts about the legion_v2 branch on our discord . I would hope that we get the port in a "completed" state by the end of the month. |
Are there any recent updates on this? How are y'all feeling about how the port is going? |
Legion ECS Evolution
Following lengthy discussion on both Discord and the Amethyst Forum (most of which, including chat logs, can be found here), we propose with this RFC to move Amethyst from SPECS to Legion, an ECS framework building on concepts in SPECS Parallel ECS, as well as lessons learned since. This proposal stems from an improved foundational flexibility in the approach of Legion which would be untenable to affect on the current SPECS crate without forcing all users of SPECS to essentially adapt to a rewrite centered on the needs of Amethyst. The flexibility in Legion is filled with tradeoffs, generally showing benefits in performance and runtime flexibility, while generally trading off some of the ergonomics of the SPECS interface. While the benefits and the impetus for seeking them is described in the "Motivations" section, the implictions of tradeoffs following those benefits will be outlined in greater detail within the "Tradeoffs" section.
There are some core parts of Amethyst which may either need to considerably change when moving to Legion, or would otherwise just benefit from substantial changes to embrace the flexibility of Legion. Notably, systems in Legion are FnMut closures, and all systems require usage of SystemDesc to construct the closure and its associated Query structure. The dispatch technique in Legion is necessarily very different from SPECS, and the parts of the engine dealing with dispatch may also be modified in terms of Legion's dispatcher. Furthermore, the platform of Legion provides ample opportunity to improve our Transform system, with improved change detection tools at our disposal. These changes as we understand them are described below in the "Refactoring" section.
The evaluation of this large transition requires undertaking a progressive port of Amethyst to Legion with a temporary synchronization shim between SPECS and Legion. This effort exists here, utilizing the Legion fork here. Currently, this progressive fork has fully transitioned the Amethyst Renderer, one of the largest and most involved parts of the engine ECS-wise, and is capable of running that demo we're all familiar with:
Not only can you take a peek at what actual code transitioned directly to Legion looks like in this fork, but the refactoring work in that fork can be utilized given this RFC is accepted while actively helping to better inform where there may be shortcomings or surprises in the present.
Motivations
The forum thread outlines the deficiencies we are facing with specs in detail. This table below is a high level summary of the problems we are having with specs, and how legion solves each one.
Immediate Benefits in a Nutshell
Significant performance gains
Scripting RFC can move forward
Queries open up many new optimizations for change detection such as culling, the transform system, etc.
More granular parallelization than we already have achieved
Resolves the dispatcher Order of Insertion design flaws
???
Tradeoffs
These are some things I have ran into that were cumbersome changes or thoughts while porting. This is by no means comprehensive. Some of these items may not make sense until you understand legion and/or read the rest of this RFC.
Systems are moved to a closure, but ergonomics are given for still maintaining state, mainly in the use of FnMut[ref] for the closure, and an alternative build_disposable [ref].
All systems are built with closures, causing some initialization design changes in regards to reference borrowing
The SystemDesc/System types have been removed.
a Trait type cannot be used for System declaration, due to the typed nature of Queries in legion. It is far more feasible and ergonomic to use a closures for type deduction. The except to this case is thread-local execution, which can still be typed for ease of use.
Refactoring
This port of amethyst from legion -> specs has aimed to keep to some of the consistencies of specs and what Amethyst users would already be familiar with. Much of the implementation of Legion and the amethyst-specific components was heavily inspired/copied from the current standing implementations.
SystemBundle, System and Dispatcher refactor
This portion of the port will have the most significant impact on users, as this is where their day-to-day coding exists. The following is an example of the same system, in both specs and legion.
High Level Changes
Systems are all now FnMut closures. This allows for easier declaration and type deduction. They can capture variables from their builder for state. Additional ‘disposable’ build types are available for more complex stateful modes.
System data declarations are now all within a builder, and not on a trait.
Component data is now accessed via "queries" instead of “component storages”
Component addition/removal is now deferred, in line with entity creation/removal
Default resource allocation is removed, all world resource access now return Option<Ref>
System registration explicit dependecies are removed, now execution is ordered based on "Stages", which can be explicit priorities, but all system execution is flattened into a single data-dependent execution.
Following is an example of a basic system in both specs and legion
Specs
Legion
Example bundle Changes
RenderBundle - Specs
RenderBundle - Legion
Parallelization of mutable queries
One of the major benefits of legion is its granularity with queries. Specs is not capable of performing a parralel join of Transform currently, because FlaggedStorage is not thread safe. Additionally, a mutable join such as above automatically flags all Transform components as mutated, meaning any readers will get N(entities) events.
In legion, however, we get this short syntax: query.par_for_each(|(entity, (mut transform, orbit))| { Under the hood, this code actually accomplishes more than what ParJoin may in specs. This method threads on a per-chunk basis on legion, meaning similiar data is being linearly iterated, and all components of those entities are in cache.
Transform Refactor (legion_transform)
Legion transform implementation
@AThilenius has taken on the task of refactoring the core Transform system. This system had some faults of its own, which were also exacerbated by specs. The system itself is heavily tied in with how specs operates, so a rewrite of the transform system was already in the cards for this migration.
Hierarchy
This refactor is aimed towards following the Unity design, where the source-of-truth for the hierarchy (hot data) is stored in Parent components (ie. a child has a parent). This has the added benefit of ensuring only tree structures can be formed at the API level. Along with the Parent component, the transform system will create/update a Children component on each parent entity. This is necessary for efficient root->leaf iteration of trees, which is a needed operation for many systems but it should be noted that the Children component is only guaranteed valid after the transform systems have run and before any hierarchy edits have been made. Several other methods of storing the hierarchy were considered and prototyped, including an implicit linked-list, detailed here. Given all the tradeoffs and technical complexity of various methods (and because a very large game engine company has come to the same conclusion) the current method was chosen. More info can be found in the readme of legion_transform.
Transform
The original Amethyst transform was problematic for several reasons, largely because it was organically grown:
The local_to_world matrix was stored in the same component as the affine-transform values.
The component was a full Affine transform (for some reason split between an Isometry and a non-uniform scale stored as a Vector3).
Much of the nalgebra API for 3D space transform creation/manipulation was replicated with little benefit.
Given the drawbacks of the original transform, it was decided to start from a clean slate, again taking inspiration from the new Unity ECS. User defined space transforms come in the form of the following components:
Translation (Vector3 XYZ translation)
Rotation (UnityQuaternion rotation)
Scale (single f32, used for uniform scaling, ie. where scale x == y == z)
NonUniformScale (a Vector3 for non-uniform scaling, which should be avoided when possible)
Any valid combinatoric of these components can also be added (although Scale and NonUniformScale are mutually exclusive). For example, if your entity only needs to translate, you need only pay the cost of storing and computing the translation and can skip the cost of storing and computing (into the final homogenius matrix4x4) the Rotation and Scale.
The final homogeneous matrix is stored in the LocalToWorld component, which described the space transform from entity->world space regardless of hierarchy membership. In the event that an entity is a member of a Hierarchy, an additional LocalToParent components (also a homogeneous matrix4x4) will be computed first and used for the final LocalToWorld update. This has the benefits of:
The LocalToWorld matrix will always exist and be updated for any entity with a space transform (ex any entity that should be rendered) regardless of hierarchy membership.
Any entity that is static (or is part of a static hierarchy) can have it’s LocalToWorld matrix pre-baked and the other transform components need not be stored.
No other system that doesn’t explicitly care about the hierarchy needs to know anything about it (ex rendering needs only the LocalToWorld) component.
Dispatcher refactor
The Dispatcher has been rewritten to utilize the built-in legion StageExecutor, while layering amethyst needs on top of it.
The builder and registration process still looks fairly similar; the main difference being naming is now debug only, and explicit dependencies have been removed in favor of inferred insertion order via Stages, and then full parallel execution. ThreadLocal execution still also exists, to execute the end of any given frame in the local game thread. This means that a Stage or RelativeStage can be used to infer the insertion order of the system, but it will still execute based on its data dependencies, and not strictly its place "in line".
Migration Story
World Synchronization Middleware
Because of the fundemental changes inherent in this migration, significant effort has gone into making transitioning, and using both "old" and “new” systems as seamless as possible. This does come with significant performance cost, but should allow people to utilize the a mix of specs and legion while testing migration.
The engine provides a "LegionSyncer" trait; this is dispatched to configure and handle syncing of resources and components between specs and legion
Underneath these LegionSyncer traits, lies a set of automatic syncer implementations for the common use cases. This includes resource and component synchronization between the two worlds
Dispatching already exists for both worlds; dispatch occurs as:
Run specs
Sync world Specs -> Legion
Run Legion
Sync world Legion -> Specs
Helper functions have been added to the GameDataBuilder and DispatcherBuilder to streamline this process.
In the current design, syncers are not enabled by default and must be explicitly selected by the user via the game data builder. For example:
The above will explicitly synchronize the specifed resources and components. Additionally, the "Sync bundles" are provided for synchronizing the features out of any given crate.
This synchronization use can be seen in the current examples, as they still utilize a large amount of unported specs systems.
Proposed Timeline
With the synchronization middleware available, it gives users the ability to slowly transition to the new systems while actively testing their project. I propose the following release timeline for this, allowing users to skip versions as we go and work between:
The current implementation is feature gated behind "legion-ecs" feature. This can be released as a new version of Amethyst to begin migration
The next release performs a "hard swap", from “specs default” to “Legion default”. This would rename all the migration_with_XXX functions to specs, and make legion default. This would also include ironing out the legion world defaulting in the dispatchers and builders.
The next release removes specs entirely, leaving legion in its place.
Brain-fart of changes needed by users
Render Plugins/Passes
Change renderer::bundle::* imports to renderer::legion::bundle
Change renderer::submodules::* imports to renderer::legion::submodules
All resource access changes from world to world.resources
fetch/fetch_mut change to get/get_mut
Read and Write remove lifetime
ReadExpect/WriteExpect change to Read/Write
You can still use the same <(Data)>::fetch(&world.resources) syntax
Resources need to cache their own queries
amethyst/amethyst-imgui@06c1a58 a commit showing a port
The text was updated successfully, but these errors were encountered: