[Compiler-V2] [Feature] Positional Fields #14000

fEst1ck · 2024-07-13T17:10:02Z

Description

This PR introduces positional fields to Move 2. Example usage:

// declarations
struct S(u8);

enum E {
  V(bool, u8)
}

// value construction with anonymous fields
fun construct() {
  let x = S(42);
  let y = E::V(true, 42);
}

// value destruction with anonymous fields
fun destruct(x: S, y: E) {
  x.0;
  let S(_x) = x;
  match (y) {
    E::V(_x, _y) => {},
  }
}

Implementation

Declarations such as

struct S(u8);

is desugared into

struct S { 0: u8 }

Similarly, S(42) is desugared into S {0 : 42 }, let S(x) = y; into let S { 0: x } = y; and so on. This happens on the parser AST, making minimal changes to the compiler.

We also need to resolve call expressions such as S(42) to pack expressions when building the build.

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

third_party/move/move-compiler-v2/tests/checking/anonymous_fields

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-07-13T17:10:06Z

⏱️ 10h 36m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
test-fuzzers	6h 6m	🟩 🟩 🟩 🟩 🟩 (+5 more)
rust-move-tests	17m	🟩
general-lints	16m	🟩 🟩 🟩 🟩 🟩 (+5 more)
rust-cargo-deny	16m	🟩 🟩 🟩 🟩 🟩 (+5 more)
rust-move-tests	15m	🟩
rust-move-tests	15m	🟩
rust-move-tests	15m	🟩
rust-move-tests	14m	🟩
rust-move-unit-coverage	14m	🟩
rust-move-unit-coverage	13m	🟩
rust-move-unit-coverage	13m	🟩
rust-move-unit-coverage	13m	🟩
rust-move-unit-coverage	13m	🟩
check-dynamic-deps	12m	🟩 🟩 🟩 🟩 🟩 (+8 more)
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-unit-coverage	9m	🟩
rust-move-unit-coverage	9m	🟩
rust-move-unit-coverage	9m	🟩
rust-move-unit-coverage	8m	🟩
rust-move-tests	8m	🟩
semgrep/ci	6m	🟩 🟩 🟩 🟩 🟩 (+8 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+7 more)
file_change_determinator	2m	🟩 🟩 ⬜ 🟩 🟩 (+8 more)
permission-check	44s	🟩 🟩 🟩 🟩 🟩 (+8 more)
permission-check	42s	🟩 🟩 🟩 🟩 🟩 (+8 more)
permission-check	40s	🟩 🟩 🟩 🟩 🟩 (+8 more)
permission-check	37s	🟩 🟩 🟩 🟩 🟩 (+8 more)
rust-move-unit-coverage	17s	⬜
rust-move-tests	15s	⬜
Backport PR	4s	🟥
permission-check	3s	🟩
rust-move-unit-coverage	1s	⬜
rust-move-tests	1s	⬜
rust-move-tests	1s	⬜
rust-move-unit-coverage	1s	⬜
rust-move-tests	1s	⬜

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

codecov · 2024-07-16T19:44:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.0%. Comparing base (235402e) to head (485fe79).
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #14000       +/-   ##
===========================================
- Coverage    70.8%    59.0%    -11.9%     
===========================================
  Files        2324      824     -1500     
  Lines      459502   198347   -261155     
===========================================
- Hits       325634   117075   -208559     
+ Misses     133868    81272    -52596

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

brmataptos · 2024-07-22T18:34:39Z

Please add a test case for "assign anonymous field", e.g., perhaps:

    module ... {
        struct S(u64, bool);
        fun ... () {
            let x = S(3, true);
            while (x.1) {
                 x = S(x.0 - 1, x.0 >= 1);
             }
         }

Or even one with the modified var as a parameter:

    module ... {
        struct S(u64, bool);
        fun ... (x: S): u64 {
            while (x.1) {
                 let count = 0;
                 let y = if (x.0 > 0) { x.0 - 1 } else { 0 };
                 x = S(y, y >=1);
                 count += 1;
             }
             count
         }

brmataptos · 2024-07-22T18:38:22Z

Also tests of accesses into nested structs, both anonymous and not, e.g.:

module ... {
    struct A { 
        x: u64
    };
    struct B(A, u64, A);
    struct C(B, A, B);
    fun ... {
         let x:C = ...;
         x.0.1.x + x.2.1 + x.2.1.x;
     }
}
```
etc.

third_party/move/move-compiler/src/parser/syntax.rs

brmataptos · 2024-07-22T18:55:03Z

...rty/move/move-compiler-v2/tests/checking/anonymous_fields/named_tuple_construct_invalid.move

+    struct S(u8);
+
+    fun arity_mismatch0(): S {
+        S()


Add a similar test but with S having 2 fields and you providing 1. Also for an assignment. 1/0 might be a special case, test enough for confidence that both corner cases and typical cases will work. A single test case isn't enough to see a rule.

fEst1ck · 2024-07-25T05:51:57Z

Please add a test case for "assign anonymous field", e.g., perhaps:

    module ... {
        struct S(u64, bool);
        fun ... () {
            let x = S(3, true);
            while (x.1) {
                 x = S(x.0 - 1, x.0 >= 1);
             }
         }

Or even one with the modified var as a parameter:

    module ... {
        struct S(u64, bool);
        fun ... (x: S): u64 {
            while (x.1) {
                 let count = 0;
                 let y = if (x.0 > 0) { x.0 - 1 } else { 0 };
                 x = S(y, y >=1);
                 count += 1;
             }
             count
         }

done. third_party/move/move-compiler-v2/tests/checking/anonymous_fields/assign_field.move
(We don't have += in Move though.)
@brmataptos

fEst1ck · 2024-07-25T05:52:17Z

Also tests of accesses into nested structs, both anonymous and not, e.g.:

module ... {
    struct A { 
        x: u64
    };
    struct B(A, u64, A);
    struct C(B, A, B);
    fun ... {
         let x:C = ...;
         x.0.1.x + x.2.1 + x.2.1.x;
     }
}

etc.

done third_party/move/move-compiler-v2/tests/checking/anonymous_fields/assign_field.move
@brmataptos

brmataptos

It would be nice to have at least a few transactional tests. Maybe something based on the assignment tests.

brmataptos · 2024-07-26T17:35:50Z

third_party/move/move-compiler/src/expansion/ast.rs

+        }
+    }
+
+    pub fn is_struct_or_schema_name(&self) -> bool {


I would keep the function name same as the original here, despite being wordy. It's more precise, and is_valid... implies that the form is ok but it might not be a struct or schema.

brmataptos · 2024-07-26T17:53:03Z

third_party/move/move-compiler-v2/tests/checking/variants/variants_check_err.exp

@@ -18,6 +18,12 @@ error: variants not allowed in this context
 22 │         0x815::m::missplaced_variant::Red();
   │                                       ^^^

+error: the function takes 1 argument but 0 were provided


Why do we have a new error here? Get rid of it, please.

vineethk

More general comments to be addressed:

We need several transactional tests showcasing end-to-end functionality. I tried adding a simple one, and it seems to fail with a compiler "bug" because 0 is not a identifier (so there may be restrictions for even later in the compiler process that disallows certain field names).
Following are some transactional tests that would be nice:

Showcases both generics (I don't think there were cases with generic structs with positional fields instantiated) and the tuple swap bugs we have had before:

struct Tup<T, U>(T, U);

fun baz(x: u64, y: u64): u64 {
	let Tup(y, x) = Tup(x, y);
	y - x
}

A case where we have enum Blah with multiple variants using positional fields, and a "cross-variant common access" (like x.0, where x: Foo).

enum Blah {
  A(u8),
  B(u8),
};

fun foo(x: Blah): u8 {
  x.0
}

vineethk · 2024-07-27T00:38:16Z

third_party/move/move-compiler-v2/tests/checking/anonymous_fields/assign_field.move

@@ -0,0 +1,65 @@
+module 0x42::test {


Let's please rename this directory as positional_fields.

vineethk · 2024-07-27T00:38:43Z

third_party/move/move-compiler-v2/tests/checking/anonymous_fields/assign_field.move

+        x: u8,
+    }
+
+    struct S1(u64, bool);


Add a test where one of the positional fields is a reference.

done (in decl_invalid.move)

vineethk · 2024-07-27T00:51:53Z

...party/move/move-compiler-v2/tests/checking/anonymous_fields/bind_anonymous_field_invalid.exp

@@ -0,0 +1,31 @@
+
+Diagnostics:
+error: missing field `0`


nit.

Would something like below be clearer to users who may not know the internal representation?

missing field 0 -> positional fields arity mismatch: expected 1, actual 0.
field 1 not declared -> positional fields arity mismatch: expected 1, actual 2.

This is done in my next PR for ".." syntax.

vineethk · 2024-07-27T00:54:53Z

third_party/move/move-compiler-v2/tests/checking/anonymous_fields/decl_ok.exp

@@ -0,0 +1,7 @@
+
+Diagnostics:


nit: it is a bit misleading to name this file decl_ok when it actually contains compile-time invalid code.

vineethk · 2024-07-27T00:57:23Z

third_party/move/move-compiler-v2/tests/checking/anonymous_fields/decl_ok.exp

@@ -0,0 +1,7 @@
+
+Diagnostics:
+error: expected struct constructor `test::E1{ ... }` for struct constructor `test::E1`


This error message is misleading and confusing: test::E1{ ... } is not a valid constructor, test::E1::V2 {...} is.

vineethk · 2024-07-27T02:19:24Z

third_party/move/move-compiler/src/parser/syntax.rs

@@ -2796,6 +2868,28 @@ fn parse_field_annot(context: &mut Context) -> Result<(Field, Type), Box<Diagnos
    Ok((f, st))
 }

+/// Parse a comma list of types surrounded by parenthesis into a vector of `(Field, Type)` pairs
+/// where the fields are named "0", "1", ... with location the location of the type in the second field


duplicate "location" term in sentence

vineethk · 2024-07-27T02:23:48Z

third_party/move/move-compiler/src/parser/syntax.rs

@@ -2796,6 +2868,28 @@ fn parse_field_annot(context: &mut Context) -> Result<(Field, Type), Box<Diagnos
    Ok((f, st))
 }

+/// Parse a comma list of types surrounded by parenthesis into a vector of `(Field, Type)` pairs


Would be nice to also give the pseudo-grammar, keeping up with the tradition of the rest of file.

vineethk · 2024-07-27T02:30:52Z

third_party/move/move-model/src/builder/exp_builder.rs

+                ))
+                .expect("unique field names");
+                let unpack_ = EA::LValue_::Unpack(maccess.clone(), generics.clone(), fields);
+                let unpack = Spanned::new(lv.loc.clone(), unpack_);


Suggested change

let unpack = Spanned::new(lv.loc.clone(), unpack_);

let unpack = Spanned::new(lv.loc, unpack_);

vineethk · 2024-07-27T02:37:04Z

third_party/move/move-model/src/builder/exp_builder.rs

+        )?;
+        let field_decls = field_decls.clone();
+        if is_positional_constructor != expected_positional_constructor {
+            let struct_name_display = struct_name.display(self.env());


We are ignoring the variant information here (when available), which is part of the constructor.

is_positional_constructor is computed from the struct and also the variant, if this is what you are asking?

vineethk · 2024-07-27T02:38:21Z

third_party/move/move-model/src/builder/model_builder.rs

@@ -133,7 +133,7 @@ pub(crate) struct StructEntry {

 #[derive(Debug, Clone)]
 pub(crate) enum StructLayout {
-    Singleton(BTreeMap<Symbol, FieldData>),
+    Singleton(BTreeMap<Symbol, FieldData>, bool),


Describe what the bool is for in a comment.

Consider whether it would be clearer to create a distinct variant here, perhaps Positional(...) to avoid the confusing bool. You could even potentially change the BTreeMap to a Vec but maybe being able to share code with the Singleton case would be useful.

github-actions · 2024-08-02T16:52:57Z

✅ Forge suite `compat` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `9182255629aa9b5a988eebd5ed85f942172a1083`

Compatibility test results for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 9182255629aa9b5a988eebd5ed85f942172a1083 (PR)
1. Check liveness of validators at old version: 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5
compatibility::simple-validator-upgrade::liveness-check : committed: 6732.39 txn/s, latency: 4199.67 ms, (p50: 3700 ms, p90: 5600 ms, p99: 17700 ms), latency samples: 283480
2. Upgrading first Validator to new version: 9182255629aa9b5a988eebd5ed85f942172a1083
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6474.05 txn/s, latency: 4305.34 ms, (p50: 4400 ms, p90: 6100 ms, p99: 6300 ms), latency samples: 117320
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7506.61 txn/s, latency: 4243.07 ms, (p50: 4400 ms, p90: 4700 ms, p99: 4800 ms), latency samples: 253800
3. Upgrading rest of first batch to new version: 9182255629aa9b5a988eebd5ed85f942172a1083
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6479.47 txn/s, latency: 4129.80 ms, (p50: 4100 ms, p90: 5700 ms, p99: 5900 ms), latency samples: 124100
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6851.77 txn/s, latency: 4508.14 ms, (p50: 4500 ms, p90: 6900 ms, p99: 7200 ms), latency samples: 234260
4. upgrading second batch to new version: 9182255629aa9b5a988eebd5ed85f942172a1083
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10306.56 txn/s, latency: 2692.30 ms, (p50: 2800 ms, p90: 3400 ms, p99: 3800 ms), latency samples: 190180
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 9987.97 txn/s, latency: 3231.24 ms, (p50: 3100 ms, p90: 4300 ms, p99: 4700 ms), latency samples: 340900
5. check swarm health
Compatibility test for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 9182255629aa9b5a988eebd5ed85f942172a1083 passed
Test Ok

github-actions · 2024-08-02T16:56:20Z

✅ Forge suite `framework_upgrade` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `9182255629aa9b5a988eebd5ed85f942172a1083`

Compatibility test results for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 9182255629aa9b5a988eebd5ed85f942172a1083 (PR)
Upgrade the nodes to version: 9182255629aa9b5a988eebd5ed85f942172a1083
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1127.74 txn/s, submitted: 1129.69 txn/s, failed submission: 1.95 txn/s, expired: 1.95 txn/s, latency: 2928.40 ms, (p50: 2100 ms, p90: 6300 ms, p99: 10600 ms), latency samples: 92380
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 974.36 txn/s, submitted: 976.22 txn/s, failed submission: 1.86 txn/s, expired: 1.86 txn/s, latency: 3381.52 ms, (p50: 2400 ms, p90: 6900 ms, p99: 10700 ms), latency samples: 83720
5. check swarm health
Compatibility test for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 9182255629aa9b5a988eebd5ed85f942172a1083 passed
Upgrade the remaining nodes to version: 9182255629aa9b5a988eebd5ed85f942172a1083
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1060.75 txn/s, submitted: 1062.75 txn/s, failed submission: 2.00 txn/s, expired: 2.00 txn/s, latency: 2971.64 ms, (p50: 2100 ms, p90: 5400 ms, p99: 12100 ms), latency samples: 95480
Test Ok

github-actions · 2024-08-02T17:13:23Z

✅ Forge suite `realistic_env_max_load` success on `9182255629aa9b5a988eebd5ed85f942172a1083`

two traffics test: inner traffic : committed: 11627.70 txn/s, latency: 3420.66 ms, (p50: 3300 ms, p90: 4000 ms, p99: 6200 ms), latency samples: 4421140
two traffics test : committed: 99.90 txn/s, latency: 2801.70 ms, (p50: 2600 ms, p90: 3600 ms, p99: 9100 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.246, avg: 0.228", "QsPosToProposal: max: 0.558, avg: 0.492", "ConsensusProposalToOrdered: max: 0.341, avg: 0.334", "ConsensusOrderedToCommit: max: 0.630, avg: 0.576", "ConsensusProposalToCommit: max: 0.971, avg: 0.910"]
Max round gap was 1 [limit 4] at version 2493834. Max no progress secs was 7.235482 [limit 15] at version 2493834.
Test Ok

fEst1ck self-assigned this Jul 13, 2024

fEst1ck added the compiler-v2 label Jul 13, 2024

fEst1ck force-pushed the anonymous-fields branch 2 times, most recently from 06df31f to 2131fe6 Compare July 16, 2024 05:50

fEst1ck marked this pull request as ready for review July 16, 2024 05:51

fEst1ck requested review from wrwg, brmataptos, vineethk and rahxephon89 July 16, 2024 05:51

fEst1ck force-pushed the anonymous-fields branch from 2131fe6 to 485fe79 Compare July 16, 2024 19:33

fEst1ck marked this pull request as draft July 19, 2024 04:08

fEst1ck force-pushed the anonymous-fields branch 2 times, most recently from 094a397 to b2919c5 Compare July 19, 2024 06:40

fEst1ck marked this pull request as ready for review July 19, 2024 19:10

fEst1ck force-pushed the anonymous-fields branch from b2919c5 to e02216a Compare July 20, 2024 02:29

brmataptos reviewed Jul 22, 2024

View reviewed changes

third_party/move/move-compiler/src/parser/syntax.rs Outdated Show resolved Hide resolved

brmataptos reviewed Jul 22, 2024

View reviewed changes

fEst1ck force-pushed the anonymous-fields branch from b14be48 to 86e5f61 Compare July 25, 2024 05:45

fEst1ck requested a review from brmataptos July 25, 2024 05:53

fEst1ck changed the title ~~[Compiler-V2] [Feature] Anonymous Fields~~ [Compiler-V2] [Feature] Positional Fields Jul 25, 2024

brmataptos reviewed Jul 26, 2024

View reviewed changes

vineethk reviewed Jul 27, 2024

View reviewed changes

fEst1ck marked this pull request as draft July 27, 2024 03:04

fEst1ck force-pushed the anonymous-fields branch from 6b0c1b4 to 09a63bc Compare July 29, 2024 23:23

Zekun Wang added 16 commits August 2, 2024 12:01

update tests

ab5d079

update tests

dde138f

bug fix

995cd9c

format

e163eb7

update tests

2cf75bd

format

ab16601

refactor

f8537e1

refactor

b709d8a

comment

8f639c4

update test

fc6081b

comment

0cb9484

add tests

0f8f38e

comment

631c01d

update tests

1ad3d06

update test

17ca414

format

3ace642

fEst1ck force-pushed the anonymous-fields branch from a26a886 to 3ace642 Compare August 2, 2024 16:01

This comment has been minimized.

Sign in to view

add exp file

9182255

This comment has been minimized.

Sign in to view

fEst1ck merged commit 1df4d9b into aptos-labs:main Aug 2, 2024
51 of 54 checks passed

		@@ -0,0 +1,7 @@

		Diagnostics:
		error: expected struct constructor `test::E1{ ... }` for struct constructor `test::E1`

	let unpack = Spanned::new(lv.loc.clone(), unpack_);
	let unpack = Spanned::new(lv.loc, unpack_);

[Compiler-V2] [Feature] Positional Fields #14000

[Compiler-V2] [Feature] Positional Fields #14000

Conversation

fEst1ck commented Jul 13, 2024 • edited Loading

Description

Implementation

Type of Change

Which Components or Systems Does This Change Impact?

How Has This Been Tested?

Checklist

trunk-io bot commented Jul 13, 2024 • edited Loading

codecov bot commented Jul 16, 2024

Codecov Report

brmataptos commented Jul 22, 2024

brmataptos commented Jul 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fEst1ck commented Jul 25, 2024 • edited Loading

fEst1ck commented Jul 25, 2024 • edited Loading

brmataptos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vineethk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Aug 2, 2024

✅ Forge suite compat success on 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 9182255629aa9b5a988eebd5ed85f942172a1083

This comment has been minimized.

github-actions bot commented Aug 2, 2024

✅ Forge suite framework_upgrade success on 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 9182255629aa9b5a988eebd5ed85f942172a1083

This comment has been minimized.

github-actions bot commented Aug 2, 2024

✅ Forge suite realistic_env_max_load success on 9182255629aa9b5a988eebd5ed85f942172a1083

fEst1ck commented Jul 13, 2024 •

edited

Loading

trunk-io bot commented Jul 13, 2024 •

edited

Loading

fEst1ck commented Jul 25, 2024 •

edited

Loading

fEst1ck commented Jul 25, 2024 •

edited

Loading

✅ Forge suite `compat` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `9182255629aa9b5a988eebd5ed85f942172a1083`

✅ Forge suite `framework_upgrade` success on `1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5` ==> `9182255629aa9b5a988eebd5ed85f942172a1083`

✅ Forge suite `realistic_env_max_load` success on `9182255629aa9b5a988eebd5ed85f942172a1083`