Generate config Go code from schema #10694

ptodev · 2024-07-22T14:56:10Z

Description

This is an attempt to generate the config.go files from Json schema. The goal of the PR is to delete all the config.go files for each component and replace them with some sort of Yaml or Json schema, which will most likely be sourced from the metadata.yaml file for each component. This PR uses an earlier PR as a starting point.

There are many unknowns related to this project. I will start threads on various problems via file comments and line comments, so that we can keep the discussion organised.

Link to tracking issue

Fixes #9769

cmd/mdatagen/testdata/config_gen/expected_golang_output/batch

ptodev · 2024-07-23T08:58:54Z

cmd/mdatagen/testdata/config_gen/expected_golang_output/batch

+	SendBatchMaxSize int `json:"send_batch_max_size,omitempty" yaml:"send_batch_max_size,omitempty" mapstructure:"send_batch_max_size,omitempty"`
+
+	// SendBatchSize corresponds to the JSON schema field "send_batch_size".
+	SendBatchSize int `json:"send_batch_size,omitempty" yaml:"send_batch_size,omitempty" mapstructure:"send_batch_size,omitempty"`


go-jsonschema doesn't seem to use uint. In theory, the minimum: 0 in the schema would be a good enough workaround but it looks like it's not supported right now.

Would you be happy if instances of uint are replaced with int + a validation that the number is >= 0?

github-actions · 2024-08-07T03:16:15Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

ptodev · 2024-08-13T17:40:45Z

processor/batchprocessor/config.go

+	if v, ok := raw["timeout"]; !ok || v == nil {
+		var err error
+		plain.Timeout, err = time.ParseDuration("200ms")
+		if err != nil {
+			return err
+		}
 	}


Here I cheated a little bit. The original code was:

if v, ok := raw["timeout"]; !ok || v == nil { plain.Timeout = "200ms" }

I edited it a bit so that it compiles. I will have to update the upstream PR for time.Duration support to generate this kind of code correctly.

ptodev · 2024-08-13T17:42:07Z

processor/batchprocessor/config_helper.go

I propose that we add files like this whenever we need to augment the auto-generated code with some additional behaviour. For example, in this file we added Validate() and createDefaultConfig() functions.

ptodev · 2024-08-13T17:43:58Z

processor/batchprocessor/metadata.yaml

+    send_batch_size:
+      type: integer
+      minimum: 0
+      default: 8192


Setting a default value is important, because otherwise g--jsonschema will autogenerate an *int. Similarly, for string properties which don't have a default value it'd generate a *string. If we do set a default value, we will get an int and a string, with no pointers.

ptodev · 2024-08-13T17:44:58Z

processor/batchprocessor/config.go

-	Timeout time.Duration `mapstructure:"timeout"`
+	// MetadataCardinalityLimit corresponds to the JSON schema field
+	// "metadata_cardinality_limit".
+	MetadataCardinalityLimit int `mapstructure:"metadata_cardinality_limit,omitempty"`


I need to check whether having the omitempty is a problem. OTel doesn't use it much right now, but it looks like go-jsonschema always puts it in.

I don't think omitempty is used by mapstructure at all.

ptodev · 2024-08-13T17:46:37Z

processor/batchprocessor/config.go

-	// Default value is 0, that means no maximum size.
-	SendBatchMaxSize uint32 `mapstructure:"send_batch_max_size"`
+	// SendBatchMaxSize corresponds to the JSON schema field "send_batch_max_size".
+	SendBatchMaxSize int `mapstructure:"send_batch_max_size,omitempty"`


Do we care much that the uint is now an int?
I suppose we don't, because we can add a check to make sure it's not negative. Also, I doubt that the integers people configure are so large that an int wouldn't fit them.

We do - at the very least we cannot change types like this. We need to deprecate and move over with a set of changes over multiple releases.

I could update my fork of go-jsonschema to use uint whenever there is a minimum: 0 or an exclusiveMinimum: 0? It seems like a reasonable feature - maybe it'll also get accepted upstream.

ptodev · 2024-08-13T17:49:12Z

processor/batchprocessor/metadata.yaml

+    send_batch_max_size:
+      type: integer
+      default: 0
+      minimum: 0


At the moment, go-jsonschema's main branch doesn't do anything when it sees constraints such as minimum and maximum. However, I patched my own fork of go-jsonschema so that it supports it. You can see in the config.go that there are checks to make sure some integers are not negative.

Maybe we can keep uint32 and we won't have to set such constraints?

Json schema itself doesn't seem to have an unsigned integer type. It only has integer and number (float) types, which can have various constraints.

ptodev · 2024-08-13T17:55:17Z

exporter/otlphttpexporter/config.go


-// Config defines configuration for OTLP/HTTP exporter.
-type Config struct {
-	confighttp.ClientConfig `mapstructure:",squash"`     // squash ensures fields are correctly decoded in embedded struct.


Unfortunately, I can't find a good way to schematise squashed fields. This is my main blocker right now.

I was hoping that we can use something like anyOf, but judging by some examples, it also looks awkward.

For now, I just listed all the squashed properties explicitly in metadata.yaml. Does anyone know of a better way? I'm not sure what folks working with Go code normally do in such situations.

Hi, @atoulme! Good news! 🥳 I was able to find a way to create squashed structs with go-jsonschema. I will update this PR next week, once I polish my code a bit. The new generated code will look a lot more like the current OTel code.

ptodev · 2024-08-13T18:00:27Z

cmd/mdatagen/go.mod

@@ -100,3 +113,5 @@ replace go.opentelemetry.io/collector/internal/globalgates => ../../internal/glo
 replace go.opentelemetry.io/collector/consumer/consumerprofiles => ../../consumer/consumerprofiles

 replace go.opentelemetry.io/collector/consumer/consumertest => ../../consumer/consumertest
+
+replace github.com/atombender/go-jsonschema => github.com/ptodev/go-jsonschema v0.0.0-20240813163654-5518ba93ee84


I had to fork go-jsonschema, because it didn't contain some of the features which I needed. I hope we can upstream those.

ptodev · 2024-08-13T18:33:36Z

config/confighttp/confighttp.json

I need to rewrite these shared configs in yaml.

atoulme · 2024-08-13T20:28:08Z

exporter/otlpexporter/otlp.go

@@ -65,7 +65,7 @@ func (e *baseExporter) start(ctx context.Context, host component.Host) (err erro
 	e.metricExporter = pmetricotlp.NewGRPCClient(e.clientConn)
 	e.logExporter = plogotlp.NewGRPCClient(e.clientConn)
 	headers := map[string]string{}
-	for k, v := range e.config.ClientConfig.Headers {
+	for k, v := range e.config.Headers {


Please revert this change

processor/batchprocessor/config_test.go

atoulme · 2024-08-13T20:31:20Z

processor/batchprocessor/config.go

-	if cfg.SendBatchMaxSize > 0 && cfg.SendBatchMaxSize < cfg.SendBatchSize {
-		return errors.New("send_batch_max_size must be greater or equal to send_batch_size")
+// UnmarshalJSON implements json.Unmarshaler.
+func (j *Config) UnmarshalJSON(b []byte) error {


We don't use UnmarshalJSON at all. We use the Validate function to validate.

atoulme · 2024-08-13T20:32:42Z

exporter/otlpexporter/config.go

 }
-
-func (c *Config) sanitizedEndpoint() string {


how is this functionality achieved?

At the moment it is not. The migration of otlpexporter and otlphttpexporter to Json schema isn't complete, due to a blocker I hit with migrating squashed fields. I've been hoping we can find a solution to that blocker before we I fix the functionality of otlpexporter and otlphttpexporter.

But in general, we could put functions like this in .go helper files which are not generated by Json schema.

atoulme · 2024-08-13T20:33:55Z

This PR is too big - please start with something much smaller.

ptodev

This PR is too big - please start with something much smaller.

Hi, @atoulme, thank you for your review! I agree that the PR is very large. The main reason for this is to demonstrate how using shared configuration could work between components. This was raised by @yurishkuro in the previous PR. The HTTP server/client settings are the primary example of shared config that I could find in the codebase.

Unfortunately, while working on shared configs, I realised that I don't know how to represent squashed fields in a clean way. I've been hoping that some of the OTel maintainers may have experience with Json schema and that the solution may be obvious to those folks.

ptodev · 2024-08-19T12:16:09Z

exporter/otlpexporter/config.go

 }
-
-func (c *Config) sanitizedEndpoint() string {


At the moment it is not. The migration of otlpexporter and otlphttpexporter to Json schema isn't complete, due to a blocker I hit with migrating squashed fields. I've been hoping we can find a solution to that blocker before we I fix the functionality of otlpexporter and otlphttpexporter.

But in general, we could put functions like this in .go helper files which are not generated by Json schema.

ptodev · 2024-08-19T12:19:27Z

processor/batchprocessor/config.go

-	// Default value is 0, that means no maximum size.
-	SendBatchMaxSize uint32 `mapstructure:"send_batch_max_size"`
+	// SendBatchMaxSize corresponds to the JSON schema field "send_batch_max_size".
+	SendBatchMaxSize int `mapstructure:"send_batch_max_size,omitempty"`


I could update my fork of go-jsonschema to use uint whenever there is a minimum: 0 or an exclusiveMinimum: 0? It seems like a reasonable feature - maybe it'll also get accepted upstream.

ptodev · 2024-08-19T12:22:27Z

processor/batchprocessor/metadata.yaml

+    send_batch_max_size:
+      type: integer
+      default: 0
+      minimum: 0


Json schema itself doesn't seem to have an unsigned integer type. It only has integer and number (float) types, which can have various constraints.

ptodev force-pushed the json-schema-new-fork branch from 25932da to 81cb055 Compare July 22, 2024 15:32

ptodev commented Jul 22, 2024

View reviewed changes

cmd/mdatagen/testdata/config_gen/expected_golang_output/batch Outdated Show resolved Hide resolved

ptodev commented Jul 23, 2024

View reviewed changes

ptodev mentioned this pull request Aug 5, 2024

Add a "duration" string format omissis/go-jsonschema#257

Draft

github-actions bot added the Stale label Aug 7, 2024

Generate config Go code from schema

4c62d16

ptodev force-pushed the json-schema-new-fork branch from 81cb055 to 4c62d16 Compare August 13, 2024 17:36

ptodev commented Aug 13, 2024

View reviewed changes

ptodev mentioned this pull request Aug 13, 2024

Auto Generation of Component Configuration from metadata.yaml open-telemetry/opentelemetry-collector-contrib#27003

Closed

ptodev commented Aug 13, 2024

View reviewed changes

config/confighttp/confighttp.json Outdated

Copy link

Author

ptodev Aug 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to rewrite these shared configs in yaml.

atoulme reviewed Aug 13, 2024

View reviewed changes

processor/batchprocessor/config_test.go Outdated Show resolved Hide resolved

atoulme reviewed Aug 13, 2024

View reviewed changes

github-actions bot removed the Stale label Aug 14, 2024

ptodev mentioned this pull request Aug 14, 2024

Splunk HEC Exporter Integration for Grafana Alloy grafana/alloy#1226

Open

Remove unnecessary comment.

b6a3daf

ptodev commented Aug 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate config Go code from schema #10694

Generate config Go code from schema #10694

ptodev commented Jul 22, 2024

ptodev Jul 23, 2024

github-actions bot commented Aug 7, 2024

ptodev Aug 13, 2024

ptodev Aug 13, 2024

ptodev Aug 13, 2024

ptodev Aug 13, 2024

atoulme Aug 13, 2024

ptodev Aug 13, 2024

atoulme Aug 13, 2024

ptodev Aug 19, 2024

ptodev Aug 13, 2024

atoulme Aug 13, 2024

ptodev Aug 19, 2024

ptodev Aug 13, 2024

ptodev Aug 30, 2024

yurishkuro Aug 30, 2024

ptodev Aug 13, 2024

ptodev Aug 13, 2024

atoulme Aug 13, 2024

atoulme Aug 13, 2024

atoulme Aug 13, 2024

ptodev Aug 19, 2024

atoulme commented Aug 13, 2024

ptodev left a comment

ptodev Aug 19, 2024

ptodev Aug 19, 2024

ptodev Aug 19, 2024

Generate config Go code from schema #10694

Are you sure you want to change the base?

Generate config Go code from schema #10694

Conversation

ptodev commented Jul 22, 2024

Description

Link to tracking issue

Choose a reason for hiding this comment

github-actions bot commented Aug 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atoulme commented Aug 13, 2024

ptodev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment