otlphttp Preparing to make HTTP request, Nothing after that. Traces not going out.

[manojnc]

Describe the bug
I configured the otlp exporter to send traces to honeycomb refinery residing in the same k8 cluster in the same namespace. However, the call is not going out to destination. I see a log line Preparing to make HTTP request in debug logs but nothing after that.

Steps to reproduce
My Exporter configuration

exporters:
      otlp:
        endpoint: "honeycomb-refinery.monitoring.svc.cluster.local:4317"
        tls:
          insecure: true
          insecure_skip_verify: true
        headers:
          "x-honeycomb-team": "${HONEYCOMB_API_KEY}"
          "x-honeycomb-dataset": "apps"
    service:
      extensions: [zpages, memory_ballast]
      pipelines:
        traces/1:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [otlp]
      telemetry:
          logs:
            level: "DEBUG"
            development: true
            encoding: "json"

What did you expect to see?
Traces should have been submitted to the refinery app over grpc

What did you see instead?
{"level":"debug","ts":1721409906.8589747,"caller":"[email protected]/otlp.go:153","msg":"Preparing to make HTTP request","kind":"exporter","data_type":"traces","name":"otlphttp","url":"http://refinery.monitoring.svc.cluster.local:80/v1/traces"}

What version did you use?
Main 0.105.0

What config did you use?
Already shared above

Environment
AWS EKS

Additional context
honeycomb service is running and port 80 is exposed. A get service command will report 80 is exposed and it is mapped to container port 8080. Honeycomb agent is sending the data over http to honeycomb refinery. So, I do not think there is any issue with the refinery app. Also, same error can be reproduced even if I point it to other backend ( Grafana and honeycomb)

[codeboten]

Thanks for opening the issue @manojnc, can you confirm the configuration you have? The one in the description only includes the otlp exporter, not the otlphttp exporter shown in the log line. Additionally the url for the config provided doesn't match the url in the log:

honeycomb-refinery.monitoring.svc.cluster.local:4317
refinery.monitoring.svc.cluster.local:80/v1/traces

[manojnc]

Thanks for taking a look at this @codeboten

Here is my latest exporter configuration for otlphttp. I have verified the network connectivity on this service and port combination. It would be helpful if we can add a debug log that logs the actual request and response from the server hosting the endpoint.

   exporters:
      otlphttp:
        endpoint: "http://refinery.monitoring.svc.cluster.local:6000/"
        tls:
          insecure: true
          insecure_skip_verify: true
        headers:
          "x-honeycomb-team": "${HONEYCOMB_API_KEY}"
          "x-honeycomb-dataset": "apps"

Refinery is another service in the same namespace and I have configured it to listen to port 6000 over http protocol. If I do a simple curl the refinery service responds with hello message back. So, I just want to understand what is happening to calls from otel to refinery. The mismatch in the url in debug logs is because of several iterations I was trying and during my last edit of the issue, i did not update the logs from my latest run. But the experience is same.

[yujianweilai]

When I use otel, I also encountered a similar problem, just prompting "Preparing to make HTTP request", and then there is no follow-up, backend apm (I use deepflow) can not receive any trace information generated by the application

my otel-collector config:

otel-version: 0.85.0

Is the configuration file wrong?