-
Notifications
You must be signed in to change notification settings - Fork 133
[BUG] - KoP CPU usage is 10 times higher than Pulsar on the broker side #1652
Comments
Please refer to the flame graph HTML file. |
What's your KoP version? It looks like the However, the public static String sanitizeMetricName(String metricName) {
int length = metricName.length();
char[] sanitized = new char[length];
for(int i = 0; i < length; ++i) {
char ch = metricName.charAt(i);
if (ch != ':' && (ch < 'a' || ch > 'z') && (ch < 'A' || ch > 'Z') && (i <= 0 || ch < '0' || ch > '9')) {
sanitized[i] = '_';
} else {
sanitized[i] = ch;
}
}
return new String(sanitized);
} |
KOP- 2.10.1.11 |
It seems to be a bug of Pulsar itself. I checked the prometheus dependency of Pulsar 2.10.2, the dependency is still 0.5.0, which has a inefficient implementation of apache-pulsar-2.10.2$ ls lib | grep prometheus
io.prometheus-simpleclient-0.5.0.jar The prometheus dependency was upgraded from apache/pulsar#13785, which cannot be cherry-picked to branch-2.10. I think we need to implement the method manually to avoid the upgrade. |
Fixes streamnative#1652 ### Motivation Pulsar 2.10 or earlier depends on io.prometheus:simpleclient:0.5.0, whose implementation of `Collectors.sanitizeMetricName` uses regex, which is inefficient and costs much CPU. ### Modifications Migrate the implementation of `sanitizeMetricName` from io:prometheus:simpleclient:0.16.0 to branch-2.10 or earlier.
hmm, I double-check the pulsar(broker) lib folder and find the
|
…me (#1654) Fixes #1652 ### Motivation Pulsar 2.10 or earlier depends on io.prometheus:simpleclient:0.5.0, whose implementation of `Collectors.sanitizeMetricName` uses regex, which is inefficient and costs much CPU. ### Modifications Migrate the implementation of `sanitizeMetricName` from io:prometheus:simpleclient:0.16.0 to branch-2.10 or earlier.
…me (streamnative#1654) Fixes streamnative#1652 ### Motivation Pulsar 2.10 or earlier depends on io.prometheus:simpleclient:0.5.0, whose implementation of `Collectors.sanitizeMetricName` uses regex, which is inefficient and costs much CPU. ### Modifications Migrate the implementation of `sanitizeMetricName` from io:prometheus:simpleclient:0.16.0 to branch-2.10 or earlier.
After I have upgraded(I cherry-pick the code from this MR - apache/pulsar#13785) the Prometheus client version from 0.5.0 to 0.15.0, the CPU usage still exceeds 30 and the Prometheus-related code costs more 24% than before(0.5.0). P1 - The CPU usage with Prometheus version 0.5.0 P2 - The CPU usage with Prometheus version 0.15.0. P3 - Prometheus version 0.5.0. The Prometheus-related code costs 37.93% CPU. P4 - Prometheus version 0.15.0. The Prometheus-related code costs 62.13% CPU. The broker Prometheus libs
|
The |
Summary: After upgrading the Prometheus version to 0.16.0 in both Pulsar and KoP. The Prometheus-related code CPU usage is 0.58% now based on the Flame graph. The Broker's CPU usage reduces by around 16.3%. (Before 33%, Now 16.7%) |
Describe the bug
KoP CUP usage is 10 times higher than Pulsar on the broker side
To Reproduce
Steps to reproduce the behavior:
Expected behavior
KoP CPU usage should be greater than or equal to the Pulsar itself.
In this case, when I use the Pulsar client, the broker CPU usage is around 3. When I use Kafka client, the broker CPU usage is around 30.
Expected result: When I use Kafka client, the broker CPU usage >=3, but not too high.
Screenshots
![image](https://user-images.githubusercontent.com/3076703/209929906-9f801467-7c6c-4546-8e24-bc2133865165.png)
![image](https://user-images.githubusercontent.com/3076703/209929945-571dfb52-db8c-4773-a1bc-5e1c50609327.png)
The broker(KoP) flame graph
Pulsar client vs. Kafka client(CUP usage)
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: