-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ability to add more specific pod annotations #777
Comments
+1. This occurs when using istio. It injects a sidecar that uses JSON format logging, where the 'main' container does not. Kubernetes allows a single slash in an annotation, so the above suggested format is not legal.
So it could use a dash if we dropped the stdout/stderr: I'm not sure how you'd get the stdout/stderr since the log file might be merged. |
This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser-container`).
This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser-container`).
This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser-container`). Signed-off-by: Don Bowman <[email protected]>
for comment and test, #789 I made this bit fluentbit.io/parser[-container]: parsername. |
This addresses fluent/fluent-bit#777 to include `fluentbit.io/parser[-container]: parser`. Signed-off-by: Don Bowman <[email protected]>
In my opinion, being able to specify a different parser for stdout and stderr is a must: Apache access logs and error logs are fundamentally different for example. If we cannot use '/' as a delimiter, maybe we can use '_' as it is not allowed in container names if I remember correctly. |
can't use _, its part of the current parsing strategy.
i could make it super complex like:
fluentbit.io/parser: { container: { stdout_parser: x, stderr_parser: y }
or i can add another field on the end:
fluentbit.io/parser[-container][-stream]
i guess (?)
other suggestions?
it would be nicer if the parser had the logic (like grok) so that if (start
regex) then (end-regex1 else end-regex2). because the reality is the entire
stdout stream is not in the same format either (nor is stderr).
…On Fri, 21 Sep 2018 at 16:23, Yann Soubeyrand ***@***.***> wrote:
In my opinion, being able to specify a different parser for stdout and
stderr is a must: Apache access logs and error logs are fundamentally
different for example. If we cannot use '/' as a delimiter, maybe we can
use '_' as it is not allowed in container names if I remember correctly.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#777 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AE5OkybMVHSHoJtQerRGIjvJOaB2NwEJks5udUrbgaJpZM4WwoIH>
.
|
shows the rules. I guess we could do fluentbit.io/parser[-stdout|stderr][-container] @ the cost of an ambiguity if there is a container called stdout or stderr in a pod. |
it's bad that we are restricted in the format by k8s spec... I just did a quick look but what about:
where
|
@bigkraig @homer6 @StevenACoffman, would you please review this proposal and provide some feedback from an usability point of view ? |
I don't think
So I can use So |
@donbowman good point, yeah, 'all' sounds good |
I went a similar path recently in the ALB Ingress controller. Should the container name come before the stream? It should go from least to most specific and I think having the option of not providing a stream makes sense. I don’t know if you would configure parsers on things other than containers in an annotation, but if so you may want to add the ‘container’ word as an identifier as well |
fluentbit.io/parsers.containers.CONTAINERNAME fluentbit.io/parsers.containers.CONTAINERNAME.stdout Sorry for the lack of quotes. I’m on a phone on my way to a soccer game 😃 |
This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser-container`). Signed-off-by: Don Bowman <[email protected]>
This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser-container`). Signed-off-by: Don Bowman <[email protected]>
i don't think it makes sense to have a log parser on anything other than
the container-level or pod-level (what else would print logs? logs are
inherently per container).
so it sounds like `fluentbit.io/parsers[.CONTAINERNAME][.STREAM]`, with the
restriction that if you want per-stream you must also specify per-container.
…On Sat, 22 Sep 2018 at 13:37, Kraig Amador ***@***.***> wrote:
fluentbit.io/parsers.containers.CONTAINERNAME
fluentbit.io/parsers.containers.CONTAINERNAME.stdout
Sorry for the lack of quotes. I’m on a phone on my way to a soccer game 😃
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#777 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AE5OkzWwQZCUtP2DYGk6IG4cOV5xQtyfks5udnVmgaJpZM4WwoIH>
.
|
Getting the stream is not that simple. flb_filter_do() calls the filters, but the msgpack there has only 2 entries(time, log). It no longer has access to the raw docker log (which had stream: stdout in it). I tried adding Still looking. |
Just want to add that being able to differentiate between |
This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser[.container][.stream]`). Stream should be one of stdout|stderr. So example annotations could be: ``` fluentbit.io/parser.istio-init.stdout": "init_stdout" "fluentbit.io/parser.session-db": "redis" ``` Signed-off-by: Don Bowman <[email protected]>
This addresses fluent/fluent-bit#777 to include `fluentbit.io/parser[-container]: parser`. Signed-off-by: Don Bowman <[email protected]>
OK I have committed this change to the PR #789 this supports: fluentbit.io/parser[.container][.stream] as to the early comment about nginx, this will work for nginx in kubernetes. |
there's a side-affect of this. Elasticsearch does not like a mix - match of It ends up wanting, in the 2nd case, to parse that path.
|
I don't think that's an issue since labels are a better selector for such
cases
…On Tue, Sep 25, 2018, 07:05 Don Bowman ***@***.***> wrote:
there's a side-affect of this. Elasticsearch does not like a mix - match of
fluentbit.io/parser: foo
fluentbit.io/parser.bob: foo
It ends up wanting, in the 2nd case, to parse that path.
curl -s 'elasticsearch.logging:9200/logstash-2018.09.24/_mapping/flb_type/field/kubernetes.annotations.fluent*' | python3 -m json.tool
{
"logstash-2018.09.24": {
"mappings": {
"flb_type": {
"kubernetes.annotations.fluentbit.io/parser.keyword": {
"full_name": "kubernetes.annotations.fluentbit.io/parser.keyword",
"mapping": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"kubernetes.annotations.fluentbit.io/parser": {
"full_name": "kubernetes.annotations.fluentbit.io/parser",
"mapping": {
"io/parser": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#777 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAWkNi8bG8tTg5Qv7vkoPXYIccwrU_Teks5ueiovgaJpZM4WwoIH>
.
|
its absolutely an issue. elastic stops working. it confuses it that a field is sometimes a scalar and sometimes an object. you see the mapping: io/parser above? So i think we cannot use a 'dot' as a delimiter in this way. this is a more general issue, e.g. if anyone (other than us) has a field which is somtimes A, sometimes A.B, it will lock up. fluentbit keeps trying to log it, elastic keeps returning an error.
|
A work-around is:
so, what is the suggested solution?
One solution is:
Other suggestions? |
why? the current method works for all container names without restriction. Kubernetes has a bit of an opinion on this internal to it. Its log-file format: shipping-notify-64db5dd76b-r9mb8_sock-shop_istio-init-d22b9937e84aca64ab211756e2f7e86c1150ebfb13f1f30a4cfe77abb91e1473.log is effectively since we don't need namespace or pod (its implicit in the meta data already), we can use the pattern of _ if you want to be consistent. its disallowed as a first-char in a container name. |
do you mean like this ?:
|
or all _ should work too. i don't think the 2nd last can be done. its again ambiguous. i think just: if you want the 2nd last, i've currently got it set so that if stderr parser is not set it uses stdout, but we could create an 'all' stream: fluentbit.io/parser_all_container. i'll ask again, the current implementation, that's not ok? its a lot of semantic discussion over the order of where 'stdout' goes :) |
ok, from slack, proposal (final unless someone complains):
the first case is the current case, so no change. or, simplified:
There is no ambiguity since a container cannot start w/ _ nor - comments please? I would like to finish this today. |
Sorry to be insistent, but to me, it seems more logical to order annotation parts by specificity and to use _ everywhere to be consistent:
even if it forbids having a container named stderr or stdout (wouldn't it be weird anyway?). |
this would mean a container name starting w/ stderr/stdout would be treated incorrectly. |
No, just two container names would be problematic: stdout and stderr. _ is forbidden in container names (to my knowledge). If a container name could contain _, it would be problematic for the log filename nomenclature ;-) |
so why break this?
and then the value is separated by a - why should we make a restriction on container names that is trivially worked around by using the -? if that's the case, we can use what i currently have implemented, the stdout-parser, stderr-parser instead. |
the thinking about the nomenclature is: Fluent Bit annotation will have 4 components:
so config key metadata follows after a -, e.g: fluentbit.io/parser_stderr-mymetadata |
I surveyed what the rest of the kubernetes ecosystem does, and I can find no examples mixing both underscores and dashes, and my instinct is to immediately try to correct it. Just look at this odd duck:
The contribution by @donbowman is a genuine improvement as is:
The only prominent alternatives in the kubernetes ecosystem are Istio and Prometheus.
I strongly suggest using the @donbowman solution as is, and if the descriptive power of annotations is insufficient in the future, then follow the example of either Prometheus or Istio. |
OK, I get the point with config key and config key metadata ;-) |
@StevenACoffman thanks for the comments. My only concern with stdout-parser and stderr-parser is that they do not start with parser, which I think makes easier to understand the annotation context (I am open to restrict containers starting with stdout and stderr) |
BTW, I looked more at how Amabassador with Istio uses annotations with json values and it looks like that would require a lot of careful thought to get it right if fluent-bit went down that path. If each pod is allowed to opt-in to self-service configuration of parser selection for it's logs via an annotation value of json something like:
This would potentially preclude containers named "container" or "streams" or "stdout" or "stderr". |
if that means that potentially we need to add a Yaml parser I would prefer to avoid such situation. |
i believe it would come in as json (given we are using the json API). That being said, can we get some consensus here? I'm strongly opposed to any solution that doesn't work with 100% of container names (given the current one does and it doesn't seem to be compelling why we should have some restriction). The current implementation... Whats wrong with it other than some don't like the stream on the left side of the word parser? Its all 'dash' based, so that objection is gone, it works with all container names. Failing that, if I do the complex annotation like istio uses, would that be ok? I really want to get this finished, i've spent far more time on it than I would have liked. |
If this option do not restrict any container I am happy with it:
disclaimer: I care a lot about readability and context (specially about configs), having the config name on the left is a must for me. |
…container] Discussion occured in [777](fluent#777 (comment)) This allows the user to select either a global behaviour for the pod, or a per stream behaviour, or a per-stream + per-container behaviour. Signed-off-by: Don Bowman <[email protected]>
PR is updated w/ the commit above |
thanks @donbowman for the hard work on this and community for their continuous feedback. @donbowman so PR #789 is updated ? |
yes PR #789 789 is updated |
…netes. (#789) * Add container-specific annotation parser for Kubernetes. This addresses #777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser[.container][.stream]`). Stream should be one of stdout|stderr. So example annotations could be: ``` fluentbit.io/parser.istio-init.stdout": "init_stdout" "fluentbit.io/parser.session-db": "redis" ``` Signed-off-by: Don Bowman <[email protected]> * Change Kubernetes parser annotation to fluentbit.io/parser[_stream][-container] Discussion occured in [777](#777 (comment)) This allows the user to select either a global behaviour for the pod, or a per stream behaviour, or a per-stream + per-container behaviour. Signed-off-by: Don Bowman <[email protected]>
got it. #789 merged, thanks! |
…netes. (fluent#789) * Add container-specific annotation parser for Kubernetes. This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser[.container][.stream]`). Stream should be one of stdout|stderr. So example annotations could be: ``` fluentbit.io/parser.istio-init.stdout": "init_stdout" "fluentbit.io/parser.session-db": "redis" ``` Signed-off-by: Don Bowman <[email protected]> * Change Kubernetes parser annotation to fluentbit.io/parser[_stream][-container] Discussion occured in [777](fluent#777 (comment)) This allows the user to select either a global behaviour for the pod, or a per stream behaviour, or a per-stream + per-container behaviour. Signed-off-by: Don Bowman <[email protected]>
@donbowman Thanks for your work! @edsiper Could you please reopen this issue since only the parser annotation has been addressed and the issue was about the exclude annotation as well? Or do you prefer to track it in a new issue? |
@yann-soubeyrand new issue please. |
…netes. (fluent#789) * Add container-specific annotation parser for Kubernetes. This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser[.container][.stream]`). Stream should be one of stdout|stderr. So example annotations could be: ``` fluentbit.io/parser.istio-init.stdout": "init_stdout" "fluentbit.io/parser.session-db": "redis" ``` Signed-off-by: Don Bowman <[email protected]> * Change Kubernetes parser annotation to fluentbit.io/parser[_stream][-container] Discussion occured in [777](fluent#777 (comment)) This allows the user to select either a global behaviour for the pod, or a per stream behaviour, or a per-stream + per-container behaviour. Signed-off-by: Don Bowman <[email protected]>
…netes. (fluent#789) * Add container-specific annotation parser for Kubernetes. This addresses fluent#777. Annotations on a Pod currently include `fluentbit.io/parser`. This adds an optional -container (e.g. `fluentbit.io/parser[.container][.stream]`). Stream should be one of stdout|stderr. So example annotations could be: ``` fluentbit.io/parser.istio-init.stdout": "init_stdout" "fluentbit.io/parser.session-db": "redis" ``` Signed-off-by: Don Bowman <[email protected]> * Change Kubernetes parser annotation to fluentbit.io/parser[_stream][-container] Discussion occured in [777](fluent#777 (comment)) This allows the user to select either a global behaviour for the pod, or a per stream behaviour, or a per-stream + per-container behaviour. Signed-off-by: Don Bowman <[email protected]>
…andbox` (fluent#777) Signed-off-by: Felix Seifert <[email protected]>
Is your feature request related to a problem? Please describe.
When making use of the Kubernetes filter plugin, one can use a pod's annotation to specify a parser which Fluent Bit could use to parse this pod's logs. One can also use a pod's annotation to exclude this pod's logs. However, a pod can be composed of several containers each of which can use different log formats. Moreover, the containers can log to their stdout and stderr streams which can also use different log formats.
Describe the solution you'd like
To solve this problem, Fluent Bit could support more specific pod annotations. For example, a pod could specify the following annotations:
The order of precedence for the pod's annotations could be (from the highest precedence to the lowest):
Describe alternatives you've considered
As a workaround, I send raw logs to Fluentd and do static parsing there (static meaning without making use of pod's annotations).
Additional context
What I'm trying to accomplish is parsing and filtering Kubernetes logs with Fluent Bit prior to sending it to Elasticsearch or Fluentd. See #776 for more context.
The text was updated successfully, but these errors were encountered: