forked from haproxy/haproxy
-
Notifications
You must be signed in to change notification settings - Fork 0
/
SPOE.txt
1255 lines (918 loc) · 45.8 KB
/
SPOE.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
-----------------------------------------------
Stream Processing Offload Engine (SPOE)
Version 1.2
( Last update: 2020-06-13 )
-----------------------------------------------
Author : Christopher Faulet
Contact : cfaulet at haproxy dot com
SUMMARY
--------
0. Terms
1. Introduction
2. SPOE configuration
2.1. SPOE scope
2.2. "spoe-agent" section
2.3. "spoe-message" section
2.4. "spoe-group" section
2.5. Example
3. SPOP specification
3.1. Data types
3.2. Frames
3.2.1. Frame capabilities
3.2.2. Frame types overview
3.2.3. Workflow
3.2.4. Frame: HAPROXY-HELLO
3.2.5. Frame: AGENT-HELLO
3.2.6. Frame: NOTIFY
3.2.7. Frame: ACK
3.2.8. Frame: HAPROXY-DISCONNECT
3.2.9. Frame: AGENT-DISCONNECT
3.3. Events & messages
3.4. Actions
3.5. Errors & timeouts
4. Logging
0. Terms
---------
* SPOE : Stream Processing Offload Engine.
A SPOE is a filter talking to servers managed by a SPOA to offload the
stream processing. An engine is attached to a proxy. A proxy can have
several engines. Each engine is linked to an agent and only one.
* SPOA : Stream Processing Offload Agent.
A SPOA is a service that will receive info from a SPOE to offload the
stream processing. An agent manages several servers. It uses a backend to
reference all of them. By extension, these servers can also be called
agents.
* SPOP : Stream Processing Offload Protocol, used by SPOEs to talk to SPOA
servers.
This protocol is used by engines to talk to agents. It is an in-house
binary protocol described in this documentation.
1. Introduction
----------------
SPOE is a feature introduced in HAProxy 1.7. It makes possible the
communication with external components to retrieve some info. The idea started
with the problems caused by most ldap libs not working fine in event-driven
systems (often at least the connect() is blocking). So, it is hard to properly
implement Single Sign On solution (SSO) in HAProxy. The SPOE will ease this
kind of processing, or we hope so.
Now, the aim of SPOE is to allow any kind of offloading on the streams. First
releases won't do lot of things. As we will see, there are few handled events
and even less actions supported. Actually, for now, the SPOE can offload the
processing before "tcp-request content", "tcp-response content", "http-request"
and "http-response" rules. And it only supports variables definition. But, in
spite of these limited features, we can easily imagine to implement SSO
solution, ip reputation or ip geolocation services.
Some example implementations in various languages are linked to from the
HAProxy Wiki page dedicated to this mechanism:
https://github.com/haproxy/wiki/wiki/SPOE:-Stream-Processing-Offloading-Engine
2. SPOE configuration
----------------------
Because SPOE is implemented as a filter, To use it, you must declare a "filter
spoe" line in a proxy section (frontend/backend/listen) :
frontend my-front
...
filter spoe [engine <name>] config <file>
...
The "config" parameter is mandatory. It specifies the SPOE configuration
file. The engine name is optional. It can be set to declare the scope to use in
the SPOE configuration. So it is possible to use the same SPOE configuration
for several engines. If no name is provided, the SPOE configuration must not
contain any scope directive.
We use a separate configuration file on purpose. By commenting SPOE filter
line, you completely disable the feature, including the parsing of sections
reserved to SPOE. This is also a way to keep the HAProxy configuration clean.
A SPOE configuration file must contains, at least, the SPOA configuration
("spoe-agent" section) and SPOE messages/groups ("spoe-message" or "spoe-group"
sections) attached to this agent.
IMPORTANT : The configuration of a SPOE filter must be located in a dedicated
file. But the backend used by a SPOA must be declared in HAProxy configuration
file.
2.1. SPOE scope
-------------------------
If you specify an engine name on the SPOE filter line, then you need to define
scope in the SPOE configuration with the same name. You can have several SPOE
scope in the same file. In each scope, you must define one and only one
"spoe-agent" section to configure the SPOA linked to your SPOE and several
"spoe-message" and "spoe-group" sections to describe, respectively, messages and
group of messages sent to servers managed by your SPOA.
A SPOE scope starts with this kind of line :
[<name>]
where <name> is the same engine name specified on the SPOE filter line. The
scope ends when the file ends or when another scope is found.
Example :
[my-first-engine]
spoe-agent my-agent
...
spoe-message msg1
...
spoe-message msg2
...
spoe-group grp1
...
spoe-group grp2
...
[my-second-engine]
...
If no engine name is provided on the SPOE filter line, no SPOE scope must be
found in the SPOE configuration file. All the file is considered to be in the
same anonymous and implicit scope.
The engine name must be uniq for a proxy. If no engine name is provided on the
SPOE filter line, the SPOE agent name is used by default.
2.2. "spoe-agent" section
--------------------------
For each engine, you must define one and only one "spoe-agent" section. In this
section, you will declare SPOE messages and the backend you will use. You will
also set timeouts and options to customize your agent's behaviour.
spoe-agent <name>
Create a new SPOA with the name <name>. It must have one and only one
"spoe-agent" definition by SPOE scope.
Arguments :
<name> is the name of the agent section.
following keywords are supported :
- groups
- log
- maxconnrate
- maxerrrate
- max-frame-size
- max-waiting-frames
- messages
- [no] option async
- [no] option dontlog-normal
- [no] option pipelining
- [no] option send-frag-payload
- option continue-on-error
- option force-set-var
- option set-on-error
- option set-process-time
- option set-total-time
- option var-prefix
- register-var-names
- timeout hello|idle|processing
- use-backend
groups <grp-name> ...
Declare the list of SPOE groups that an agent will handle.
Arguments :
<grp-name> is the name of a SPOE group.
Groups declared here must be found in the same engine scope, else an error is
triggered during the configuration parsing. You can have many "groups" lines.
See also: "spoe-group" section.
log global
log <address> [len <length>] [format <format>] <facility> [<level> [<minlevel>]]
no log
Enable per-instance logging of events and traffic.
Prefix :
no should be used when the logger list must be flushed.
See the HAProxy Configuration Manual for details about this option.
maxconnrate <number>
Set the maximum number of connections per second to <number>. The SPOE will
stop to open new connections if the maximum is reached and will wait to
acquire an existing one. So it is important to set "timeout hello" to a
relatively small value.
maxerrrate <number>
Set the maximum number of errors per second to <number>. The SPOE will stop
its processing if the maximum is reached.
max-frame-size <number>
Set the maximum allowed size for frames exchanged between HAProxy and SPOA.
It must be in the range [256, tune.bufsize-4] (4 bytes are reserved for the
frame length). By default, it is set to (tune.bufsize-4).
max-waiting-frames <number>
Set the maximum number of frames waiting for an acknowledgement on the same
connection. This value is only used when the pipelinied or asynchronous
exchanges between HAProxy and SPOA are enabled. By default, it is set to 20.
messages <msg-name> ...
Declare the list of SPOE messages that an agent will handle.
Arguments :
<msg-name> is the name of a SPOE message.
Messages declared here must be found in the same engine scope, else an error
is triggered during the configuration parsing. You can have many "messages"
lines.
See also: "spoe-message" section.
option async
no option async
Enable or disable the support of asynchronous exchanges between HAProxy and
SPOA. By default, this option is enabled.
option continue-on-error
Do not stop the events processing when an error occurred on a stream.
By default, for a specific stream, when an abnormal/unexpected error occurs,
the SPOE is disabled for all the transaction. So if you have several events
configured, such error on an event will disabled all following. For TCP
streams, this will disable the SPOE for the whole session. For HTTP streams,
this will disable it for the transaction (request and response).
When set, this option bypass this behaviour and only the current event will
be ignored.
option dontlog-normal
no option dontlog-normal
Enable or disable logging of normal, successful processing.
Arguments : none
See also: "log" and section 4 about logging.
option force-set-var
By default, SPOE filter only register already known variables (mainly from
parsing of the configuration), and process-wide variables (those of scope
"proc") cannot be created. If you want that haproxy trusts the agent and
registers all variables (ex: can be useful for LUA workload), activate this
option.
Caution : this option opens to a variety of attacks such as a rogue SPOA that
asks to register too many variables.
option pipelining
no option pipelining
Enable or disable the support of pipelined exchanges between HAProxy and
SPOA. By default, this option is enabled.
option send-frag-payload
no option send-frag-payload
Enable or disable the sending of fragmented payload to SPOA. By default, this
option is enabled.
option set-on-error <var name>
Define the variable to set when an error occurred during an event processing.
Arguments :
<var name> is the variable name, without the scope. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
This variable will only be set when an error occurred in the scope of the
transaction. As for all other variables define by the SPOE, it will be
prefixed. So, if your variable name is "error" and your prefix is
"my_spoe_pfx", the variable will be "txn.my_spoe_pfx.error".
When set, the variable is an integer representing the error reason. For values
under 256, it represents an error coming from the engine. Below 256, it
reports a SPOP error. In this case, to retrieve the right SPOP status code,
you must remove 256 to this value. Here are possible values:
* 1 a timeout occurred during the event processing.
* 2 an error was triggered during the resources allocation.
* 3 the frame payload exceeds the frame size and it cannot be
fragmented.
* 4 the fragmentation of a payload is aborted.
* 5 The frame processing has been interrupted by HAProxy.
* 255 an unknown error occurred during the event processing.
* 256+N a SPOP error occurred during the event processing (see section
"Errors & timeouts").
Note that if "option continue-on-error" is set, the variable is not
automatically removed between events processing.
See also: "option continue-on-error", "option var-prefix".
option set-process-time <var name>
Define the variable to set to report the processing time of the last event or
group.
Arguments :
<var name> is the variable name, without the scope. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
This variable will be set in the scope of the transaction. As for all other
variables define by the SPOE, it will be prefixed. So, if your variable name
is "process_time" and your prefix is "my_spoe_pfx", the variable will be
"txn.my_spoe_pfx.process_time".
When set, the variable is an integer representing the delay to process the
event or the group, in milliseconds. From the stream point of view, it is the
latency added by the SPOE processing for the last handled event or group.
If several events or groups are processed for the same stream, this value
will be overrideen.
See also: "option set-total-time".
option set-total-time <var name>
Define the variable to set to report the total processing time SPOE for a
stream.
Arguments :
<var name> is the variable name, without the scope. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
This variable will be set in the scope of the transaction. As for all other
variables define by the SPOE, it will be prefixed. So, if your variable name
is "total_time" and your prefix is "my_spoe_pfx", the variable will be
"txn.my_spoe_pfx.total_time".
When set, the variable is an integer representing the sum of processing times
for a stream, in milliseconds. From the stream point of view, it is the
latency added by the SPOE processing.
If several events or groups are processed for the same stream, this value
will be updated.
See also: "option set-process-time".
option var-prefix <prefix>
Define the prefix used when variables are set by an agent.
Arguments :
<prefix> is the prefix used to limit the scope of variables set by an
agent.
To avoid conflict with other variables defined by HAProxy, all variables
names will be prefixed. By default, the "spoe-agent" name is used. This
option can be used to customize it.
The prefix will be added between the variable scope and its name, separated
by a '.'. It may only contain characters 'a-z', 'A-Z', '0-9', '.' and '_', as
for variables name. In HAProxy configuration, you need to use this prefix as
a part of the variables name. For example, if an agent define the variable
"myvar" in the "txn" scope, with the prefix "my_spoe_pfx", then you should
use "txn.my_spoe_pfx.myvar" name in your HAProxy configuration.
By default, an agent will never set new variables at runtime: It can only set
new value for existing ones. If you want a different behaviour, see
force-set-var option and register-var-names directive.
register-var-names <var name> ...
Register some variable names. By default, an agent will not be allowed to set
new variables at runtime. This rule can be totally relaxed by setting the
option "force-set-var". If you know all the variables you will need, this
directive is a good way to register them without letting an agent doing what
it want. This is only required if these variables are not referenced anywhere
in the HAProxy configuration or the SPOE one.
Arguments:
<var name> is a variable name without the scope. The name may only
contain characters 'a-z', 'A-Z', '0-9', '.' and '_'.
The prefix will be automatically added during the registration. You can have
many "register-var-names" lines.
See also: "option force-set-var", "option var-prefix".
timeout hello <timeout>
Set the maximum time to wait for an agent to receive the AGENT-HELLO frame.
It is applied on the stream that handle the connection with the agent.
Arguments :
<timeout> is the timeout value specified in milliseconds by default, but
can be in any other unit if the number is suffixed by the unit,
as explained at the top of this document.
This timeout is an applicative timeout. It differ from "timeout connect"
defined on backends.
timeout idle <timeout>
Set the maximum time to wait for an agent to close an idle connection. It is
applied on the stream that handle the connection with the agent.
Arguments :
<timeout> is the timeout value specified in milliseconds by default, but
can be in any other unit if the number is suffixed by the unit,
as explained at the top of this document.
timeout processing <timeout>
Set the maximum time to wait for a stream to process an event, i.e to acquire
a stream to talk with an agent, to encode all messages, to send the NOTIFY
frame, to receive the corresponding acknowledgement and to process all
actions. It is applied on the stream that handle the client and the server
sessions.
Arguments :
<timeout> is the timeout value specified in milliseconds by default, but
can be in any other unit if the number is suffixed by the unit,
as explained at the top of this document.
use-backend <backend>
Specify the backend to use. It must be defined.
Arguments :
<backend> is the name of a valid "backend" section.
2.3. "spoe-message" section
----------------------------
To offload the stream processing, SPOE will send messages with specific
information at a specific moment in the stream life and will wait for
corresponding replies to know what to do.
spoe-message <name>
Create a new SPOE message with the name <name>.
Arguments :
<name> is the name of the SPOE message.
Here you define a message that can be referenced in a "spoe-agent"
section. Following keywords are supported :
- acl
- args
- event
See also: "spoe-agent" section.
acl <aclname> <criterion> [flags] [operator] <value> ...
Declare or complete an access list.
See section 7 about ACL usage in the HAProxy Configuration Manual.
args [name=]<sample> ...
Define arguments passed into the SPOE message.
Arguments :
<sample> is a sample expression.
When the message is processed, if a sample expression is not available, it is
set to NULL. Arguments are processed in their declaration order and added in
the message in that order. It is possible to declare named arguments.
For example:
args frontend=fe_id src dst
event <name> [ { if | unless } <condition> ]
Set the event that triggers sending of the message. It may optionally be
followed by an ACL-based condition, in which case it will only be evaluated
if the condition is true. A SPOE message can only be sent on one event. If
several events are defined, only the last one is considered.
ACL-based conditions are executed in the context of the stream that handle
the client and the server connections.
Arguments :
<name> is the event name.
<condition> is a standard ACL-based condition.
Supported events are:
- on-client-session
- on-server-session
- on-frontend-tcp-request
- on-backend-tcp-request
- on-tcp-response
- on-frontend-http-request
- on-backend-http-request
- on-http-response
See section "Events & Messages" for more details about supported events.
See section 7 about ACL usage in the HAProxy Configuration Manual.
2.4. "spoe-group" section
--------------------------
This section can be used to declare a group of SPOE messages. Unlike messages
referenced in a "spoe-agent" section, messages inside a group are not sent on a
specific event. The sending must be triggered by TCP or HTTP rules, from the
HAProxy configuration.
spoe-group <name>
Create a new SPOE group with the name <name>.
Arguments :
<name> is the name of the SPOE group.
Here you define a group of SPOE messages that can be referenced in a
"spoe-agent" section. Following keywords are supported :
- messages
See also: "spoe-agent" and "spoe-message" sections.
messages <msg-name> ...
Declare the list of SPOE messages belonging to the group.
Arguments :
<msg-name> is the name of a SPOE message.
Messages declared here must be found in the same engine scope, else an error
is triggered during the configuration parsing. Furthermore, a message belongs
at most to a group. You can have many "messages" lines.
See also: "spoe-message" section.
2.5. Example
-------------
Here is a simple but complete example that sends client-ip address to a ip
reputation service. This service can set the variable "ip_score" which is an
integer between 0 and 100, indicating its reputation (100 means totally safe
and 0 a blacklisted IP with no doubt).
###
### HAProxy configuration
frontend www
mode http
bind *:80
filter spoe engine ip-reputation config spoe-ip-reputation.conf
# Reject connection if the IP reputation is under 20
tcp-request content reject if { var(sess.iprep.ip_score) -m int lt 20 }
default_backend http-servers
backend http-servers
mode http
server http A.B.C.D:80
backend iprep-servers
mode tcp
balance roundrobin
timeout connect 5s # greater than hello timeout
timeout server 3m # greater than idle timeout
server iprep1 A1.B1.C1.D1:12345
server iprep2 A2.B2.C2.D2:12345
####
### spoe-ip-reputation.conf
[ip-reputation]
spoe-agent iprep-agent
messages get-ip-reputation
option var-prefix iprep
timeout hello 2s
timeout idle 2m
timeout processing 10ms
use-backend iprep-servers
spoe-message get-ip-reputation
args ip=src
event on-client-session if ! { src -f /etc/haproxy/whitelist.lst }
3. SPOP specification
----------------------
3.1. Data types
----------------
Here is the bytewise representation of typed data:
TYPED-DATA : <TYPE:4 bits><FLAGS:4 bits><DATA>
Supported types and their representation are:
TYPE | ID | DESCRIPTION
-----------------------------+-----+----------------------------------
NULL | 0 | NULL : <0>
Boolean | 1 | BOOL : <1+FLAG>
32bits signed integer | 2 | INT32 : <2><VALUE:varint>
32bits unsigned integer | 3 | UINT32 : <3><VALUE:varint>
64bits signed integer | 4 | INT64 : <4><VALUE:varint>
32bits unsigned integer | 5 | UNIT64 : <5><VALUE:varint>
IPV4 | 6 | IPV4 : <6><STRUCT IN_ADDR:4 bytes>
IPV6 | 7 | IPV6 : <7><STRUCT IN_ADDR6:16 bytes>
String | 8 | STRING : <8><LENGTH:varint><BYTES>
Binary | 9 | BINARY : <9><LENGTH:varint><BYTES>
10 -> 15 unused/reserved | - | -
-----------------------------+-----+----------------------------------
Variable-length integer (varint) are encoded using Peers encoding:
0 <= X < 240 : 1 byte (7.875 bits) [ XXXX XXXX ]
240 <= X < 2288 : 2 bytes (11 bits) [ 1111 XXXX ] [ 0XXX XXXX ]
2288 <= X < 264432 : 3 bytes (18 bits) [ 1111 XXXX ] [ 1XXX XXXX ] [ 0XXX XXXX ]
264432 <= X < 33818864 : 4 bytes (25 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*2 [ 0XXX XXXX ]
33818864 <= X < 4328786160 : 5 bytes (32 bits) [ 1111 XXXX ] [ 1XXX XXXX ]*3 [ 0XXX XXXX ]
...
For booleans, the value (true or false) is the first bit in the FLAGS
bitfield. if this bit is set to 0, then the boolean is evaluated as false,
otherwise, the boolean is evaluated as true.
3.2. Frames
------------
Exchange between HAProxy and agents are made using FRAME packets. All frames
must be prefixed with their size encoded on 4 bytes in network byte order:
<FRAME-LENGTH:4 bytes> <FRAME>
A frame always starts with its type, on one byte, followed by metadata
containing flags, on 4 bytes and a two variable-length integer representing the
stream identifier and the frame identifier inside the stream:
FRAME : <FRAME-TYPE:1 byte> <METADATA> <FRAME-PAYLOAD>
METADATA : <FLAGS:4 bytes> <STREAM-ID:varint> <FRAME-ID:varint>
Then comes the frame payload. Depending on the frame type, the payload can be
of three types: a simple key/value list, a list of messages or a list of
actions.
FRAME-PAYLOAD : <LIST-OF-MESSAGES> | <LIST-OF-ACTIONS> | <KV-LIST>
LIST-OF-MESSAGES : [ <MESSAGE-NAME> <NB-ARGS:1 byte> <KV-LIST> ... ]
MESSAGE-NAME : <STRING>
LIST-OF-ACTIONS : [ <ACTION-TYPE:1 byte> <NB-ARGS:1 byte> <ACTION-ARGS> ... ]
ACTION-ARGS : [ <TYPED-DATA>... ]
KV-LIST : [ <KV-NAME> <KV-VALUE> ... ]
KV-NAME : <STRING>
KV-VALUE : <TYPED-DATA>
FLAGS :
Flags are a 32 bits field. They are encoded on 4 bytes in network byte
order, where the bit 0 is the LSB.
0 1 2-31
+---+---+----------+
| | A | |
| F | B | |
| I | O | RESERVED |
| N | R | |
| | T | |
+---+---+----------+
FIN: Indicates that this is the final payload fragment. The first fragment
may also be the final fragment.
ABORT: Indicates that the processing of the current frame must be
cancelled. This bit should be set on frames with a fragmented
payload. It can be ignore for frames with an unfragemnted
payload. When it is set, the FIN bit must also be set.
Frames cannot exceed a maximum size negotiated between HAProxy and agents
during the HELLO handshake. Most of time, payload will be small enough to send
it in one frame. But when supported by the peer, it will be possible to
fragment huge payload on many frames. This ability is announced during the
HELLO handshake and it can be asynmetric (supported by agents but not by
HAProxy or the opposite). The following rules apply to fragmentation:
* An unfragemnted payload consists of a single frame with the FIN bit set.
* A fragemented payload consists of several frames with the FIN bit clear and
terminated by a single frame with the FIN bit set. All these frames must
share the same STREAM-ID and FRAME-ID. The first frame must set the right
FRAME-TYPE (e.g, NOTIFY). The following frames must have an unset type (0).
Beside the support of fragmented payload by a peer, some payload must not be
fragmented. See below for details.
IMPORTANT : The maximum size supported by peers for a frame must be greater
than or equal to 256 bytes.
3.2.1. Frame capabilities
--------------------------
Here are the list of official capabilities that HAProxy and agents can support:
* fragmentation: This is the ability for a peer to support fragmented
payload in received frames. This is an asymmectical
capability, it only concerns the peer that announces
it. This is the responsibility to the other peer to use it
or not.
* pipelining: This is the ability for a peer to decouple NOTIFY and ACK
frames. This is a symmectical capability. To be used, it must
be supported by HAProxy and agents. Unlike HTTP pipelining, the
ACK frames can be send in any order, but always on the same TCP
connection used for the corresponding NOTIFY frame.
* async: This ability is similar to the pipelining, but here any TCP
connection established between HAProxy and the agent can be used to
send ACK frames. if an agent accepts connections from multiple
HAProxy, it can use the "engine-id" value to group TCP
connections. See details about HAPROXY-HELLO frame.
Unsupported or unknown capabilities are silently ignored, when possible.
NOTE: HAProxy does not support the fragmentation for now. This means it is not
able to handle fragmented frames. However, if an agent announces the
fragmentation support, HAProxy may choose to send fragemented frames.
3.2.2. Frame types overview
----------------------------
Here are types of frame supported by SPOE. Frames sent by HAProxy come first,
then frames sent by agents :
TYPE | ID | DESCRIPTION
-----------------------------+-----+-------------------------------------
UNSET | 0 | Used for all frames but the first when a
| | payload is fragmented.
-----------------------------+-----+-------------------------------------
HAPROXY-HELLO | 1 | Sent by HAProxy when it opens a
| | connection on an agent.
| |
HAPROXY-DISCONNECT | 2 | Sent by HAProxy when it want to close
| | the connection or in reply to an
| | AGENT-DISCONNECT frame
| |
NOTIFY | 3 | Sent by HAProxy to pass information
| | to an agent
-----------------------------+-----+-------------------------------------
AGENT-HELLO | 101 | Reply to a HAPROXY-HELLO frame, when
| | the connection is established
| |
AGENT-DISCONNECT | 102 | Sent by an agent just before closing
| | the connection
| |
ACK | 103 | Sent to acknowledge a NOTIFY frame
-----------------------------+-----+-------------------------------------
Unknown frames may be silently skipped.
3.2.3. Workflow
----------------
* Successful HELLO handshake:
HAPROXY AGENT SRV
| HAPROXY-HELLO |
| (healthcheck: false) |
| --------------------------> |
| |
| AGENT-HELLO |
| <-------------------------- |
| |
* Successful HELLO healthcheck:
HAPROXY AGENT SRV
| HAPROXY-HELLO |
| (healthcheck: true) |
| --------------------------> |
| |
| AGENT-HELLO + close() |
| <-------------------------- |
| |
* Error encountered by agent during the HELLO handshake:
HAPROXY AGENT SRV
| HAPROXY-HELLO |
| --------------------------> |
| |
| DISCONNECT + close() |
| <-------------------------- |
| |
* Error encountered by HAProxy during the HELLO handshake:
HAPROXY AGENT SRV
| HAPROXY-HELLO |
| --------------------------> |
| |
| AGENT-HELLO |
| <-------------------------- |
| |
| DISCONNECT |
| --------------------------> |
| |
| DISCONNECT + close() |
| <-------------------------- |
| |
* Notify / Ack exchange (unfragmented payload):
HAPROXY AGENT SRV
| NOTIFY |
| --------------------------> |
| |
| ACK |
| <-------------------------- |
| |
* Notify / Ack exchange (fragmented payload):
HAPROXY AGENT SRV
| NOTIFY (frag 1) |
| --------------------------> |
| |
| UNSET (frag 2) |
| --------------------------> |
| ... |
| UNSET (frag N) |
| --------------------------> |
| |
| ACK |
| <-------------------------- |
| |
* Aborted fragmentation of a NOTIFY frame:
HAPROXY AGENT SRV
| ... |
| UNSET (frag X) |
| --------------------------> |
| |
| ACK/ABORT |
| <-------------------------- |
| |
| UNSET (frag X+1) |
| -----------X |
| |
| |
* Connection closed by haproxy:
HAPROXY AGENT SRV
| DISCONNECT |
| --------------------------> |
| |
| DISCONNECT + close() |
| <-------------------------- |
| |
* Connection closed by agent:
HAPROXY AGENT SRV
| DISCONNECT + close() |
| <-------------------------- |
| |
3.2.4. Frame: HAPROXY-HELLO
----------------------------
This frame is the first one exchanged between HAProxy and an agent, when the
connection is established. The payload of this frame is a KV-LIST. It cannot be
fragmented. STREAM-ID and FRAME-ID are must be set 0.
Following items are mandatory in the KV-LIST:
* "supported-versions" <STRING>
Last SPOP major versions supported by HAProxy. It is a comma-separated list
of versions, following the format "Major.Minor". Spaces must be ignored, if
any. When a major version is announced by HAProxy, it means it also support
all previous minor versions.
Example: "2.0, 1.5" means HAProxy supports SPOP 2.0 and 1.0 to 1.5
* "max-frame-size" <UINT32>
This is the maximum size allowed for a frame. The HAPROXY-HELLO frame must
be lower or equal to this value.
* "capabilities" <STRING>
This a comma-separated list of capabilities supported by HAProxy. Spaces
must be ignored, if any.
Following optional items can be added in the KV-LIST:
* "healthcheck" <BOOLEAN>
If this item is set to TRUE, then the HAPROXY-HELLO frame is sent during a
SPOE health check. When set to FALSE, this item can be ignored.
* "engine-id" <STRING>
This is a uniq string that identify a SPOE engine.
To finish the HELLO handshake, the agent must return an AGENT-HELLO frame with
its supported SPOP version, the lower value between its maximum size allowed
for a frame and the HAProxy one and capabilities it supports. If an error
occurs or if an incompatibility is detected with the agent configuration, an
AGENT-DISCONNECT frame must be returned.
3.2.5. Frame: AGENT-HELLO
--------------------------
This frame is sent in reply to a HAPROXY-HELLO frame to finish a HELLO
handshake. As for HAPROXY-HELLO frame, STREAM-ID and FRAME-ID are also set
0. The payload of this frame is a KV-LIST and it cannot be fragmented.
Following items are mandatory in the KV-LIST:
* "version" <STRING>
This is the SPOP version the agent supports. It must follow the format
"Major.Minor" and it must be lower or equal than one of major versions
announced by HAProxy.
* "max-frame-size" <UINT32>
This is the maximum size allowed for a frame. It must be lower or equal to
the value in the HAPROXY-HELLO frame. This value will be used for all
subsequent frames.
* "capabilities" <STRING>
This a comma-separated list of capabilities supported by agent. Spaces must
be ignored, if any.
At this time, if everything is ok for HAProxy (supported version and valid
max-frame-size value), the HELLO handshake is successfully completed. Else,
HAProxy sends a HAPROXY-DISCONNECT frame with the corresponding error.
If "healthcheck" item was set to TRUE in the HAPROXY-HELLO frame, the agent can
safely close the connection without DISCONNECT frame. In all cases, HAProxy
will close the connection at the end of the health check.
3.2.6. Frame: NOTIFY
---------------------
Information are sent to the agents inside NOTIFY frames. These frames are
attached to a stream, so STREAM-ID and FRAME-ID must be set. The payload of
NOTIFY frames is a LIST-OF-MESSAGES and, if supported by agents, it can be