It is not because things are difficult that we do not dare; it is because we do not dare that things are difficult.
Seneca
HAProxy provides Prometheus metrics by default. The provided metrics group HTTP status codes as
2xx
, 3xx
, 4xx
, and 5xx
. This is intentional, as these groupings are often sufficient at the
reverse proxy layer.
For more detailed information, logs can be utilised. HAProxy supports customised log formats, offering numerous options.
Managing and storing logs can be costly. Metrics are efficient and cheap store and easier to process.
mtail is a tool that extracts metrics from logs using regular expressions.
HAProxy does not support writing logs directly to the filesystem. To save logs to a filesystem,
a specialised tool is required. rsyslog can handle the task easily.
The rsyslog
configuration is simple:
# Load UDP module
module(load="imudp")
input(type="imudp" port="5113")
# Define template for log file path
template(name="Everything" type="string" string="/opt/log/everything.log")
# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
/opt/log/haproxy.log
stop
}
# Log everything to the specified template
*.* ?Everything
# Provide logging through standard error, useful when running in Docker
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$FileCreateMode 0644
$DirCreateMode 0755
$Umask 0022
# Ensure all log files have the correct permissions
$FileCreateMode 0644
$DirCreateMode 0755
$Umask 0022
The HAProxy log format used is as follows:
%ci:%cp %ft %b/%s %ST %B %Tr %ac/%fc/%bc/%sc/%rc %sq/%bq %{+Q}r
HAProxy supports an extremely wide range of log formats. The above example is a simple yet functional one.
The mtail
configuration is as follows:
counter mtail_line_count by process, pid
counter mtail_haproxy_http_requests_total by frontend_name_transport, backend_name, status_code, request_method, http_version
counter mtail_haproxy_http_nosrv_total by frontend_name_transport, backend_name, status_code, request_method, http_version
histogram mtail_haproxy_http_response_duration_ms buckets 0.0, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0 by backend_name, server_name
hidden gauge response_time_ms
def syslog {
/^(?P<date>(?P<legacy_date>\w+\s+\d+\s+\d+:\d+:\d+)|(?P<rfc3339_date>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+\d{2}:\d{2}))/ +
/\s+(?P<hostname>[\w\.-]+)\s+(?P<application>[\w\.-]+)(?:\[(?P<pid>\d+)\])?:\s+(?P<message>.*)/ {
len($legacy_date) > 0 {
strptime($2, "Jan _2 15:04:05")
}
len($rfc3339_date) > 0 {
strptime($rfc3339_date, "2006-01-02T15:04:05-07:00")
}
mtail_line_count[$application][$pid]++
next
}
}
@syslog {
/\d+(\.\d+){3}:\d+ / +
/(?P<frontend_name_transport>http-\w+) / +
/(?P<backend_name>\w+)\/<?(?P<server_name>[<>\w\.-]+)>? / +
/(?P<status_code>\d{3}) / +
/(?P<bytes_read>\d+) / +
/(?P<response_time>-?\d+) / +
/(?P<actconn>-?\d+)\// +
/(?P<feconn>-?\d+)\// +
/(?P<beconn>-?\d+)\// +
/(?P<srv_conn>\d+)\// +
/(?P<retries>\d+) / +
/(?P<srv_queue>\d+)\/(?P<backend_queue>\d+) / +
/"(?P<request_method>[A-Z]+) (?P<URI>\S+) (?P<http_version>HTTP\/[0-9\.]+)"/ +
/$/ {
# Handle response_time (%Tr):
$response_time =~ /^-?\d+$|^0$/ {
response_time_ms = 0.0
} else {
response_time_ms = $response_time / 1000.0
}
mtail_haproxy_http_requests_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version]++
/<NOSRV>/ {
mtail_haproxy_http_nosrv_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version]++
} else {
mtail_haproxy_http_response_duration_ms[$backend_name][$server_name] = response_time_ms
}
}
}
The @syslog
decorator is likely unnecessary here. The rest is a regular expression that matches the HAProxy log format.
In this case, we are extracting mtail_haproxy_http_requests_total
and a few other metrics. With mtail
, we can extract
any type of metrics we need from logs.
The setup can be visualized as follows:
+------------------+ +------------------+ +------------------+
| | | | | |
| HAProxy | logs to UDP | rsyslogd | saves to file | mtail pod |
| | port 5113/udp | (port 5113) | /opt/log/haproxy.log | transforms |
| |--------------->| | | logs into metrics|
| | | |<----------------------| exposed at port |
| | | | | 3903 |
+------------------+ +------------------+ +------------------+
The metrics can be scraped by Prometheus. The mtail
metrics are exposed on port 3903
by default.