LogQL: Log query language on Grafana Labs

Log queries

Thu, 10 Apr 2025 12:15:54 +0000

Log queries

All LogQL queries contain a log stream selector.

Optionally, the log stream selector can be followed by a log pipeline. A log pipeline is a set of stage expressions that are chained together and applied to the selected log streams. Each expression can filter out, parse, or mutate log lines and their respective labels.

The following example shows a full log query in action:

{container="query-frontend",namespace="loki-dev"} |= "metrics.go" | logfmt | duration > 10s and throughput_mb < 500

The query is composed of:

a log stream selector {container="query-frontend",namespace="loki-dev"} which targets the query-frontend container in the loki-dev namespace.
a log pipeline |= "metrics.go" | logfmt | duration > 10s and throughput_mb < 500 which will filter out log that contains the word metrics.go, then parses each log line to extract more labels and filter with them.

To avoid escaping special characters you can use the `(backtick) instead of " when quoting strings. For example `\w+` is the same as "\\w+". This is specially useful when writing a regular expression which contains multiple backslashes that require escaping.

Log stream selector

The stream selector determines which log streams to include in a query’s results. A log stream is a unique source of log content, such as a file. A more granular log stream selector then reduces the number of searched streams to a manageable volume. This means that the labels passed to the log stream selector will affect the relative performance of the query’s execution.

The log stream selector is specified by one or more comma-separated key-value pairs. Each key is a log label and each value is that label’s value. Curly braces ({ and }) delimit the stream selector.

Consider this stream selector:

{app="mysql",name="mysql-backup"}

All log streams that have both a label of app whose value is mysql and a label of name whose value is mysql-backup will be included in the query results. A stream may contain other pairs of labels and values, but only the specified pairs within the stream selector are used to determine which streams will be included within the query results.

The same rules that apply for Prometheus Label Selectors apply for Grafana Loki log stream selectors.

The = operator after the label name is a label matching operator. The following label matching operators are supported:

=: exactly equal
!=: not equal
=~: regex matches
!~: regex does not match

Regex log stream examples:

{name =~ "mysql.+"}
{name !~ "mysql.+"}
{name !~ `mysql-\d+`}

Note: Unlike the line filter regex expressions, the =~ and !~ regex operators are fully anchored. This means that the regex expression must match against the entire string, including newlines. The regex . character does not match newlines by default. If you want the regex dot character to match newlines you can use the single-line flag, like so: (?s)search_term.+ matches search_term\n. Alternatively, you can use the \s (match whitespaces, including newline) in combination with \S (match not whitespace characters) to match all characters, including newlines. Refer to Google’s RE2 syntax for more information.

Regex log stream newlines:

{name =~ ".*mysql.*"}: does not match log label values with newline character
{name =~ "(?s).*mysql.*}: match log label values with newline character
{name =~ "[\S\s]*mysql[\S\s]*}: match log label values with newline character

Log pipeline

A log pipeline can be appended to a log stream selector to further process and filter log streams. It is composed of a set of expressions. Each expression is executed in left to right sequence for each log line. If an expression filters out a log line, the pipeline will stop processing the current log line and start processing the next log line.

Some expressions can mutate the log content and respective labels, which will be then be available for further filtering and processing in subsequent expressions. An example that mutates is the expression

| line_format "{{.status_code}}"

Log pipeline expressions fall into one of four categories:

Filtering expressions: line filter expressions and label filter expressions
Parsing expressions
Formatting expressions: line format expressions and label format expressions
Labels expressions: drop labels expression and keep labels expression

Line filter expression

The line filter expression does a distributed grep over the aggregated logs from the matching log streams. It searches the contents of the log line, discarding those lines that do not match the case-sensitive expression.

Each line filter expression has a filter operator followed by text or a regular expression. These filter operators are supported:

|=: Log line contains string
!=: Log line does not contain string
|~: Log line contains a match to the regular expression
!~: Log line does not contain a match to the regular expression

Note: Unlike the label matcher regex operators, the |~ and !~ regex operators are not fully anchored. This means that the . regex character matches all characters, including newlines.

Line filter expression examples:

Keep log lines that have the substring “error”:
```
|= "error"
```
A complete query using this example:
```
{job="mysql"} |= "error"
```
Discard log lines that have the substring “kafka.server:type=ReplicaManager”:
```
!= "kafka.server:type=ReplicaManager"
```
A complete query using this example:
```
{instance=~"kafka-[23]",name="kafka"} != "kafka.server:type=ReplicaManager"
```
Keep log lines that contain a substring that starts with tsdb-ops and ends with io:2003. A complete query with a regular expression:
```
{name="kafka"} |~ "tsdb-ops.*io:2003"
```
Keep log lines that contain a substring that starts with error=, and is followed by 1 or more word characters. A complete query with a regular expression:
```
{name="cassandra"} |~  `error=\w+`
```

Filter operators can be chained. Filters are applied sequentially. Query results will have satisfied every filter. This complete query example will give results that include the string error, and do not include the string timeout.

{job="mysql"} |= "error" != "timeout"

When using |~ and !~, Go (as in Golang) RE2 syntax regex may be used. The matching is case-sensitive by default. Switch to case-insensitive matching by prefixing the regular expression with (?i).

While line filter expressions could be placed anywhere within a log pipeline, it is almost always better to have them at the beginning. Placing them at the beginning improves the performance of the query, as it only does further processing when a line matches. For example, while the results will be the same, the query specified with

{job="mysql"} |= "error" | json | line_format "{{.err}}"

will always run faster than

{job="mysql"} | json | line_format "{{.message}}" |= "error"

Line filter expressions are the fastest way to filter logs once the log stream selectors have been applied.

Line filter expressions have support matching IP addresses. See Matching IP addresses for details.

Removing color codes

Line filter expressions support stripping ANSI sequences (color codes) from the line:

{job="example"} | decolorize

Label filter expression

Label filter expression allows filtering log line using their original and extracted labels. It can contain multiple predicates.

A predicate contains a label identifier, an operation and a value to compare the label with.

For example with cluster="namespace" the cluster is the label identifier, the operation is = and the value is “namespace”. The label identifier is always on the left side of the operation.

We support multiple value types which are automatically inferred from the query input.

String is double quoted or backticked such as "200" or `us-central1`.
Duration is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “300ms”, “1.5h” or “2h45m”. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”. The value of the label identifier used for comparison must be a string with a unit suffix to be parsed correctly, such as “0.10ms” or “1h30m”. Optionally, label_format can be used to modify the value and append the unit before making the comparison.
Number are floating-point number (64bits), such as250, 89.923.
Bytes is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “42MB”, “1.5KiB” or “20B”. Valid bytes units are “B”, “kB”, “MB”, “GB”, “TB”, “KB”, “KiB”, “MiB”, “GiB”, “TiB”.

String type work exactly like Prometheus label matchers use in log stream selector. This means you can use the same operations (=,!=,=~,!~).

The string type is the only one that can filter out a log line with a label __error__.

Using Duration, Number and Bytes will convert the label value prior to comparision and support the following comparators:

== or = for equality.
!= for inequality.
> and >= for greater than and greater than or equal.
< and <= for lesser than and lesser than or equal.

For instance, logfmt | duration > 1m and bytes_consumed > 20MB

If the conversion of the label value fails, the log line is not filtered and an __error__ label is added. To filters those errors see the pipeline errors section.

You can chain multiple predicates using and and or which respectively express the and and or binary operations. and can be equivalently expressed by a comma, a space or another pipe. Label filters can be place anywhere in a log pipeline.

This means that all the following expressions are equivalent:

| duration >= 20ms or size == 20KB and method!~"2.."
| duration >= 20ms or size == 20KB | method!~"2.."
| duration >= 20ms or size == 20KB , method!~"2.."
| duration >= 20ms or size == 20KB  method!~"2.."

The precedence for evaluation of multiple predicates is left to right. You can wrap predicates with parenthesis to force a different precedence.

These examples are equivalent:

| duration >= 20ms or method="GET" and size <= 20KB
| ((duration >= 20ms or method="GET") and size <= 20KB)

To evaluate the logical and first, use parenthesis, as in this example:

| duration >= 20ms or (method="GET" and size <= 20KB)

Label filter expressions are the only expression allowed after the unwrap expression. This is mainly to allow filtering errors from the metric extraction.

Label filter expressions have support matching IP addresses. See Matching IP addresses for details.

Parser expression

Parser expression can parse and extract labels from the log content. Those extracted labels can then be used for filtering using label filter expressions or for metric aggregations.

Extracted label keys are automatically sanitized by all parsers, to follow Prometheus metric name convention.(They can only contain ASCII letters and digits, as well as underscores and colons. They cannot start with a digit.)

For instance, the pipeline | json will produce the following mapping:

{ "a.b": {c: "d"}, e: "f" }

{a_b_c="d", e="f"}

In case of errors, for instance if the line is not in the expected format, the log line won’t be filtered but instead will get a new __error__ label added.

If an extracted label key name already exists in the original log stream, the extracted label key will be suffixed with the _extracted keyword to make the distinction between the two labels. You can forcefully override the original label using a label formatter expression. However, if an extracted key appears twice, only the first label value will be kept.

Loki supports JSON, logfmt, pattern, regexp and unpack parsers.

It’s easier to use the predefined parsers json and logfmt when you can. If you can’t, the pattern and regexp parsers can be used for log lines with an unusual structure. The pattern parser is easier and faster to write; it also outperforms the regexp parser. Multiple parsers can be used by a single log pipeline. This is useful for parsing complex logs. There are examples in Multiple parsers.

JSON

The json parser operates in two modes:

without parameters:

Adding | json to your pipeline will extract all json properties as labels if the log line is a valid json document. Nested properties are flattened into label keys using the _ separator.

Note: Arrays are skipped.

For example the json parsers will extract from the following document:

{
    "protocol": "HTTP/2.0",
    "servers": ["129.0.1.1","10.2.1.3"],
    "request": {
        "time": "6.032",
        "method": "GET",
        "host": "foo.grafana.net",
        "size": "55",
        "headers": {
          "Accept": "*/*",
          "User-Agent": "curl/7.68.0"
        }
    },
    "response": {
        "status": 401,
        "size": "228",
        "latency_seconds": "6.031"
    }
}

The following list of labels:

"protocol" => "HTTP/2.0"
"request_time" => "6.032"
"request_method" => "GET"
"request_host" => "foo.grafana.net"
"request_size" => "55"
"response_status" => "401"
"response_size" => "228"
"response_latency_seconds" => "6.031"

with parameters:

Using | json label="expression", another="expression" in your pipeline will extract only the specified json fields to labels. You can specify one or more expressions in this way, the same as label_format; all expressions must be quoted.

Currently, we only support field access (my.field, my["field"]) and array access (list[0]), and any combination of these in any level of nesting (my.list[0]["field"]).

For example, | json first_server="servers[0]", ua="request.headers[\"User-Agent\"] will extract from the following document:
JSON
```
{
    "protocol": "HTTP/2.0",
    "servers": ["129.0.1.1","10.2.1.3"],
    "request": {
        "time": "6.032",
        "method": "GET",
        "host": "foo.grafana.net",
        "size": "55",
        "headers": {
          "Accept": "*/*",
          "User-Agent": "curl/7.68.0"
        }
    },
    "response": {
        "status": 401,
        "size": "228",
        "latency_seconds": "6.031"
    }
}
```
The following list of labels:
kv
```
"first_server" => "129.0.1.1"
"ua" => "curl/7.68.0"
```
If an array or an object returned by an expression, it will be assigned to the label in json format.

For example, | json server_list="servers", headers="request.headers" will extract:
kv
```
"server_list" => `["129.0.1.1","10.2.1.3"]`
"headers" => `{"Accept": "*/*", "User-Agent": "curl/7.68.0"}`
```
If the label to be extracted is same as the original JSON field, expression can be written as just | json <label>

For example, to extract servers fields as label, expression can be written as following

| json servers will extract:
kv
```
"servers" => `["129.0.1.1","10.2.1.3"]`
```
Note that | json servers is same as | json servers="servers"

logfmt

The logfmt parser can operate in two modes:

without parameters:

The logfmt parser can be added using | logfmt and will extract all keys and values from the logfmt formatted log line.

For example the following log line:
logfmt
```
at=info method=GET path=/ host=grafana.net fwd="124.133.124.161" service=8ms status=200
```
will result in having the following labels extracted:
kv
```
"at" => "info"
"method" => "GET"
"path" => "/"
"host" => "grafana.net"
"fwd" => "124.133.124.161"
"service" => "8ms"
"status" => "200"
```
with parameters:

Similar to JSON, using | logfmt label="expression", another="expression" in the pipeline will result in extracting only the fields specified by the labels.

For example, | logfmt host, fwd_ip="fwd" will extract the labels host and fwd from the following log line:
logfmt
```
at=info method=GET path=/ host=grafana.net fwd="124.133.124.161" service=8ms status=200
```
And rename fwd to fwd_ip:
kv
```
"host" => "grafana.net"
"fwd_ip" => "124.133.124.161"
```

The logfmt parser also supports the following flags:

--strict to enable strict parsing

With strict parsing enabled, the logfmt parser immediately stops scanning the log line and returns early with an error when it encounters any poorly formatted key/value pair.
```
// accepted key/value pairs
key=value key="value in double quotes"

// invalid key/value pairs
=value // no key
foo=bar=buzz
fo"o=bar
```
Without the --strict flag the parser skips invalid key/value pairs and continues parsing the rest of the log line. Non-strict mode offers the flexibility to parse semi-structed log lines, though note that this is only best-effort.
--keep-empty to retain standalone keys with empty value

With --keep-empty flag set, the logfmt parser retains standalone keys(keys without a value) as labels with value set to empty string. If the standalone key is explicitly requested using label extraction parameters, there is no need to add this flag.

Note: flags if any should appear right after logfmt and before label extraction parameters

| logfmt --strict
| logfmt --strict host, fwd_ip="fwd"
| logfmt --keep-empty --strict host

Pattern

The pattern parser allows the explicit extraction of fields from log lines by defining a pattern expression (| pattern "<pattern-expression>"). The expression matches the structure of a log line.

Consider this NGINX log line.

0.191.12.2 - - [10/Jun/2021:09:14:29 +0000] "GET /api/plugins/versioncheck HTTP/1.1" 200 2 "-" "Go-http-client/2.0" "13.76.247.102, 34.120.177.193" "TLSv1.2" "US" ""

This log line can be parsed with the expression

<ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>

to extract these fields:

"ip" => "0.191.12.2"
"method" => "GET"
"uri" => "/api/plugins/versioncheck"
"status" => "200"
"size" => "2"
"agent" => "Go-http-client/2.0"

A pattern expression is composed of captures and literals.

A capture is a field name delimited by the < and > characters. <example> defines the field name example. An unnamed capture appears as <_>. The unnamed capture skips matched content.

Captures are matched from the line beginning or the previous set of literals, to the line end or the next set of literals. If a capture is not matched, the pattern parser will stop.

Literals can be any sequence of UTF-8 characters, including whitespace characters.

By default, a pattern expression is anchored at the start of the log line. If the expression starts with literals, then the log line must also start with the same set of literals. Use <_> at the beginning of the expression if you don’t want to anchor the expression at the start.

Consider the log line

level=debug ts=2021-06-10T09:24:13.472094048Z caller=logging.go:66 traceID=0568b66ad2d9294c msg="POST /loki/api/v1/push (204) 16.652862ms"

To match msg=", use the expression:

<_> msg="<method> <path> (<status>) <latency>"

A pattern expression is invalid if

It does not contain any named capture.
It contains two consecutive captures not separated by whitespace characters.

Regular expression

Unlike the logfmt and json, which extract implicitly all values and takes no parameters, the regexp parser takes a single parameter | regexp "<re>" which is the regular expression using the Golang RE2 syntax.

The regular expression must contain a least one named sub-match (e.g (?P<name>re)), each sub-match will extract a different label.

For example the parser | regexp "(?P<method>\\w+) (?P<path>[\\w|/]+) \$(?P<status>\\d+?)\$ (?P<duration>.*)" will extract from the following line:

POST /api/prom/api/v1/query_range (200) 1.5s

those labels:

"method" => "POST"
"path" => "/api/prom/api/v1/query_range"
"status" => "200"
"duration" => "1.5s"

unpack

The unpack parser parses a JSON log line, unpacking all embedded labels from Promtail’s pack stage. A special property _entry will also be used to replace the original log line.

For example, using | unpack with the log line:

{
  "container": "myapp",
  "pod": "pod-3223f",
  "_entry": "original log message"
}

extracts the container and pod labels; it sets original log message as the new log line.

You can combine the unpack and json parsers (or any other parsers) if the original embedded log line is of a specific format.

Line format expression

The line format expression can rewrite the log line content by using the text/template format. It takes a single string parameter | line_format "{{.label_name}}", which is the template format. All labels are injected variables into the template and are available to use with the {{.label_name}} notation.

For example the following expression:

{container="frontend"} | logfmt | line_format "{{.query}} {{.duration}}"

Will extract and rewrite the log line to only contains the query and the duration of a request.

You can use double quoted string for the template or backticks `{{.label_name}}` to avoid the need to escape special characters.

line_format also supports math functions. Example:

If we have the following labels ip=1.1.1.1, status=200 and duration=3000(ms), we can divide the duration by 1000 to get the value in seconds.

{container="frontend"} | logfmt | line_format "{{.ip}} {{.status}} {{div .duration 1000}}"

The above query will give us the line as 1.1.1.1 200 3

See template functions to learn about available functions in the template format.

Labels format expression

The | label_format expression can rename, modify or add labels. It takes as parameter a comma separated list of equality operations, enabling multiple operations at once.

When both side are label identifiers, for example dst=src, the operation will rename the src label into dst.

The right side can alternatively be a template string (double quoted or backtick), for example dst="{{.status}} {{.query}}", in which case the dst label value is replaced by the result of the text/template evaluation. This is the same template engine as the | line_format expression, which means labels are available as variables and you can use the same list of functions.

In both cases, if the destination label doesn’t exist, then a new one is created.

The renaming form dst=src will drop the src label after remapping it to the dst label. However, the template form will preserve the referenced labels, such that dst="{{.src}}" results in both dst and src having the same value.

A single label name can only appear once per expression. This means | label_format foo=bar,foo="new" is not allowed but you can use two expressions for the desired effect: | label_format foo=bar | label_format foo="new"

Drop Labels expression

Syntax: |drop name, other_name, some_name="some_value"

The | drop expression will drop the given labels in the pipeline. For example, for the query {job="varlogs"}|json|drop level, method="GET", with below log line

{"level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

the result will be

{host="grafana.net", path="/", status="200"} {"level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

Similary, this expression can be used to drop __error__ labels as well. For example, for the query {job="varlogs"}|json|drop __error__, with below log line

INFO GET / loki.net 200

the result will be

{} INFO GET / loki.net 200

Example with regex and multiple names

For the query {job="varlogs"}|json|drop level, path, app=~"some-api.*", with below log lines

{"app": "some-api-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{"app": "other-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

the result will be

{host="grafana.net", job="varlogs", method="GET", status="200"} {"app": "some-api-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{app="other-service", host="grafana.net", job="varlogs", method="GET", status="200"} {"app": "other-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

Keep Labels expression

Syntax: |keep name, other_name, some_name="some_value"

The | keep expression will keep only the specified labels in the pipeline and drop all the other labels.

Note
The keep stage will not drop the error or error_details labels added by Loki at query time. To drop these labels, refer to drop stage.

Query examples:

For the query {job="varlogs"}|json|keep level, method="GET", with the following log lines:

{"level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{"level": "info", "method": "POST", "path": "/", "host": "grafana.net", "status": "200"}

the result will be

{level="info", method="GET"} {"level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{level="info"} {"level": "info", "method": "POST", "path": "/", "host": "grafana.net", "status": "200"}

For the query {job="varlogs"}|json|keep level, tenant, app=~"some-api.*", with the following log lines:

{"app": "some-api-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{"app": "other-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

the result will be

{app="some-api-service", level="info"} {"app": "some-api-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}
{level="info"} {"app": "other-service", "level": "info", "method": "GET", "path": "/", "host": "grafana.net", "status": "200"}

Metric queries

Thu, 10 Apr 2025 12:15:54 +0000

Metric queries

Metric queries extend log queries by applying a function to log query results. This powerful feature creates metrics from logs.

Metric queries can be used to calculate the rate of error messages or the top N log sources with the greatest quantity of logs over the last 3 hours.

Combined with parsers, metric queries can also be used to calculate metrics from a sample value within the log line, such as latency or request size. All labels, including extracted ones, will be available for aggregations and generation of new series.

Range Vector aggregation

LogQL shares the range vector concept of Prometheus. In Grafana Loki, the selected range of samples is a range of selected log or label values.

The aggregation is applied over a time duration. Loki defines Time Durations with the same syntax as Prometheus.

Loki supports two types of range vector aggregations: log range aggregations and unwrapped range aggregations.

Log range aggregations

A log range aggregation is a query followed by a duration. A function is applied to aggregate the query over the duration. The duration can be placed after the log stream selector or at end of the log pipeline.

The functions:

rate(log-range): calculates the number of entries per second
count_over_time(log-range): counts the entries for each log stream within the given range.
bytes_rate(log-range): calculates the number of bytes per second for each stream.
bytes_over_time(log-range): counts the amount of bytes used by each log stream for a given range.
absent_over_time(log-range): returns an empty vector if the range vector passed to it has any elements and a 1-element vector with the value 1 if the range vector passed to it has no elements. (absent_over_time is useful for alerting on when no time series and logs stream exist for label combination for a certain amount of time.)

Examples:

Count all the log lines within the last five minutes for the MySQL job.
logql
```
count_over_time({job="mysql"}[5m])
```
This aggregation includes filters and parsers. It returns the per-second rate of all non-timeout errors within the last minutes per host for the MySQL job and only includes errors whose duration is above ten seconds.
logql
```
sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m]))
```

Offset modifier

The offset modifier allows changing the time offset for individual range vectors in a query.

For example, the following expression counts all the logs within the last ten minutes to five minutes rather than last five minutes for the MySQL job. Note that the offset modifier always needs to follow the range vector selector immediately.

count_over_time({job="mysql"}[5m] offset 5m) // GOOD
count_over_time({job="mysql"}[5m]) offset 5m // INVALID

Unwrapped range aggregations

Unwrapped ranges uses extracted labels as sample values instead of log lines. However to select which label will be used within the aggregation, the log query must end with an unwrap expression and optionally a label filter expression to discard errors.

The unwrap expression is noted | unwrap label_identifier where the label identifier is the label name to use for extracting sample values.

Since label values are string, by default a conversion into a float (64bits) will be attempted, in case of failure the __error__ label is added to the sample. Optionally the label identifier can be wrapped by a conversion function | unwrap <function>(label_identifier), which will attempt to convert the label value from a specific format.

We currently support the functions:

duration_seconds(label_identifier) (or its short equivalent duration) which will convert the label value in seconds from the go duration format (e.g 5m, 24s30ms).
bytes(label_identifier) which will convert the label value to raw bytes applying the bytes unit (e.g. 5 MiB, 3k, 1G).

Supported function for operating over unwrapped ranges are:

rate(unwrapped-range): calculates per second rate of the sum of all values in the specified interval.
rate_counter(unwrapped-range): calculates per second rate of the values in the specified interval and treating them as “counter metric”
sum_over_time(unwrapped-range): the sum of all values in the specified interval.
avg_over_time(unwrapped-range): the average value of all points in the specified interval.
max_over_time(unwrapped-range): the maximum value of all points in the specified interval.
min_over_time(unwrapped-range): the minimum value of all points in the specified interval
first_over_time(unwrapped-range): the first value of all points in the specified interval
last_over_time(unwrapped-range): the last value of all points in the specified interval
stdvar_over_time(unwrapped-range): the population standard variance of the values in the specified interval.
stddev_over_time(unwrapped-range): the population standard deviation of the values in the specified interval.
quantile_over_time(scalar,unwrapped-range): the φ-quantile (0 ≤ φ ≤ 1) of the values in the specified interval.
absent_over_time(unwrapped-range): returns an empty vector if the range vector passed to it has any elements and a 1-element vector with the value 1 if the range vector passed to it has no elements. (absent_over_time is useful for alerting on when no time series and logs stream exist for label combination for a certain amount of time.)

Except for sum_over_time,absent_over_time, rate and rate_counter, unwrapped range aggregations support grouping.

<aggr-op>([parameter,] <unwrapped-range>) [without|by (<label list>)]

Which can be used to aggregate over distinct labels dimensions by including a without or by clause.

without removes the listed labels from the result vector, while all other labels are preserved the output. by does the opposite and drops labels that are not listed in the by clause, even if their label values are identical between all elements of the vector.

See Unwrap examples for query examples that use the unwrap expression.

Built-in aggregation operators

Like PromQL, LogQL supports a subset of built-in aggregation operators that can be used to aggregate the element of a single vector, resulting in a new vector of fewer elements but with aggregated values:

sum: Calculate sum over labels
avg: Calculate the average over labels
min: Select minimum over labels
max: Select maximum over labels
stddev: Calculate the population standard deviation over labels
stdvar: Calculate the population standard variance over labels
count: Count number of elements in the vector
topk: Select largest k elements by sample value
bottomk: Select smallest k elements by sample value
sort: returns vector elements sorted by their sample values, in ascending order.
sort_desc: Same as sort, but sorts in descending order.

The aggregation operators can either be used to aggregate over all label values or a set of distinct label values by including a without or a by clause:

<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]

parameter is required when using topk and bottomk. topk and bottomk are different from other aggregators in that a subset of the input samples, including the original labels, are returned in the result vector.

by and without are only used to group the input vector. The without clause removes the listed labels from the resulting vector, keeping all others. The by clause does the opposite, dropping labels that are not listed in the clause, even if their label values are identical between all elements of the vector.

See vector aggregation examples for query examples that use vector aggregation expressions.

Functions

LogQL supports a set of built-in functions.

vector(s scalar): returns the scalar s as a vector with no labels. This behaves identically to the Prometheus vector() function. vector is mainly used to return a value for a series that would otherwise return nothing; this can be useful when using LogQL to define an alert.

Examples:

Count all the log lines within the last five minutes for the traefik namespace.

sum(count_over_time({namespace="traefik"}[5m])) # will return nothing
  or
vector(0) # will return 0

Template functions

Wed, 10 Jan 2024 20:44:02 +0000

Template functions

The text template format used in | line_format and | label_format support the usage of functions.

All labels are added as variables in the template engine. They can be referenced using they label name prefixed by a .(e.g .label_name). For example the following template will output the value of the path label:

{{ .path }}

Additionally you can also access the log line using the __line__ function and the timestamp using the __timestamp__ function.

You can take advantage of pipeline to join together multiple functions. In a chained pipeline, the result of each command is passed as the last argument of the following command.

Example:

{{ .path | replace " " "_" | trunc 5 | upper }}

For function that returns a bool such as contains, eq, hasPrefix and hasSuffix, you can apply AND / OR and nested if logic.

Example:

{{ if and (contains "he" "hello") (contains "llo" "hello") }} yes {{end}}
{{ if or (contains "he" "hello") (contains("llo" "hello") }} yes {{end}}
{{ if contains .err "ErrTimeout" }} timeout {{else if contains "he" "hello"}} yes {{else}} no {{end}}

line

This function returns the current log line.

Signature: line() string

Examples:

"{{ __line__ | lower }}"
`{{ __line__ }}`

timestamp

This function returns the current log lines timestamp.

Signature: timestamp() time.Time

"{{ __timestamp__ }}"
`{{ __timestamp__ | date "2006-01-02T15:04:05.00Z-07:00" }}`
`{{ __timestamp__ | unixEpoch }}`

See the blog: Parsing and formatting date/time in Go for more information.

regexReplaceAll and regexReplaceAllLiteral

regexReplaceAll returns a copy of the input string, replacing matches of the Regexp with the replacement string replacement. Inside string replacement, $ signs are interpreted as in Expand, so for instance $1 represents the text of the first sub-match. See the golang Regexp.replaceAll documentation for more examples.

Example:

`{{ regexReplaceAll "(a*)bc" .some_label "${1}a" }}`

regexReplaceAllLiteral function returns a copy of the input string and replaces matches of the Regexp with the replacement string replacement. The replacement string is substituted directly, without using Expand.

Example:

`{{ regexReplaceAllLiteral "(ts=)" .timestamp "timestamp=" }}`

lower

Use this function to convert to lower case.

Signature: lower(string) string

Examples:

"{{ .request_method | lower }}"
`{{ lower  "HELLO"}}`

The last example will return hello.

upper

Use this function to convert to upper case.

Signature: upper(string) string

Examples:

"{{ .request_method | upper }}"
`{{ upper  "hello"}}`

This results in HELLO.

title

Convert to title case.

Signature: title(string) string

Examples:

"{{.request_method | title}}"
`{{ title "hello world"}}`

The last example will return Hello World.

trunc

Truncate a string and add no suffix.

Signature: trunc(count int,value string) string

Examples:

"{{ .path | trunc 2 }}"
`{{ trunc 5 "hello world"}}`   // output: hello
`{{ trunc -5 "hello world"}}`  // output: world

substr

Get a substring from a string.

Signature: substr(start int,end int,value string) string

If start is < 0, this calls value[:end]. If start is >= 0 and end < 0 or end bigger than s length, this calls value[start:] Otherwise, this calls value[start, end].

Examples:

"{{ .path | substr 2 5 }}"
`{{ substr 0 5 "hello world"}}`  // output: hello
`{{ substr 6 11 "hello world"}}` // output: world

replace

This function performs simple string replacement.

Signature: replace(old string, new string, src string) string

It takes three arguments:

old string to replace
new string to replace with
src source string

Examples:

{{ .cluster | replace "-cluster" "" }}
{{ replace "hello" "world" "hello world" }}

The last example will return world world.

trim

The trim function removes space from either side of a string.

Signature: trim(string) string

Examples:

{{ .ip | trim }}
{{ trim "   hello    " }} // output: hello

trimAll

Use this function to remove given characters from the front or back of a string.

Signature: trimAll(chars string,src string) string

Examples:

{{ .path | trimAll "/" }}
{{ trimAll "$" "$5.00" }} // output: 5.00

trimSuffix

Use this function to trim just the suffix from a string.

Signature: trimSuffix(suffix string, src string) string

Examples:

{{  .path | trimSuffix "/" }}
{{ trimSuffix "-" "hello-" }} // output: hello

trimPrefix

Use this function to trim just the prefix from a string.

Signature: trimPrefix(prefix string, src string) string

Examples:

{{  .path | trimPrefix "/" }}
{{ trimPrefix "-" "-hello" }} // output: hello

alignLeft

Use this function to format a string to a fixed with, aligning the content to the left.

Signature: alignLeft(count int, src string) string

Examples:

`{{ alignLeft 5 "hello world"}}` // output: "hello"
`{{ alignLeft 5 "hi"}}`          // output: "hi   "

alignRight

Use this function to format a string to a fixed with, aligning the content to the right.

Signature: alignRight(count int, src string) string

Examples:

`{{ alignRight 5 "hello world"}}` // output: "world"
`{{ alignRight 5 "hi"}}`          // output: "   hi"

indent

The indent function indents every line in a given string to the specified indent width. This is useful when aligning multi-line strings.

Signature: indent(spaces int,src string) string

Example:

{{ indent 4 .query }}

This indents each line contained in the .query by four (4) spaces.

nindent

The nindent function is the same as the indent function, but prepends a new line to the beginning of the string.

Signature: nindent(spaces int,src string) string

Example:

{{ nindent 4 .query }}

This will indent every line of text by 4 space characters and add a new line to the beginning.

repeat

Use this function to repeat a string multiple times.

Signature: repeat(c int,value string) string

Example:

{{ repeat 3 "hello" }} // output: hellohellohello

contains

Use this function to test to see if one string is contained inside of another.

Signature: contains(s string, src string) bool

Examples:

{{ if contains .err "ErrTimeout" }} timeout {{end}}
{{ if contains "he" "hello" }} yes {{end}}

eq

Use this function to test to see if one string has exact matching inside of another.

Signature: eq(s string, src string) bool

Examples:

{{ if eq .err "ErrTimeout" }} timeout {{end}}
{{ if eq "he" "hello" }} yes {{end}}

hasPrefix and hasSuffix

The hasPrefix and hasSuffix functions test whether a string has a given prefix or suffix.

Signatures:

hasPrefix(prefix string, src string) bool
hasSuffix(suffix string, src string) bool

Examples:

{{ if hasSuffix .err "Timeout" }} timeout {{end}}
{{ if hasPrefix "he" "hello" }} yes {{end}}

add

Sum numbers. Supports multiple numbers

Signature: func(i ...interface{}) int64

Example:

{{ add 3 2 5 }} // output: 10

sub

Subtract numbers.

Signature: func(a, b interface{}) int64

Example:

{{ sub 5 2 }} // output: 3

mul

Mulitply numbers. Supports multiple numbers.

Signature: func(a interface{}, v ...interface{}) int64

Example:

{{ mul 5 2 3}} // output: 30

div

Integer divide numbers.

Signature: func(a, b interface{}) int64

Example:

{{ div 10 2}} // output: 5

addf

Sum numbers. Supports multiple numbers.

Signature: func(i ...interface{}) float64

Example:

{{ addf 3.5 2 5 }} // output: 10.5

subf

Subtract numbers. Supports multiple numbers.

Signature: func(a interface{}, v ...interface{}) float64

Example:

{{ subf  5.5 2 1.5 }} // output: 2

mulf

Mulitply numbers. Supports multiple numbers

Signature: func(a interface{}, v ...interface{}) float64

Example:

{{ mulf 5.5 2 2.5 }} // output: 27.5

divf

Divide numbers. Supports multiple numbers.

Signature: func(a interface{}, v ...interface{}) float64

Example:

{{ divf 10 2 4}} // output: 1.25

mod

Modulo wit mod.

Signature: func(a, b interface{}) int64

Example:

{{ mod 10 3}} // output: 1

max

Return the largest of a series of integers:

Signature: max(a interface{}, i ...interface{}) int64

Example:

{{ max 1 2 3 }} //output 3

min

Return the smallest of a series of integers.

Signature: min(a interface{}, i ...interface{}) int64

Example:

{{ min 1 2 3 }} //output 1

maxf

Return the largest of a series of floats:

Signature: maxf(a interface{}, i ...interface{}) float64

Example:

{{ maxf 1 2.5 3 }} //output 3

minf

Return the smallest of a series of floats.

Signature: minf(a interface{}, i ...interface{}) float64

Example:

{{ minf 1 2.5 3 }} //output 1.5

ceil

Returns the greatest float value greater than or equal to input value

Signature: ceil(a interface{}) float64

Example:

{{ ceil 123.001 }} //output 124.0

floor

Returns the greatest float value less than or equal to input value

Signature: floor(a interface{}) float64

Example:

{{ floor 123.9999 }} //output 123.0

round

Returns a float value with the remainder rounded to the given number of digits after the decimal point.

Signature: round(a interface{}, p int, rOpt ...float64) float64

Example:

{{ round 123.555555 3 }} //output 123.556

We can also provide a roundOn number as third parameter

Example:

{{ round 123.88571428571 5 .2 }} //output 123.88572

With default roundOn of .5 the above value would be 123.88571

int

Convert value to an int.

Signature: toInt(v interface{}) int

Example:

{{ "3" | int }} //output 3

float64

Convert to a float64.

Signature: toFloat64(v interface{}) float64

Example:

{{ "3.5" | float64 }} //output 3.5

fromJson

Decodes a JSON document into a structure. If the input cannot be decoded as JSON the function will return an empty string.

Signature: fromJson(v string) interface{}

Example:

fromJson "{\"foo\": 55}"

Example of a query to print a newline per queries stored as a json array in the log line:

{job="loki/querier"} |= "finish in prometheus" | logfmt | line_format "{{ range $q := fromJson .queries }} {{ $q.query }} {{ end }}"

now

Returns the current time in the local timezone of the Loki server.

Signature: Now() time.Time

Example:

{{ now }}

toDate

Parses a formatted string and returns the time value it represents using the local timezone of the server running Loki.

For more consistency between Loki installations, it’s recommended to use toDateInZone

The format string must use the exact date as defined in the golang datetime layout

Signature: toDate(fmt, str string) time.Time

Examples:

{{ toDate "2006-01-02" "2021-11-02" }}
{{ .foo | toDate "2006-01-02T15:04:05.999999999Z" }}

toDateInZone

Parses a formatted string and returns the time value it represents in the provided timezone.

The format string must use the exact date as defined in the golang datetime layout

The timezone value can be Local, UTC, or any of the IANA Time Zone database values

Signature: toDateInZone(fmt, zone, str string) time.Time

Examples:

{{ toDateInZone "2006-01-02" "UTC" "2021-11-02" }}
{{ .foo | toDateInZone "2006-01-02T15:04:05.999999999Z" "UTC" }}

date

Returns a textual representation of the time value formatted according to the provided golang datetime layout.

Signature: date(fmt string, date interface{}) string

Example:

{{ date "2006-01-02" now }}

unixEpoch

Returns the number of seconds elapsed since January 1, 1970 UTC.

Signature: unixEpoch(date time.Time) string

Examples:

{{ unixEpoch now }}
{{ .foo | toDateInZone "2006-01-02T15:04:05.999999999Z" "UTC" | unixEpoch }}

Example of a query to filter Loki querier jobs which create time is 1 day before:

{job="loki/querier"} | label_format nowEpoch=`{{(unixEpoch now)}}`,createDateEpoch=`{{unixEpoch (toDate "2006-01-02" .createDate)}}` | label_format dateTimeDiff="{{sub .nowEpoch .createDateEpoch}}" | dateTimeDiff > 86400

unixEpochMillis

Returns the number of milliseconds elapsed since January 1, 1970 UTC.

Signature: unixEpochMillis(date time.Time) string

Examples:

{{ unixEpochMillis now }}
{{ .foo | toDateInZone "2006-01-02T15:04:05.999999999Z" "UTC" | unixEpochMillis }}

unixEpochNanos

Returns the number of nanoseconds elapsed since January 1, 1970 UTC.

Signature: unixEpochNanos(date time.Time) string

Examples:

{{ unixEpochNanos now }}
{{ .foo | toDateInZone "2006-01-02T15:04:05.999999999Z" "UTC" | unixEpochNanos }}

unixToTime

Converts the string epoch to the time value it represents. Epoch times in days, seconds, milliseconds, microseconds and nanoseconds are supported.

Signature: unixToTime(epoch string) time.Time

Examples:

Consider the following log line {"from": "1679577215","to":"1679587215","message":"some message"}. To print the from field as human readable add the following at the end of the LogQL query:

... | json | line_format `from="{{date "2006-01-02" (unixToTime .from)}}"`

default

Checks whether the string(src) is set, and returns default(d) if not set.

Signature: default(d string, src string) string

Examples:

{{ default "-" "" }} // output: -
{{ default "" "foo" }} // output: foo

Example of a query to print a - if the http_request_headers_x_forwarded_for label is empty:

{job="access_log"} | json | line_format `{{.http_request_headers_x_forwarded_for | default "-"}}`

count

Counts occurrences of the regex (regex) in (src).

Signature: count(regex string, src string) int

Examples:

{{ count "a|b" "abab" }} // output: 4
{{ count "o" "foo" }}    // output: 2

Example of a query to print how many times XYZ occurs in a line:

{job="xyzlog"} | line_format `{{ __line__ | count "XYZ"}}`

urlencode

Use this function to urlencode a string.

Signature: urlencode(string) string

Examples:

"{{ .request_url | urlencode }}"
`{{ urlencode  .request_url}}`

urldecode

Use this function to urldecode a string.

Signature: urldecode(string) string

Examples:

"{{ .request_url | urldecode }}"
`{{ urldecode  .request_url}}`

b64enc

Base64 encode a string.

Signature: b64enc(string) string

Examples:

"{{ .foo | b64enc }}"
`{{ b64enc  .foo }}`

b64dec

Base64 decode a string.

Signature: b64dec(string) string

Examples:

"{{ .foo | b64dec }}"
`{{ b64dec  .foo }}`

bytes

Convert a humanized byte string to bytes using go-humanize

Signature: bytes(string) string

Examples:

"{{ .foo | bytes }}"
`{{ bytes .foo }}`

duration

An alias for duration_seconds

Examples:

"{{ .foo | duration }}"
`{{ duration .foo }}`

duration_seconds

Convert a humanized time duration to seconds using time.ParseDuration

Signature: duration_seconds(string) float64

Examples:

"{{ .foo | duration_seconds }}"
`{{ duration_seconds .foo }}`

Matching IP addresses

Wed, 06 Sep 2023 12:47:18 +0000

Matching IP addresses

LogQL supports matching IP addresses.

With logs such as

3.180.71.3 - - [17/May/2015:08:05:32 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"
80.91.33.133 - - [17/May/2015:08:05:14 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.16)"
46.4.66.76 - - [17/May/2015:08:05:45 +0000] "GET /downloads/product_1 HTTP/1.1" 404 318 "-" "Debian APT-HTTP/1.3 (1.0.1ubuntu2)"
93.180.71.3 - - [17/May/2015:08:05:26 +0000] "GET /downloads/product_1 HTTP/1.1" 404 324 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"

the LogQL line filter is not sufficient. A line filter such as

{job_name="myapp"} |= "3.180.71.3"

also matches example IP addresses such as 93.180.71.3. A better choice uses a regexp: |~"^3.180.71.3". This regexp does not handle IPv6 addresses, and it does not match a range of IP addresses.

The LogQL support for matching IP addresses handles both IPv4 and IPv6 single addresses, as well as ranges within IP addresses and CIDR patterns.

Match IP addresses with the syntax: ip("<pattern>"). The <pattern> can be:

A single IP address. Examples: ip("192.0.2.0"), ip("::1")
A range within the IP address. Examples: ip("192.168.0.1-192.189.10.12"), ip("2001:db8::1-2001:db8::8")
A CIDR specification. Examples: ip("192.51.100.0/24"), ip("2001:db8::/32")

The IP matching can be used in both line filter and label filter expressions. When specifying line filter expressions, only the |= and != operations are allowed. When specifying label filter expressions, only the = and != operations are allowed.

Line filter examples

{job_name="myapp"} |= ip("192.168.4.5/16")

Return log lines that do not match with an IPv4 range:

{job_name="myapp"} != ip("192.168.4.5-192.168.4.20")

Label filter examples
logql
```
{job_name="myapp"}
	| logfmt
	| remote_addr = ip("2001:db8::1-2001:db8::8")
	| level = "error"
```
Filters can also be chained. This example matches log lines with all IPv4 subnet values 192.168.4.5/16 except IP address 192.168.4.2:
logql
```
{job_name="myapp"}
	| logfmt
	| addr = ip("192.168.4.5/16")
	| addr != ip("192.168.4.2")
```
This example use the conditional or and matches log lines with either, all IPv4 subnet values 192.168.4.0/24 OR all IPv4 subnet values 10.10.15.0/24:
logql
```
{job_name="myapp"}
	| logfmt
	| addr = ip("192.168.4.0/24") or addr = ip("10.10.15.0/24")
```

Query examples

Fri, 12 Jan 2024 20:04:43 +0000

Query examples

These LogQL query examples have explanations of what the queries accomplish.

Log query examples

Examples that filter on IP address

Return log lines that are not within a range of IPv4 addresses:
logql
```
{job_name="myapp"} != ip("192.168.4.5-192.168.4.20")
```
This example matches log lines with all IPv4 subnet values 192.168.4.5/16 except IP address 192.168.4.2:
logql
```
{job_name="myapp"}
	| logfmt
	| addr = ip("192.168.4.5/16")
	| addr != ip("192.168.4.2")
```

Examples that aid in security evaluation

Extract the user and IP address of failed logins from Linux /var/log/secure

{job="security"} 
    |~ "Invalid user.*"
    | regexp "(^(?P<user>\\S+ {1,2}){8})"
    | regexp "(^(?P<ip>\\S+ {1,2}){10})"
    | line_format "IP = {{.ip}}\tUSER = {{.user}}"

Get successful logins from Linux /var/log/secure

{job="security"}
    != "grafana_com"
    |= "session opened"
    != "sudo: "
    | regexp "(^(?P<user>\\S+ {1,2}){11})"
    | line_format "USER = {{.user}}"

Metrics query examples

Return the per-second rate of all non-timeout errors within the last minutes per host for the MySQL job, and only include errors whose duration is above ten seconds.
```
sum by (host) (rate({job="mysql"}
    |= "error" != "timeout"
    | json
    | duration > 10s [1m]))
```

Multiple filtering stages examples

Query results are gathered by successive evaluation of parts of the query from left to right. To make querying efficient, order the filtering stages left to right:

stream selector
line filters
label filters

Consider the query:

{cluster="ops-tools1", namespace="loki-dev", job="loki-dev/query-frontend"} |= "metrics.go" != "out of order" | logfmt | duration > 30s or status_code != "200"

Within this query, the stream selector is

{cluster="ops-tools1", namespace="loki-dev", job="loki-dev/query-frontend"}

There are two line filters: |= "metrics.go" and !="out of order". Of the log lines identified with the stream selector, the query results include only those log lines that contain the string “metrics.go” and do not contain the string “out of order”.

The logfmt parser produces the duration and status_code labels, such that they can be used by a label filter.

The label filter | duration > 30s or status_code!="200" further filters out log lines. It includes those log lines that contain a status_code label with any value other than the value 200, as well as log lines that contain a duration label with a value greater than 30 sections,

While every query will have a stream selector, not all queries will have line and label filters.

Examples that use multiple parsers

Consider this logfmt log line. To extract the method and the path of this logfmt log line,

level=debug ts=2020-10-02T10:10:42.092268913Z caller=logging.go:66 traceID=a9d4d8a928d8db1 msg="POST /api/prom/api/v1/query_range (200) 1.5s"

To extract the method and the path, use multiple parsers (logfmt and regexp):

{job="loki-ops/query-frontend"} | logfmt | line_format "{{.msg}}" | regexp "(?P<method>\\w+) (?P<path>[\\w|/]+) \\((?P<status>\\d+?)\\) (?P<duration>.*)"

This is possible because the | line_format reformats the log line to become POST /api/prom/api/v1/query_range (200) 1.5s which can then be parsed with the | regexp ... parser.

Log line formatting examples

The following query shows how you can reformat a log line to make it easier to read on screen.

{cluster="ops-tools1", name="querier", namespace="loki-dev"}
  |= "metrics.go" != "loki-canary"
  | logfmt
  | query != ""
  | label_format query="{{ Replace .query \"\\n\" \"\" -1 }}"
  | line_format "{{ .ts}}\t{{.duration}}\ttraceID = {{.traceID}}\t{{ printf \"%-100.100s\" .query }} "

Label formatting is used to sanitize the query while the line format reduce the amount of information and creates a tabular output.

For these given log lines:

level=info ts=2020-10-23T20:32:18.094668233Z caller=metrics.go:81 org_id=29 traceID=1980d41501b57b68 latency=fast query="{cluster=\"ops-tools1\", job=\"loki-ops/query-frontend\"} |= \"query_range\"" query_type=filter range_type=range length=15m0s step=7s duration=650.22401ms status=200 throughput_mb=1.529717 total_bytes_mb=0.994659
level=info ts=2020-10-23T20:32:18.068866235Z caller=metrics.go:81 org_id=29 traceID=1980d41501b57b68 latency=fast query="{cluster=\"ops-tools1\", job=\"loki-ops/query-frontend\"} |= \"query_range\"" query_type=filter range_type=range length=15m0s step=7s duration=624.008132ms status=200 throughput_mb=0.693449 total_bytes_mb=0.432718

The result would be:

2020-10-23T20:32:18.094668233Z	650.22401ms	    traceID = 1980d41501b57b68	{cluster="ops-tools1", job="loki-ops/query-frontend"} |= "query_range"
2020-10-23T20:32:18.068866235Z	624.008132ms	traceID = 1980d41501b57b68	{cluster="ops-tools1", job="loki-ops/query-frontend"} |= "query_range"

It’s possible to strip ANSI sequences from the log line, making it easier to parse it further:

{job="example"} | decolorize

This way this log line:

[2022-11-04 22:17:57.811] \033[0;32http\033[0m: GET /_health (0 ms) 204

turns into:

[2022-11-04 22:17:57.811] http: GET /_health (0 ms) 204

Unwrap examples

Calculate the p99 of the nginx-ingress latency by path:

quantile_over_time(0.99,
  {cluster="ops-tools1",container="ingress-nginx"}
    | json
    | __error__ = ""
    | unwrap request_time [1m]) by (path)

Calculate the quantity of bytes processed per organization ID:

sum by (org_id) (
  sum_over_time(
  {cluster="ops-tools1",container="loki-dev"}
      |= "metrics.go"
      | logfmt
      | unwrap bytes_processed [1m])
  )

Vector aggregation examples

Get the top 10 applications by the highest log throughput:

topk(10,sum(rate({region="us-east1"}[5m])) by (name))

Get the count of log lines for the last five minutes for a specified job, grouping by level:

sum(count_over_time({job="mysql"}[5m])) by (level)

Get the rate of HTTP GET requests to the /home endpoint for NGINX logs by region:

avg(rate(({job="nginx"} |= "GET" | json | path="/home")[10s])) by (region)

LogQL Analyzer

Thu, 10 Apr 2025 12:15:54 +0000

LogQL Analyzer

The LogQL Analyzer is an inline tool for experimenting with writing LogQL queries.

Chose the log line format with the radio buttons. A set of example log lines are included for each format.

Use the provided example log lines, or copy and paste your own log lines into the example log lines box.

Use the provided example query, or enter your own query. The log stream selector remains fixed for all possible example queries. Modify the remainder of the log line and click on the Run query button to run the entered query against the example log lines.

The results output provides details for each example log line. Clicking on a line in the results pane expands the details, showing why the line is or is not included in the query result set.

Log line format: logfmt JSON Unstructured text

{job="analyze"}

Query:

{job="analyze"}

LogCLI

Thu, 10 Apr 2025 12:15:54 +0000

LogCLI

LogCLI is the command-line interface to Grafana Loki. It facilitates running LogQL queries against a Loki instance.

Installation

Binary (Recommended)

Download the logcli binary from the Loki releases page.

Build LogCLI from source

Clone the Loki repository and build logcli from source:

git clone https://github.com/grafana/loki.git
cd loki
make logcli

Optionally, move the binary into a directory that is part of your $PATH.

cp cmd/logcli/logcli /usr/local/bin/logcli

Set up command completion

You can set up tab-completion for logcli with one of the two options, depending on your shell:

For bash, add this to your ~/.bashrc file:
```
eval "$(logcli --completion-script-bash)"
```
For zsh, add this to your ~/.zshrc file:
```
eval "$(logcli --completion-script-zsh)"
```

LogCLI usage

Grafana Cloud example

If you are running on Grafana Cloud, use:

export LOKI_ADDR=https://logs-us-west1.grafana.net
export LOKI_USERNAME=<username>
export LOKI_PASSWORD=<password>

Otherwise you can point LogCLI to a local instance directly without needing a username and password:

export LOKI_ADDR=http://localhost:3100

Note: If you are running Loki behind a proxy server and you have authentication configured, you will also have to pass in LOKI_USERNAME and LOKI_PASSWORD, LOKI_BEARER_TOKEN or LOKI_BEARER_TOKEN_FILE accordingly.

$ logcli labels job
https://logs-dev-ops-tools1.grafana.net/api/prom/label/job/values
loki-ops/consul
loki-ops/loki-gw
...

$ logcli query '{job="loki-ops/consul"}'
https://logs-dev-ops-tools1.grafana.net/api/prom/query?query=%7Bjob%3D%22loki-ops%2Fconsul%22%7D&limit=30&start=1529928228&end=1529931828&direction=backward&regexp=
Common labels: {job="loki-ops/consul", namespace="loki-ops"}
2018-06-25T12:52:09Z {instance="consul-8576459955-pl75w"} 2018/06/25 12:52:09 [INFO] raft: Snapshot to 475409 complete
2018-06-25T12:52:09Z {instance="consul-8576459955-pl75w"} 2018/06/25 12:52:09 [INFO] raft: Compacting logs from 456973 to 465169
...

$ logcli series -q --match='{namespace="loki",container_name="loki"}'
{app="loki", container_name="loki", controller_revision_hash="loki-57c9df47f4", filename="/var/log/pods/loki_loki-0_8ed03ded-bacb-4b13-a6fe-53a445a15887/loki/0.log", instance="loki-0", job="loki/loki", name="loki", namespace="loki", release="loki", statefulset_kubernetes_io_pod_name="loki-0", stream="stderr"}

Batched queries

LogCLI sends queries to Loki such that query results arrive in batches.

The --limit option for a logcli query command caps the quantity of log lines for a single query. When not set, --limit defaults to 30. The limit protects the user from overwhelming the system for cases in which the specified query would have returned a large quantity of log lines. The limit also protects the user from unexpectedly large responses.

The quantity of log line results that arrive in each batch is set by the --batch option in a logcli query command. When not set, --batch defaults to 1000.

Setting a --limit value larger than the --batch value causes the requests from LogCLI to Loki to be batched. Loki has a server-side limit that defaults to 5000 for the maximum quantity of lines returned for a single query. The batching of requests allows you to query for a results set that is larger than the server-side limit, as long as the --batch value is less than the server limit.

Query metadata is output to stderr for each batch. Set the --quiet option on the logcli query command line to suppress the output of the query metadata.

Configuration

Configuration values are considered in the following order (lowest to highest):

Environment variables
Command-line options

LogCLI command reference

The output of logcli help:

usage: logcli [<flags>] <command> [<args> ...]

A command-line for loki.

Flags:
      --help             Show context-sensitive help (also try --help-long and
                         --help-man).
      --version          Show application version.
  -q, --quiet            Suppress query metadata
      --stats            Show query statistics
  -o, --output=default   Specify output mode [default, raw, jsonl]. raw
                         suppresses log labels and timestamp.
  -z, --timezone=Local   Specify the timezone to use when formatting output
                         timestamps [Local, UTC]
      --cpuprofile=""    Specify the location for writing a CPU profile.
      --memprofile=""    Specify the location for writing a memory profile.
      --stdin            Take input logs from stdin
      --addr="http://localhost:3100"
                         Server address. Can also be set using LOKI_ADDR env
                         var.
      --username=""      Username for HTTP basic auth. Can also be set using
                         LOKI_USERNAME env var.
      --password=""      Password for HTTP basic auth. Can also be set using
                         LOKI_PASSWORD env var.
      --ca-cert=""       Path to the server Certificate Authority. Can also be
                         set using LOKI_CA_CERT_PATH env var.
      --tls-skip-verify  Server certificate TLS skip verify.
      --cert=""          Path to the client certificate. Can also be set using
                         LOKI_CLIENT_CERT_PATH env var.
      --key=""           Path to the client certificate key. Can also be set
                         using LOKI_CLIENT_KEY_PATH env var.
      --org-id=""        adds X-Scope-OrgID to API requests for representing
                         tenant ID. Useful for requesting tenant data when
                         bypassing an auth gateway.

Commands:
  help [<command>...]
    Show help.

  query [<flags>] <query>
    Run a LogQL query.

    The "query" command is useful for querying for logs. Logs can be returned in
    a few output modes:

      raw: log line
      default: log timestamp + log labels + log line
      jsonl: JSON response from Loki API of log line

    The output of the log can be specified with the "-o" flag, for example, "-o
    raw" for the raw output format.

    The "query" command will output extra information about the query and its
    results, such as the API URL, set of common labels, and set of excluded
    labels. This extra information can be suppressed with the --quiet flag.

    By default we look over the last hour of data; use --since to modify or
    provide specific start and end times with --from and --to respectively.

    Notice that when using --from and --to then ensure to use RFC3339Nano time
    format, but without timezone at the end. The local timezone will be added
    automatically or if using --timezone flag.

    Example:

      logcli query
         --timezone=UTC
         --from="2021-01-19T10:00:00Z"
         --to="2021-01-19T20:00:00Z"
         --output=jsonl
         'my-query'

    The output is limited to 30 entries by default; use --limit to increase.

    While "query" does support metrics queries, its output contains multiple
    data points between the start and end query time. This output is used to
    build graphs, similar to what is seen in the Grafana Explore graph view. If
    you are querying metrics and just want the most recent data point (like what
    is seen in the Grafana Explore table view), then you should use the
    "instant-query" command instead.

  instant-query [<flags>] <query>
    Run an instant LogQL query.

    The "instant-query" command is useful for evaluating a metric query for a
    single point in time. This is equivalent to the Grafana Explore table view;
    if you want a metrics query that is used to build a Grafana graph, you
    should use the "query" command instead.

    This command does not produce useful output when querying for log lines; you
    should always use the "query" command when you are running log queries.

    For more information about log queries and metric queries, refer to the
    LogQL documentation:

    https://grafana.com/docs/loki/<LOKI_VERSION>/logql/

  labels [<flags>] [<label>]
    Find values for a given label.

  series [<flags>] <matcher>
    Run series query.

    The "series" command will take the provided label matcher and return all the
    log streams found in the time window.

    It is possible to send an empty label matcher '{}' to return all streams.

    Use the --analyze-labels flag to get a summary of the labels found in all
    streams. This is helpful to find high cardinality labels.

LogCLI query command reference

The output of logcli help query:

usage: logcli query [<flags>] <query>

Run a LogQL query.

The "query" command is useful for querying for logs. Logs can be returned in a few output modes:

  raw: log line
  default: log timestamp + log labels + log line
  jsonl: JSON response from Loki API of log line

The output of the log can be specified with the "-o" flag, for example, "-o raw" for the raw output format.

The "query" command will output extra information about the query and its results, such as the API URL, set of common labels, and set of excluded labels. This extra information can be
suppressed with the --quiet flag.

By default we look over the last hour of data; use --since to modify or provide specific start and end times with --from and --to respectively.

Notice that when using --from and --to then ensure to use RFC3339Nano time format, but without timezone at the end. The local timezone will be added automatically or if using --timezone
flag.

Example:

  logcli query
     --timezone=UTC
     --from="2021-01-19T10:00:00Z"
     --to="2021-01-19T20:00:00Z"
     --output=jsonl
     'my-query'

The output is limited to 30 entries by default; use --limit to increase.

While "query" does support metrics queries, its output contains multiple data points between the start and end query time. This output is used to build graphs, similar to what is seen
in the Grafana Explore graph view. If you are querying metrics and just want the most recent data point (like what is seen in the Grafana Explore table view), then you should use the
"instant-query" command instead.

Parallelization:

You can download an unlimited number of logs in parallel, there are a few flags which control this behaviour:

  --parallel-duration
  --parallel-max-workers
  --part-path-prefix
  --overwrite-completed-parts
  --merge-parts
  --keep-parts

Refer to the help for each flag for details about what each of them do.

Example:

  logcli query
     --timezone=UTC
     --from="2021-01-19T10:00:00Z"
     --to="2021-01-19T20:00:00Z"
     --output=jsonl
     --parallel-duration="15m"
     --parallel-max-workers="4"
     --part-path-prefix="/tmp/my_query"
     --merge-parts
     'my-query'

This example will create a queue of jobs to execute, each being 15 minutes in duration. In this case, that means, for the 10-hour total duration, there will be forty 15-minute jobs.
The --limit flag is ignored.

It will start four workers, and they will each take a job to work on from the queue until all the jobs have been completed.

Each job will save a "part" file to the location specified by the --part-path-prefix. Different prefixes can be used to run multiple queries at the same time. The timestamp of the start and
end of the part is in the file name. While the part is being downloaded, the filename will end in ".part", when it is complete, the file will be renamed to remove this ".part" extension.
By default, if a completed part file is found, that part will not be downloaded again. This can be overridden with the --overwrite-completed-parts flag.

Part file example using the previous command, adding --keep-parts so they are not deleted:

Since we don't have the --forward flag, the parts will be downloaded in reverse. Two of the workers have finished their jobs (last two files), and have picked up the next jobs in the queue.
Running ls, this is what we should expect to see.

$ ls -1 /tmp/my_query* /tmp/my_query_20210119T183000_20210119T184500.part.tmp /tmp/my_query_20210119T184500_20210119T190000.part.tmp /tmp/my_query_20210119T190000_20210119T191500.part.tmp
/tmp/my_query_20210119T191500_20210119T193000.part.tmp /tmp/my_query_20210119T193000_20210119T194500.part /tmp/my_query_20210119T194500_20210119T200000.part

If you do not specify the --merge-parts flag, the part files will be downloaded, and logcli will exit, and you can process the files as you wish. With the flag specified, the part files
will be read in order, and the output printed to the terminal. The lines will be printed as soon as the next part is complete, you don't have to wait for all the parts to download before
getting output. The --merge-parts flag will remove the part files when it is done reading each of them. To change this, you can use the --keep-parts flag, and the part files will not be
removed.

Flags:
      --help                    Show context-sensitive help (also try --help-long and --help-man).
      --version                 Show application version.
  -q, --quiet                   Suppress query metadata
      --stats                   Show query statistics
  -o, --output=default          Specify output mode [default, raw, jsonl]. raw suppresses log labels and timestamp.
  -z, --timezone=Local          Specify the timezone to use when formatting output timestamps [Local, UTC]
      --cpuprofile=""           Specify the location for writing a CPU profile.
      --memprofile=""           Specify the location for writing a memory profile.
      --stdin                   Take input logs from stdin
      --addr="http://localhost:3100"
                                Server address. Can also be set using LOKI_ADDR env var.
      --username=""             Username for HTTP basic auth. Can also be set using LOKI_USERNAME env var.
      --password=""             Password for HTTP basic auth. Can also be set using LOKI_PASSWORD env var.
      --ca-cert=""              Path to the server Certificate Authority. Can also be set using LOKI_CA_CERT_PATH env var.
      --tls-skip-verify         Server certificate TLS skip verify. Can also be set using LOKI_TLS_SKIP_VERIFY env var.
      --cert=""                 Path to the client certificate. Can also be set using LOKI_CLIENT_CERT_PATH env var.
      --key=""                  Path to the client certificate key. Can also be set using LOKI_CLIENT_KEY_PATH env var.
      --org-id=""               adds X-Scope-OrgID to API requests for representing tenant ID. Useful for requesting tenant data when bypassing an auth gateway. Can also be set using
                                LOKI_ORG_ID env var.
      --query-tags=""           adds X-Query-Tags http header to API requests. This header value will be part of `metrics.go` statistics. Useful for tracking the query. Can also be set
                                using LOKI_QUERY_TAGS env var.
      --bearer-token=""         adds the Authorization header to API requests for authentication purposes. Can also be set using LOKI_BEARER_TOKEN env var.
      --bearer-token-file=""    adds the Authorization header to API requests for authentication purposes. Can also be set using LOKI_BEARER_TOKEN_FILE env var.
      --retries=0               How many times to retry each query when getting an error response from Loki. Can also be set using LOKI_CLIENT_RETRIES env var.
      --min-backoff=0           Minimum backoff time between retries. Can also be set using LOKI_CLIENT_MIN_BACKOFF env var.
      --max-backoff=0           Maximum backoff time between retries. Can also be set using LOKI_CLIENT_MAX_BACKOFF env var.
      --auth-header="Authorization"
                                The authorization header used. Can also be set using LOKI_AUTH_HEADER env var.
      --proxy-url=""            The http or https proxy to use when making requests. Can also be set using LOKI_HTTP_PROXY_URL env var.
      --limit=30                Limit on number of entries to print. Setting it to 0 will fetch all entries.
      --since=1h                Lookback window.
      --from=FROM               Start looking for logs at this absolute time (inclusive)
      --to=TO                   Stop looking for logs at this absolute time (exclusive)
      --step=STEP               Query resolution step width, for metric queries. Evaluate the query at the specified step over the time range.
      --interval=INTERVAL       Query interval, for log queries. Return entries at the specified interval, ignoring those between. **This parameter is experimental, see Issue 1779**
      --batch=1000              Query batch size to use until 'limit' is reached
      --parallel-duration=1h    Split the range into jobs of this length to download the logs in parallel. This will result in the logs being out of order. Use --part-path-prefix to create
                                a file per job to maintain ordering.
      --parallel-max-workers=1  Max number of workers to start up for parallel jobs. A value of 1 will not create any parallel workers. When using parallel workers, limit is ignored.
      --part-path-prefix=PART-PATH-PREFIX
                                When set, each server response will be saved to a file with this prefix. Creates files in the format: 'prefix-utc_start-utc_end.part'. Intended to be used
                                with the parallel-* flags so that you can combine the files to maintain ordering based on the filename. Default is to write to stdout.
      --overwrite-completed-parts
                                Overwrites completed part files. This will download the range again, and replace the original completed part file. Default will skip a range if it's part
                                file is already downloaded.
      --merge-parts             Reads the part files in order and writes the output to stdout. Original part files will be deleted with this option.
      --keep-parts              Overrides the default behaviour of --merge-parts which will delete the part files once all the files have been read. This option will keep the part files.
      --forward                 Scan forwards through logs.
      --no-labels               Do not print any labels
      --exclude-label=EXCLUDE-LABEL ...
                                Exclude labels given the provided key during output.
      --include-label=INCLUDE-LABEL ...
                                Include labels given the provided key during output.
      --labels-length=0         Set a fixed padding to labels
      --store-config=""         Execute the current query using a configured storage from a given Loki configuration file.
      --remote-schema           Execute the current query using a remote schema retrieved using the configured storage in the given Loki configuration file.
      --colored-output          Show output with colored labels
  -t, --tail                    Tail the logs
  -f, --follow                  Alias for --tail
      --delay-for=0             Delay in tailing by number of seconds to accumulate logs for re-ordering

Args:
  <query>  eg '{foo="bar",baz=~".*blip"} |~ ".*error.*"'

LogCLI labels command reference

The output of logcli help labels:

usage: logcli labels [<flags>] [<label>]

Find values for a given label.

Flags:
      --help             Show context-sensitive help (also try --help-long and
                         --help-man).
      --version          Show application version.
  -q, --quiet            Suppress query metadata
      --stats            Show query statistics
  -o, --output=default   Specify output mode [default, raw, jsonl]. raw
                         suppresses log labels and timestamp.
  -z, --timezone=Local   Specify the timezone to use when formatting output
                         timestamps [Local, UTC]
      --cpuprofile=""    Specify the location for writing a CPU profile.
      --memprofile=""    Specify the location for writing a memory profile.
      --stdin            Take input logs from stdin
      --addr="http://localhost:3100"
                         Server address. Can also be set using LOKI_ADDR env
                         var.
      --username=""      Username for HTTP basic auth. Can also be set using
                         LOKI_USERNAME env var.
      --password=""      Password for HTTP basic auth. Can also be set using
                         LOKI_PASSWORD env var.
      --ca-cert=""       Path to the server Certificate Authority. Can also be
                         set using LOKI_CA_CERT_PATH env var.
      --tls-skip-verify  Server certificate TLS skip verify.
      --cert=""          Path to the client certificate. Can also be set using
                         LOKI_CLIENT_CERT_PATH env var.
      --key=""           Path to the client certificate key. Can also be set
                         using LOKI_CLIENT_KEY_PATH env var.
      --org-id=""        adds X-Scope-OrgID to API requests for representing
                         tenant ID. Useful for requesting tenant data when
                         bypassing an auth gateway.
      --since=1h         Lookback window.
      --from=FROM        Start looking for labels at this absolute time
                         (inclusive)
      --to=TO            Stop looking for labels at this absolute time
                         (exclusive)

Args:
  [<label>]  The name of the label.

LogCLI series command reference

The output of logcli help series:

usage: logcli series [<flags>] <matcher>

Run series query.

The "series" command will take the provided label matcher and return all the log
streams found in the time window.

It is possible to send an empty label matcher '{}' to return all streams.

Use the --analyze-labels flag to get a summary of the labels found in all
streams. This is helpful to find high cardinality labels.

Flags:
      --help             Show context-sensitive help (also try --help-long and
                         --help-man).
      --version          Show application version.
  -q, --quiet            Suppress query metadata
      --stats            Show query statistics
  -o, --output=default   Specify output mode [default, raw, jsonl]. raw
                         suppresses log labels and timestamp.
  -z, --timezone=Local   Specify the timezone to use when formatting output
                         timestamps [Local, UTC]
      --cpuprofile=""    Specify the location for writing a CPU profile.
      --memprofile=""    Specify the location for writing a memory profile.
      --stdin            Take input logs from stdin
      --addr="http://localhost:3100"
                         Server address. Can also be set using LOKI_ADDR env
                         var.
      --username=""      Username for HTTP basic auth. Can also be set using
                         LOKI_USERNAME env var.
      --password=""      Password for HTTP basic auth. Can also be set using
                         LOKI_PASSWORD env var.
      --ca-cert=""       Path to the server Certificate Authority. Can also be
                         set using LOKI_CA_CERT_PATH env var.
      --tls-skip-verify  Server certificate TLS skip verify.
      --cert=""          Path to the client certificate. Can also be set using
                         LOKI_CLIENT_CERT_PATH env var.
      --key=""           Path to the client certificate key. Can also be set
                         using LOKI_CLIENT_KEY_PATH env var.
      --org-id=""        adds X-Scope-OrgID to API requests for representing
                         tenant ID. Useful for requesting tenant data when
                         bypassing an auth gateway.
      --since=1h         Lookback window.
      --from=FROM        Start looking for logs at this absolute time
                         (inclusive)
      --to=TO            Stop looking for logs at this absolute time (exclusive)
      --analyze-labels   Printout a summary of labels including count of label
                         value combinations, useful for debugging high
                         cardinality series

Args:
  <matcher>  eg '{foo="bar",baz=~".*blip"}'

LogCLI `--stdin` usage

You can consume log lines from your stdin instead of Loki servers.

Say you have log files in your local, and just want to do run some LogQL queries for that, --stdin flag can help.

NOTE: Currently it doesn’t support any type of metric queries

You may have to use stdin flag for several reasons

Quick way to check and validate a LogQL expressions.
Learn basics of LogQL with just Log files and LogCLItool ( without needing set up Loki servers, Grafana etc.)
Easy discussion on public forums. Like Q&A, Share the LogQL expressions.

NOTES on Usage

--limits flag doesn’t have any meaning when using --stdin (use pager like less for that)
Be aware there are no labels when using --stdin
- So stream selector in the query is optional e.g just |="timeout"|logfmt|level="error" is same as {foo="bar"}|="timeout|logfmt|level="error"

Examples

Line filter - cat mylog.log | logcli --stdin query '|="too many open connections"'
Label matcher - echo 'msg="timeout happened" level="warning"' | logcli --stdin query '|logfmt|level="warning"'
Different parsers (logfmt, json, pattern, regexp) - cat mylog.log | logcli --stdin query '|pattern <ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>'
Line formatters - cat mylog.log | logcli --stdin query '|logfmt|line_format "{{.query}} {{.duration}}"'