OpenTelemetry-Go Contrib vulnerable to denial of service in otelhttp due to unbound cardinality metrics

Description

Summary

This handler wrapper https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5f7e6ad5a49b45df45f61a1deb29d7f1158032df/instrumentation/net/http/otelhttp/handler.go#L63-L65
out of the box adds labels

  • http.user_agent
  • http.method

that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent to it.

Details

HTTP header User-Agent or HTTP method for requests can be easily set by an attacker to be random and long. The library internally uses httpconv.ServerRequest that records every value for HTTP method and User-Agent.

PoC

Send many requests with long randomly generated HTTP methods or/and User agents (e.g. a million) and observe how memory consumption increases during it.

Impact

In order to be affected, the program has to configure a metrics pipeline, use otelhttp.NewHandler wrapper, and does not filter any unknown HTTP methods or User agents on the level of CDN, LB, previous middleware, etc.

Others

It is similar to already reported vulnerabilities
- https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-5r5m-65gx-7vrh (open-telemetry/opentelemetry-go-contrib)
- https://github.com/advisories/GHSA-cg3q-j54f-5p7p (prometheus/client_golang)

Workaround for affected versions

As a workaround to stop being affected otelhttp.WithFilter() can be used, but it requires manual careful configuration to not log certain requests entirely.

For convenience and safe usage of this library, it should by default mark with the label unknown non-standard HTTP methods and User agents to show that such requests were made but do not increase cardinality. In case someone wants to stay with the current behavior, library API should allow to enable it.

The other possibility is to disable HTTP metrics instrumentation by passing otelhttp.WithMeterProvider option with noop.NewMeterProvider.

Solution provided by upgrading

In PR https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4277, released with package version 0.44.0, the values collected for attribute http.request.method were changed to be restricted to a set of well-known values and other high cardinality attributes were removed.

References

  • https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4277
  • https://github.com/open-telemetry/opentelemetry-go-contrib/releases/tag/v1.19.0

Basic information

Type
reviewed
Severity
high
Advisory on GitHub
Open advisory ↗
Repository advisory
Open repository advisory ↗
Source code
Browse source ↗
Published (advisory)
2023-10-16 14:01:54 UTC
Updated
2024-02-19 03:30:24 UTC
GitHub reviewed
2023-10-16 14:01:54 UTC
NVD published
2023-10-12

EPSS Score

Score Percentile
1.16% 78.51%

CVSS Scores

Base score Version Severity Vector
7.5 3.1
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H Click to expand
Attack vector (AV:N)
Could be attacked over the internet or any normal routed network—not just someone sitting at the machine.
Attack complexity (AC:L)
Once they can reach the bug, pulling it off is straightforward—no weird race conditions or rare setup.
Privileges required (PR:N)
No account or special rights needed—anonymous or random user is enough.
User interaction (UI:N)
Nobody has to click “OK” or open a trap file; it can work without a victim helping.
Scope (S:U)
Damage stays in the same “trust bubble” as the broken component—no big spill into unrelated systems.
Confidentiality (C:N)
Doesn’t really leak secrets in a meaningful way.
Integrity (I:N)
Data isn’t meaningfully altered or forged.
Availability (A:H)
Could take the service down hard or make it unusable for people who depend on it.

Identifiers

CWEs

CWE id Name
CWE-770 Allocation of Resources Without Limits or Throttling

Credits

  • programmer04 (reporter)
  • MadVikingGod (remediation_developer)
  • arminru (coordinator)
  • pellared (remediation_reviewer)

Affected packages (7)

Vulnerable version ranges and first patched releases as published by GitHub.

Ecosystem Package Vulnerable range First patched Vulnerable functions
go go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp < 0.44.0 0.44.0
go go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful < 0.44.0 0.44.0
go go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin < 0.44.0 0.44.0
go go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux < 0.44.0 0.44.0
go go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho < 0.44.0 0.44.0
go go.opentelemetry.io/contrib/instrumentation/gopkg.in/macaron.v1/otelmacaron < 0.44.0 0.44.0
go go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace < 0.44.0 0.44.0

References

cvelogic Threat Intelligence