net-imap vulnerable to denial of service via high iteration count for `SCRAM-*` authentication

Summary

When authenticating a connection with SCRAM-SHA1 or SCRAM-SHA256, a hostile server can perform a computational denial-of-service attack on the client process by sending a big iteration count value.

Details

A hostile IMAP server can send an arbitrarily large PBKDF2 iteration count in the SCRAM server-first-message, causing the client to perform an expensive OpenSSL::KDF.pbkdf2_hmac call. Because the PBKDF2 function is a blocking C extension and holds onto Ruby’s Global VM Lock, it can freeze the entire Ruby VM for the duration of the computation.

OpenSSL enforces an effective maximum by using a 32-bit signed integer for the iteration count, Depending on hardware capabilities and OpenSSL version, this iteration count may be sufficient for to block all Ruby threads in the process for over seven minutes.

This is listed as one of the "Security Considerations", in RFC 7804:
> A hostile server can perform a computational denial-of-service attack on clients by sending a big iteration count value. In order to defend against that, a client implementation can pick a maximum iteration count that it is willing to use and reject any values that exceed that threshold (in such cases, the client, of course, has to fail the authentication).

Impact

During SCRAM authentication to a hostile server, the entire Ruby VM will be locked for the duration of the computation. Depending on hardware capabilities and OpenSSL version, this may take many minutes.

OpenSSL::KDF.pbkdf2_hmac is a blocking C function, so Timeout cannot be used to guard against this. And it retains the Global VM lock, so other ruby threads will also be unable to run.

Mitigation

• Upgrade to a patched version of net-imap that adds the max_iterations option to the SASL-* authenticators, and call Net::IMAP#authenticate with a max_iterations keyword argument.

NOTE: The default max_iterations is 2³¹ - 1, the maximum signed 32 bit integer, the maximum allowed by OpenSSL.
To prevent a denial of service attack, this must be set to a safe value, depending on hardware and version of OpenSSL.
It is the user's responsibility to enforce minimum and maximum iteration counts that are appropriate for their security context.
* Alternatively, avoid SCRAM-* mechanisms when authenticating to untrusted servers.