MDEV-40141 PAM authentication fails when started with --default-auth=dialog

[vaintroub]

https://jira.mariadb.org/browse/MDEV-40141

When the client selects the dialog plugin itself (--default-auth=dialog),
the server sends no AuthSwitchRequest and hands the pam plugin the client's
initial reply, which is empty (dialog sends no data upfront). The plugin
decided the client had already supplied a password by testing only the first
byte of that reply, which points at stale buffer data, and so sent an empty
password to PAM instead of prompting.

Fixed by also checking the reply length, in both the v2 (auth_pam.c) and
v1 (auth_pam_v1.c) pam plugins. Adds a regression test to plugins.pam
and plugins.pam_v1.

[Copilot]

Pull request overview

Fixes MDEV-40141 where PAM authentication could incorrectly treat an empty initial client reply (when the client starts with --default-auth=dialog) as already containing password data due to inspecting only the first byte, leading to an empty password being sent to PAM instead of prompting.

Changes:

• Add explicit length checks before interpreting the first byte of the client reply in both PAM plugins (v2 and v1).
• Add regression coverage to ensure --default-auth=dialog triggers the expected interactive prompt flow for both plugin variants.
• Skip the PAM tests early when the dialog client plugin is unavailable.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
plugin/auth_pam/auth_pam.c	Prevents stale/empty initial reply from being treated as a valid client response by checking pkt_len.
plugin/auth_pam/auth_pam_v1.c	Prevents dereferencing/using an empty cached initial packet by checking cached_len.
mysql-test/suite/plugins/t/pam.test	Adds regression test case for --default-auth=dialog and skips when dialog client plugin isn’t present.
mysql-test/suite/plugins/t/pam_v1.test	Adds the same regression coverage for the v1 PAM plugin and skips when dialog client plugin isn’t present.
mysql-test/suite/plugins/r/pam.result	Updates expected output for the new regression scenario.
mysql-test/suite/plugins/r/pam_v1.result	Updates expected output for the new regression scenario in v1.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.