Stop globally registering RSA key methods

[mtrojnar]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Fixes #672

Current Behavior

On OpenSSL 3.x, PKCS11_get_private_key() registers libp11's custom RSA
EVP_PKEY_METHOD process-wide with EVP_PKEY_meth_add0().

That registration changes method selection for unrelated software RSA keys.
Provider-backed keys created before the PKCS#11 key can subsequently fail in
operations such as X509_sign() with:

error:03000093:digital envelope routines:default_check:command not supported

Software RSA keys created afterward can also silently fall back to legacy
implementations instead of remaining backed by the default provider.

The registration also crosses an important architecture boundary: the
libp11 library initializes deprecated OpenSSL method integration that belongs
to the ENGINE component.

New Behavior

The public libp11 API constructs PKCS#11-backed RSA keys without globally
registering an OpenSSL EVP_PKEY_METHOD. Loading such a key no longer changes
RSA method selection elsewhere in the process, so existing and newly created
software RSA keys remain provider-backed and continue to work.

RSA-PSS method selection remains in the existing ENGINE integration. The
ENGINE tests verify both:

• OpenSSL PSS encoding followed by raw signing through CKM_RSA_X_509 when
  native PSS is unavailable;
• native signing through CKM_RSA_PKCS_PSS when raw RSA is unavailable.

The OpenSSL provider continues to use its separate provider implementation.

Scope of Changes

• Remove process-wide RSA EVP_PKEY_METHOD registration from the libp11
  library.
• Remove the associated global method state and cleanup.
• Keep ENGINE method selection in the existing eng_* integration instead of
  invoking deprecated ENGINE APIs from p11_* sources.
• Add a public-API regression test covering provider-backed software RSA keys
  created before and after PKCS11_get_private_key().
• Exercise raw-RSA-only and native-PSS-only SoftHSM configurations through the
  ENGINE interface.

No public API or ABI is changed.

Testing

• Existing tests
• New tests added
• Manual testing

Built with OpenSSL 3.6.3 using strict configuration and ran:

./configure --enable-strict
make
make check

Result: 51 total, 46 passed, 5 skipped due to unavailable ML-DSA/ML-KEM
features, 0 failed.

Focused engine/provider RSA-PSS and RSA-OAEP tests pass using
/usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so.

A clean strict build with CPPFLAGS=-DOPENSSL_NO_ENGINE also passed the new
public-API software-key isolation test. In that configuration the ENGINE
RSA-PSS scenarios are skipped because the ENGINE component is unavailable.

Additional Notes

The custom RSA PKEY method is an ENGINE integration detail. Applications
using the public libp11 API directly retain the key-specific RSA_METHOD and
raw-RSA fallback, but do not receive ENGINE-only parameter callbacks for
native-PSS-only or parameterized-OAEP operations. Applications requiring
those OpenSSL interfaces should use the ENGINE or provider component.

License Declaration

• I hereby agree to license my contribution under the project's license.

[mtrojnar]

Moved the OpenSSL ENGINE method-selection and lifetime management out of
src/p11_rsa.c into src/p11_pkey.c, alongside the existing EVP_PKEY and
ENGINE method integration. p11_rsa.c is again limited to RSA key material
and RSA_METHOD operations.

Revalidated the strict build, focused engine/provider RSA tests, and a clean
OPENSSL_NO_ENGINE strict build with both raw-only and native-PSS-only
SoftHSM scenarios.

[mtrojnar]

Reworked the fix to preserve the architecture boundary:

• Removed all newly introduced ENGINE calls and state from p11_* sources.
• The public libp11 API now fixes EVP_PKEY_meth_add0 #645 issue reopen #672 simply by not registering a global RSA
  EVP_PKEY_METHOD.
• Existing ENGINE method selection remains in eng_front.c; native-PSS-only
  and raw-RSA-only tests now explicitly run through the ENGINE executable.
• The provider continues to use its independent provider implementation.
• Added the three-component architecture and mandatory separation rules to
  AGENTS.md.

The branch remains a single squashed commit (9ab98f8). Full make check
and the clean OPENSSL_NO_ENGINE build pass.