Make Transformers optional

[hallerite]

Summary

• replace the mandatory transformers dependency with a small structural tokenizer contract and a Rust tokenizers adapter
• keep opaque chat templates, custom remote-code tokenizers, AutoConfig probing, and AutoProcessor loading behind the optional renderers[hf] extra
• update that optional integration and the parity-test environment to Transformers 5.14.1
• retain offset-based body/scaffold attribution and add explicit ID + Unicode-offset parity coverage
• fail clearly for unsupported core-only cases instead of silently selecting a text renderer for an unknown VLM

Why

Concrete renderers only need token IDs, decoding, special-token lookup, and character offsets. Importing and installing all of Transformers for that surface made the core package larger, created downstream version conflicts, and coupled every renderer annotation to an implementation detail.

The standalone backend cannot replace three HF-only capabilities: apply_chat_template for unknown models, custom tokenizers that do not publish tokenizer.json (notably Kimi), and multimodal AutoProcessor loading. Those remain supported through renderers[hf] or caller-supplied compatible objects.

Transformers 5.14.1 constrains Tokenizers to <=0.23.0; because 0.23.1 is above that cap, the core uses the newest published compatible Tokenizers release, 0.22.2.

Validation

• ruff check .
• full suite on Transformers 5.14.1: 2728 passed, 128 skipped, 1 xfailed
• final multimodal/render/roundtrip slice: 562 passed, 54 skipped, 1 xfailed
• focused backend/config/security slice: 50 passed
• ID and offset parity checked across Qwen3, Llama 3.2, and DeepSeek V3, including emoji, accented text, CJK, and special-token scaffolding
• built wheel metadata contains tokenizers in core and transformers only under the hf extra
• clean temporary core install contained no Transformers, imported every renderer, loaded Qwen3 with TokenizersTokenizer, and rendered a Unicode prompt

Note

Make transformers an optional dependency, adding a standalone tokenizers backend

• Moves transformers from a core dependency to an optional renderers[hf] extra in pyproject.toml, with tokenizers (Rust) as the new core dependency.
• Adds a new renderers/tokenizer.py module with TokenizerLike, ChatTemplateTokenizerLike protocols and a TokenizersTokenizer adapter for the Rust tokenizers library.
• Extends load_tokenizer in renderers/base.py with backend ('auto', 'tokenizers', 'transformers') and revision parameters; 'auto' uses the Rust backend when transformers is absent.
• VLM renderers (Qwen3VL, Qwen35, KimiK25) guard AutoProcessor imports and raise ImportError with instructions to install renderers[hf] when transformers is missing.
• DefaultRenderer now raises TypeError at construction time if the tokenizer lacks apply_chat_template, pointing users to renderers[hf] or a model-specific renderer.
• Behavioral Change: code that previously imported renderers without transformers would fail at import; it now succeeds but raises at runtime when HF-specific features are used.

^{Macroscope summarized 3bd9f6e. (Automatic summaries will resume when PR exits draft mode or review begins).}