FEAT add SATA masking converter - #2404
FEAT add SATA masking converter#2404Alireza Aminzadeh (alireza-aminzadeh) wants to merge 2 commits into
Conversation
|
@microsoft-github-policy-service agree |
| Build the converter identifier with SATA masking parameters. | ||
|
|
||
| Returns: | ||
| ComponentIdentifier: The identifier for this converter. |
There was a problem hiding this comment.
This identifier can be misleading when a custom strategy is supplied: it still records num_masks and skip_first even though those values are ignored. It also omits task_template, so converters that produce different framed output can have identical identifiers. Please make the identifier describe the behavior actually in use, including the framing template when applicable.
There was a problem hiding this comment.
The identifier now always records mask_token and the selection-strategy class. num_masks, skip_first, and min_word_length are included only for the default strategy, since a custom strategy ignores them. Framing is no longer applied inside this converter, so there is no template parameter to list.
Add a dependency-free converter that selects content words and replaces them with [MASK] so SATA can be composed with TaskFramingConverter.
Keep TaskFramingConverter composition outside this converter, reject mixed selection_strategy configuration, and identify only the parameters that affect output.
336e688 to
8c065a8
Compare
2 participants
Summary
SATAMaskingConverterand a reusableContentWordSelectionStrategythat selects content words with a deterministic, dependency-free heuristic (no NLTK or POS-tagger download).TaskFramingConverterviaSATA_TASK_TEMPLATE(the wiki-infill prompt from the SATA paper). It also works withSelectiveTextConverterif you want to apply a different sub-converter to the same word selection.I chose the dependency-free selector so this stays easy to test and does not add model-data downloads. Feedback welcome if maintainers would rather plug in an optional POS tagger later.
Test plan
pytest tests/unit/converter/test_sata_masking_converter.py tests/unit/converter/test_text_selection_strategy.py tests/unit/converter/test_task_framing_converter.py tests/unit/converter/test_selective_text_converter.py tests/unit/docs/test_converter_documentation.py(148 passed)doc/code/converters/1_text_to_text_converters.py