Telugu TN: Cardinal Semiotic Class

[nharidas]

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Before your PR is "Ready for review"

Pre checks:

• Have you signed your commits? Use git commit -s to sign.
• Do all unittests finish successfully before sending PR?
  1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
  2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
• If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
• Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
• Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
• Have you added the correct license header Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
• If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
• Remove import guards (try import: ... except: ...) if not already done.
• If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
• Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

• New Feature
• Bugfix
• Documentation
• Test

If you haven't finished some of the above items you can still open "Draft" PR.

[github-actions]

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

[folivoramanh]

make sure both pytest and sparrowhawk test pass, and fix the problem of blank space

[nharidas]

Thanks for the review. I’ve addressed the comments on this PR.

• Removed Telugu post-processing (no PostProcessingFst for te; tests use post_process=False)
• Dropped custom add_weight; competing *1 forms are split instead of using a -0.2 bias
• Deleted teens_and_ties_thousand.tsv; combining ఒక is composed from ties
• magnitudes.tsv is loaded with load_labels
• Digit/zero classes come from digit.tsv / zero.tsv (no hardcoded NEMO_TE_DIGIT)
• Word tagger simplified to NEMO_NOT_SPACE
• Jenkinsfile: removed the commented TE ITN stage
• Tests: assert pred == expected (no .strip())

pytest for Telugu cardinals is passing. Sparrowhawk tests are still pending; I’ll update this PR once they are run.

[folivoramanh]

• add init file in data/ (currently only in data/numbers/)

[folivoramanh]

check this example and see that if you can build a reusable component for thousand/million/billion/trillion
https://github.com/NVIDIA/NeMo-text-processing/blob/main/nemo_text_processing/text_normalization/es/taggers/cardinal.py

[nharidas]

Thanks for the pointer. I reviewed, es/taggers/cardinal.py.

Spanish works with one reusable block because it uses the Western scale: each step adds 6 zeros (thousand → million → billion → trillion) and reuses the same 3-digit pattern with a new magnitude word.

Telugu uses the Indian numbering system, which is structured differently:

• 1,000 → thousand
• 1,00,000 → lakh (not “hundred thousand”)
• 1,00,00,000 → crore
• higher tiers (arab, kharab, nil, padma, shankh) with different digit groupings

On top of that, the spoken form is not one fixed word per magnitude. It depends on position in the number and what follows, for example:

• singular vs plural / “before” forms (e.g. వేల vs వేలు, కోటి vs కోట్లు)
• special “one” form when combining (ఒక in 21, 31, … vs ఒకటి for standalone 1)
• prefix variants (e.g. నూట vs వంద for hundreds)
• special lakh form (లక్షా)

So we cannot drop in one ES-style thousand/million/billion component, both the scale and the word forms change at each tier.

This current code already reuses shared logic where the grammar allows it:

• build_group, create_graph_suffix, create_larger_number_graph
• magnitude ladders (thousand → lakh → crore → …)
• shared strings in magnitudes.tsv

The longer code reflects real Telugu morphology, not unnecessary duplication. Collapsing it into one Spanish-style block would risk wrong readings (e.g. 21 → ఇరవై ఒక, not ఇరవై ఒకటి).

[mgrafu]

if it uses the Indian numbering system, are we building the graph in the same way that the Hindi graph does?

[nharidas]

@mgrafu Yes, same Indian numbering system and the same overall graph structure as Hindi.
Both use the tiered scale (hundred → thousand → lakh → crore → arab/kharab/nil/…) with the same create_graph_suffix / create_larger_number_graph pattern and the same 2+3 digit grouping.
Telugu extends that base for language-specific morphology: singular/plural/before magnitude forms, special “one” handling (ఒక vs ఒకటి), and hundred prefixes (నూట vs వంద), driven by TSVs and build_group ladders. Hindi uses a simpler one-suffix-per-tier model.

[mgrafu]

ok, thanks! one more comment though: graph_arabs / graph_kharabs / graph_padmas / graph_shankhs name tiers that produce no such word. The file already uses crore-relative naming for its prefix helpers (hundred_crore_prefix, lakh_crore_count_prefix, ten_lakh_crore_prefix), so this is internally inconsistent too. Renaming to that existing convention — graph_hundred_crores (10^9), graph_ten_thousand_crores (10^11), graph_lakh_crores (10^13), graph_crore_crores (10^15) — would make the structure self-describing.

[nharidas]

@mgrafu Done. Renamed higher tiers to the crore-relative convention already used in the file:
graph_arabs → graph_hundred_crores
graph_ten_arabs → graph_thousand_crores
graph_kharabs → graph_ten_thousand_crores
graph_ten_kharabs → graph_lakh_crores
graph_ten_nils → graph_ten_lakh_crores
graph_padmas → graph_crore_crores
graph_ten_padmas → graph_ten_crore_crores
graph_shankhs → graph_hundred_crore_crores

[nharidas]

Thanks for the follow-up review.

Changes Made:

• Added nemo_text_processing/text_normalization/te/data/init.py
• Simplified Jenkinsfile to a single L0: TE TN grammars stage (removed the parallel Create TE TN/ITN Grammars wrapper)
• Expanded cardinal tests to cover large magnitudes, including 999999999999999 (15 digits) and up to 19-digit shankh-scale numbers, plus Telugu digits, in-sentence, leading zeros, and negatives
• Negative numbers: verbalizer now outputs “మైనస్” for the sign (Telugu localized form, same pattern as Spanish “menos”), instead of English “minus”. Test cases updated accordingly.

Testing:

• Cardinal pytest test cases passed
• Sparrowhawk “testTNCardinal” passed

Re: Spanish reusable component: replied in the thread with details.

[nharidas]

Thanks for the review.

Changes Made:

• Expanded test_cases_cardinal.txt with internal-zero and separator coverage (Indian and Western grouping)
• Added separator handling in cardinal.py (indian_grouping / western_grouping) to strip commas before normalization
• Moved hardcoded inputs to TSV files.
• Removed unused code in test_sparrowhawk_normalization.sh

Testing:

• Cardinal pytest test cases passed
• Sparrowhawk testTNCardinal passed

[nharidas]

@mgrafu Thanks for the review.

Changes Made:

• Documented max magnitude coverage in the CardinalFst docstring (up to 19 digits / shankh / శంఖ, 10¹⁷; larger values via composed కోటి groups)
• Fixed singular/plural కోటి vs కోట్ల for large numbers.
• Expanded test_cases_cardinal.txt with the suggested large-number, mixed-script, separator, and edge cases
• Updated Jenkins: guarded TE L0 grammar build (when + failFast + parallel) and L1 TE pytest stage with TE_TN_CACHE
• Added word and punctuation unit tests (test_word.py, test_punctuation.py + corresponding test_cases_*.txt)
• Updated test_sparrowhawk_normalization.sh to run word and punctuation Sparrowhawk suites

Testing:

• Cardinal, word, and punctuation pytest cases passed
• Sparrowhawk testTNCardinal, testTNWord, and testTNPunctuation passed

[mgrafu]

please follow the style of English word test cases

[mgrafu]

also, are specific English test cases necessary?

[nharidas]

@mgrafu Done. updated the TE word tests to follow the English style (empty/whitespace, leading/trailing space, sentences with punct, hyphenated words, digit+punct spacing, etc.), with Telugu examples.
EN cases that need measure/money grammars, or English-only letter+digit splits, aren’t covered yet since those aren’t in TE.

[mgrafu]

let's test just Telugu for now instead of English. Please remove lines like $ and 5% or %~$ and ఐదు % or % or X!~X!

[mgrafu]

let's review and add these test cases

51010000000~ఐదు వేల నూట ఒక కోటి
34018285364~మూడు వేల నాలుగు వందల ఒక కోటి ఎనభై రెండు లక్షల ఎనభై ఐదు వేల మూడు వందల అరవై నాలుగు
711010000000~డెబ్బై ఒక వేల నూట ఒక కోటి
465017762066~నలభై ఆరు వేల ఐదు వందల ఒక కోటి డెబ్బై ఏడు లక్షల అరవై రెండు వేల అరవై ఆరు
3164010000000~మూడు లక్షల పదహారు వేల నాలుగు వందల ఒక కోటి
97931010000000~తొంభై ఏడు లక్షల తొంభై మూడు వేల నూట ఒక కోటి
90140000000000000~తొమ్మిది వందల ఒక కోటి నలభై లక్షల కోట్లు
340100000000000000~మూడు వేల నాలుగు వందల ఒక కోటి కోట్లు
340182853640000000~మూడు వేల నాలుగు వందల ఒక కోటి ఎనభై రెండు లక్షల ఎనభై ఐదు వేల మూడు వందల అరవై నాలుగు కోట్లు
4650177620660000000~నలభై ఆరు వేల ఐదు వందల ఒక కోటి డెబ్బై ఏడు లక్షల అరవై రెండు వేల అరవై ఆరు కోట్లు
51020000000~ఐదు వేల నూట రెండు కోట్లు
34028285364~మూడు వేల నాలుగు వందల రెండు కోట్ల ఎనభై రెండు లక్షల ఎనభై ఐదు వేల మూడు వందల అరవై నాలుగు
90240000000000000~తొమ్మిది వందల రెండు కోట్ల నలభై లక్షల కోట్లు
90100000000000000~తొమ్మిది వందల ఒక కోటి కోట్లు
10010000000~వెయ్యి ఒక కోటి
20010000000~రెండు వేల ఒక కోటి
50010000000~ఐదు వేల ఒక కోటి
10001010000000~పది లక్షల నూట ఒక కోటి
10000000101~వెయ్యి కోట్ల నూట ఒకటి
10000000201~వెయ్యి కోట్ల రెండు వందల ఒకటి
220000000000000~రెండు కోట్ల ఇరవై లక్షల కోట్లు
230000000000000~రెండు కోట్ల ముప్పై లక్షల కోట్లు
410000000000000~నాలుగు కోట్ల పది లక్షల కోట్లు
941302654706146~తొమ్మిది కోట్ల నలభై ఒక లక్షల ముప్పై వేల రెండు వందల అరవై ఐదు కోట్ల నలభై ఏడు లక్షల ఆరు వేల నూట నలభై ఆరు
851750346234890~ఎనిమిది కోట్ల యాభై ఒక లక్షల డెబ్బై ఐదు వేల ముప్పై నాలుగు కోట్ల అరవై రెండు లక్షల ముప్పై నాలుగు వేల ఎనిమిది వందల తొంభై
0౫~0౫
౦5~౦5
00౭~00౭
0౦5~0౦5
05~సున్నా ఐదు
౦౫~సున్నా ఐదు
007~సున్నా సున్నా ఏడు
౦౦౭~సున్నా సున్నా ఏడు

[nharidas]

@mgrafu Done. Added all the test cases you listed. A few didn’t match at first (…01 కోటి, ఒక vs ఒకటి before లక్షల, and mixed-script zeros). Fixed those; everything passes now.

[nharidas]

@mgrafu Thanks for the review.

Changes Made:

• Fixed cardinal edge cases: …01 → ఒక కోటి, X1 లక్షల → ఒక (not ఒకటి), mixed-script leading zeros pass-through
• Renamed arab/kharab/padma/shankh tiers to crore-relative names.
• Fixed FST ambiguity so Sparrowhawk and Pytest agree on large numbers (e.g. 3164010000000)
• Added EN-style word and punctuation tests (Telugu examples), plus punctuation match-input via punct_post_process=True

Testing:

• Cardinal, word, and punctuation pytest cases passed
• Sparrowhawk testTNCardinal, testTNWord, and testTNPunctuation passed