Make the LLM redactor just a redactor

[infrared0]

We still need redaction capabilities but probably need to take the LLM out of the loop. Then we can remove the 20 minute edit lock on tickets.

[gw1108]

I'm not sure what redaction capabilities are planned? And where the 20 minute edit lock is? @infrared0

[philion]

We need clarification from Esther. No idea what this means.

[infrared0]

Currently the PII (personally identifiable information) redactor is LLM-based and only has maybe 85% recall (it misses some PII identification in Redmine tickets). It also takes 20 minutes to perform each redaction, and tickets are "locked" while that is happening. After discussion with some SCN folks with strong opinions, I think it is probably best to not use a LLM for this and instead just write code that catches PII and redacts it, creating a redacted public version without PII that is saved alongside the non-redacted version (as implemented now). We also want responses in Redmine tickets to be redacted, not only ticket descriptions. Currently we have LLM prompts that define what type of information we wanted to redact, which we can ask Rudra for.

Paul- could you pls point us at the current redactor code?
Here is the research paper we wrote about the LLM project: https://www.overleaf.com/read/nzrckzgdhzpr#146cac

[philion]

The netbot code is in https://github.com/Local-Connectivity-Lab/netbot/tree/llm-redactor.

I don't know if the LLM code is committed anywhere. It can be found on the LLM proxmox host: ~scn/llm-redaction-app/. (creds in VW)

Regarding "code that catches PII and redacts it": https://github.com/LeapBeyond/scrubadub . I found a lot of recent projects that use LLMs. This older one doesn't (after cursory review).

Some design questions, assuming this (or similar) library works as expected:

• store the redacted PII or remove it completely?
• if storing (assumed): Store as a chunk of unredacted text (the original email), or as fields that are substituted back into the text?: { EMAIL:"...", PHONE:"..." }
• should this be applied to comments (any text entered in the discord ticket thread)?

Assuming roughly current behavior:

• When a new ticket is created, via email or discord command, redact the original text using an old-school NLP redactor.
• Store the original text in custom field in the ticket record in redmine.
• Command to show original text is only available to admins/special roles, and custom field is admin-only.

[philion]

Noting: The "llm-redactor" branch can be merged with main. I made sure that the redactor can be not-configured or not-running and processing can still proceed. The same framework for redaction can be used for a local redaction library.

See: https://github.com/Local-Connectivity-Lab/netbot/blob/llm-redactor/threader/imap.py

[gw1108]

I can't seem to view the https://github.com/Local-Connectivity-Lab/netbot/blob/llm-redactor/threader/imap.py or the https://github.com/Local-Connectivity-Lab/netbot/tree/llm-redactor.

[AperExanimus]

https://github.com/Local-Connectivity-Lab/netbot/blob/main/threader/imap.py seems like there was a name change or other change that broke previous links. I found the code by searching LCL's repository directly.