Adding custom PII and ROT rules to Shinydocs Pro Control Center | Shinydocs Pro

Shinydocs Pro finds PII and ROT with rules. Each rule is a JSON file, and you can add your own next to the ones that ship with the product. For the shipped rules, see Default PII Rules and Default ROT Rules.

Where the rule files are

On the Shinydocs Pro server, in the Control Center install folder (by default C:\Program Files\Shinydocs Professional\ControlCenter):

PII: Pii Rules\Potential PII\
ROT: Rot Rules\Obsolete\, Rot Rules\Redundant\ and Rot Rules\Trivial\

Save one rule per file. The shipped files are named <field name>-<value>.json, for example potential_pii-Email_address.json; use a name of your own that won't clash with them. Keep a copy of your custom rules somewhere else too, so you can put them back after a reinstall.

Write a rule

RawQuery is an OpenSearch query (Query DSL) that picks the documents the rule tags. Test it before you apply the rule; see Testing PII and ROT rules.

PII rule fields

Field	Required	What it does
`Name`	Yes	The name shown in the rule lists under Settings > Analysis > PII.
`Description`	No	Can be shown under the name in the rule lists, and the lists' search matches it.
`DisplayGroup`	No	Groups rules in the results, for example `Driver's licenses`.
`FieldName`	Yes	`potential_pii`.
`FieldValue`	Yes	The value written to `potential_pii` for a match. Make it unique to the rule.
`Sensitivity`	Yes	`0` low, `1` medium or `2` high.
`RegexPattern`	No	A regular expression checked against the text of each document the query finds. Only documents with a match are tagged, and the matched text is saved in `pii_matches`. Try it at regex101.com.
`LuhnValidate`	No	`"true"` to keep only matches whose check digit passes the Luhn algorithm, as credit card and SIN numbers do.
`RawQuery`	Yes	The query that finds candidate documents, usually documents with text (`fullText`) that mention a related keyword.

Example: potential_pii-canadian_postal_code.json

JSON

{
  "Name": "Canadian postal code",
  "DisplayGroup": "Postal codes",
  "Description": "",
  "FieldName": "potential_pii",
  "FieldValue": "Canadian Postal Code",
  "Sensitivity": 0,
  "RegexPattern": "\\b[a-zA-Z]\\d[a-zA-Z][ -]{0,3}\\d[a-zA-Z]\\d\\b",
  "LuhnValidate": "false",
  "RawQuery": {
    "bool": {
      "must": [ { "exists": { "field": "fullText" } } ],
      "should": [
        { "match": { "fullText": "address" } },
        { "match": { "fullText": "canada" } },
        { "term": { "fullText": "CA" } }
      ],
      "minimum_should_match": 1,
      "must_not": [
        { "exists": { "field": "potential_pii" } },
        { "match_phrase": { "tag-Exclude-from-PII-detection": "true" } }
      ]
    }
  }
}

ROT rule fields

Field	Required	What it does
`Name`	Yes	The name shown in the rule lists under Settings > Analysis > ROT.
`Description`	No	Can be shown under the name in the rule lists.
`FieldName`	Yes	`rot_obsolete`, `rot_redundant` or `rot_trivial`.
`FieldValue`	Yes	The value written to that field for a match.
`RawQuery`	Yes	The query that picks the documents to tag.

Example: rot_obsolete-log.json

JSON

{
  "Name": "Log",
  "Description": "Old log files",
  "FieldName": "rot_obsolete",
  "FieldValue": "log",
  "RawQuery": {
    "bool": {
      "must": [ { "terms": { "extension": [ "log", "LOG" ] } } ],
      "filter": { "range": { "lastWriteTimeUtc": { "lte": "now-6M/d" } } },
      "must_not": [
        { "term": { "rot_obsolete": "log" } },
        { "match_phrase": { "tag-Exclude-from-ROT-detection": "true" } }
      ]
    }
  }
}

Tips:

Add a must_not on the rule's own field and value, as in the examples, so documents already tagged aren't processed again.
Add the must_not on tag-Exclude-from-PII-detection or tag-Exclude-from-ROT-detection, so files people excluded in the Catalog stay excluded. The rule's query is what skips them.
Extensions are stored in lowercase without the dot (log, not .log), though some connectors keep the original case.

Apply the rule

Restart the Shinydocs Control Center Windows service so it loads the new file. Otherwise it can take an hour or more to appear.
In the Control Center, go to Settings > Analysis and open the PII or ROT tab.
In Available rules, select your rule and select Add.
Select Save changes, check the list of changes, and select Save changes again. Save each tab on its own.

Applied rules are used by every source, the next time each source runs.

Run it on your sources

Go to Sources and open a source.
In Analysis, make sure Personal & sensitive information (PII) or Non-valuable content (ROT) is on. Personal & sensitive information also needs Reading file contents.
Select Run now, or wait for the source's next scheduled update. Each card shows its progress.

Expand a card to see the rule sets applied to it. Manage rule sets opens the PII or ROT tab.

Review the results

Overview shows PII and ROT across your sources. See Getting to Know Overview.
Catalog: filter on PII or ROT (Any PII, Any ROT, or a single rule's value) and combine them with other filters. The text a PII rule matched is in pii_matches. See Getting to Know Content Catalog.

Stop using a rule

Go to Settings > Analysis and open the PII or ROT tab.
In Applied rules, select the rule and select Remove.
Select Save changes and confirm.

The rule's tags are removed from each source's documents the next time that source runs.