Shinydocs Pro

CLI: Migration: File Share to SharePoint Online

Software: Cognitive Toolkit (CognitiveToolkit.exe)

Operation: Migrate

Use the Cognitive Toolkit’s Migrate operation to copy files that Shinydocs Pro has crawled from a file share into a SharePoint Online document library, keeping their folder structure.

  • File contents are copied, and each file gets the created and modified dates of the source file.

  • Permissions aren’t carried over. The files get the library’s permissions.

  • Columns can be filled from index fields with a mapping file.

  • The source files stay where they are.

Before you start

  • The Cognitive Toolkit is in the Control Center folder, by default C:\Program Files\Shinydocs Professional\ControlCenter. Activate it once with your Shinydocs Pro license file, as the Windows account that will run the migration: CognitiveToolkit.exe activate -p "C:\Shinydocs\license.xml". Keep the license file where it is: the Cognitive Toolkit reads it from that path each time. Your license must include migration.

  • The source must already be crawled by Shinydocs Pro. To find its index name, open the source in Control Center and look under Index information: you can use its Name or its Alias, and each has a copy button.

  • Register an application in Microsoft Entra ID that signs in with a certificate, and grant it SharePoint application permissions that let it add files to the destination site, with admin consent. Microsoft’s guide uses Sites.FullControl.All; to limit the app to chosen sites, use Sites.Selected and give it write access to the destination site. See Microsoft’s Granting access via Entra ID App-Only. SharePoint Online only accepts certificates for app-only access.

  • Put the certificate’s .pfx file on the machine that runs Migrate, and note the app’s application (client) ID and your directory (tenant) ID.

  • The account that runs Migrate needs to read the source files.

  • Give every file as a full path.

Steps

1. Create the destination settings file

Save this as, for example, C:\Shinydocs\sources\sharepoint-online.json:

JSON
{
  "sourceType": "sharepoint-online",
  "sourceConnectionSettings": {
    "site-url": "https://contoso.sharepoint.com/sites/Records",
    "app-id": "00000000-0000-0000-0000-000000000000",
    "tenant-id": "00000000-0000-0000-0000-000000000000",
    "cert-path": "C:\\Shinydocs\\Certificates\\contoso-migrate.pfx",
    "cert-password": "~spo-cert-password"
  }
}

Key

Value

site-url

The URL of the site to migrate into.

app-id

The application (client) ID of the app registration.

tenant-id

Your directory (tenant) ID.

cert-path

Full path to the certificate’s .pfx file. In JSON, write each backslash twice.

cert-password

The .pfx file’s password, if it has one. Use a tilde (~) followed by a value you saved with CognitiveToolkit.exe savevalue; see CLI: Encrypting repository passwords with SaveValue. Leave the key out if the file has no password.

The key names are case-sensitive, and sourceType must be sharepoint-online.

2. Create the query file

The query file holds only the query, with no "query" wrapper. To migrate everything in the index, save this as C:\Shinydocs\queries\match_all.json:

JSON
{ "match_all": {} }

For more examples, see CLI: Common Query Types.

3. Run Migrate

Open Command Prompt as the account that will run the migration, go to the Control Center folder and run Migrate. For example:

CognitiveToolkit.exe Migrate --index-server-url "http://localhost:9200" --index-name "finance-share" --query "C:\Shinydocs\queries\match_all.json" --destination-source-settings "C:\Shinydocs\sources\sharepoint-online.json" --path-prefix-to-remove "\\FS01\Finance" --start-location "Shared Documents"

--start-location is the library’s address within the site, not its display name. For the default Documents library, that’s Shared Documents. With these values, \\FS01\Finance\Budgets\2026\plan.xlsx goes to the Budgets/2026 folder of that library.

To check your settings first, add --dry-run: Migrate goes through the items without uploading anything or updating the index.

Check the results

Migrate shows its progress as it goes, and ends with an exit code:

Exit code

Meaning

0

Every item the query found was migrated.

1

Some items failed, or a required parameter was missing.

5

A license problem, for example a license without migration.

-1

Migrate stopped, for example because a settings file isn’t valid JSON or sign-in to the destination failed.

Details of each failure are in the Cognitive Toolkit’s log file, CognitiveToolkit<date>.log, in its logs folder.

Migrate also records the result on each item in the index, so you can find failures with the Catalog’s advanced search, for example last-migration-status:error:

Field

Value

last-migration-status

success or error

last-migration-date

When the item was last migrated.

error-string

Why the item failed. Empty when it succeeded.

prop-content-id

Where the item went: the file’s server-relative URL in SharePoint. Set only when it succeeded.

Running Migrate again processes every item the query finds, so files already uploaded are uploaded again and replaced. To migrate only the items that failed or haven’t been migrated yet, use this query file:

JSON
{
  "bool": {
    "must_not": [
      { "match": { "last-migration-status": "success" } }
    ]
  }
}

File and folder names

Migrate changes names that SharePoint Online doesn’t allow:

  • Each of " * : < > ? / \ | and the text _vti_ is replaced with a hyphen (-).

  • A hyphen is added in front of a name that SharePoint reserves, such as CON, PRN, AUX, NUL, COM1, LPT1, .lock or desktop.ini.

  • Repeated hyphens are then reduced to one, including hyphens that were already in the name.

See also CLI Migration: Characters to Avoid.

Files that already exist

By default, Migrate replaces a file with the same name in the destination folder. To keep existing files, add --disable-over-write: those items aren’t uploaded and are reported as errors.

Filling columns

To fill library columns from index fields, give Migrate a mapping file with --metadata-mapping. Each key is a column’s display name, matched exactly, including capitals, and each value is an index field name:

JSON
[
  {
    "values": {
      "Department": "department",
      "Document Type": "doc-type"
    }
  }
]

Command parameters

Parameter

Required

Description

--index-server-url, -u

Yes

The Shinydocs Pro search engine URL, for example http://localhost:9200.

--index-name, -i

Yes

The source’s index name or alias.

--query, -q

Yes

Full path to the query file.

--destination-source-settings

Yes

Full path to the destination settings file.

--start-location

Yes

The library’s address within the site, for example Shared Documents.

--path-prefix-to-remove

No

The part of each source path to leave out, usually the folder that was crawled. Without it, only the drive letter or server name is removed.

--disable-over-write

No

Don’t replace files that already exist. Give it with no value.

--metadata-mapping

No

Full path to the column mapping file.

--name-field-name

No

An index field that holds each item’s file name, used instead of its current name.

--threads, -t

No

How many items to migrate at once. The default is 1.

--dry-run

No

Go through the items without uploading anything or updating the index.

--location-field isn’t supported for SharePoint Online.