Automate Loading Data from Semaphore

[jung-thomas]

Rather than a batch data load from Semaphore, can we automate this?
https://wiki.one.int.sap/wiki/spaces/SAPONEDX/pages/1894943972/Semaphore+API+-+Various+Ways+to+Consume+and+Update

[jung-thomas]

Feasibility: yes — automate via Semaphore's SES REST API

Confirmed we can replace the manual batch load with an automated sync. The SES allterms.json endpoint (e.g. https://sap.data.progress.cloud/semantic/prodses/SAPCore/en/allterms.json, filterable with ?FILTER=CL=…) returns the model's terms as JSON — the same shape we already ingest. So this is a substitution of the input source, not new plumbing.

Current state (no live Semaphore connection today)

• The Tags entity already carries semaphoreId, isActualTag, isInterestItem — no schema change needed.
• Real semaphoreId values arrived once, via the legacy HANA→HANA migration from IMS_TAG.
• Ongoing loads go through the Admin-UI bulk importer, but it writes only name + titlePath and drops semaphoreId. That manual step is what this issue removes.
• Topic clusters / featured topics / channel atlas are Knowledge-Graph– and Channels-derived, not Semaphore — out of scope.

Proposed design

1. srv/lib/semaphore-client.js (mirrors the existing NGDS client) calling SES allterms.json, optionally per term-class via FILTER.
2. A scheduled job under srv/jobs/ wired through the existing CronService/scheduler — periodic sync instead of a batch upload.
3. Extend the existing tag-import upsert engine to also persist semaphoreId / isActualTag / isInterestItem (closes a current gap; the manual path benefits too).
4. Auth via a CAP requires destination, Service-Account token held in the BTP Credential Store (not env).
5. DB-feature-flag gated, fail-open, DEV-first — standard rollout shape.

One prerequisite (being requested in parallel)

Unattended SES access needs a Service Account with an API User role + auth token for SAP's Semaphore tenant. Draft email to DL Semaphore Modelling Support below.

---

Draft email — Service Account request

To: DL Semaphore Modelling Support
Subject: Service Account request — automated read-only SES API access (developers.sap.com tutorials platform)

Hello,

We are automating taxonomy ingestion for the developers.sap.com tutorials platform (replacing a manual batch load). We'd like to consume the SES (Semantic Enhancement Server) REST API on the SAP tenant unattended — read-only — to keep our tag catalogue in sync with the SAPCore model.

Could you please set up:

1. A Service Account for productive/unattended use (not tied to an individual user).
2. The API User role on that account.
3. An authentication token for it, with the setup steps.
4. Confirmation of read-only SES access to the SAPCore model (e.g. .../semantic/prodses/SAPCore/en/allterms.json).

Scope is read-only runtime consumption (SES allterms/terms/filter commands) — no model-management (KMM) access needed. We'll store the token securely in the SAP BTP Credential Store; it will not be embedded in source.

Please let us know if you need any additional details (tenant, intended call volume/cadence, source IPs).

Thanks,
[name / team]

I'll proceed with design + development against DEV in parallel; will wire the token in once the Service Account is issued.

[jung-thomas]

Futher details 1)A service account is created with API user role. I will store the API key in PassVault. Do you have a PassVault Group? I will send the information via a Jira Ticket.

2. You can generate a token by submitting a POST request to the token service at: https://sap.data.progress.cloud/token/

3. https://sap.data.progress.cloud/semantic/prodses/SAPCore/en/allterms.json

4. Here’s some general guidance which should help to regenerate the API programmatically

---

Connect to PDC ("POST /token" using service account API key saved locally in a configuration file or something like this).
Check to see if the API key is about to expire ("GET /api/account/apikey", examine the "expiryDate" returned).
If it is "about to expire" (one week ahead, one day ahead, whatever is needed) then do the following:
Generate a new API key ("PUT /api/account/apikey" which returns the API key in the response but disconnects the session).
Save the new API key returned in the previous step for use the next time the process runs (in the same configuration file mentioned above).
Re-connect to PDC using the new API key ("POST /token").
Continue processing.

The swagger functions would look something similar to:

Connect to the Progress Data Cloud using service account API key saved locally in a configuration file
How: Make a POST request to /token with the current API key from the config file

Check if the key is nearing expiration.
How: Make a GET request to /api/account/apikey

Check expiration date field- Ex. "expiryDate"
If the expiry date is within a predefined threshold (e.g., 7 days), proceed to Step 3

Generate a new API key and save
How: Make a PUT request to /api/account/apikey using the current access_token in the Authorization header

Note: This action disconnects your current session — re-authentication is needed

Save the new key
How: Overwrite the old API key in your config file or secret store

Re-authenticate using the new key for future use.
How: Make a POST request to /token with the current API key from the config file (step 1)

Note: After generating a new API key, all future automated tasks should reference the updated key from your config file or secret store to prevent authentication failures