Merge remote-tracking branch 'origin/staging' into pr7624

Author: waleedlatif1

.github/workflows/test-build.yml

@@ -153,6 +153,15 @@ jobs:
           lib/table/rows/secret-provenance.postgres.test.ts
           lib/memory/message-provenance.postgres.test.ts
 
+      - name: Verify Search progress and pagination in PostgreSQL
+        working-directory: apps/sim
+        env:
+          KNOWLEDGE_ACL_TEST_DATABASE_URL: postgresql://postgres:postgres@127.0.0.1:5432/sim_auth_scim
+        run: >-
+          bunx vitest run --mode integration
+          lib/knowledge/__integration__/search-source-progress.integration.ts
+          lib/knowledge/__integration__/search-source-pagination.integration.ts
+
   test-build:
     name: Lint and Test
     runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}

apps/docs/content/docs/search/confluence.mdx

@@ -11,16 +11,16 @@ Search pages and blog posts from selected Confluence Cloud spaces. A Sim organiz
 
 Search indexes each page's own text, including supported local callouts and code blocks. It does not expand Include Page, Excerpt Include, or third-party macros into that page. Referenced pages can be indexed separately with their own access rules.
 
-Admin setup uses your organization's **Settings → Integrations** page. Teammates connect from **Integrations** in the main sidebar. For workspace Search, use **Search → Add source** instead; **Create & Invite** is the workspace equivalent of **Add source**.
+Admin setup uses your organization's **Settings → Sources** page. Teammates connect from **Integrations** in the main sidebar. For workspace Search, use **Search → Add source** instead; **Create & Invite** is the workspace equivalent of **Add source**.
 
-## Choose a connection method
+## Choose a setup
 
 | Method | Who supplies the content? | What teammates do |
 | --- | --- | --- |
-| **Admin or service account** | One account syncs content, space permissions, page restrictions, and group membership. | Connect their own Confluence account so Sim can match their Atlassian identity to those permissions. |
+| **Central account** | One account syncs content, space permissions, page restrictions, and group membership. | Connect their own Confluence account so Sim can match their Atlassian identity to those permissions. |
 | **Member accounts** | Sim syncs content separately through connected members' accounts. | Connect their own Confluence account to establish which pages they can access. |
 
-Use **Admin or service account** when you have a dedicated account that can read the intended spaces and their permissions. Use **Member accounts** when each person should supply their own connection. Available methods depend on your organization's enabled features.
+Adding a source from **Settings → Sources → Confluence → Sources** opens central account setup. Use this when one account can read the intended spaces and their permissions. For a member source, start **Connect account** from **Integrations** in the main sidebar after an admin adds Confluence. Available methods depend on your organization's enabled features.
 
 **Everyone still connects in both methods.** With a central account, teammates supply their identity; they do not configure another central crawl or choose spaces again.
 
@@ -33,23 +33,21 @@ Use **Admin or service account** when you have a dedicated account that can read
 
 On hosted Sim, personal connections authorize the existing Sim app. Teammates do not create OAuth apps or service-account tokens. Self-hosted deployments need the [shared OAuth configuration](#self-hosted-operator-setup) even when a service account supplies the content.
 
-## Set up the source
+## Set up a central source
 
 <Steps>
 <Step>
 
 ### Choose Confluence
 
-Open **Settings → Integrations → Providers**, approve **Confluence**, then select **Set up**. Select your **Connection method**.
+Open **Settings → Sources → Add integration** and choose **Confluence**. On its **Sources** tab, select **Add source**. If Confluence is already listed, open it directly. **Setup guide** opens this guide from the source form.
 
 </Step>
 <Step>
 
 ### Select an account
 
-For **Admin or service account**, open **Account** and select an existing account, choose **Connect Confluence account** for OAuth, or add a service account using the [steps below](#using-a-service-account).
-
-For **Member accounts**, **Browse with** supplies an account for the space picker only. Select or connect an account, or switch **Spaces** to manual input to enter space keys without a browsing account. Browsing does not connect that account to Search or share its access with teammates.
+Under **Indexing account**, select an existing account, choose **Connect Confluence account** for OAuth, or add a service account using the [steps below](#using-a-service-account). The account must be able to read the content and its permissions.
 
 </Step>
 <Step>
@@ -58,27 +56,33 @@ For **Member accounts**, **Browse with** supplies an account for the space picke
 
 Enter **Confluence Domain**, then choose one or more **Spaces**. The picker shows spaces accessible to the selected account. Use the switch beside the field to enter comma-separated **Space Keys**, such as `ENG, PRODUCT`.
 
-Keep **Content Type** at its default for pages, or choose blog posts or both. Leave **Filter by Label** empty unless you want a smaller scope. **Document details (optional)** contains metadata tag settings.
+Open **More options** to change **Content Type**, **Filter by Label**, or **Metadata tags**. The default includes pages; choose **All content** for pages and blog posts. Leave the label filter empty unless you want a smaller scope.
 
 <Image
   src="/static/search/confluence-setup.jpg"
-  alt="Confluence Search source configuration showing connection method, account, domain, and spaces"
-  width={1280}
-  height={720}
+  alt="Confluence central source setup with an indexing account, domain, and spaces"
+  width={520}
+  height={397}
 />
 
 </Step>
 <Step>
 
 ### Save and connect your identity
 
-Click **Connect & Sync** for a central account, or **Add source** for member accounts. Back in Integrations, click **Connect account** on the Confluence row and finish the connection in the new tab. Sign in using the Atlassian email that matches your verified Sim email, and authorize the configured site.
+Click **Connect & Sync**. Then open **Integrations** in the main sidebar, click **Connect account** on the Confluence source, and finish the connection in the new tab. Sign in using the Atlassian email that matches your verified Sim email, and authorize the configured site.
 
 Each teammate completes this last step. A previously authorized account may already be connected. Return to Integrations to see indexing status and your searchable document count.
 
 </Step>
 </Steps>
 
+## Connect member accounts
+
+After an admin adds Confluence, open **Integrations** in the main sidebar and select **Connect account**. If there is no source yet, enter **Confluence Domain** and **Space Keys**, then select **Connect** and authorize your account. For another site or space scope, use **Add source** beside **Add another Confluence source**.
+
+An admin can open the resulting source under **Settings → Sources → Confluence → Sources** and use **Settings** to adjust its filters. **Account for browsing** helps populate the space picker; it does not connect that account for Search. Manual space keys work without a browsing account.
+
 ## Using a service account
 
 Sim's Atlassian service account form accepts a **scoped API token** and **site domain**.
@@ -135,12 +139,12 @@ read:group:confluence
 
 ### Add the token to Sim
 
-In the Search setup's **Account** menu, choose the service-account option. Paste the **API token** and enter **Site domain**. Optionally add a display name and description, then click **Add service account**. Continue in the original source modal, using the same domain in both forms.
+Under **Indexing account**, choose the service-account connection action. Paste the **API token** and enter **Site domain**. Optionally add a display name and description, then click **Add service account**. Continue in the original source modal, using the same domain in both forms.
 
 </Step>
 </Steps>
 
-Scopes do not grant access to spaces or pages by themselves. Keep the account's Confluence permissions and its token scopes aligned. When a token expires or needs different scopes, create a replacement in Atlassian. In Sim, open **Integrations**, select the saved service account, and click **Reconnect** to enter the new token and the same site domain.
+Scopes do not grant access to spaces or pages by themselves. Keep the account's Confluence permissions and its token scopes aligned. When a token expires or needs different scopes, create a replacement in Atlassian. Add the replacement service account in the source's **Settings**, then use **Change indexing account** to apply it.
 
 <Callout type="info">
 Personal OAuth uses Sim's shared Confluence integration and requests a broader set of permissions, including writes. Search reads content and permissions; it does not edit your Confluence pages. Older OAuth connections need to reconnect to grant the group-read permission used by central permission syncing.
@@ -154,7 +158,7 @@ Personal OAuth uses Sim's shared Confluence integration and requests a broader s
 | **Spaces / Space Keys** | Required spaces to index. The picker and manual key input are two ways to set the same scope. |
 | **Content Type** | **Pages only** by default. **All content** means pages and blog posts; it does not include comments or attachment contents. |
 | **Filter by Label** | Optional comma-separated labels. Content can match any listed label. |
-| **Document details** | Optional labels, version, and last-modified metadata tags. |
+| **Metadata tags** | Optional labels, version, and last-modified tags. In the add-source form, these are under **More options**. |
 
 Search manages the schedule and hides item limits. Published/current content is indexed; archived and trashed content is excluded.
 
@@ -164,7 +168,7 @@ Existing organization members see the configured Confluence source and their own
 
 With a central account, Sim applies space access together with the page's restrictions and inherited ancestor restrictions. Group membership is refreshed in the background. With member accounts, each person's provider listing determines the pages available to them. A Sim organization admin does not automatically receive access to every Confluence document.
 
-New content and permission changes require a sync and processing before Search reflects them. Open **Manage** on the source to inspect errors, edit its configuration, or trigger a sync. If your own account needs authorization again, use **Reconnect** on the source row.
+New content and permission changes require a sync and processing before Search reflects them. Open **Settings → Sources → Confluence → Sources → [source]** to inspect **Documents**, edit **Settings**, or review **Sync history**. If your own account needs authorization again, use **Reconnect** in the main Integrations page. Where available, the provider's **Accounts → Request connections** sends account connection requests; these do not invite people to the Sim organization.
 
 ## Troubleshooting
 
@@ -180,7 +184,7 @@ New content and permission changes require a sync and processing before Search r
 
 ### Check access in Confluence
 
-Open a missing page in Confluence with the affected teammate's account. On the page, **Share → General access** shows whether access comes from the space, a parent, or an explicit restriction. A space admin can inspect restricted pages under **Space settings → Content → Restricted**. Check both the teammate and central crawling account when using **Admin or service account**. See Atlassian's [content access guide](https://support.atlassian.com/confluence-cloud/docs/add-or-remove-page-restrictions/).
+Open a missing page in Confluence with the affected teammate's account. On the page, **Share → General access** shows whether access comes from the space, a parent, or an explicit restriction. A space admin can inspect restricted pages under **Space settings → Content → Restricted**. Check both the teammate and central crawling account when using a central source. See Atlassian's [content access guide](https://support.atlassian.com/confluence-cloud/docs/add-or-remove-page-restrictions/).
 
 On Confluence Premium, **Inspect permissions** can show where a user's access is denied across the page, its ancestors, the space, and the product. Check **Can view**, resolve the relevant permission, then run a sync in Sim. See [Atlassian's permission inspection guide](https://support.atlassian.com/confluence-cloud/docs/inspect-a-users-permissions/).
 
@@ -192,6 +196,6 @@ Configure one shared Confluence OAuth integration for your deployment. This powe
 2. Under **Authorization → OAuth 2.0 (3LO)**, add `https://<your-sim-domain>/api/auth/oauth2/callback/confluence` to **Callback URLs**, keep existing callbacks used by the deployment, and save.
 3. Under **Permissions**, add the Confluence API and configure the full `confluence` scope list for your release in [Sim's OAuth configuration](https://github.com/simstudioai/sim/blob/staging/apps/sim/lib/oauth/oauth.ts), including `read:group:confluence`. Also add **User Identity API** with `read:me`. Sim requests `offline_access` for refresh tokens. The service-account read scopes above do not replace the broader shared OAuth scope set.
 4. Enable sharing under **Distribution**. Set `CONFLUENCE_CLIENT_ID` and `CONFLUENCE_CLIENT_SECRET` from the app's **Settings**, verify `NEXT_PUBLIC_APP_URL`, and restart Sim.
-5. Start authorization from Search and select the configured site. Reconnect old accounts after adding scopes so the new permission grant takes effect.
+5. Start authorization from **Integrations** and select the configured site. Reconnect old accounts after adding scopes so the new permission grant takes effect.
 
 A callback mismatch needs a corrected callback URL; a connection that works only for the app owner needs sharing enabled. See Atlassian's [OAuth configuration guide](https://developer.atlassian.com/cloud/confluence/oauth-2-3lo-apps/) and Sim's [deployment reference](/platform/self-hosting/integrations-oauth).

apps/docs/content/docs/search/connect-your-account.mdx

@@ -7,7 +7,7 @@ import { Callout } from 'fumadocs-ui/components/callout'
 import { Step, Steps } from 'fumadocs-ui/components/steps'
 import { Image } from '@/components/ui/image'
 
-Your admin approves the provider and can configure shared source filters. You connect your own account so Sim can establish what you are allowed to search.
+Your admin adds the integration and can configure shared source filters. You connect your own account so Sim can establish what you are allowed to search. Admins connect their own accounts through the same flow.
 
 <Steps>
 <Step>
@@ -21,11 +21,12 @@ Accept your Sim organization invitation or sign in through your organization's S
 
 ## Open Integrations
 
-Open **Integrations**, find the provider or source, and select **Connect account**. Your first connection may ask for required source settings, such as repository or project names. If the provider is missing, ask an organization admin to approve it.
+Open **Integrations** in the main sidebar, find the provider or source, and select **Connect account**. Your first connection may ask for a GitHub repository, Confluence domain and space keys, or Jira domain and project keys. Enter the required fields and select **Connect**. If the provider is missing, ask an organization admin to add it under **Settings → Sources**.
 
-<Image src="/static/search/connect-account.png" alt="Google Drive source row with Connect account" width={1280} height={720} />
+Use **Add source** beside **Add another [provider] source** when you need another supported repository, site, or project scope. Connecting an existing source does not ask you to configure it again.
+
+<Image src="/static/search/connect-account.png" alt="Organization Integrations showing approved providers and Connect account actions" width={738} height={680} />
 
-*Workspace Search is shown here. Organization sources are managed in Settings → Integrations.*
 
 </Step>
 <Step>
@@ -34,7 +35,9 @@ Open **Integrations**, find the provider or source, and select **Connect account
 
 In the new tab, select **Connect** and complete the provider's authorization. Choose the account associated with your verified Sim email. The provider may require your organization's SSO or app approval.
 
-Return to Integrations when the connection completes. If the popup was blocked or closed, allow popups and select **Connect account** again. While authorization is pending, use **Open again**.
+Return to Integrations when the connection completes, or select **Return to Search** to open your organization’s Search page. Your account is saved when authorization completes; there is no separate submit step. If the popup was blocked or closed, allow popups and select **Connect account** again. While authorization is pending, use **Open again**.
+
+<Image src="/static/search/member-enrollment.jpg" alt="Gmail account connection with Connect and Return to Search actions" width={680} height={305} />
 
 </Step>
 <Step>
@@ -59,6 +62,18 @@ For a source configured inside a workspace, join that workspace and use its **Se
 
 Connecting one Google service does not connect all of them. Gmail, Calendar, and Drive each have their own Search connection.
 
+## If you received a connection request
+
+Open the link from your admin and sign in to Sim with the invited email. A provider-specific request opens that provider's connection directly. Verify your Sim email if prompted, then reopen the original link and authorize the account.
+
+An account connection request does not invite you into the Sim organization. You can contribute an account without organization membership, but you need membership and enabled Search access to search the organization's documents. The connection page offers **Return to Search** when you have that access; otherwise it offers **Your connected accounts**.
+
+## Manage your connected accounts
+
+Select **Your accounts** on the main Integrations page to open your personal **Connected accounts** settings. Use **Reconnect** to renew an organization account connection or **Disconnect** to withdraw it. Disconnecting stops that account from being used for organization indexing and workflows, and removes Search access that depends on it.
+
+Admins can select **Manage sources** to open organization setup, then open an integration's **Sources** and **Accounts** tabs. This does not grant the admin access to every document.
+
 ## If you get stuck
 
 | Status | What to do |
@@ -67,7 +82,10 @@ Connecting one Google service does not connect all of them. Gmail, Calendar, and
 | **Reconnect** | Authorize the same source account again. |
 | **Finish connecting in the other tab** | Finish authorization, or use **Open again**. Allow popups for Sim. |
 | No results | Check the source's filters and sync status with your admin. Confirm you can open the document at the source. |
-| Needs admin attention | Ask your admin to inspect **Manage** for the source error. |
+| **Verify email** | Verify your Sim email, then reopen the connection link. |
+| Expired or cancelled authorization | Return to the original connection page and start again. If the invitation itself expired, ask the admin for a new request. |
+| Access revoked | Ask the organization admin to restore your account contribution access before reconnecting. |
+| Needs admin attention | Ask your admin to open **Settings → Sources → [integration] → Sources → [source]** to inspect its error. |
 
 <Callout type="info">
   Your Sim role does not override document access at the source. Connecting a different account or receiving a Search link does not share someone else's mailbox, private calendar, or restricted documents with you.