Troubleshooting background work

Background work is everything xMatix runs on your behalf without you waiting on screen: scheduled jobs, imports and exports, integration sync, dataset refreshes, subscription emails, outbound messages, workflows and document actions that run in the background. When something "didn't happen", the answer is almost never on the record that didn't change. It is in the history of the mechanism that was supposed to change it. This page tells you which history to open, and how to read it, retry from it and schedule from it. Most of it is about Setup → Platform → Monitoring → Jobs, the one screen that lists runs from most of these mechanisms.

Overview

Every kind of background work keeps its own history, and every history entry has a status and an error message. Use this table to find the right one.

How xMatix handles it

One run ledger, several kinds of run. The Runs tab on the Jobs screen reads a single ledger of runs. Its Type column says what kind of run each row is:

• Tenant job — a job from the job catalog, such as Provision GST returns, Settle claim or Schedule Run automation script. These are the only runs you can cancel from this screen.
• Federated run — a summary row for work that another part of the product owns: integration flow runs, AI agent executions, automation scripts and subscription families. They appear here so you can see everything in one list, but you retry them where they belong (the integration app, Automation or AI Studio).
• Data job types, such as Import job — imports, exports and document actions. Import History is still the better place to work with imports.

Where a run comes from. The Trigger column shows what started the run: Schedule (a row on the Schedules tab), Api (Run now, a Retry, or an external call), Code (the product itself), BusinessRule, Workflow, Script or Event. A scheduled run's Username is "Schedule:" followed by the schedule's name. A run you start shows your name.

Statuses. A run moves through Queued (or Scheduled when it was asked to start later) → Processing → one of Completed, PartiallyCompleted, Failed or Cancelled. A few more statuses explain how a run ended or was handed on:

• RetryQueued (the Retried filter) — someone retried the run, and a new run took its place.
• DeadLettered — the run ended after it could not be delivered or completed.
• PublishFailed — the run never reached the queue.

Automatic retries. Each job type has a number of attempts. When an attempt fails with an error that may clear up by itself, the job is tried again after a delay. By default the delay starts at 30 seconds and doubles with each attempt, up to an hour. No job gets more than five attempts. The Attempts column shows where the run is, for example 2/3. An error that would only happen again is not retried. The run goes straight to Failed.

Runs that stop reporting are closed for you. A running job reports that it is alive every 30 seconds. If a Processing run has not reported for 15 minutes, or has run far longer than any job may, it is marked Failed with a message that starts "Orphaned:". If a run is still Queued 30 minutes after it was created and no worker ever picked it up, it is marked Failed with a message that starts "Never claimed:".

Duplicates are collapsed. If a schedule, rule, workflow, script or the product itself asks for a job while an identical job (same job type, same payload) is still waiting to be picked up, xMatix does not queue a second copy. Runs you start by hand are always queued.

Licensing. A job that belongs to a product your organization does not hold (for example Beat intelligence, which needs Field Sales) is refused when you start it. Its schedule does not fire. It starts firing again once the product is licensed and switched on.

Prerequisites

• View jobs to open the Jobs screen and read runs and schedules.
• Manage jobs to use Retry, Cancel, Run now, New schedule, Edit and Delete. Without it, those buttons are hidden.
• Some job types act with the organization's system identity and need one more permission to start or schedule. If you lack it, the screen tells you which permission is missing.

See Permissions for where these are granted.

Procedure

Step 1 — Open the Jobs screen

Go to Setup → Platform → Monitoring → Jobs. The screen has two tabs: Runs (the default) and Schedules. The page header reads "Monitor background job runs and manage recurring job schedules."

Step 2 — Narrow the runs to the one you need

The second row of the Runs table holds three filters:

1. The status strip: All, Queued, Scheduled, Processing, Completed, Partial, Failed, Cancelled and Retried.
2. The job picker: All jobs, or one job by name.
3. The type picker: All types, Tenant job, Federated run, plus any other type in the loaded rows.

These three filters ask the server. The Search box in the first row searches only the rows already loaded, across type, entity, action, job, trigger, username, status, result, error and Job ID. Use Refresh to load the latest runs.

If the table says "No job runs match these filters", set the status to All and both pickers back to All jobs and All types.

Step 3 — Open the run and read the error

Select any row to open the Job detail drawer. It shows the run's Status, Type, Job type, Trigger, Entity, Action, Username, Retry count, Attempt, the timestamps (Created, Started, Heartbeat, Updated, Completed) and Duration, then four sections when they have content:

• Result — what the job reports it did, for example how many records it settled or created.
• Error — the message to act on.
• Stack trace — technical detail for support.
• Payload — the input the job ran with.

A run with a small arrow at the start of its row has been retried. Expand it to see the retry runs underneath. You can open each of them the same way.

Step 4 — Retry a failed run

Fix the cause first (the Error section usually names it). Then select Retry in the run's row and confirm "Retry this job?".

• Retry is available on original runs that are Failed, PublishFailed or Cancelled. It is greyed out on federated runs and on retry runs. To retry those, retry the original run, or use the owning screen.
• xMatix creates a new run with the same job type and payload. You see "Job re-queued" with the new Job ID. The original run changes to RetryQueued and its Retry Count goes up by one. The new run appears under it when you expand the row.
• For an import, prefer Retry in Import History. It replays only the rows that failed.

Step 5 — Cancel a run that should not continue

Select Cancel in the run's row and confirm "Cancel this job?" with Cancel job. You can cancel only Tenant job runs that are Queued, Scheduled, Processing or RetryQueued.

• A run that has not started yet becomes Cancelled at once ("Job cancelled").
• A run that is Processing gets "Cancellation requested". It stops at its next checkpoint, which is usually within about 30 seconds. Until then the drawer shows Cancellation requested.

Step 6 — Create a recurring schedule

1. Open the Schedules tab and select New schedule in the page header.
2. Enter a Name (required) and, if you like, a Description.
3. Pick the Job type. Only jobs that can be started by hand are listed, grouped by category.
4. Choose the Frequency (Minute, Hourly, Daily, Weekly or Monthly) and the controls that appear for it. Enter the Time in UTC. This dialog does not convert from your local time.
5. Fill the payload:
   • For Schedule Run automation script, choose the Script and, optionally, Arguments (JSON, optional).
   • For any other job, enter the Payload JSON if that job needs one.
6. Leave Active on, and select Create.

You see "Schedule created". The new row shows the schedule in words, Active, and the Next due time. When the schedule fires, its run appears on the Runs tab with trigger Schedule.

Step 7 — Run, change, pause or delete a schedule

• Run now starts the job immediately, independently of the timetable ("Job enqueued" with the Job ID). It does not change Last fired or Next due.
• Edit opens the same dialog. Turn Active off to pause the schedule.
• Delete removes the schedule after confirmation. Runs it already started stay in the Runs tab.

Fields

Runs table

Schedule dialog

The Schedules list also shows Last fired (the timetable slot it last fired for), Next due (blank when the schedule is inactive) and Created.

Business rules

• Schedules are checked every few minutes, in UTC. A schedule is due when its next slot after Last fired has passed. It runs within a few minutes of that slot. A brand-new schedule does not replay slots from before it was created.
• Re-activating a schedule does not reset it. The scheduler works forward from Last fired, so slots that passed while the schedule was off can run, one per check, until the schedule has caught up. If you do not want those catch-up runs, create a new schedule instead of turning the old one back on.
• Saving checks the timetable and the job. A schedule whose timetable never fires, or whose job type is unknown or cannot be started by hand, is refused. For an automation-script schedule, the script must still exist. A disabled script is accepted, but its runs skip.
• Some schedules ship with xMatix and come back if you delete them. If your organization holds the product, xMatix creates these active schedules for you:
  • Verify inventory cost (nightly) — every day at 21:45 UTC (03:15 IST). Finance.
  • Beat intelligence (nightly) — every day at 04:00 UTC (09:30 IST). Field Sales.
  • Assignment history retention (nightly) — every day at 02:30 UTC (08:00 IST). Deletes assignment run history older than 90 days.

  Once a day xMatix checks for these, and it re-creates any that has no schedule row. To stop one, edit it and turn Active off. Do not delete it.

  • Retry makes a new run. The original keeps its error for the record and changes to RetryQueued. A retry run cannot itself be retried. Retry the original again.
  • Cancel only applies to tenant jobs. Data jobs, document actions and federated runs are stopped or retried from their own screens.

  Example

  A finance administrator notices that no overdue interest was accrued last night.

  1. They open Setup → Platform → Monitoring → Jobs, choose Failed in the status strip and pick Accrue interest on overdue payments in the job picker.
  2. One run from 20:30 UTC is listed with trigger Schedule. Its Attempts column reads 1/3.
  3. They open it. The Error section names a configuration problem that would fail again on a retry. That is why the run went straight to Failed without using its other two attempts.
  4. They correct the setting the error points to, then select Retry on the run. "Job re-queued" appears with a new Job ID. The original changes to RetryQueued.
  5. They switch the status strip to All, expand the original row and watch the new run go from Queued to Processing to Completed. Its Result shows what was accrued.
  6. On the Schedules tab, the schedule still shows Active and the next nightly slot under Next due, so tonight's run will go ahead as usual.

  Training

  Practice exercise

  In a sandbox, with Manage jobs:

  1. On Schedules, create a schedule named Practice – assignment history for Assignment history retention, Daily at a time a few minutes from now in UTC, with Active on.
  2. Check that Next due shows the time you chose.
  3. Select Run now on it, then open the Runs tab and find the run (trigger Api, with your name as Username).
  4. Wait for the scheduled slot to pass and find the second run (trigger Schedule, Username "Schedule: Practice – assignment history").
  5. Edit the schedule, turn Active off, and check that Next due is now blank. Then delete it.

  Expected result: two Completed runs of Assignment history retention, one per trigger, and no practice schedule left. The shipped Assignment history retention (nightly) schedule is untouched.

  Quick reference

  • One screen for most background runs: Setup → Platform → Monitoring → Jobs.
  • Filter by status, job and type first. Search only searches what is loaded.
  • Open a row to read Error, Result and Payload.
  • Retry creates a new run. Fix the cause first.
  • Cancel stops a running tenant job at its next checkpoint.
  • Times on the Schedules dialog are UTC.
  • Pause a schedule with Active off. Deleting a shipped schedule brings it back.
  • "Orphaned:" and "Never claimed:" errors mean the platform closed a run that stalled. Retry it.
  • Imports: retry from Import History, which replays only the failed rows.

  Permissions

  The Jobs screen is controlled by two capabilities in the Platform Diagnostics feature. Grant them on a security profile's Entitlements tab. See Security profiles.

  Capability	What it allows	Without it
  View jobs	Open Setup → Platform → Monitoring → Jobs, read runs, run details and schedules	The Jobs menu item is hidden
  Manage jobs	Retry, Cancel, Run now, New schedule, Edit, Delete	The buttons are hidden. The empty Schedules list says schedules "appear here once an administrator creates one"
  View message log	Open Message Log	The menu item is hidden

  Some job types also need an extra capability to start or schedule by hand. Without it, saving or running is refused with a message that names the missing permission. The other history screens (Import, Export, Sync Monitor, Data Studio, workflows) have their own permissions, described on their pages.

  Configuration

  What you can configure	Where	What it affects
  Recurring job timetables	Setup → Platform → Monitoring → Jobs → Schedules	When each job runs and with what payload
  Pausing a shipped nightly job	Same place: edit the schedule and turn Active off	Stops the job without xMatix re-creating the schedule
  Product licences and module switches	Your licences and Setup → Feature Hub	A job of a product you do not hold, or whose module is switched off, does not run or fire
  Dataset, model and archive schedules	Data Studio and Setup → Platform → Data → Archive rules	See Refresh scheduling and monitoring
  Integration sync schedules	Setup → Platform → Sync profiles and the integration app	See Sync health, conflicts and failed runs
  Which actions run in the background	The entity action's Execution setting	See Entity behavior
  Report and dashboard subscriptions	The Subscribe dialog on the report or dashboard	See Subscriptions

  Common problems

  A scheduled job ran at the wrong hour

  The Time in a job schedule is UTC. A run that is off by exactly your UTC offset (5 hours 30 minutes in India) was entered as local time. Edit the schedule and enter the UTC time. The Subscribe dialog for reports and dashboards is different: it takes your local time zone, which is shown under its Time field.

  A schedule never runs

  Check, in order:

  • Is it Active? An inactive schedule shows no Next due.
  • Does your organization hold the job's product, with its module switched on? Schedules of unlicensed jobs do not fire.
  • Does Next due show the slot you expect, in UTC?
  • For a Data Studio, archive or sync schedule: a blank schedule means manual only. An expression saved before the screens checked expressions, or written by a package or the API, is skipped silently if it is invalid. Re-save the schedule on its screen to have it checked.

  "Save" on a schedule says the cron expression is invalid or never fires

  The timetable has no future date, for example 30 February. Pick another day or frequency.

  "Job type '…' cannot be scheduled" or the job is missing from the picker

  That job is internal and cannot be started by hand. It runs only from the product, rules, workflows or scripts.

  "… was not run" when I select Run now

  The job's product is not licensed for your organization, or the organization cannot accept new work at the moment. The message gives the reason. The same refusal can appear on Retry.

  A run failed with "Orphaned: …"

  The worker running the job stopped reporting for 15 minutes or ran past the time limit, typically because it restarted. Nothing is still running. Check the Result and your data for what was already done, then Retry.

  A run failed with "Never claimed: …"

  No worker picked the run up within 30 minutes. Retry it. If new runs keep ending this way, contact support with a Job ID.

  Retry or Cancel is greyed out

  Retry needs an original run that is Failed, PublishFailed or Cancelled. Retry runs and federated runs cannot be retried here. Cancel works only on tenant jobs that have not finished. If neither button is shown at all, you need Manage jobs.

  I deleted a nightly schedule and it came back

  Verify inventory cost (nightly), Beat intelligence (nightly) and Assignment history retention (nightly) are re-created once a day when no schedule for them exists. Edit the schedule and turn Active off instead.

  A document is locked with "processing" and never unlocks

  A background document action is still running. Open View progress on the record. If the job stopped reporting for 15 minutes, the next attempt to change the document marks the job Failed with an "Orphaned:" message and unlocks the document. Run the action again from the document.

  An import is stuck at Processing

  An Import Now run depends on its browser tab. If the tab closed, nothing finishes the history row. Rows already written stay. Run large files with Run Async instead. See Import data.

  Common questions

  Is a Partial status a failure I need to re-run in full?

  No. PartiallyCompleted means the good part committed: an import has written its good rows, and a sync cycle has moved on where it succeeded. Re-running everything can apply the successful part twice. Work on the failed subset. Import Retry replays only the rows that failed, and the Download Errors and fix-file options give you just those rows. For a job, read its Result to see what it did before retrying.

  Do failed jobs retry themselves?

  Some do, inside the same run. Each job type has a number of attempts, shown in the Attempts column. A failure that may clear up by itself is retried after a growing delay, and one that would only happen again goes straight to Failed. Document actions that run in the background are never retried automatically: use Retry on the document. Integration runs follow their own retry rules, and rejected credentials are never retried until someone fixes the credential.

  What is the difference between Retry, Replay and Resend?

  Each history retries on its own terms. A Jobs Retry queues a new run with the same payload. An import Retry replays only the failed rows of the stored file. An integration Replay reprocesses an archived payload through the flow's current version, and only for finished runs. A Message Log Resend re-queues one failed or held message. A faulted workflow is fixed, republished and triggered again. Read which one you are using before you click it, because the wrong one can duplicate work.

  Does Run now count as the schedule's run?

  No. Run now starts an extra run immediately. The schedule's Last fired and Next due do not change, so the next timetabled run still happens. If the job should not run twice, wait for the slot or pause the schedule.

  Why does the Duration look longer than the job took?

  Duration runs from Submitted to Completed, so it includes time spent waiting in the queue and the delays between automatic attempts. Compare Started with Completed in the run's detail to see how long the last attempt took.

  Related topics

  • Troubleshooting xMatix
  • Import data
  • Export data
  • Sync health, conflicts and failed runs
  • Troubleshooting integrations
  • Refresh scheduling and monitoring
  • Messaging channels and the message log
  • Subscribe to reports and dashboards by email
  • Entity behavior
  • Monitoring mobile sync failures