Webhooks

Instead of polling a job until it finishes, register an endpoint and NexqData sends it a POST when the job is done. Add endpoints in the dashboard under Profile → Webhooks, or with the API below.

Endpoints belong to your user, like API keys. You get events for the jobs, pipeline runs, group runs, and schedules you started or created, including runs your schedules start.

Events

Group and pipeline runs start one job per agent, so each of those jobs also sends its own job.* event. If you only care about the group as a whole, subscribe to group_run.completed and not the job.* events.

What you receive

Each event is a JSON POST:

{
  "id": "evt_3fa85f6457174562b3fc2c963f66afa6",
  "type": "job.succeeded",
  "createdAt": "2026-09-23T09:52:00Z",
  "data": {
    "agentJobHistoryId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "scraperAgentId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "agentName": "Court records · TX",
    "agentExternalId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "status": 3,
    "statusName": "Succeeded",
    "isTerminal": true,
    "triggerSource": "Schedule",
    "scheduleId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "searchGroupRunId": null,
    "pipelineRunId": null,
    "startTime": "2026-09-23T09:40:12+00:00",
    "endTime": "2026-09-23T09:51:47+00:00",
    "completedRunCount": 2,
    "totalRunCount": 2,
    "totalDataCount": 1840,
    "totalErrors": 0,
    "errorMessage": null
  }
}

A job event's data uses the same field names as a job from GET /api/scraper-agents/jobs. Use agentJobHistoryId to download its data.

A pipeline event's data has pipelineRunId, pipelineId, pipelineName, status (Completed or Failed), stepCount, startedAt, finishedAt, and steps. Each step lists its stepIndex, agentName, agentJobHistoryId, status, and error.

{
  "id": "evt_3fa85f6457174562b3fc2c963f66afa6",
  "type": "group_run.completed",
  "createdAt": "2026-09-23T10:14:03Z",
  "data": {
    "searchGroupRunId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "searchGroupId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "groupName": "County court records · TX",
    "status": "PartiallySucceeded",
    "memberCount": 3,
    "succeededCount": 1,
    "failedCount": 1,
    "notStartedCount": 1,
    "totalDataCount": 1840,
    "executedAt": "2026-09-23T10:02:11Z",
    "completedAt": "2026-09-23T10:14:03Z",
    "jobs": [
      { "agentJobHistoryId": "3fa85f64-…", "agentName": "Harris County", "status": 3, "statusName": "Succeeded", "totalDataCount": 1840, "errorMessage": null },
      { "agentJobHistoryId": "3fa85f64-…", "agentName": "Dallas County", "status": 5, "statusName": "Failed", "totalDataCount": 0, "errorMessage": "The court portal didn't respond." }
    ],
    "notStarted": [
      { "scraperAgentId": "3fa85f64-…", "agentName": "Travis County", "error": "The agent is at its session limit." }
    ]
  }
}

A group run's status is Succeeded when every member started and succeeded, Failed when none succeeded, and PartiallySucceeded otherwise. jobs has one entry per member that started. Download each one's data with its agentJobHistoryId. notStarted lists the members that never started, when that was recorded; notStartedCount always has the number.

A schedule event's data has scheduleId, targetType (Agent, Pipeline, or Group), targetName, the target's id (scraperAgentId and agentExternalId, pipelineId, or searchGroupId), reason, errorMessage, dueAt, firedAt, and nextRunTime (null if the schedule won't run again). reason is one of:

The request headers:

Respond with a 2xx

Any 2xx response counts as delivered. Anything else, or no response within 10 seconds, is a failed attempt. We don't follow redirects.

Respond first and do the work afterwards: put the event on a queue, return 200, then download the data.

Verify the signature

Anyone can send a request to a public URL, so check the signature before you trust a request. When you add an endpoint you get a signing secret starting with whsec_. It's shown once.

The v1 value is the hex HMAC-SHA256 of <t>.<raw body>, keyed with your secret. To check it:

1. Take t and v1 from NexqData-Signature.
2. Compute the HMAC over t, a ., and the raw request body, exactly as received. Parsing the JSON and re-serializing it changes the bytes.
3. Compare with v1 in constant time.
4. Reject the request if t is more than 5 minutes from now. That stops a captured request from being replayed.

import crypto from "node:crypto"

export function verifyNexqData(secret, header, rawBody, toleranceSeconds = 300) {
  const parts = Object.fromEntries(header.split(",").map((part) => part.split("=", 2)))
  const t = Number(parts.t)
  if (!parts.v1 || !Number.isInteger(t)) return false
  if (Math.abs(Date.now() / 1000 - t) > toleranceSeconds) return false

  const expected = crypto.createHmac("sha256", secret).update(`${parts.t}.${rawBody}`).digest()
  const given = Buffer.from(parts.v1, "hex")
  return given.length === expected.length && crypto.timingSafeEqual(given, expected)
}

// Express: read the body raw, not through express.json().
app.post("/webhooks/nexqdata", express.raw({ type: "application/json" }), (req, res) => {
  const ok = verifyNexqData(
    process.env.NEXQDATA_WEBHOOK_SECRET,
    req.get("NexqData-Signature") ?? "",
    req.body.toString("utf8")
  )
  if (!ok) return res.sendStatus(400)
  res.sendStatus(200)
  // …then process JSON.parse(req.body) asynchronously
})

Retries and duplicates

A failed delivery is retried 1 minute, 5 minutes, 30 minutes, 2 hours, 6 hours, and 12 hours after the attempt before it: 7 attempts over about 21 hours. Every attempt sends the same body with the same event id.

The same event can reach you more than once. Your response can be lost after you've handled it, you can redeliver it yourself, or a restarted job can finish again (that sends a new event with a new id). Record each id you process and skip ones you've already seen.

Events aren't guaranteed to arrive in order. Use endTime or finishedAt if order matters.

If every delivery to an endpoint fails for about a day, the endpoint is disabled and you get a notification in the dashboard. Fix the receiver, enable the endpoint again, then redeliver anything you missed from its delivery log.

Manage endpoints

Your endpoints. The signing secret is never included, only its last four characters as secretHint.

{
  "url": "https://example.com/webhooks/nexqdata",
  "description": "Production data sync",
  "events": ["job.succeeded", "job.failed"]
}

Returns the endpoint and its secret. Store the secret now; it isn't shown again. The URL must use https:// and resolve to a public address. You can have up to 5 endpoints.

Change any of url, description, events, and isActive, and leave out the ones you're not changing. Setting isActive to true enables an endpoint again after it was disabled.

Issues a new secret, returned once. The old secret stops working immediately.

Sends a ping event right away and returns succeeded, statusCode, durationMs, and error. It counts toward the same rate limit as starting jobs.

The delivery log, newest first, with each payload. Takes page and pageSize (up to 100). Deliveries are kept for 30 days.

Sends a delivery again within a few seconds, with a fresh set of retries. The endpoint must be enabled.

Deletes the endpoint and its delivery log.