API Reference

{
  model:             string,   // Model used (may differ if default was applied)
  created_at:        string,   // Timestamp in app timezone (Asia/Kuala_Lumpur)
  response:          string,   // Model text reply
  done:              true,     // Always true (non-streaming)
  context:           number[], // Ollama context tokens; [] for cloud providers
  total_duration:    number,   // Duration in nanoseconds (ms x 1,000,000)
  load_duration:     number,   // Ollama model load ns; 0 for cloud providers
  prompt_eval_count: number,   // Input token count
  eval_count:        number    // Output token count
}

• Cache key is MD5 of { model, prompt, system, context, images, options }. Any field change forces a cache miss.
• Model status is updated to "unavailable" in memory if this call fails (except 429 rate limits — those are transient).
• Response is sent before Redis cache write and MySQL log to keep client-visible latency minimal.

// Primary model succeeded (or use_fallback=false):
{
  model:             string,
  created_at:        string,
  response:          string,
  done:              true,
  context:           number[],
  total_duration:    number,
  load_duration:     number,
  prompt_eval_count: number,
  eval_count:        number
}

// Fallback model was used (primary failed, fallback succeeded):
{
  model:             string,   // originally requested model
  actual_model:      string,   // fallback model that actually ran
  fallback_used:     true,
  fallback_reason:   string,   // primary-model error that triggered fallback
  created_at:        string,
  response:          string,
  done:              true,
  context:           number[],
  total_duration:    number,
  load_duration:     number,
  prompt_eval_count: number,
  eval_count:        number
}

• API key lookup order: Authorization: Bearer <key> | X-Api-Key: <key> | api_key body field.
• use_fallback defaults to true — unlike /api/generate_async which defaults to false (opt-in).
• markModelUnavailable() is called for the primary model on any failure (except 429 / policy errors), even when the fallback succeeds.
• Fallback result is cached under the fallback model's own cache key, not the primary model's key.
• When both primary and fallback fail, api_logs.fallback_reason contains both errors: "primary[<model>]: <err> | fallback[<model>]: <err>".
• api_key_id is written to the api_logs row; use the Admin Console Token Usage tab to view per-key stats.

{
  job_id:   string,  // UUID — use this to poll GET /api/jobs/:id
  status:   "queued",
  model:    string,
  poll_url: string   // convenience URL: /api/jobs/
}

• Callback POST body shape: { job_id, status, model, requested_model, fallback_used, fallback_reason, provider, response, error, prompt_tokens, completion_tokens, duration_ms, completed_at } — model is the actual model that ran; requested_model is non-null only when fallback was used.
• If both primary and fallback fail, job status is "failed". fallback_reason contains both errors: "primary[<model>]: <err> | fallback[<model>]: <err>". error contains only the fallback error message.
• Callback retry is controlled by two proxy_settings rows: callback_retry_max (Y — max retries, default 0 = no retry) and callback_retry_delay_sec (X — seconds between retries, default 30). callback_attempts in async_jobs records how many delivery attempts were made. Retries run detached from the job worker loop so they never block new jobs.
• Worker concurrency: 3 (ASYNC_JOB_CONCURRENCY env). Queue Redis key: async:jobs:queue
• API key lookup: Authorization: Bearer <key> | X-Api-Key: <key> | api_key body field (stripped before processing).
• callback_url must be an absolute URL reachable from inside the Docker container — not localhost or 127.0.0.1 (those resolve to the container itself, not your machine). Examples: http://host.docker.internal:8080/webhook — Windows/Mac Docker Desktop (reaches host machine); http://172.17.0.1:8080/webhook — Linux Docker host (default bridge gateway IP); http://<LAN-IP>:8080/webhook — any machine on the same network; http://other-container:8080/webhook — another service on the same Docker network.

{
  job_id:            string,
  status:            "queued" | "processing" | "completed" | "failed",
  model:             string,          // actual model that ran (fallback model if fallback was used)
  requested_model:   string | null,   // original requested model — non-null only when fallback was used
  fallback_used:     boolean,         // true if fallback inference was triggered
  fallback_reason:   string | null,   // primary error; or "primary[m]: e | fallback[m]: e" when both fail
  provider:          string | null,   // OLLAMA | GEMINI | GROQ | null (while queued)
  created_at:        string,
  started_at:        string | null,
  completed_at:      string | null,
  response:          string | null,   // inference result when status=completed
  error:             string | null,   // error detail when status=failed
  prompt_tokens:     number,
  completion_tokens: number,
  duration_ms:       number,
  callback_url:      string,
  callback_failed:   boolean,         // true if all callback delivery attempts failed
  callback_attempts: number           // how many delivery attempts were made (0 = not yet attempted)
}

• Poll until status is "completed" or "failed" — processing can take seconds to minutes.
• response is null while the job is queued or processing.
• When fallback_used is true: model holds the fallback model ID, requested_model holds the original, fallback_reason holds the primary error.
• callback_attempts counts delivery attempts: 1 = first try succeeded or failed with no retries configured; 2+ = at least one retry was made. 0 means callback delivery has not been attempted yet.
• If callback_failed is true, the job result is still available here — poll GET /api/jobs/:id to retrieve it.

{
  ollama: [{ name: string, id: string, size: number, provider: "ollama", details: object }],
  gemini: [{ name: string, id: string,               provider: "gemini", details: object }],
  groq:   [{ name: string, id: string,               provider: "groq",   details: object }]
}

• To refresh after adding a new Ollama model or provider changes, restart the server.

{
  lastUpdated: string,
  summary: {
    total: number, available: number,
    unavailable: number, pending: number, ignored: number
  },
  providers: {
    ollama: [{ modelId, provider, status, latencyMs, lastChecked, error }],
    gemini: [...],
    groq:   [...]
  }
}
// status: "available" | "unavailable" | "pending" | "ignored"

• "pending" — registered but not yet probed (only possible in normal/once mode before first cycle completes).
• "ignored" — present in .providermodelignore.
• Probe uses the prompt defined by MODEL_HEALTH_CHECK_PROBE with MODEL_HEALTH_CHECK_DELAY_MS between each model probe to avoid provider rate limits.
• When MODEL_HEALTH_CHECK_INTERVAL_SEC=-1 (bypass mode): all models start as "available", no probes are ever run. Status flips to "unavailable" only when POST /api/generate fails (excluding 429s).
• When MODEL_HEALTH_CHECK_INTERVAL_SEC=0: one startup probe cycle only, no periodic re-checks.
• When MODEL_HEALTH_CHECK_INTERVAL_SEC>0: startup probe + automatic re-check every N seconds.

// Default grouped response:
{
  lastUpdated: string,
  providers: {
    ollama: [{
      modelId, provider,
      rpm,  tpm,  rpd,
      avgLatencyMs,     // rolling 60-second avg (successful only)
      avgLatencyHourMs, // rolling 1-hour avg   (successful only)
      totalRequests, totalTokens, errorCount, lastUsed
    }],
    gemini: [...], groq: [...]
  }
}
// With ?flat=1  → { lastUpdated, models: [...same fields...] }
// With ?provider=X&modelId=Y → { lastUpdated, model: {...} }

• rpm / tpm — rolling 60-second window.
• avgLatencyMs — rolling 60-second average, successful requests only.
• avgLatencyHourMs — rolling 1-hour average, successful requests only.
• rpd — requests today, resets at midnight UTC+8.
• totalRequests / totalTokens / errorCount — lifetime since last restart.

{ success: true, processed: number }  // rows rolled up this call

• Called by the Admin Console "Sync Now" button in the Token Usage tab.
• No request body is required or read.

Content-Type: text/html — the gateway diagnostic page

Content-Type: text/html — this page