REST API & Custom Server

SpyWeb runs a built-in web server on 127.0.0.1:7979 serving the dashboard UI, a built-in JSON API, and any custom endpoints you define.

1. Built-in REST API

SpyWeb provides default endpoints for interacting with your jobs and records.

Endpoint	Description
GET /	HTML records viewer (dashboard)
GET /api/records?job_id=<id>	JSON records for a job
GET /api/records?job_id=<id>&limit=50&after=<timestamp>	Paginated records
GET /api/jobs	List all configured jobs
/api/v/{name}	Custom endpoints — requires auth (defined in server/init.lua)
/api/public/{name}	Custom endpoints — always public, no auth required (defined in server/init.lua)

Authentication

If SPYWEB_API_KEY is set, all /api/* endpoints require the X-SpyWeb-Key header — except /api/public/* routes, which are always accessible without a key:

curl -H "X-SpyWeb-Key: your_secret_key" http://127.0.0.1:7979/api/jobs
curl http://127.0.0.1:7979/api/public/health

Bring Your Own UI

The default dashboard is in ui/index.html. SpyWeb serves this file dynamically from disk. Build your own with Vue, React, Svelte, or vanilla JS, as long as your build outputs index.html into ui/, SpyWeb serves it instantly with no restart.

---

2. Webhook Payloads

When the engine finishes a cycle and has new items, it can push them to a webhook. The default payload looks like this:

{
  "job_name": "Product Tracker",
  "item_count": 2,
  "items": [
    {
      "title": "Item A",
      "price": "$49.99",
      "link": "https://example.com/a",
      "keywords": ["sale"]
    }
  ]
}

Use before_webhook(payload, ctx) to reshape this JSON before it is sent, useful for Discord embeds, Slack blocks, or Pushover.

---

3. Programmable API Server

SpyWeb lets you define custom HTTP endpoints via Lua. Create a server/init.lua file in your project root.

Quick Start

Create server/init.lua:

function get:hello()
    return { status = 200, body = "Hello from SpyWeb!" }
end

function post:echo()
    return { status = 200, body = self.body, headers = { ["Content-Type"] = "text/plain" } }
end

Start SpyWeb and hit your endpoints:

./spyweb start
curl http://127.0.0.1:7979/api/v/hello
curl -X POST -d "test body" http://127.0.0.1:7979/api/v/echo

How It Works

Each HTTP request to /api/v/{name} (or /api/public/{name} for unauthenticated endpoints):

1. Reads server/init.lua from disk (with hot-reload)
2. Creates a fresh Lua VM for the request
3. Loads the script, defining route handlers
4. For /api/v/{name}, looks up get["name"] → post["name"] → all["name"]. For /api/public/{name}, looks up public.get["name"] → public.post["name"] → public.all["name"]
5. Calls the handler with self carrying the request context
6. Returns the value as an HTTP response
7. Drops the VM, so no shared state between requests

Route Registration

Private endpoints (require X-SpyWeb-Key if SPYWEB_API_KEY is set):

function get:users()
    -- handles GET /api/v/users
end

function post:users()
    -- handles POST /api/v/users
end

function all:ping()
    -- handles any method on /api/v/ping
end

Public endpoints (always accessible without a key):

function public.get:status()
    -- handles GET /api/public/status
end

function public.post:webhook()
    -- handles POST /api/public/webhook
end

function public.all:catchall()
    -- handles any method on /api/public/catchall
end

The method tables get, post, put, patch, delete, all, public.get, public.post, public.put, public.patch, public.delete, and public.all are pre-injected.

Functions defined outside method tables are private helpers, never exposed as endpoints:

function validate_input(data)
    return data and data.name
end

function get:users()
    local users = fetch_users()
    return { status = 200, body = users }
end

URL Structure

Auth-protected endpoints:

/api/v/{name}/{path_args...}

Public endpoints (no key required):

/api/public/{name}/{path_args...}

Each prefix resolves from separate handler tables — a handler defined under get.* is only reachable via /api/v/, and a handler defined under public.get.* is only reachable via /api/public/.

URL	Handler	self.path_args
/api/v/users	get.users	{}
/api/v/users/123	get.users	{"123"}
/api/v/users/123/profile	get.users	{"123", "profile"}
/api/public/status	public.get.status	{}
/api/public/status/db	public.get.status	{"db"}

Request Context (self)

Field	Type	Description
self.body	string or nil	Raw request body (POST, PUT, PATCH); silently truncated at 10MB
self.method	string	HTTP method
self.path	string	Full request path
self.path_args	table	Array of trailing path segments
self.query	table	URL query parameters
self.headers	table	Request headers (lowercase keys)
self.client_ip	string	Client IP address

Response Contract

Table (full control)

return {
    status = 200,           -- HTTP status (clamped to 100-599)
    body = "Hello!",        -- string, table (auto-JSON), number, boolean, or binary
    headers = {             -- optional extra headers
        ["X-Custom"] = "value"
    }
}

Body types:

Lua Type	Response
string	Raw body as-is
table	Auto-encoded as JSON with Content-Type: application/json
number	Converted to string
boolean	Converted to "true" or "false"
nil	Empty body

String (quick response)

return "Hello, World!"
-- Equivalent to: { status = 200, body = "Hello, World!" }

Available Globals

The server VM has the same globals as scraper hooks (HTTP client, storage, database, file I/O, and utilities.

Defer (Async Cleanup)

Unlike defer() in pipeline hooks, the server’s defer() supports async bindings.

Deferred functions run in reverse order (last registered runs first). Each function runs sequentially in the same Lua VM, so they share globals and can safely access self via closure capture. Errors are logged but don’t affect other deferred functions.

function get:resource()
    defer(function()
        sleep(1000)
        log("request finished for " .. self.path)
        http_post("https://hooks.example.com/callback", json_encode({ status = "done" }), {
            ["Content-Type"] = "application/json"
        })
    end)
    return { status = 200, body = { message = "ok" } }
end

Examples

Robust JSON API

function post:items()
    local data, err = json_decode(self.body or "")
    if not data or type(data) ~= "table" then
        return { status = 400, body = { error = "Invalid JSON payload: " .. tostring(err) } }
    end
    if not data.name then
        return { status = 400, body = { error = "Missing 'name' field" } }
    end
    return { status = 201, body = { message = "Created", item = data } }
end

Serve Scraped Data

function get:latest_prices()
    local prices = global_store_get("latest_prices")
    if not prices then
        return { status = 404, body = "No data yet" }
    end
    return { status = 200, body = prices, headers = { ["Content-Type"] = "application/json" } }
end

Proxy to External API

function get:weather()
    local city = self.query.city or "London"
    local res, err = http_get("https://api.weather.com/v1/" .. city)
    if not res then
        return { status = 502, body = { error = err } }
    end
    return { status = res.status, body = res.body, headers = { ["Content-Type"] = "application/json" } }
end

function post:webhook()
    local payload = json_decode(self.body)
    http_post("https://hooks.example.com/relay", json_encode(payload), {
        ["Content-Type"] = "application/json"
    })
    return { status = 200, body = "Relayed" }
end

Health Check

-- Private health check (/api/v/health)
function all:health()
    return { status = 200, body = "ok" }
end

-- Public health check (/api/public/health, no key required)
function public.all:health()
    return { status = 200, body = "ok" }
end

Error Handling

If a handler throws a Lua error:

1. The error is logged to the terminal
2. The error is appended to server/error.log with a timestamp
3. The client receives {"error": "Internal Server Error"} with status 500

If server/init.lua has a syntax error, all requests fail with 500 until the file is fixed.

Timeout

With Luau (the default), each request has a 30-second timeout. If a handler runs longer, it is forcefully terminated and the client receives a 500 error with "Script execution timed out".

The timeout is enforced via Luau’s bytecode interrupt hook, so no threads are leaked.

Note: Timeout is only available with Luau. Lua 5.4 builds do not have timeout protection.

Serving Binary Data

Use fs_read_binary() to serve images, PDFs, or other assets from server/ or shared/:

function get:logo()
    local data = fs_read_binary("logo.png")
    if not data then
        return { status = 404, body = "Not Found" }
    end
    return {
        status = 200,
        body = data,
        headers = { ["Content-Type"] = "image/png" }
    }
end

Available Lua Globals

The server VM has access to the same globals as scraper hooks:

Category	Functions
File I/O	fs_read(filename), fs_read_binary(filename), fs_append(filename, content), fs_overwrite(filename, content), log(message)
HTTP Client	http_get(url, [headers]), http_post(url, body, [headers]), http_request({...}), http_multipart(url, fields, [headers])
Storage	global_store_set(key, value), global_store_get(key), global_store_incr(key, default, delta), global_store_delete(key)
Database (SQLite)	db_query(sql, [params]), db_exec(sql, [params])
Utilities	json_encode(value), json_decode(string), env_get(key), defer(fn), sleep(ms), notify(title, body, [timeout])

File writes go to server/ by default. Use shared/ prefix (e.g. fs_overwrite("shared/data.json", ...)) to write to the shared folder. Reads scan server/ first, then fall back to shared/.

Authentication

The API server reuses SpyWeb’s existing authentication. If SPYWEB_API_KEY is set, all /api/* routes require the X-SpyWeb-Key header — except /api/public/* routes, which are always accessible without a key:

# Requires key
curl -H "X-SpyWeb-Key: your_key" http://127.0.0.1:7979/api/v/hello

# Always public, no key needed
curl http://127.0.0.1:7979/api/public/health

Without the header on a protected route, the server returns 401 Unauthorized.

Public endpoints are defined under the public.* namespace in init.lua — they are never reachable via /api/v/:

-- Private (requires key)
function get:status()
    return { status = 200, body = "ok" }
end

-- Public (no key required)
function public.get:health()
    return { status = 200, body = "ok" }
end

Environment Variables

Variable	Default	Description
SPYWEB_PORT	7979	Server port
SPYWEB_API_KEY	none	API authentication key
SPYWEB_DISABLE_SERVER	none	Set to any value to skip starting the web server
SPYWEB_LOG	info	Log verbosity: info (all), warn (warnings+errors), error (errors only)

The server can also be configured via CLI flags on spyweb start:

Flag	Short	Description
--port <PORT>		Customize the server port (default 7979)
--no-server	-n	Start without the web server
--quiet	-q	Suppress info logs (use -qq for errors only)

Data Directory

The server’s fs_* operations are scoped to the server/ directory:

• Directoryproject/

  • Directoryserver/

    • init.lua Route definitions
    • error.log Error logs (auto-created)
    • Directorydata/ Persistent data

      • …
  • Directoryshared/ Cross-VM shared data

    • …
  • jobs.toml
  • data Scraper database
  • Directoryui/ Dashboard files

    • …

Troubleshooting

All endpoints return 500: Check server/error.log for syntax or runtime errors.

Endpoints return 404: Verify the handler is in the correct method table for the URL prefix — /api/v/{name} uses get.{name}, post.{name}, etc.; /api/public/{name} uses public.get.{name}, public.post.{name}, etc. The all (or public.all) table is checked last.

Body is empty or truncated: Request body is capped at 10MB.

Timeout errors: Luau-only 30-second timeout per request. Check for infinite loops in handler code. Move heavy computation to background tasks using defer.