REST API & Custom Server

SpyWeb runs a built-in web server on 127.0.0.1:7979 serving the dashboard UI, a built-in JSON API, and any custom endpoints you define.

1. Built-in REST API

SpyWeb provides default endpoints for interacting with your jobs and records.

Endpoint	Description
GET /	HTML records viewer (dashboard)
GET /api/records?job_id=<id>	JSON records for a job
GET /api/records?job_id=<id>&limit=50&after=<timestamp>	Paginated records
GET /api/jobs	List all configured jobs
/api/v/{name}	Custom endpoints defined in server/init.lua

Authentication

If SPYWEB_API_KEY is set, all /api/* endpoints require the X-SpyWeb-Key header:

curl -H "X-SpyWeb-Key: your_secret_key" http://127.0.0.1:7979/api/jobs

Bring Your Own UI

The default dashboard is in ui/index.html. SpyWeb serves this file dynamically from disk. Build your own with Vue, React, Svelte, or vanilla JS, as long as your build outputs index.html into ui/, SpyWeb serves it instantly with no restart.

---

2. Webhook Payloads

When the engine finishes a cycle and has new items, it can push them to a webhook. The default payload looks like this:

{
  "job_name": "Product Tracker",
  "item_count": 2,
  "items": [
    {
      "title": "Item A",
      "price": "$49.99",
      "link": "https://example.com/a",
      "keywords": ["sale"]
    }
  ]
}

Use before_webhook(payload, ctx) to reshape this JSON before it is sent, useful for Discord embeds, Slack blocks, or Pushover.

---

3. Programmable API Server

SpyWeb lets you define custom HTTP endpoints via Lua. Create a server/init.lua file in your project root.

Quick Start

Create server/init.lua:

function get:hello()
    return { status = 200, body = "Hello from SpyWeb!" }
end

function post:echo()
    return { status = 200, body = self.body, headers = { ["Content-Type"] = "text/plain" } }
end

Start SpyWeb and hit your endpoints:

./spyweb start
curl http://127.0.0.1:7979/api/v/hello
curl -X POST -d "test body" http://127.0.0.1:7979/api/v/echo

How It Works

Each HTTP request to /api/v/{name}:

1. Reads server/init.lua from disk (with hot-reload)
2. Creates a fresh Lua VM for the request
3. Loads the script, defining route handlers
4. Looks up get["name"] or post["name"] - falls back to all["name"]
5. Calls the handler with self carrying the request context
6. Returns the value as an HTTP response
7. Drops the VM, so no shared state between requests

Route Registration

function get:users()
    -- handles GET /api/v/users
end

function post:users()
    -- handles POST /api/v/users
end

function all:ping()
    -- handles any method on /api/v/ping
end

The method tables get, post, put, patch, delete, and all are pre-injected.

Functions defined outside method tables are private helpers, never exposed as endpoints:

function validate_input(data)
    return data and data.name
end

function get:users()
    local users = fetch_users()
    return { status = 200, body = users }
end

URL Structure

/api/v/{name}/{path_args...}

URL	Handler	self.path_args
/api/v/users	get.users	{}
/api/v/users/123	get.users	{"123"}
/api/v/users/123/profile	get.users	{"123", "profile"}

Request Context (self)

Field	Type	Description
self.body	string or nil	Raw request body (POST, PUT, PATCH); silently truncated at 10MB
self.method	string	HTTP method
self.path	string	Full request path
self.path_args	table	Array of trailing path segments
self.query	table	URL query parameters
self.headers	table	Request headers (lowercase keys)
self.client_ip	string	Client IP address

Response Contract

Table (full control)

return {
    status = 200,           -- HTTP status (clamped to 100-599)
    body = "Hello!",        -- string, table (auto-JSON), number, boolean, or binary
    headers = {             -- optional extra headers
        ["X-Custom"] = "value"
    }
}

Body types:

Lua Type	Response
string	Raw body as-is
table	Auto-encoded as JSON with Content-Type: application/json
number	Converted to string
boolean	Converted to "true" or "false"
nil	Empty body

String (quick response)

return "Hello, World!"
-- Equivalent to: { status = 200, body = "Hello, World!" }

Available Globals

The server VM has the same globals as scraper hooks (HTTP client, storage, database, file I/O, and utilities.

Defer (Async Cleanup)

Unlike defer() in pipeline hooks, the server’s defer() supports async bindings.

Deferred functions run in reverse order (last registered runs first). Each function runs sequentially in the same Lua VM, so they share globals and can safely access self via closure capture. Errors are logged but don’t affect other deferred functions.

function get:resource()
    defer(function()
        sleep(1000)
        log("request finished for " .. self.path)
        http_post("https://hooks.example.com/callback", json_encode({ status = "done" }), {
            ["Content-Type"] = "application/json"
        })
    end)
    return { status = 200, body = { message = "ok" } }
end

Examples

Robust JSON API

function post:items()
    local data, err = json_decode(self.body or "")
    if not data or type(data) ~= "table" then
        return { status = 400, body = { error = "Invalid JSON payload: " .. tostring(err) } }
    end
    if not data.name then
        return { status = 400, body = { error = "Missing 'name' field" } }
    end
    return { status = 201, body = { message = "Created", item = data } }
end

Serve Scraped Data

function get:latest_prices()
    local prices = global_store_get("latest_prices")
    if not prices then
        return { status = 404, body = "No data yet" }
    end
    return { status = 200, body = prices, headers = { ["Content-Type"] = "application/json" } }
end

Proxy to External API

function get:weather()
    local city = self.query.city or "London"
    local res, err = http_get("https://api.weather.com/v1/" .. city)
    if not res then
        return { status = 502, body = { error = err } }
    end
    return { status = res.status, body = res.body, headers = { ["Content-Type"] = "application/json" } }
end

function post:webhook()
    local payload = json_decode(self.body)
    http_post("https://hooks.example.com/relay", json_encode(payload), {
        ["Content-Type"] = "application/json"
    })
    return { status = 200, body = "Relayed" }
end

Health Check

function all:health()
    return { status = 200, body = "ok" }
end

Error Handling

If a handler throws a Lua error:

1. The error is logged to the terminal
2. The error is appended to server/error.log with a timestamp
3. The client receives {"error": "Internal Server Error"} with status 500

If server/init.lua has a syntax error, all requests fail with 500 until the file is fixed.

Timeout

With Luau (the default), each request has a 30-second timeout. If a handler runs longer, it is forcefully terminated and the client receives a 500 error with "Script execution timed out".

The timeout is enforced via Luau’s bytecode interrupt hook, so no threads are leaked.

Note: Timeout is only available with Luau. Lua 5.4 builds do not have timeout protection.

Serving Binary Data

Use fs_read_binary() to serve images, PDFs, or other assets from server/ or shared/:

function get:logo()
    local data = fs_read_binary("logo.png")
    if not data then
        return { status = 404, body = "Not Found" }
    end
    return {
        status = 200,
        body = data,
        headers = { ["Content-Type"] = "image/png" }
    }
end

Available Lua Globals

The server VM has access to the same globals as scraper hooks:

Category	Functions
File I/O	fs_read(filename), fs_read_binary(filename), fs_append(filename, content), fs_overwrite(filename, content), log(message)
HTTP Client	http_get(url, [headers]), http_post(url, body, [headers]), http_request({...}), http_multipart(url, fields, [headers])
Storage	global_store_set(key, value), global_store_get(key), global_store_incr(key, default, delta), global_store_delete(key)
Database (SQLite)	db_query(sql, [params]), db_exec(sql, [params])
Utilities	json_encode(value), json_decode(string), env_get(key), defer(fn), sleep(ms), notify(title, body, [timeout])

File writes go to server/ by default. Use shared/ prefix (e.g. fs_overwrite("shared/data.json", ...)) to write to the shared folder. Reads scan server/ first, then fall back to shared/.

Authentication

The API server reuses SpyWeb’s existing authentication. If SPYWEB_API_KEY is set, all /api/* routes (including /api/v/*) require the X-SpyWeb-Key header:

curl -H "X-SpyWeb-Key: your_key" http://127.0.0.1:7979/api/v/hello

Without the header, the server returns 401 Unauthorized.

Environment Variables

Variable	Default	Description
SPYWEB_PORT	7979	Server port
SPYWEB_API_KEY	none	API authentication key

Data Directory

The server’s fs_* operations are scoped to the server/ directory:

• Directoryproject/

  • Directoryserver/

    • init.lua Route definitions
    • error.log Error logs (auto-created)
    • Directorydata/ Persistent data

      • …
  • Directoryshared/ Cross-VM shared data

    • …
  • jobs.toml
  • data Scraper database
  • Directoryui/ Dashboard files

    • …

Troubleshooting

All endpoints return 500: Check server/error.log for syntax or runtime errors.

Endpoints return 404: Verify the handler is in the correct method table (get, post, etc.) and the name matches the URL segment exactly. The all table is checked last.

Body is empty or truncated: Request body is capped at 10MB.

Timeout errors: Luau-only 30-second timeout per request. Check for infinite loops in handler code. Move heavy computation to background tasks using defer.