# Open source oversight: how to pick the mode for one action

> Open source puts the choice between human in the loop and human on the loop on each action: hold the call, supervise it, or let it run, and a rule you write enforces it. Kortix is the open-source AI Operating System.

URL: https://humanintheloopvshumanontheloop.com/how-to-choose

## Sorting a call into hold, supervise or allow

Three questions about one action, answered with the rule that enforces it.

1. Can the wrong action be undone before anyone would notice it? (No. It is a message, a payment or a deletion. / Partly, if someone catches it that day. / Yes, in the same system, within minutes.)
2. Who sees the damage first? (A customer, a partner or the public. / A teammate, in a queue we already read. / Nobody has to. The next read shows the error.)
3. Does this call cross the company boundary? (Always. The recipient sits outside. / Sometimes. It depends on the argument. / No. Everything it touches stays inside.)

Possible answers:

- **Hold the call**: Put Ask on this action, `require_approval`. The call returns `pending_approval` with an approval link, nothing runs, and the turn resumes when a project manager or the person who started the session decides. When only part of the traffic is risky, keep Allow for the safe argument and let a second rule catch the rest.
- **Supervise it**: Let it run under `policy: default_mode: risk`. Reads run, writes and destructive calls wait for a person, and every call is recorded for the supervisor to read back. Keep the rule in the repo, so whoever watches can change it without starting a session.
- **Let it run**: Put Allow on this action, `always_run`. Nothing pauses, and the record is written, so a wrong call turns up in review. Revisit the rule once the action starts reaching people outside the company.

## Two rules on one action

`kortix.yaml`

```yaml
policy:
  default_mode: risk

policies:
  - match: gmail.send_email
    action: always_run
    conditions:
      - arg: to
        match: "*@example.com"
  - match: gmail.send_email
    action: require_approval
```

Source: https://kortix.com/docs/connect/connectors

A send to an internal address runs, and every other send of the same action waits for a person.

## The test: what survives after the call goes wrong

Sort one action by what happens after it goes wrong. Three questions cover almost every call an agent makes, and none of them asks what the action costs or how important it looks. The answers decide whether the call waits for a person, runs under a rule someone watches, or runs on its own.

The first question is reversibility inside the detection window: the gap between the call and the first person who would notice it. A written CRM field is undone by the next edit, and the account owner sees the change that afternoon. A message to a customer is not undone by an edit, because the customer has already read it. Reversibility on its own says little. Reversibility before notice is the whole test.

The second question is blast radius, meaning how many rows, accounts or people one call can touch. A single refund moves money once. A bulk update moves every record the query matched, and the count is decided by the query rather than by the person who approved the agent's job.

The third question is the boundary. An action that leaves the company is read by someone you cannot correct inside the same system, so the correction arrives as an apology or a chargeback.

The line between asking and allowing follows from that test, and cost is the signal people reach for first. It points the wrong way. A small refund to a stranger can be gone for good. A large transfer between two accounts you own is fixed by an entry the same afternoon, before the bank statement reaches anyone. The refund waits and the transfer runs, because one leaves the boundary and the other is undone inside it. Importance behaves the same way: a quarterly report sounds weighty and writes one file, while a routine email to a customer list is never taken back.

The same action name splits by its arguments, which is what makes a per-action choice practical. Kortix evaluates a rule against the call as it is composed, so `gmail.send_email` can run to a colleague's address and wait for every other recipient. Conditions read the arguments by dot path, `negate: true` inverts a test, a rule holds up to 10 conditions and all of them must match. A missing argument never matches a pattern, so a negated condition on it holds. A malformed condition fails closed, and a typo narrows the gate instead of opening it. The rule grammar is documented in [Kortix's connector docs](https://kortix.com/docs/connect/connectors).

## Eight actions and the mode each gets

The modes in this table are a recommendation, applied to eight actions agent projects ask about most. Each reason names what makes the action hard to take back, and a reader who disagrees with a reason has the argument they need to overrule it.

| Action | Mode | Rule | Reason |
| --- | --- | --- | --- |
| Email a customer | Ask | `require_approval` | A sent message cannot be pulled back |
| Post to a public channel | Ask | `require_approval` | Readers outside the company see it first |
| Write a CRM field | Allow inside, Ask outside | `always_run` plus a condition | The owner edits a wrong value the same day |
| Merge to `main` | Ask | change request, default-deny for agents | A person reads the diff before it lands |
| Delete a record | Block, or Ask with a restore path | `block` | Restoring costs more than one question |
| Refund a payment | Ask above a threshold | `require_approval` plus a condition | Money leaves and the buyer keeps it |
| Run a migration | Ask, once | `require_approval` | The schema outlives the session that changed it |
| Read a report | Allow | `always_run` | Nothing changes that a person must catch |

Two rows depend on an argument rather than on the action. The email row is Ask for a customer and Allow for a colleague at your own domain, written as two rules on one action. The refund row is Allow under a threshold and Ask above it, which is the same shape with a different argument.

Reads earn Allow for a specific reason. A read changes nothing a person would have to catch, and holding it puts decision work in front of someone for no gain. The exception is a mailbox or a payment account, where a read exposes customer data. `sensitive: true` on that connector holds every remaining action, reads included.

## What each mode costs

Ask costs decisions. Every held call spends one decision from a person allowed to give it, so the work moves at the pace those people can keep. The waiting is the larger cost. A held call pauses the turn, and an approver asleep at midnight means the work stops until morning. A call nobody decides expires within 24 hours, or within the project's `approvals.ttl`, and the session does not resume; the agent asks again with a new call. There is no session-wide approval, so an action that must run unattended needs an Allow rule.

Supervision costs attention spread thin. The rule decides which calls wait and which run, and a person reads the record of what ran. That works as long as someone actually reads it, and it stops working when the review becomes a wall of green rows.

Allow costs the correction. Nothing slows the run, and a wrong call is found in review or by the person who received it. Every action is recorded. Reading, exporting and streaming the audit log depend on the plan.

Block costs almost nothing at run time and one edit at the moment someone needs the action. A blocked call never reaches a reviewer, so the cost lands on whoever has to change the rule while work waits.

## Three ways the choice goes wrong

The first is one mode for the whole agent. Approve everything and the queue becomes the bottleneck, which is the failure Decagon describes when reviewers who approve a long run of correct proposals stop reading closely. Allow everything and nobody is watching the call that mattered. Flowable's line about the decision belonging to every meaningful step in a workflow is the right granularity, and each step still has to be assigned.

The second is approval on everything. A queue that holds reads, writes and deletes alike gets read at the speed of its volume, and the reviewer learns to click through the prompt without opening the arguments. Narrow the gate and the queue stays short enough that the next prompt is read.

The third is supervision with no policy lever. A person who can deny one call and cannot change the rule that produced it gets the same call again after lunch. Supervision holds when the watcher owns the rule and can edit it, and when the edit reaches `main` through a review rather than through the running session.

## Writing the choice into kortix.yaml

Approval gates you set. Off until you set them. A new project starts on `policy.default_mode: allow_all`, which runs every action no rule matches, so the first edit is usually the `risk` default. `risk` runs reads and holds writes and destructive calls.

Project-wide `policies:` in `kortix.yaml` evaluate first and use fully qualified patterns such as `gmail.send_email` or `*.delete*`. The connector's own `policies:` come second, with patterns relative to the connector. `sensitive: true` comes third, and `default_mode` last. The first match wins, so the narrow rule that allows an internal recipient is written before the rule that holds everyone else. Because a connector rule cannot override a project rule, the gate lives in the repo the project owns.

```bash
kortix connectors policy set --default risk --apply
kortix connectors policy add "gmail.send_*" ask --condition to='*@example.com'
kortix connectors sensitive gmail on
```

Actions are always_run, require_approval, block, shown as Allow, Ask, Block. Approval holds the call; the agent's turn pauses and resumes. One approval covers one exact request: the same connector, action, arguments and account. Change an argument and the call needs a new approval. After an approval, the identical call runs once when it repeats within 15 minutes. A project manager or the person who started the session decides a held call, and a session started with an API token is decided by project managers only. That queue of approvers is the cost of Ask.

Then keep the list current. Session work reaches `main` through a change request. Merge is default-deny for agents, so the rule that governs an agent passes a person before it governs anything. Review the exceptions on a schedule of their own, because a condition written for one customer outlives the contract and a `risk` default added for a pilot outlives the pilot. [The approval workflow page](/approval-workflow) takes the held call apart, [the human-in-the-loop page](/human-in-the-loop) covers the approver's side of it, and [the human-on-the-loop page](/human-on-the-loop) covers the supervision lever that changes future runs.

## What people ask before they set a mode

Short answers to the five questions this decision raises.

### How do you choose between human in the loop and human on the loop

By the action, not by the system. Hold the call when a wrong result cannot be undone before someone notices it, supervise it when the result can be reversed while a person can still see it happen, and let it run when nothing needs catching.

### Which AI agent actions should require approval

Calls that leave the company, move money, delete data, change a schema or reach `main`. A sent message, a payment and a deletion cannot be taken back in time. Write one rule per action class instead of one rule for the whole agent.

### Should every AI action be reviewed by a person

No. A review on every call builds a queue nobody reads closely, and reviewer fatigue is the failure Decagon names. Reviews earn their place on calls that are hard to reverse. Reads and internal writes can run and be recorded instead.

### What is an approval threshold for an AI agent

The line where a call stops running on its own and waits for a person, written as a condition on its arguments: a refund above an amount, a recipient outside your domain, a delete that touches more than one row. Kortix tests arguments with `arg` and `match`.

### Can one agent use both oversight models

Yes, and most working agents do. Rules apply per action and per argument, so one session can hold a refund, run a report, and allow an email to an internal address while holding the same action to a customer.

## Write the rule before the call runs.

Start with the one action you would not want to take back.

[Try Kortix](https://kortix.com) · [Read the approval workflow](https://humanintheloopvshumanontheloop.com/approval-workflow)