---
title: 'AI trading agent risks: 6 guardrails before you trust one'
url: https://comparand.xyz/blog/ai-trading-agent-guardrails
description: 'Six checks before you trust an AI trading agent: stale data, sources, order access, spending caps, tests on unseen dates and a human read.'
updated: '2026-10-21'
category: AI agents
---

# AI trading agent risks: 6 guardrails before you trust one

Six checks before you trust an AI trading agent: stale data, sources, order access, spending caps, tests on unseen dates and a human read.

Before you trust an AI trading agent, check six things: does it refuse stale data, can you see where each number came from, can it place orders, what caps its spending, was its rule tested on dates it never saw, and did a person read the result. Each one stops a real failure. In our Bitcoin test, 83.60% of 1,000 random rules scored lower on later dates than in training, and a tool should refuse stale data outright: Comparand refuses live labels older than 120 seconds and does not bill the refused call.

Some checks below use the Sharpe score, which compares average results with how much they swing. A higher score looks better, which is why a fake one is dangerous. This is research on past cases, not a forecast, and nothing here is advice.

For the timing bug that fakes a great result, see [Look-ahead bias backtest example](https://comparand.xyz/blog/look-ahead-bias-backtest-example).

## What can go wrong when an agent tests an idea?

We tested one rule on Binance BTCUSDT, a Bitcoin perpetual future (a contract that follows the price and never expires). The rule goes long or short for the next hour based on the last hour's direction. The honest version reads the last finished hour. The broken version reads the hour it is about to trade, which is how a missing "shift" in code can slip in.

Both used 0.05% fees on each side and the same hours, from September 2019 to September 2026.

| Version | Sharpe | Average per hour after fees | Hours with a gain | Worst fall from a high |
| --- | ---: | ---: | ---: | ---: |
| Honest | -8.00 | -0.06% | 40.17% | -100.00% |
| Broken | 59.41 | +0.33% | 87.21% | -0.62% |

The broken version looks almost perfect. That is the warning sign: it is a bug. We explain how it happens in [our look-ahead bias example](https://comparand.xyz/blog/look-ahead-bias-backtest-example), and what an AI can and cannot do in a backtest in [can ChatGPT or Claude backtest a strategy?](https://comparand.xyz/blog/can-ai-backtest-a-trading-strategy). The guardrails below are the checks that would have caught it, plus the ones that guard everything around it.

*Diagram: Show two bars side by side for the same 61,901 hours of Binance BTCUSDT, one labeled honest with Sharpe -8.00 and one labeled broken with Sharpe 59.41.*
## Guardrail 1: Does the agent refuse stale data?

A price from five minutes ago can be wrong for a fast market. An agent will not notice unless the data tool tells it.

So pick tools that refuse old data instead of quietly returning it. In Comparand, the live labels tool (hl_labels_live) returns the newest per-minute Hyperliquid labels for one coin. If those labels are older than 120 seconds, the call is refused with a live_stale error. A refused call is not billed.

Ask your agent to print the time of the data it used. If a call is refused as stale, the agent should say so and stop, not look for another way to get the old number.

## Guardrail 2: Can you see where every number came from?

If you cannot see where a number came from, you cannot check it. Make the agent show the source of each figure.

Every Comparand answer carries a prov block, which lists the days, digests (short fingerprints of the files) and code revisions behind it. It also carries a meter block, which says what the call cost. Ask the agent to quote both when it reports a result. We show how to read these records in [how to check crypto data provenance](https://comparand.xyz/blog/check-crypto-data-provenance).

If it cannot point to its data, treat the number as unchecked.

## Guardrail 3: Can the agent place orders?

Reading and acting should be separate steps. An agent that can only read can give a wrong answer. An agent that can also place orders can lose money on one.

Comparand is research only. It has no order placement or other execution, and it does not publish forecasts, trade calls or advice. If you connect other tools that can trade, that part is yours to control. A common habit is to let the agent propose and a person approve.

## Guardrail 4: What caps the spending?

An agent that loops can make a lot of calls. Put a ceiling on what it can spend.

Comparand credits are prepaid. A call that needs credits is refused when none are left, and nothing is billed beyond your plan price or your pay-as-you-go purchase, plus any tax at checkout. Plans also set request limits: 60 requests per minute on Starter, 300 on Pro and 900 on Desk. The usage tool shows your key's usage and credit balance for a month, and it is free.

Check usage after any run that you did not watch.

## Guardrail 5: Was the rule tested on dates the agent never saw?

When an agent tries many rules and shows you the best, the best is often luck. This is overfitting.

We made 1,000 random rules for hourly Bitcoin and kept the one with the best score on 2019 to 2022. Then we ran it unchanged on January 2023 to September 2026.

| Measure | Training (2019-2022) | Later dates (2023-2026) |
| --- | ---: | ---: |
| Sharpe of the chosen rule | 0.75 | -0.19 |
| Return after fees | +155.22% | -51.72% |
| Share of all 1,000 rules with a positive Sharpe | 24.50% | 1.00% |

Of the 1,000 rules, 83.60% scored lower on the later dates than in training. Ask the agent to hide a stretch of dates before it starts, pick the rule without looking, and test once. Our full test is in [the 1,000 random rules study](https://comparand.xyz/blog/overfitting-1000-random-trading-rules), and [walk-forward testing](https://comparand.xyz/blog/walk-forward-testing-explained) shows how to repeat this over many periods.

Later dates do not catch a timing bug. The broken rule above scored a Sharpe of 62.74 on January 2023 to September 2026 too, because it reads the future in every period.

## Guardrail 6: Did a person read the result?

Some results are too good. Our broken test had a worst fall of only 0.62% over seven years, and a gain in 87.21% of hours. Real results almost never look like that.

A person should check four things before trusting any agent result:

- **Timing.** For each input, when was it first known, and was that before the trade?
- **Costs.** Were fees included? A simple rule can look strong before costs. See [how fees change a backtest](https://comparand.xyz/blog/crypto-trading-fees-backtest-impact).
- **One lucky stretch.** Does the result fall apart if you drop the best year or best trade?
- **Too smooth.** Is the worst fall tiny or the win rate very high? Look for a bug first.

## What do these tests not show?

We did not test any particular AI model. We tested one fixed rule on one coin, and a person can make the same timing mistake as an agent. The point is that checks matter, not that agents are worse than people.

Guardrails reduce some risks. They do not make a result right, and they do not make a rule work in the future. The hourly tests leave out funding payments, spread, slippage and delays.

## How we measured this

We used Binance BTCUSDT perpetual hourly bars. The look-ahead test covers 61,901 hours from September 2019 to September 2026. The honest and broken versions differ in one thing: whether the position comes from the previous hour's return or the coming hour's. Both pay 0.05% on each side of a trade, including the first entry and last exit.

The overfitting test generated 1,000 seeded rules with look-backs of 1 to 240 hours. We chose the rule with the best Sharpe score from September 2019 through 2022, then froze it for January 2023 through September 2026.

## Try this in Comparand

Ask your connected AI agent: "Get the newest Hyperliquid labels for BTC. Tell me the time of the data, and show the prov and meter blocks."

- **hl_labels_live** returns the latest per-minute labels for one coin from the live feed. It is refused when the data is stale, and costs $0.005 per call on pay-as-you-go.
- The **prov** block lists the days, digests and code revisions behind the answer. The **meter** block shows what the call cost.

Then ask: "Show my usage and credit balance for this month."

- **usage** returns your key's usage and credit balance for one UTC month. It is free.

See the [docs](https://comparand.xyz/docs) to connect your agent and [pricing](https://comparand.xyz/pricing) for plans. You can also [try 3 coins free](https://comparand.xyz/try).

## The short version

- One timing bug moved a Bitcoin rule's Sharpe from -8.00 to 59.41 on the same 61,901 hours.
- Refuse stale data: Comparand's live labels are refused after 120 seconds, and a refused call is not billed.
- Ask for sources on every number, keep reading and ordering separate, and cap spending with prepaid credits.
- Test on dates the agent has not seen: the best of 1,000 random rules went from +155.22% to -51.72%.
- Treat results that look too smooth as bugs until a person has checked the timing and the costs.

## Questions people ask

**What are the main risks of an AI trading agent?**

Four stand out: it can act on stale prices, it can use a fake-looking backtest, it can overfit a rule to old data, and it can spend or trade more than you meant. In our hourly Bitcoin test, one timing mistake changed a Sharpe score from -8.00 to 59.41. Six checks help: refuse stale data, show sources, keep it away from orders, cap spending, test on unseen dates and have a person read the result.

**How do I know if an AI agent is using old data?**

Ask for the time of the newest data point with every answer, and use a data tool that refuses old data. Comparand's live labels tool returns a live_stale error when its newest labels are more than 120 seconds old, and a refused call is not billed.

**Can an AI agent backtest a rule honestly?**

It can, but only if the rule uses information that was known before each trade. Our broken test used the hour's own return to pick its position and showed a positive result in 87.21% of hours. The honest version did so in 40.17%.

**Does testing on later dates make an AI-made strategy safe?**

No. It catches overfitting, but not a timing bug, and a rule that passes can still lose. The best of 1,000 random rules gained 155.22% after fees in training and lost 51.72% on later dates.

**How can I limit what an AI agent spends on data?**

Use prepaid credits and check your balance often. Comparand's credits are prepaid, a call that needs credits is refused when none are left, and the free usage tool shows your balance and usage for the month.