Abstract

Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regular agentic tasks — a phenomenon we term the cold-start safety gap. To study this systematically, we introduce SODA (Safety Over Depth for Agents), a benchmark with 16 tool-use environments and 80 scenarios that controls how many regular tasks the agent completes before encountering a safety threat. Evaluating 7 models from 4 families, we find safety improves by 9–52% as depth increases from zero to twenty. By ablating which part of the conversation matters, we identify the key driver and propose a simple zero-cost deployment strategy that closes the gap while preserving utility.

Our contributions:

1. Benchmark & Discovery: We design SODA, a benchmark with 16 tool-use environments that controls conversation depth. Evaluating 7 models from 4 families, we find safety improves by 9–52% as the number of preceding regular agentic tasks increases from zero to twenty. Representation analysis confirms hidden states gradually shift toward a safety-aligned region.
2. What drives the safety improvement: We systematically ablate warm-up conversations and find that regular agentic task requests are the primary driver of safety — the agent's own prior responses have less effect on safety but are essential for preserving utility.
3. Safety generalizes without hurting utility: The warm-up effect generalizes to external safety benchmarks (AgentHarm, Agent Safety Bench) while preserving full tool-calling utility on BFCL Multi-Turn and API-Bank.
4. Deployment recommendation: We recommend a simple zero-cost strategy: having the agent complete a few regular agentic tasks before possible exposure to safety-critical requests mitigates the cold-start safety gap.

Figure 1. Overview of the cold-start safety gap and the SODA benchmark.

SODA: Safety Over Depth for Agents

SODA controls how many regular agentic tasks an agent completes before encountering a harmful request. Each task involves real multi-turn tool interaction across 16 environments spanning banking, healthcare, social media, cloud infrastructure, and more.

Figure 2. SODA benchmark design: 16 environments with controlled depth.

Agents Are Most Vulnerable at Session Start

All 7 models are significantly more likely to comply with harmful requests at the very beginning of a conversation (D=0) than after completing regular tasks. This vulnerability is universal across model families and scales.

Representation Analysis

Hidden states at the moment a harmful request is presented migrate from the unsafe region to the safe region as depth increases.

Figure 3. PCA projections colored by safety verdict (blue=safe, red=unsafe). Dashed line: estimated boundary. At D=0, most points fall in the unsafe region; with more preceding tasks, they migrate into the safe region.

What Drives the Safety Improvement?

We isolate which part of the warm-up conversation drives safety by modifying the task requests (user side), the agent's responses (assistant side), or both:

• Full Interaction: Agent genuinely interacts with the environment (baseline).
• Fix Requests (vary responses):
  • Compliant Response: Real task requests + short agreeable reply ("Sure, I can help.")
  • Random Response: Real task requests + unrelated random text as response
  • Empty Response: Real task requests + empty response
• Fix Responses (vary requests):
  • Random Request: Random text as request + real agent response
  • Empty Request: Empty request + real agent response
• Vary Both:
  • All Random: Both sides replaced with random text
  • All Empty: Both sides empty (only chat template preserved)

Does the Warm-Up Generalize and Preserve Utility?

We test whether the warm-up effect holds on external safety benchmarks and whether it preserves tool-calling utility.

Conclusion & Recommendation

We discover the cold-start safety gap: tool-calling LLM agents are most vulnerable at the very start of a session. A brief warm-up of regular agentic tasks (full interaction) substantially closes this gap across all models tested.

We additionally tested in-context refusal demonstrations and safety fine-tuning as alternative mitigations. Both improve safety but at significant cost: ICL refusal is unstable and causes over-refusal, while safety SFT collapses tool-calling utility (e.g., BFCL drops from 64% to 17%). This reveals a fundamental helpfulness–safety tradeoff — it is not easy to close the gap without losing agent capability.

The simplest and most practical solution today is full interaction warm-up: having the agent complete a few regular tasks before possible exposure to safety-critical requests. This requires no fine-tuning, no data collection, and nearly zero computational overhead. We hope future work explores strategies that achieve high safety without this tradeoff.

BibTeX

@article{sun2026coldstart,
  title={The Cold-Start Safety Gap in LLM Agents},
  author={Sun, Chung-En and Liu, Linbo and Weng, Tsui-Wei},
  journal={arXiv preprint arXiv:2606.07867},
  year={2026},
  url={https://arxiv.org/abs/2606.07867}
}