Memory exhaustion in loop executions with agent blocks making tool calls

[aadamsx]

Summary

When running workflows with loops containing agent blocks that make many tool calls (e.g., MCP file operations), memory accumulates unbounded until system OOM. This caused my 64GB system to run out of memory during workflow execution.

Environment

• Self-hosted via Docker
• macOS with 64GB RAM
• Docker containers with no memory limits (default)

Root Cause Analysis

After investigating the codebase, I identified 6 critical memory accumulation points with no bounds:

1. allIterationOutputs in LoopScope (CRITICAL)

File: apps/sim/executor/orchestrators/loop.ts (line 144)

if (iterationResults.length > 0) {
  scope.allIterationOutputs.push(iterationResults)  // UNBOUNDED
}

Every loop iteration pushes results to this array with no size limit or pruning.

2. blockLogs array in ExecutionContext (CRITICAL)

File: apps/sim/executor/execution/block-executor.ts (lines 60-62)

blockLog = this.createBlockLog(ctx, node.id, block, node)
ctx.blockLogs.push(blockLog)  // UNBOUNDED - every block execution adds

Every block execution (including each loop iteration) pushes to this array.

3. TraceSpans accumulation

File: apps/sim/lib/logs/execution/trace-spans/trace-spans.ts

The buildTraceSpans() function creates spans for ALL block logs and tool calls, building an unbounded in-memory tree structure.

4. Tool calls array per agent

File: apps/sim/executor/handlers/agent/agent-handler.ts (lines 250-318)

Each agent execution stores all tool calls in span.toolCalls array. With 100 tool calls per agent × 1000 iterations = 100,000 ToolCall objects.

5. File extraction unbounded recursion

File: apps/sim/lib/logs/execution/logger.ts (lines 621-695)

extractFilesFromExecution() recursively walks the entire trace span tree with no depth limits.

6. Memory table JSONB concatenation

File: apps/sim/executor/handlers/agent/memory.ts (line ~580)

data: sql`${memory.data} || ${JSON.stringify([message])}::jsonb`  // Unbounded append

Memory Growth Calculation

For a workflow with:

• Loop: 1000 iterations
• Agent block per iteration with 100 tool calls (file read/write via MCP)

Memory consumption:

• blockLogs: 1000 entries × 50KB = 50MB
• allIterationOutputs: 1000 × 1MB = 1GB
• Tool calls in spans: 100,000 objects = 1GB
• TraceSpans tree traversal: O(GB) object graph
• Total: 2-10+ GB in memory, persisted to executionData JSONB

Reproduction Steps

1. Create a workflow with a Loop block (while condition)
2. Inside loop: StateTracker function → Agent with MCP tools (file read/write) → Reviewer
3. Run the workflow with multiple tasks that require many file operations
4. Monitor Docker container memory with docker stats
5. Observe memory climbing until OOM

Suggested Fixes

Short-term mitigations:

1. Add iteration limits check with memory awareness:

const MAX_MEMORY_PER_LOOP = 100 * 1024 * 1024; // 100MB
if (estimateSize(scope.allIterationOutputs) > MAX_MEMORY_PER_LOOP) {
  logger.warn('Loop memory limit reached, truncating older iterations');
  scope.allIterationOutputs = scope.allIterationOutputs.slice(-10);
}

2. Stream block logs to database instead of accumulating:

// Instead of ctx.blockLogs.push(blockLog)
await this.persistBlockLog(blockLog);

3. Truncate tool call results in spans:

const MAX_TOOL_RESULT_SIZE = 10000;
toolCall.result = truncate(toolCall.result, MAX_TOOL_RESULT_SIZE);

Long-term solutions:

1. Implement streaming execution logs (write to DB incrementally)
2. Add configurable memory limits per execution
3. Implement log rotation/pruning during long-running executions
4. Add memory pressure monitoring with graceful degradation

Workaround

Set Docker memory limits to prevent system-wide OOM:

# docker-compose.override.yml
services:
  simstudio-app:
    deploy:
      resources:
        limits:
          memory: 8G

Related Files

• apps/sim/executor/orchestrators/loop.ts
• apps/sim/executor/execution/block-executor.ts
• apps/sim/lib/logs/execution/trace-spans/trace-spans.ts
• apps/sim/lib/logs/execution/logger.ts
• apps/sim/executor/handlers/agent/agent-handler.ts
• apps/sim/executor/handlers/agent/memory.ts
• packages/db/schema.ts (executionData JSONB field)