Data-transfer timeout leaves osmo-ctrl blocked on watchdog channel sends #1408

Description

createOutCommandStream() starts a watchdog goroutine that monitors stdout inactivity. The stream function creates an unbuffered quit channel and after stdout scanning finishes, sends quit <- true followed by timeoutChan <- false.

When inactivity reaches DataTimeout, the watchdog kills the subprocess, sends true to timeoutChan and exits. Once stdout closes, the stream function attempts to send to quit, but the watchdog that would receive that value has already exited. The send blocks indefinitely, preventing the deferred waitStreamLogs.Done() call.

common.RunCommand() waits for that wait group before consuming the timeout result. Consequently, the command never returns its timeout error and the surrounding retry loop cannot advance.

Relevant code

• src/runtime/pkg/data/data.go: createOutCommandStream() and RunOSMOCommandStreamingWithRetry().
• src/runtime/pkg/common/common.go: RunCommand().
• src/runtime/pkg/data/data_runtime_test.go: existing stream tests cover ordinary scanner completion but do not exercise this timeout sequence.

Reproduction sequence

1. Start a subprocess that remains alive without emitting stdout.
2. Allow stdout inactivity to exceed DataTimeout.
3. The watchdog kills the process and writes true to the buffered timeout channel.
4. The watchdog exits and the stdout scanner finishes.
5. The stream goroutine blocks sending to the unbuffered quit channel.
6. RunCommand() remains blocked in waitStreamLogs.Wait().

Impact

Affected download/upload subprocess timeouts can leave osmo-ctrl indefinitely stuck instead of retrying or reporting the timeout. External termination may be required to stop the stuck execution.

Verification

An isolated reproduction used the existing stream function, a real sleeping subprocess and a shortened timeout. The goroutine dump showed the stream blocked on the channel send and the caller blocked in WaitGroup.Wait(), with one timeout result already queued.

Suggested resolution

Use cancellation that does not require a live receiver, such as closing a completion channel or cancelling a context. Coordinate the watchdog and scanner so exactly one terminal timeout result is sent and the watchdog exits cleanly.

Merely buffering quit is insufficient: the subsequent timeoutChan <- false would block because the capacity-one channel already contains true and its receiver waits for stream completion first. Timeout reporting must also preserve the timeout outcome when scanner completion follows process termination.

Add a regression test that forces a real timeout and verifies bounded completion with a timeout error, alongside normal-completion coverage. Synchronize shared progress timestamps when revising the watchdog.