<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Illia Khorunzhyi</title>
    <description>The latest articles on DEV Community by Illia Khorunzhyi (@aggosh).</description>
    <link>https://dev.to/aggosh</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4235303%2F04094eec-4707-4a2d-a22d-2d612a810636.jpg</url>
      <title>DEV Community: Illia Khorunzhyi</title>
      <link>https://dev.to/aggosh</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aggosh"/>
    <language>en</language>
    <item>
      <title>Our open dataset said paid streams last 11 hours. It was one customer.</title>
      <dc:creator>Illia Khorunzhyi</dc:creator>
      <pubDate>Tue, 10 Nov 2026 10:23:02 +0000</pubDate>
      <link>https://dev.to/aggosh/our-open-dataset-said-paid-streams-last-11-hours-it-was-one-customer-42aa</link>
      <guid>https://dev.to/aggosh/our-open-dataset-said-paid-streams-last-11-hours-it-was-one-customer-42aa</guid>
      <description>&lt;p&gt;On 9 November I published an open dataset: aggregate numbers from about 44,000 hours of 24/7 live streams that ran on our service in September and October. Platforms, resolutions, bitrates, uptime, how long streams stay on air. CC BY, with a CSV.&lt;/p&gt;

&lt;p&gt;The day after, one of its headline numbers turned out to describe a single account.&lt;/p&gt;

&lt;p&gt;This is how that happened, how we found it, and the check I now run before publishing any product statistic.&lt;/p&gt;

&lt;h2&gt;
  
  
  The number
&lt;/h2&gt;

&lt;p&gt;The report said paid streams run longer than free-trial ones: a median continuous session of &lt;strong&gt;11.3 hours&lt;/strong&gt; on paid plans, against &lt;strong&gt;2.2 hours&lt;/strong&gt; on the trial. It looked right. People who pay for a 24/7 streaming service should keep streams up longer than people trying it for a day. A believable number that matches what you expected is the most dangerous kind.&lt;/p&gt;

&lt;p&gt;I had also sent the report to a few people who maintain statistics pages, and several of those letters quoted "about 11 hours".&lt;/p&gt;

&lt;h2&gt;
  
  
  The smell
&lt;/h2&gt;

&lt;p&gt;The number kept bothering me: paid streams so often ending at about half a day. So I plotted how long paid runs last, in 30-minute bins. A real "how long do people stream" distribution should be smooth and long-tailed. This one had a cliff: the bins from 10.5 to 11.5 hours were about twice as tall as their neighbours, and at 12 hours the count fell by more than half.&lt;/p&gt;

&lt;p&gt;A cliff like that usually means a clock somewhere: a timeout, a token expiry, a platform limit. So the first job was to find the clock.&lt;/p&gt;

&lt;h2&gt;
  
  
  Suspects
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Our own cap.&lt;/strong&gt; The orchestrator has a maximum stream duration. It is 45 days, nothing overrides it, and the logs we checked contain no line saying a stream hit it. An 11-hour run is nowhere near it anyway.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The platform.&lt;/strong&gt; YouTube does reset ingest connections now and then, and it's plausible it does so on a schedule. To test that without trusting our own servers, we had two small control machines streaming to YouTube around the clock, one in the US and one in Germany, outside our cluster. Between 5 and 10 November YouTube reset them 13 times, after anything from 0.5 to 41 hours. Only one of the 13 fell between 10 and 14 hours. Twitch, for comparison, cut its stream at exactly 48 hours, twice. So YouTube has no 12-hour clock, in either region.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Billing.&lt;/strong&gt; A balance running out could stop streams, but not at the same hour of the day, day after day.&lt;/p&gt;

&lt;p&gt;None of these were it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The answer was a GROUP BY
&lt;/h2&gt;

&lt;p&gt;The query that solved it was embarrassingly simple: group the runs in that 10 to 14 hour spike by account.&lt;/p&gt;

&lt;p&gt;Almost all of them came from one account. That account runs many streams and restarts them on its own schedule, about twice a day. Nothing on our side was cutting anything; it was one customer's normal routine, and it was so large that it &lt;em&gt;was&lt;/em&gt; the paid median. That account alone produced more than two-thirds of all paid sessions in the period.&lt;/p&gt;

&lt;p&gt;Our paying cohort is small, as it is for most young products. In a small cohort, one heavy user's habit doesn't skew the average a little. It becomes the answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Same data, three answers
&lt;/h2&gt;

&lt;p&gt;Here's the paid median session, computed three ways from the same rows:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Method&lt;/th&gt;
&lt;th&gt;Median paid session&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Every session counts once (what we published)&lt;/td&gt;
&lt;td&gt;11.3 h&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Same, without the heaviest account&lt;/td&gt;
&lt;td&gt;19.7 h&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;One median per account, then the median of those&lt;/td&gt;
&lt;td&gt;22.2 h&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;None of these is a calculation error. They answer different questions. "What does a typical paid &lt;em&gt;session&lt;/em&gt; look like?" is dominated by whoever creates the most sessions. "What does a typical paying &lt;em&gt;customer&lt;/em&gt; do?" needs each customer to count once. The report claimed to answer the second question with the arithmetic of the first.&lt;/p&gt;

&lt;p&gt;The share of long sessions moved even more. Measured per session, 11.6% of paid sessions lasted a day or more. Measured per account, more than 8 in 10 paying accounts kept a stream up for at least a day, and more than half for at least a week.&lt;/p&gt;

&lt;p&gt;The free-trial median of 2.2 hours didn't move at all: over a thousand accounts, and none of them has more than 1% of the sessions.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Comparisons between paid plans and the trial are now &lt;strong&gt;per account&lt;/strong&gt;: "more than 8 in 10 paying accounts…", not "11.6% of sessions".&lt;/li&gt;
&lt;li&gt;The paid median is &lt;strong&gt;gone&lt;/strong&gt;. It moves by hours depending on the weighting, so it describes the weighting, not the streams.&lt;/li&gt;
&lt;li&gt;Account shares are rounded to 5% and written as words on the page, so they don't reveal how many paying accounts there are.&lt;/li&gt;
&lt;li&gt;Shares of hours on air stay as they were. They answer "what does the airtime look like?", and there no single account holds more than about 12% of any split.&lt;/li&gt;
&lt;li&gt;The methodology says so in one line, and the page shows the date it was updated.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two other splits had the same problem in a milder form. "60 fps on paid plans" and "single-video loops on paid plans" were each leaning on a handful of accounts. Both are now per account too, and the direction held: paying accounts use playlists and 60 fps more often than trial accounts.&lt;/p&gt;

&lt;h2&gt;
  
  
  The check I run now
&lt;/h2&gt;

&lt;p&gt;Before any product statistic leaves the building:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Plot the distribution, not just the median.&lt;/strong&gt; Cliffs and spikes are questions. Find the clock, or find the person.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;For every split, compute the top account's share.&lt;/strong&gt; If one account holds more than about a fifth of the rows in a group, the group's median is that account's habit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decide whether you're describing rows or people,&lt;/strong&gt; and weight accordingly. "Sessions", "hours" and "accounts" are three different populations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep a privacy floor.&lt;/strong&gt; We publish nothing for a group of fewer than five accounts, and round shares so the size of the group can't be backed out.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;When you blame the platform, run a control.&lt;/strong&gt; Two cheap servers streaming next to production settled the "is it YouTube?" question in a few days, which reading forum threads never would have.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check before you pitch.&lt;/strong&gt; I'd sent the 11-hour figure to people before checking it. The corrected page is the only fix I can offer them.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The corrected report and the CSV are here: &lt;a href="https://theloops.live/research/24-7-streaming-2026" rel="noopener noreferrer"&gt;24/7 streaming in practice&lt;/a&gt;. The CSV is also archived on Zenodo as &lt;a href="https://doi.org/10.5281/zenodo.24103783" rel="noopener noreferrer"&gt;10.5281/zenodo.24103783&lt;/a&gt;. If you find another number in it that one account is holding up, I'd like to hear about it.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I run &lt;a href="https://theloops.live" rel="noopener noreferrer"&gt;TheLoops&lt;/a&gt;, a service that keeps pre-recorded videos streaming 24/7. This post was drafted with help from an AI assistant; the data, the checks and the mistakes are ours.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>datascience</category>
      <category>analytics</category>
      <category>postmortem</category>
      <category>sql</category>
    </item>
    <item>
      <title>One stalled RTMP destination froze all the others — ffmpeg tee, use_fifo, and a doubled backslash</title>
      <dc:creator>Illia Khorunzhyi</dc:creator>
      <pubDate>Mon, 09 Nov 2026 14:19:53 +0000</pubDate>
      <link>https://dev.to/aggosh/one-stalled-rtmp-destination-froze-all-the-others-ffmpeg-tee-usefifo-and-a-doubled-backslash-4ckf</link>
      <guid>https://dev.to/aggosh/one-stalled-rtmp-destination-froze-all-the-others-ffmpeg-tee-usefifo-and-a-doubled-backslash-4ckf</guid>
      <description>&lt;p&gt;We run 24/7 live streams from pre-recorded video: one playlist, sent to several platforms at once — YouTube, Twitch and Kick, often all three. Each stream is one FFmpeg process, and the fan-out goes through the &lt;code&gt;tee&lt;/code&gt; muxer.&lt;/p&gt;

&lt;p&gt;This post is about the failure that &lt;code&gt;onfail=ignore&lt;/code&gt; does not cover, how to reproduce it on your laptop in two minutes, and the one-character escaping mistake that takes every output down when you apply the fix.&lt;/p&gt;

&lt;p&gt;Everything below was measured on ffmpeg 7.1.5 in production and reproduced on 8.1 for this post.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why &lt;code&gt;tee&lt;/code&gt; at all
&lt;/h2&gt;

&lt;p&gt;You can give ffmpeg several outputs instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ffmpeg &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt;.mp4 ... &lt;span class="nt"&gt;-f&lt;/span&gt; flv rtmp://a/... &lt;span class="nt"&gt;-f&lt;/span&gt; flv rtmp://b/...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But then every output runs its own encoders. We copy the video and re-encode the audio to AAC, and with separate outputs the AAC encoder runs once per destination — it cost us &lt;strong&gt;1.9× the CPU&lt;/strong&gt;. &lt;code&gt;tee&lt;/code&gt; encodes once and hands the same packets to every muxer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ffmpeg &lt;span class="nt"&gt;-re&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt;.mp4 &lt;span class="nt"&gt;-c&lt;/span&gt;:v copy &lt;span class="nt"&gt;-c&lt;/span&gt;:a aac &lt;span class="nt"&gt;-map&lt;/span&gt; 0:v &lt;span class="nt"&gt;-map&lt;/span&gt; 0:a &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="nb"&gt;tee&lt;/span&gt; &lt;span class="s2"&gt;"[f=flv:onfail=ignore]rtmp://a/app/KEY1|[f=flv:onfail=ignore]rtmp://b/app/KEY2"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Measured with video copied and audio re-encoded, each extra destination costs about two points of a core: 9.7% for one, 11.9% for two, 13.0% for three.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;onfail=ignore&lt;/code&gt; handles a destination that fails
&lt;/h2&gt;

&lt;p&gt;The documented default for a tee slave is &lt;code&gt;onfail=abort&lt;/code&gt;: one output fails, the whole process stops. With &lt;code&gt;onfail=ignore&lt;/code&gt;, a destination that refuses the connection or drops it is removed, and ffmpeg tells you so:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[tee @ 0x…] Slave muxer #1 failed: Connection refused, continuing with 1/2 slaves.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We set it explicitly even though our build already behaved that way without it. If you depend on a default the docs say is different, an ffmpeg upgrade can take every multi-destination stream off the air at once.&lt;/p&gt;

&lt;h2&gt;
  
  
  …but not a destination that stops reading
&lt;/h2&gt;

&lt;p&gt;The bad case is a server that &lt;strong&gt;keeps the connection open and stops reading&lt;/strong&gt;. Nothing fails. &lt;code&gt;tee&lt;/code&gt; writes to every slave from one thread, so once the kernel's socket buffers to the stuck peer are full, that write blocks — and the healthy destinations stop getting packets too.&lt;/p&gt;

&lt;p&gt;There is no error, no "slave failed" line, and the process is alive. From the outside, every destination still looks connected, and none of them is receiving video.&lt;/p&gt;

&lt;h3&gt;
  
  
  Reproduce it
&lt;/h3&gt;

&lt;p&gt;A TCP peer that reads for 8 seconds, then holds the socket open and goes quiet:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# stall_peer.py — read for N seconds, then keep the socket open and stop reading
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;read_for&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setsockopt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SOL_SOCKET&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SO_REUSEADDR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;bind&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;127.0.0.1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;listen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;accept&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;read_for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;settimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;recv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;65536&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3600&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then one healthy output (a local file) and one output to the stalling peer, in real time (&lt;code&gt;-re&lt;/code&gt;) for 70 seconds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 stall_peer.py 19351 8 &amp;amp;
ffmpeg &lt;span class="nt"&gt;-re&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; lavfi &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="nv"&gt;testsrc2&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1280x720:rate&lt;span class="o"&gt;=&lt;/span&gt;30 &lt;span class="nt"&gt;-f&lt;/span&gt; lavfi &lt;span class="nt"&gt;-i&lt;/span&gt; sine &lt;span class="nt"&gt;-t&lt;/span&gt; 70 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-c&lt;/span&gt;:v libx264 &lt;span class="nt"&gt;-preset&lt;/span&gt; ultrafast &lt;span class="nt"&gt;-b&lt;/span&gt;:v 4M &lt;span class="nt"&gt;-g&lt;/span&gt; 60 &lt;span class="nt"&gt;-c&lt;/span&gt;:a aac &lt;span class="nt"&gt;-map&lt;/span&gt; 0:v &lt;span class="nt"&gt;-map&lt;/span&gt; 1:a &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-progress&lt;/span&gt; progress.txt &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="nb"&gt;tee&lt;/span&gt; &lt;span class="s2"&gt;"[f=flv:onfail=ignore]healthy.flv|[f=flv:onfail=ignore]tcp://127.0.0.1:19351"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Watch &lt;code&gt;healthy.flv&lt;/code&gt; and &lt;code&gt;out_time&lt;/code&gt; in &lt;code&gt;progress.txt&lt;/code&gt;. Results, sampled every 10 seconds:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;slave options&lt;/th&gt;
&lt;th&gt;healthy output after 80 s&lt;/th&gt;
&lt;th&gt;stream time reached&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;onfail=ignore&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;frozen at 4.6 MB&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;stuck at 8.5 s&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;onfail=ignore:use_fifo=1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;frozen at 4.8 MB&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;stuck at 9.3 s&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;onfail=ignore:use_fifo=1:fifo_options=queue_size=600\\:drop_pkts_on_overflow=1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;34.9 MB, growing all the way&lt;/td&gt;
&lt;td&gt;68.8 s of 70&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In the first two runs, ffmpeg printed nothing until I killed the peer 80 seconds later; only then did &lt;code&gt;Slave muxer #1 failed: Broken pipe&lt;/code&gt; appear. Bigger buffers only delay the freeze: in our first test of this, the stream ran 51.7 seconds before stopping dead. It still stopped.&lt;/p&gt;

&lt;p&gt;Note the middle row: &lt;strong&gt;&lt;code&gt;use_fifo=1&lt;/code&gt; on its own does not help.&lt;/strong&gt; It moves each slave into its own thread with its own queue, but the queue blocks when it is full, so you've only added a buffer before the same stall. The queue has to be allowed to drop.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="n"&gt;use_fifo&lt;/span&gt;=&lt;span class="m"&gt;1&lt;/span&gt;:&lt;span class="n"&gt;fifo_options&lt;/span&gt;=&lt;span class="n"&gt;queue_size&lt;/span&gt;=&lt;span class="m"&gt;600&lt;/span&gt;\\:&lt;span class="n"&gt;drop_pkts_on_overflow&lt;/span&gt;=&lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each destination gets its own thread and a queue of 600 packets — roughly ten seconds of a 1080p stream, long enough to ride out a hiccup and small enough to afford per destination on a box running twenty-odd streams. When a destination stops reading, its queue fills and starts dropping packets; the other destinations never notice.&lt;/p&gt;

&lt;p&gt;The stalled destination gets a stream with holes in it, which is the right trade when the alternative is all of them going dark. With &lt;strong&gt;one&lt;/strong&gt; destination we don't use any of this: there is nothing to protect, and blocking is better there — the stream stalls, our health check notices the frozen clock and restarts it cleanly, instead of feeding a platform a broken stream it may not recover from.&lt;/p&gt;

&lt;h2&gt;
  
  
  The doubled backslash
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;fifo_options&lt;/code&gt; is a dictionary &lt;em&gt;inside&lt;/em&gt; the slave's option list, and both use &lt;code&gt;:&lt;/code&gt; as a separator. So the inner &lt;code&gt;:&lt;/code&gt; has to be escaped — and &lt;code&gt;tee&lt;/code&gt; unescapes the slave options once, and the nested dictionary once more. One backslash is not enough:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# single backslash&lt;/span&gt;
&lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="nb"&gt;tee&lt;/span&gt; &lt;span class="s1"&gt;'[f=flv:onfail=ignore:use_fifo=1:fifo_options=queue_size=600\:drop_pkts_on_overflow=1]a.flv|…'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[flv @ 0x…] Unknown option 'drop_pkts_on_overflow'
[tee @ 0x…] Slave muxer #0 failed: Invalid argument, continuing with 1/2 slaves.
[tee @ 0x…] All tee outputs failed.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exit code 234, and both files stop at 383 bytes — the header. The single-escaped &lt;code&gt;drop_pkts_on_overflow&lt;/code&gt; escapes the fifo dictionary and gets handed to the flv muxer, which rejects it, and &lt;code&gt;onfail=ignore&lt;/code&gt; cannot save you because &lt;em&gt;every&lt;/em&gt; slave carries the same bad option. With &lt;code&gt;\\&lt;/code&gt; the same command writes both files to the end.&lt;/p&gt;

&lt;p&gt;In Python, build the argument as a raw string and pass it to the process without a shell, so nothing else gets a chance to eat a backslash:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;FLV&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;f=flv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;flvflags=+no_duration_filesize&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;onfail=ignore&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;FIFO&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;use_fifo=1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fifo_options=queue_size=600\\:drop_pkts_on_overflow=1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;tee_spec&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;opts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;FLV&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;FIFO&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;[]))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;]&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# asyncio.create_subprocess_exec("ffmpeg", ..., "-f", "tee", tee_spec(urls))
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A user-supplied URL is part of the tee syntax
&lt;/h2&gt;

&lt;p&gt;The whole spec is one string shaped &lt;code&gt;[opts]url1|[opts]url2&lt;/code&gt;. A &lt;code&gt;|&lt;/code&gt; inside a URL starts another slave:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ffmpeg &lt;span class="nt"&gt;-f&lt;/span&gt; lavfi &lt;span class="nt"&gt;-i&lt;/span&gt; testsrc &lt;span class="nt"&gt;-t&lt;/span&gt; 1 &lt;span class="nt"&gt;-c&lt;/span&gt;:v libx264 &lt;span class="nt"&gt;-map&lt;/span&gt; 0:v &lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="nb"&gt;tee&lt;/span&gt; &lt;span class="s1"&gt;'[f=flv:onfail=ignore]x1.flv|x2.flv'&lt;/span&gt;
&lt;span class="c"&gt;# writes x1.flv AND x2.flv&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If your users paste their own ingest URLs, that is a way to push to a destination you never counted or reported on. Reject &lt;code&gt;|&lt;/code&gt;, &lt;code&gt;[&lt;/code&gt;, &lt;code&gt;]&lt;/code&gt;, &lt;code&gt;\&lt;/code&gt; and whitespace in destination URLs before they reach the spec — and accept only &lt;code&gt;rtmp://&lt;/code&gt; and &lt;code&gt;rtmps://&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Knowing which destination died
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;tee&lt;/code&gt; gives no per-destination byte counts (&lt;code&gt;total_size=N/A&lt;/code&gt; in &lt;code&gt;-progress&lt;/code&gt;), so the log lines are the signal. These are the ones worth parsing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[tee @ 0x…] Slave muxer #1 failed: Connection refused, continuing with 1/2 slaves.
[fifo @ 0x…] Error opening rtmp://…: Input/output error
[tee @ 0x…] All tee outputs failed.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mask the stream key before you log any of them: the address in these lines contains it.&lt;/p&gt;




&lt;p&gt;I run &lt;a href="https://theloops.live" rel="noopener noreferrer"&gt;TheLoops&lt;/a&gt;, a service that streams pre-recorded video 24/7 to YouTube, Twitch and Kick from the cloud. Every multi-destination stream it runs goes through the spec above. Questions about ffmpeg fan-out are welcome in the comments.&lt;/p&gt;

</description>
      <category>ffmpeg</category>
      <category>streaming</category>
      <category>video</category>
      <category>devops</category>
    </item>
  </channel>
</rss>
