in_tail: Ensure to discard TailWatcher with missing target when follow_inodes

[daipom]

Which issue(s) this PR fixes:
Not related to a particular issue, but it can have something to do with reported log duplication or resource leak phenomenon about in_tail.

What this PR does / why we need it:
Ensure to discard TailWatcher with missing target when follow_inodes.

For example, when a rotation process is slow, there is a small time lag between moving and adding files.
There is a possibility that StatWatcher notifies too quickly to the TailWatcher before the new file is moved to that target path. From the TailWatcher, it appears as if the file is resurrected once it disappeared.

In this case, refresh_watcher can't recognize it, so TailWatcher needs to discard self correctly.
In the previous implementation, it was not done.
So it caused the handle leak and log duplication.
(Please check the added test case)

Because of the problem #4237 (comment), log duplication can still occur during rotate_wait.
We need to set rotate_wait 0 for a workaround.
(Duplication will occur if refresh_watcher is called during the rotate_wait. We need the mechanism not to add duplicated TailWathcer with detaching TailWatcher.)

Docs Changes:
Not needed.

Release Note:
Same as the title.

How to Reproduce:

<source>
  @type tail
  tag test
  path /path/to/test.log*
  pos_file /test/fluentd/pos/pos
  read_from_head true
  follow_inodes true
  refresh_interval 15s
  <parse>
    @type none
  </parse>
</source>

<match test.**>
  @type stdout
</match>

• A: initial start

test.log

• B: move the file

test.log1

• C: add the new current file

test.log
test.log1

• D: wait rerfresh_watcher TWICE
  • first: add TailWatcher for test.log (new inode)
    • this overwrites @tails[test.log] so old inode information is lost from @tails at this timing
  • second: add TailWatcher for test.log1 (old inode)
• E: append logs to test.log1
  • Because now there are 2 TailWatchers for test.log1, log duplication occurs.
    • TailWatcher(test.log, old inode) (orphan) (This actually opens the file handle of test.log1)
    • TailWatcher(test.log1, old inode)

When StatWatcher (or TimeTrigger) notifies between B and C, then the TailWatcher becomes an orphan, and a handle leak happens.

[daipom]

I will check this

• https://github.com/fluent/fluentd/actions/runs/5539804706/jobs/10111142203?pr=4239

[daipom]

I will check this

* https://github.com/fluent/fluentd/actions/runs/5539804706/jobs/10111142203?pr=4239

I think this is just a list order problem. I will improve the test.

[daipom]

It seems that log duplication and handle leak still can occur on macOS...
I will check it 😢

https://github.com/fluent/fluentd/actions/runs/5539804708/jobs/10111141849?pr=4239
https://github.com/fluent/fluentd/actions/runs/5539804708/jobs/10111141988?pr=4239

[daipom]

It seems that log duplication and handle leak still can occur on macOS... I will check it 😢

https://github.com/fluent/fluentd/actions/runs/5539804708/jobs/10111141849?pr=4239 https://github.com/fluent/fluentd/actions/runs/5539804708/jobs/10111141988?pr=4239

It seems that this can happen only on macOS.
I can reproduce this on my macOS (Ventura, Apple M1) too, but I can't reproduce it on my Windows 10.

[daipom]

It seems that log duplication and handle leak still can occur on macOS... I will check it 😢
https://github.com/fluent/fluentd/actions/runs/5539804708/jobs/10111141849?pr=4239 https://github.com/fluent/fluentd/actions/runs/5539804708/jobs/10111141988?pr=4239

It seems that this can happen only on macOS. I can reproduce this on my macOS (Ventura, Apple M1) too, but I can't reproduce it on my Windows 10.

It can happen on Windows and macOS because StatWatcher does not work on those platforms (specification).
So this is merely a matter of the interval time between moving and adding files.
I will change the sleep time a bit longer for those platform to make the test valid for those platforms.

[daipom]

The above problems are fixed.

[daipom]

I'm sorry. There was an easy mistake.

https://github.com/fluent/fluentd/actions/runs/5539804706/jobs/10111142203?pr=4239

Fixed by https://github.com/fluent/fluentd/compare/5a4c9b34b0f62b5d3f024f93b6803fc962859bbf..0c13c9fe6a9b540362cd5e48f61f44d2f1a683ad

[daipom]

Rebased to the latest master.

[ashie]

Thanks!

[daipom]

Thanks for your review!