in_tail: Ensure to detach correct watcher on rotation with follow_inodes

[daipom]

Which issue(s) this PR fixes:

• Partially fixes Fluentd in_tail "unreadable" file causes "following tail of <file>" to stop and no logs pushed #3614
• Fixes [in_tail][follow_inode enabled] found a racing bug that new created watcher be unwatched unexpectedly and thus log stuck #4190

What this PR does / why we need it:
If refresh_watchers run before update_watcher, the old implementation of update_watcher detach wrongly the new TailWatcher which is added in refresh_watcher. This causes the problem of stopping tailing log and handle leak.

The test case test_updateTW_after_refreshTW reproduces this problem.
This PR solves it.

NOTE: There is another BUG about unwatching. (I adjusted some expected values of the tests for this BUG.)
When refresh_watcher find the rotated old file AFTER unwatching it (rotate_wait), then the logs will be collected in duplicate.
If refresh_watcher find it BEFORE unwatching it (rotate_wait), this problem doesn't occur because the position entry is still alive.
We need to fix this and fix the adjusted expected values.

• => in_tail: prevent wrongly unwatching with follow_inodes #4221

Docs Changes:
Not needed.

Release Note:
Same as the title.

OTHERS:
This fix is based on the content and discussion of the following issue and PR:

• Fluentd in_tail "unreadable" file causes "following tail of <file>" to stop and no logs pushed #3614
• in_tail: Use inode for key of TailWatcher when follow_inodes #4185
• in_tail: Check detaching inode when follow_inodes #4191

Thanks to those who have given me much insight, especially @kattz-kawa, @masaki-hatada, and @garyzjq!

[daipom]

The scenario of this problem (basically from #4185 (comment))
Please refer to the test case test_updateTW_after_refreshTW.

When follow_inodes is enabled.

• Files: file.log (inode-A)
  • @tails[file.log] => Watcher(file.log, inode-A)
• Rotation!
  • file.log is moved to file.log.1.
  • A new file is created as file.log.
• Files: file.log (inode-B), file.log.1 (inode-A)
• refresh_watcher and update_watcher can be processed at the same time...

If refresh_watcher is processed before update_watcher

1. refresh_watcher
   • Find file.log (inode-B) as a new target.
   • OVERWRITE @tails[file.log] by Watcher(file.log, inode-B)
     • @tails[file.log] already has a watcher. We can't overwrite it here.
   • Result:
     • @tails[file.log] => Watcher(file.log, inode-B)
     • Watcher(file.log, inode-A) remains, but it does NOT EXIST in @tails.
       • This was causing handle leaks.
2. update_watcher
   • Watcher(file.log, inode-A) will call it's update_watcher
     • This will update the inode of file.log to inode-B.
     • Old implementation
       • Detach @tails[file.log] (Watcher(file.log, inode-B)) WRONG
         • This stops tailing the new file!!
     • This PR
       • Detach self (Watcher(file.log, inode-A)). CORRECT
       • This solves the problem both of stopping tailing and handle leaks.

[kattz-kawa]

Hi @daipom

Thank you for the disuccsing regarding #3614.
And I appreciate the adding co-authored sign on this PR's commit message.

@masaki-hatada and Me have checked and tested heavily last week.
It looks good.
There is no concern for us.
I hope that this PR merged in the future release for fixing incorrect file handle issue.

[masaki-hatada]

Looks good for me.

[daipom]

@kattz-kawa @masaki-hatada Thanks for your review!

[daipom]

@ashie Thanks for your review! I'll check it.

[ashie]

LGTM (above my comments are nitpicking)

[ashie]

Thanks to all contributors who help investigating this issue!

[ashie]

TODO:

• Apply it to v1.16 branch
• How about follow_inode false?

[daipom]

Thanks to all contributors who have given me much insight, especially @kattz-kawa, @masaki-hatada, and @garyzjq!
And thanks for your review!