Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port parser filter from fluent-plugin-parser. fix #1189 #1191

Merged
merged 19 commits into from
Nov 10, 2016
Merged

Port parser filter from fluent-plugin-parser. fix #1189 #1191

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
96664ac
Port parser filter from fluent-plugin-parser. fix #1189
repeatedly Aug 29, 2016
0f14fa6
Apply v0.14 style. Use plugin helpers.
repeatedly Aug 30, 2016
a11d4fb
Add missing parser parameters to compat_parameters
repeatedly Aug 31, 2016
a0c8f7e
Call Timecop.return in teardown to restore Time
repeatedly Sep 1, 2016
b166787
Use filter_with_time API
repeatedly Sep 2, 2016
9359eea
Remove duplicated compat parameters
repeatedly Nov 4, 2016
8e8b92d
Use error stream instead of warning logs
repeatedly Nov 4, 2016
c993be5
Replace time_parse with reserve_time
repeatedly Nov 4, 2016
5ffcb87
Update test to use new configuration format
repeatedly Nov 4, 2016
2c82887
Use String#scrub instead of own routine
repeatedly Nov 10, 2016
4aef481
Remove useless config_section definition
repeatedly Nov 10, 2016
33cac26
Improve invalid string handling to reduce duplicated code
repeatedly Nov 10, 2016
d08a34d
Recent update removes ValuesParser from v0.14 parsers
repeatedly Nov 10, 2016
5cb93a4
Use default_tag for driver run
repeatedly Nov 10, 2016
cb3dee6
Remove deleted paramters tests
repeatedly Nov 10, 2016
163dc5c
Rename suppress warning test to unmached pattern
repeatedly Nov 10, 2016
71d700c
Improve tests
repeatedly Nov 10, 2016
82953c5
Forget to update test to use event_time instead of to_i
repeatedly Nov 10, 2016
f64bb77
Remove useless shutdown option from run
repeatedly Nov 10, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use default_tag for driver run
  • Loading branch information
@repeatedly
repeatedly committed Nov 10, 2016
commit 5cb93a4244864e029547e26514d9a0de28dd8af7
78 changes: 39 additions & 39 deletions test/plugin/test_filter_parser.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,12 +118,12 @@ def test_configure
def test_filter
d1 = create_driver(CONFIG)
time = Time.parse("2012-01-02 13:14:15").to_i
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use EventTime instance as time for v0.14 plugins: event_time("2012-01-02 13:14:15")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a test code especially for integer time, test case name should show it.

d1.run do
d1.feed(@tag, time, {'message' => '12 20120402182059'})
d1.feed(@tag, time, {'message' => '34 20120402182100'})
d1.feed(@tag, time, {'message' => '56 20120402182100'})
d1.feed(@tag, time, {'message' => '78 20120402182101'})
d1.feed(@tag, time, {'message' => '90 20120402182100'})
d1.run(default_tag: @tag) do
d1.feed(time, {'message' => '12 20120402182059'})
d1.feed(time, {'message' => '34 20120402182100'})
d1.feed(time, {'message' => '56 20120402182100'})
d1.feed(time, {'message' => '78 20120402182101'})
d1.feed(time, {'message' => '90 20120402182100'})
end
filtered = d1.filtered
assert_equal 5, filtered.length
Expand Down Expand Up @@ -159,9 +159,9 @@ def test_filter
format /^(?<x>.)(?<y>.) (?<t>.+)$/
])
time = Fluent::EventTime.from_time(@default_time) # EventTime emit test
d2.run do
d2.feed(@tag, time, {'data' => '12 20120402182059'})
d2.feed(@tag, time, {'data' => '34 20120402182100'})
d2.run(default_tag: @tag) do
d2.feed(time, {'data' => '12 20120402182059'})
d2.feed(time, {'data' => '34 20120402182100'})
end
filtered = d2.filtered
assert_equal 2, filtered.length
Expand All @@ -188,10 +188,10 @@ def test_filter
</parse>
])
time = Time.parse("2012-04-02 18:20:59").to_i
d3.run do
d3.feed(@tag, time, {'data' => '12 20120402182059'})
d3.feed(@tag, time, {'data' => '34 20120402182100'})
d3.feed(@tag, time, {'data' => 'xy 20120402182101'})
d3.run(default_tag: @tag) do
d3.feed(time, {'data' => '12 20120402182059'})
d3.feed(time, {'data' => '34 20120402182100'})
d3.feed(time, {'data' => 'xy 20120402182101'})
end
filtered = d3.filtered
assert_equal 2, filtered.length
Expand All @@ -205,10 +205,10 @@ def test_filter
</parse>
])
time = Time.parse("2012-04-02 18:20:59").to_i
d3x.run do
d3x.feed(@tag, time, {'data' => '12 20120402182059'})
d3x.feed(@tag, time, {'data' => '34 20120402182100'})
d3x.feed(@tag, time, {'data' => 'xy 20120402182101'})
d3x.run(default_tag: @tag) do
d3x.feed(time, {'data' => '12 20120402182059'})
d3x.feed(time, {'data' => '34 20120402182100'})
d3x.feed(time, {'data' => 'xy 20120402182101'})
end
filtered = d3x.filtered
assert_equal 3, filtered.length
Expand All @@ -220,9 +220,9 @@ def test_filter
</parse>
])
time = Time.parse("2012-04-02 18:20:59").to_i
d4.run do
d4.feed(@tag, time, {'data' => '{"xxx":"first","yyy":"second"}', 'xxx' => 'x', 'yyy' => 'y'})
d4.feed(@tag, time, {'data' => 'foobar', 'xxx' => 'x', 'yyy' => 'y'})
d4.run(default_tag: @tag) do
d4.feed(time, {'data' => '{"xxx":"first","yyy":"second"}', 'xxx' => 'x', 'yyy' => 'y'})
d4.feed(time, {'data' => 'foobar', 'xxx' => 'x', 'yyy' => 'y'})
end
filtered = d4.filtered
assert_equal 1, filtered.length
Expand All @@ -235,9 +235,9 @@ def test_filter
</parse>
])
time = @default_time.to_i
d4x.run do
d4x.feed(@tag, time, {'data' => '{"xxx":"first","yyy":"second"}', 'xxx' => 'x', 'yyy' => 'y'})
d4x.feed(@tag, time, {'data' => 'foobar', 'xxx' => 'x', 'yyy' => 'y'})
d4x.run(default_tag: @tag) do
d4x.feed(time, {'data' => '{"xxx":"first","yyy":"second"}', 'xxx' => 'x', 'yyy' => 'y'})
d4x.feed(time, {'data' => 'foobar', 'xxx' => 'x', 'yyy' => 'y'})
end
filtered = d4x.filtered
assert_equal 2, filtered.length
Expand Down Expand Up @@ -271,9 +271,9 @@ def test_filter
def test_filter_ltsv
d = create_driver(CONFIG_LTSV)
time = @default_time.to_i
d.run do
d.feed(@tag, time, {'data' => "xxx:first\tyyy:second", 'xxx' => 'x', 'yyy' => 'y'})
d.feed(@tag, time, {'data' => "xxx:first\tyyy:second2", 'xxx' => 'x', 'yyy' => 'y'})
d.run(default_tag: @tag) do
d.feed(time, {'data' => "xxx:first\tyyy:second", 'xxx' => 'x', 'yyy' => 'y'})
d.feed(time, {'data' => "xxx:first\tyyy:second2", 'xxx' => 'x', 'yyy' => 'y'})
end
filtered = d.filtered
assert_equal 2, filtered.length
Expand All @@ -292,9 +292,9 @@ def test_filter_ltsv

d = create_driver(CONFIG_LTSV + %[reserve_data yes])
time = @default_time.to_i
d.run do
d.feed(@tag, time, {'data' => "xxx:first\tyyy:second", 'xxx' => 'x', 'yyy' => 'y'})
d.feed(@tag, time, {'data' => "xxx:first\tyyy:second2", 'xxx' => 'x', 'yyy' => 'y'})
d.run(default_tag: @tag) do
d.feed(time, {'data' => "xxx:first\tyyy:second", 'xxx' => 'x', 'yyy' => 'y'})
d.feed(time, {'data' => "xxx:first\tyyy:second2", 'xxx' => 'x', 'yyy' => 'y'})
end
filtered = d.filtered
assert_equal 2, filtered.length
Expand Down Expand Up @@ -479,10 +479,10 @@ def test_filter_inject_hash_value_field
def test_time_should_be_reserved(conf)
t = Time.now.to_i
d = create_driver(conf)
d.run do
d.feed(@tag, t, {'data' => '{"time":1383190430, "f1":"v1"}'})
d.feed(@tag, t, {'data' => '{"time":"1383190430", "f1":"v1"}'})
d.feed(@tag, t, {'data' => '{"time":"2013-10-31 12:34:03 +0900", "f1":"v1"}'})
d.run(default_tag: @tag) do
d.feed(t, {'data' => '{"time":1383190430, "f1":"v1"}'})
d.feed(t, {'data' => '{"time":"1383190430", "f1":"v1"}'})
d.feed(t, {'data' => '{"time":"2013-10-31 12:34:03 +0900", "f1":"v1"}'})
end
filtered = d.filtered
assert_equal 3, filtered.length
Expand Down Expand Up @@ -511,9 +511,9 @@ def test_filter_invalid_time_data
time = Time.now.to_i
d = create_driver(CONFIG_INVALID_TIME_VALUE)
assert_nothing_raised {
d.run do
d.feed(@tag, time, {'data' => '{"time":[], "f1":"v1"}'})
d.feed(@tag, time, {'data' => '{"time":"thisisnottime", "f1":"v1"}'})
d.run(default_tag: @tag) do
d.feed(time, {'data' => '{"time":[], "f1":"v1"}'})
d.feed(time, {'data' => '{"time":"thisisnottime", "f1":"v1"}'})
end
}
filtered = d.filtered
Expand Down Expand Up @@ -721,9 +721,9 @@ class EnabledSuppressParseErrorLogTest < self
def test_nothing_raised
flexmock(@d.instance.router).should_receive(:emit_error_event).
with(String, Integer, Hash, ParserError).once
@d.run do
@d.feed(@tag, Fluent::EventTime.now.to_i, {'message' => INVALID_MESSAGE})
@d.feed(@tag, Fluent::EventTime.now.to_i, {'message' => VALID_MESSAGE})
@d.run(default_tag: @tag) do
@d.feed(Fluent::EventTime.now.to_i, {'message' => INVALID_MESSAGE})
@d.feed(Fluent::EventTime.now.to_i, {'message' => VALID_MESSAGE})
end
end
end
Expand Down