Create tool that load dumped event

[ganmacs]

fluent-binlog-reader is a command line tool to read the data which is saved as Message Pack format.

Usage of fluent-binlog-reader's Commands

head

head is used to display the beginning of a text file.

$ fluent-binlog-reader head packed.log
2016-08-12T17:24:18+09:00       packed.log      {"message":"dummy"}
2016-08-12T17:24:18+09:00       packed.log      {"message":"dummy"}
2016-08-12T17:24:18+09:00       packed.log      {"message":"dummy"}
2016-08-12T17:24:18+09:00       packed.log      {"message":"dummy"}
2016-08-12T17:24:18+09:00       packed.log      {"message":"dummy"}

You can specify the number of events to display.

$ fluent-binlog-reader head -n 1 packed.log
2016-08-12T17:24:18+09:00       packed.log      {"message":"dummy"}

cat

cat reads files sequentially, writing them to standard output.

$ fluent-binlog-reader cat packed.log
2016-08-12T17:24:18+09:00       packed.log      {"message":"dummy"}

...

2016-08-12T17:24:18+09:00       packed.log      {"message":"dummy"}

formats

formats displays plugins that you can use.

$ fluent-binlog-reader formats
out_file
stdout
json
hash
msgpack
ltsv
csv
single_value

Global Options

--format

You can specify the formatter as shown below.

$ fluent-binlog-reader cat packed.log --format=json
{"message":"dummy"}
...
{"message":"dummy"}

Format params

formatter has configurable params.(e.g. CsvFormatter has delimiter, force_quotes, and fields)
You can pass these params by using -e option.

$ fluent-binlog-reader cat --formats=csv  -e fields=message packed.log
"dummy"
...
"dummy"

--plugin

You can add new plugins path.

# $ touch my_plugins/fomatter_csv2.rb
# $ vi my_plugins/fomatter_csv2.rb

$ fluent-binlog-reader cat formats=csv2 --plugin=my_plugins
...

TODO

• Subcommands
  • head
  • cat
  • formats
• adding plugins path
• specifed format
• specifed format params
• add tests

[tagomoris]

typo

[ganmacs]

head and cat can share code

Done.

[tagomoris]

nonsense def (just doing super).

[tagomoris]

I added some minor comments.
The last major problem is naming...
@repeatedly do you have any idea about naming?

[repeatedly]

Hmm... if this tool focuses on binary format, fluent-binlog-reader is good for me.
dumped seems too general because other text files, e.g. result of out_file, are also dumped.

[tagomoris]

looks good > fluent-binlog-reader

[ganmacs]

@tagomoris
I changed a command name unpacker -> binlog-reader

@repeatedly Thank you for your help!

[repeatedly]

Two directories for test formatter plugins are needed?
test/scripts/fluent/plugin/formatter/{formatter_test1,formatter_test2}.rb seems enough.

[repeatedly]

This check seems verbose.

if !@options[:plugin].empty?
  if dir = @options[:plugin].find { |d| !Dir.exist?(d) }
     usage "Directory #{dir} doesn't exist"
  else
    @options[:plugin].each do |d|
      Fluent::Plugin.add_plugin_dir(d)
    end
  end
end

[ganmacs]

@repeatedly

There are two tests
( https://github.com/fluent/fluentd/pull/1165/files#diff-5e97ad93fa2a92bc851090105261df21R329
and https://github.com/fluent/fluentd/pull/1165/files#diff-5e97ad93fa2a92bc851090105261df21R339) for -p option'.

The first one is a test when two directories are passed.
The second one is a test when a directory is passed.
So I think it is need two directories for a test.

[repeatedly]

@ganmacs Ah, I see. Got it.

[ganmacs]

@tagomoris @repeatedly
I updated. Please review the code again.

[tagomoris]

LGTM.