RFilter : as Awq, but in Ruby !

Posted on 21 August 2019 Updated on 30 October 2020

A little tool (400 LOC) for working on UNIX piping filtering.

No gem ! get file at Github : https://github.com/glurp/rfilter

Usage by pipe or by filenames arguments:

cat ddd | rfilter ‘expression’
or
rfilter [-F regexp-field-separator] ‘expression’ *.txt

With
expression = ruby code, special fonctions can be use : sum(arg),tohcount(a,1),plot()…
special variables are : _ (line splited); _0 (all line as string), _1 (first word), _2 , nol (no of line),

.

Speciale globals variables :
         $pending | at the end if filtering, pretty-print if defined (used by sum(),tohcount(),plot())….
         $line    | content of current line
         $oldline | contant of last line readed

Exemples:

Count number of file which have size bigger than 1K:

ls -l | rfilter ‘p _0 if _5.to_i>1024’ | rfilter ‘sum 1’

Count total size of files which have size bigger than 1K:

ls -l | rfilter ‘sell {_5.to_i>1024}’ | rfilter ‘sum _5’

Count LOC of shell-script files

find . -type f -name ‘.sh’ -exec cat {} \; | rfilter ‘sell {_0!=/^\s#/ && _0.size>0}’ | rfilter ‘sum 1’

Format lines if condition

ls -l a*.rb | rfilter “format_if(‘%15s | %10s’,_9,_5) { _5.to_i>1}”
a.rb | 864
abool.rb | 888
allways.rb | 698
anac.rb | 5777

Plot curve of http consommation by hours via nginx logs :

# First, examinaize the structude of log:

egrep “GET|POST” ssl_acces.log | rfilter show
_1 => ‘47.187.112.390’
_2 => ‘-‘
_3 => ‘-‘
_4 => ‘[18/Zey/2019:02:19:44’
_5 => ‘+0200]’
_6 => ‘”GET’
_7 => ‘/appliserveur/http/ddddd.cgi’
_8 => ‘HTTP/1.1″‘
_9 => ‘200’
_10 => ’74’
_11 => ‘”-“‘
_12 => ‘”Java/3.2.0_1″‘
…

# so hour is in field _4 :

egrep “GET|POST” ssl_acces.log | tail -10000 |
rfilter ‘st=_4[12..-1] ; t=Time.parse(st) ; tohcount(t.hour,1)’ |
rfilter ‘plot(_4)’