Quick links: Flags Verbs Functions Glossary Release docs

Scripting with Miller¶

Suppose you are often doing something like

mlr --icsv --opprint \
filter '$quantity != 20' \
then count-distinct -f shape \
then fraction -f count \
example.csv

shape    count count_fraction
triangle 3     0.3
square   4     0.4
circle   3     0.3

Typing this out can get a bit old, if the only thing that changes for you is the filename. Some options include:

On Linux/Mac/etc you can make a script with #!/bin/sh which invokes Miller as part of the shell-script body.
On Linux/Mac/etc you can make a script with #!/usr/bin/env mlr -s which invokes Miller.
On any platform you can put the reusable part of your command line into a text file (say myflags.txt), then mlr -s myflags-txt filename-which-varies.csv.

Let's look at examples of each.

Shell scripts¶

A shell-script option:

cat example-shell-script

#!/bin/bash
mlr --c2p \
  filter '$quantity != 20' \
  then count-distinct -f shape \
  then fraction -f count \
  -- "$@"

Key points here:

Use -- before "$@" at the end so that main-flags like --json won't be confused for options to the fraction verb.
Use "$@" at the end which means "all remaining arguments to the script".
Use chmod +x example-shell-script after you create one of these.

Then you can do

./example-shell-script example.csv

shape    count count_fraction
triangle 3     0.3
square   4     0.4
circle   3     0.3

cat example.csv | ./example-shell-script

shape    count count_fraction
triangle 3     0.3
square   4     0.4
circle   3     0.3

./example-shell-script --ojson example.csv

[
{
  "shape": "triangle",
  "count": 3,
  "count_fraction": 0.3
},
{
  "shape": "square",
  "count": 4,
  "count_fraction": 0.4
},
{
  "shape": "circle",
  "count": 3,
  "count_fraction": 0.3
}
]

./example-shell-script --ojson then filter '$count == 3' example.csv

[
{
  "shape": "triangle",
  "count": 3,
  "count_fraction": 0.3
},
{
  "shape": "circle",
  "count": 3,
  "count_fraction": 0.3
}
]

etc.

Miller scripts¶

Here instead of putting #!/bin/bash on the first line, we can put mlr directly:

cat ./example-mlr-s-script

#!/usr/bin/env mlr -s
--c2p
filter '$quantity != 20' # Here is a comment
then count-distinct -f shape
then fraction -f count

Points:

This is largely the same as a shell script.
Use chmod +x example-mlr-s-script after you create one of these.
You leave off the initial mlr since that's present on line 1.
You don't need all the backslashing for line-continuations.
You don't need the explicit -- or "$@".
All text from # to end of line is stripped out. If for any reason you need to suppress this, please use mlr --s-no-comment-strip in place of mlr -s.

Then you can do

./example-mlr-s-script example.csv

shape    count count_fraction
triangle 3     0.3
square   4     0.4
circle   3     0.3

cat example.csv | ./example-mlr-s-script

shape    count count_fraction
triangle 3     0.3
square   4     0.4
circle   3     0.3

./example-mlr-s-script --ojson example.csv

[
{
  "shape": "triangle",
  "count": 3,
  "count_fraction": 0.3
},
{
  "shape": "square",
  "count": 4,
  "count_fraction": 0.4
},
{
  "shape": "circle",
  "count": 3,
  "count_fraction": 0.3
}
]

./example-mlr-s-script --ojson then filter '$count == 3' example.csv

[
{
  "shape": "triangle",
  "count": 3,
  "count_fraction": 0.3
},
{
  "shape": "circle",
  "count": 3,
  "count_fraction": 0.3
}
]

Miller scripts on Windows¶

Both the previous options require executable mode with chmod, and a shebang line with #!..., which are unixisms.

One of the nice features of mlr -s is it can be done without a shebang line, and this works fine on Windows. For example:

cat example-mlr-s-script-no-shebang

--c2p
filter '$quantity != 20'
then count-distinct -f shape
then fraction -f count

Points:

Same as above, where the #! line isn't needed. (But you can include a #! line; mlr -s will simply see it as a comment line.).
As above, you don't need all the backslashing for line-continuations.
As above, you don't need the explicit -- or "$@".

Then you can do

mlr -s example-mlr-s-script-no-shebang example.csv

shape    count count_fraction
triangle 3     0.3
square   4     0.4
circle   3     0.3

mlr -s example-mlr-s-script-no-shebang --ojson example.csv

[
{
  "shape": "triangle",
  "count": 3,
  "count_fraction": 0.3
},
{
  "shape": "square",
  "count": 4,
  "count_fraction": 0.4
},
{
  "shape": "circle",
  "count": 3,
  "count_fraction": 0.3
}
]

mlr -s example-mlr-s-script-no-shebang --ojson then filter '$count == 3' example.csv

[
{
  "shape": "triangle",
  "count": 3,
  "count_fraction": 0.3
},
{
  "shape": "circle",
  "count": 3,
  "count_fraction": 0.3
}
]

and so on. See also Miller on Windows.