bash_tricks

Tricks I learned for BASH scripting during my masters thesis

21.9.2017

• miller plugin for bash

# convert a csv to a tabular format
mlr --icsv --opprint cat example.csv

# use cut to only show specific fields, call the field by typing the column name!
$ mlr --icsv --opprint cut -f columnname,shape example.csv

• whenever you want to parse out a section of a header, column name, etc... an effective method is to use sed to find and replace a part of the header with a tab then cut out the new field

sed -e 's/thingtocut/\t/' | cut -fx

# sed -e is for finding and replacing an expression
# \t is for tabular
# cut -fx -> x is for the field you would like to cut

• LC=all learn about this... it makes characters on your keyboard smaller bites of informationso that job run faster...

26.9.2017

• process subsitution allows for mulitple process to be inputed in another command, here is an example:

# before variable X and Y are joined, they are sorted by certain columns
# within each <() a process can take place before it is fed into another function
join -1 2 -2 4 -v2 <(sort -k2,2 $X) <(sort -k3,3 $Y)

28.9.2017

• parallel examples

29.9.2017

• good explanation for rsync
  • you can copy files and directories to and from remotes servers from a local

 rsync -avze ssh user@remote.server:/path/to/directory/or/file/ /new/place/
 # -a: copy files recursively and maintain grou adn user ownershio
 # -v: print whats going on
 # -h: human readable
 # -z: compress the data sent
 # -e: us ssh

rsync -avze ssh mschecht@bigmem-1:/scratch/mschecht/clusters_no_rejects/GU_clusters_no_rejects.ts

2.10.16

-randomly suffle a file then pick the first 100 lines:

sort -R input | head -n 100 > output

shuf -n N input > output