Fat Stacks

Bash patterns I use

Bash patterns I use

Sebastian Dressler's photo
Sebastian Dressler
·

2 min read

This post will be a collection of some patterns I typically use when working with Bash. There's not much in here yet, as I just now got the idea to share them ;-)

Execute in parallel but with worker limit

Suppose you have to load a bunch of CSV files into your database. Of course, you want to do this in parallel. But, if you have way more files than CPU cores you can easily overload the system. I typically use GNU parallels (see [1]) to get the job done. Note, that you possibly have to install it (CentOS 8: dnf install parallel).

JOBS=24
PG_DB="psql -U postgres -h localhost -d some_database"
DATA="/path/to/data"

for F in $DATA/*.csv; do
    sem -j $JOBS "$PG_DB -c '\COPY my_table FROM $F CSV'"
done
sem --wait

cat via SSH

Since I am on macOS, I am used to do cat file.txt | pbcopy quite often to then paste it somewhere. However, most of the time I have some SSH open to some server I'm working on. It can get a bit difficult to copy files out to the clipboard. So, here is a shortcut I use:

ssh user@the-server.i-am.on "cat ~/that/open/file.txt" | pbcopy

Of course with SSH keys ;-)

Notes, Citations and Changelog

  • Photo by Iva Muškić from Pexels

  • 2020/10/06 - Added "cat via SSH"

  • [1]: O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881, DOI doi.org/10.5281/zenodo.1146014

Bashbasicspatterns