This post will be a collection of some patterns I typically use when working with Bash. There's not much in here yet, as I just now got the idea to share them ;-)
Execute in parallel but with worker limit
Suppose you have to load a bunch of CSV files into your database. Of course, you want to do this in parallel. But, if you have way more files than CPU cores you can easily overload the system. I typically use GNU parallels (see [1]) to get the job done. Note, that you possibly have to install it (CentOS 8: dnf install parallel
).
JOBS=24
PG_DB="psql -U postgres -h localhost -d some_database"
DATA="/path/to/data"
for F in $DATA/*.csv; do
sem -j $JOBS "$PG_DB -c '\COPY my_table FROM $F CSV'"
done
sem --wait
cat via SSH
Since I am on macOS, I am used to do cat file.txt | pbcopy
quite often to then paste it somewhere. However, most of the time I have some SSH open to some server I'm working on. It can get a bit difficult to copy files out to the clipboard. So, here is a shortcut I use:
ssh user@the-server.i-am.on "cat ~/that/open/file.txt" | pbcopy
Of course with SSH keys ;-)
Notes, Citations and Changelog
- Photo by Iva Muškić from Pexels
2020/10/06 - Added "cat via SSH"
[1]: O. Tange (2018): GNU Parallel 2018, Mar 2018, ISBN 9781387509881, DOI doi.org/10.5281/zenodo.1146014