diff options
Diffstat (limited to 'README')
| -rw-r--r-- | README | 146 |
1 files changed, 146 insertions, 0 deletions
@@ -0,0 +1,146 @@ + + GNU Parallel + + https://www.gnu.org/software/parallel/ + + += Presentation of GNU Parallel = + +GNU Parallel is a shell tool for executing jobs in parallel using one +or more computers. A job can be a single command or a small script +that has to be run for each of the lines in the input. The typical +input is a list of files, a list of hosts, a list of users, a list of +URLs, or a list of tables. A job can also be a command that reads from +a pipe. GNU Parallel can then split the input and pipe it into +commands in parallel. + +If you use xargs and tee today you will find GNU Parallel very easy to +use as GNU Parallel is written to have the same options as xargs. If +you write loops in shell, you will find GNU Parallel may be able to +replace most of the loops and make them run faster by running several +jobs in parallel. + +GNU Parallel makes sure output from the commands is the same output as +you would get had you run the commands sequentially. This makes it +possible to use output from GNU Parallel as input for other programs. + +See the cheat sheet for examples that cover most of the daily usage: +www.gnu.org/s/parallel/parallel_cheat.pdf + + += 10 seconds installation = + +For security reasons it is recommended you use your package manager to +install. But if you cannot do that then you can use this 10 seconds +installation. + +The 10 seconds installation will try to do a full installation; if +that fails, a personal installation; if that fails, a minimal +installation. + + $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \ + fetch -o - http://pi.dk/3 ) > install.sh + $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a + 12345678 883c667e 01eed62f 975ad28b 6d50e22a + $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0 + cc21b4c9 43fd03e9 3ae1ae49 e28573c0 + $ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf + 79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224 + fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35 + $ bash install.sh + +This will literally install faster than reading the rest of this +document. + + += Full installation = + +Full installation of GNU Parallel is as simple as: + + wget https://ftpmirror.gnu.org/parallel/parallel-20240222.tar.bz2 + wget https://ftpmirror.gnu.org/parallel/parallel-20240222.tar.bz2.sig + gpg parallel-20240222.tar.bz2.sig + bzip2 -dc parallel-20240222.tar.bz2 | tar xvf - + cd parallel-20240222 + ./configure && make && sudo make install + + += Personal installation = + +If you are not root you can add ~/bin to your path and install in +~/bin and ~/share: + + wget https://ftpmirror.gnu.org/parallel/parallel-20240222.tar.bz2 + wget https://ftpmirror.gnu.org/parallel/parallel-20240222.tar.bz2.sig + gpg parallel-20240222.tar.bz2.sig + bzip2 -dc parallel-20240222.tar.bz2 | tar xvf - + cd parallel-20240222 + ./configure --prefix=$HOME && make && make install + +Or if your system lacks 'make' you can simply copy src/parallel +src/sem src/niceload src/sql to a dir in your path. + + += Minimal installation = + +If you just need parallel and do not have 'make' installed (maybe the +system is old or Microsoft Windows): + + wget https://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel + chmod 755 parallel + cp parallel sem + mv parallel sem dir-in-your-$PATH/bin/ + + += Installation on compute servers = + +If you are developing your script to run on a remote server, that does +not have GNU Parallel installed, but you have it installed on you +development machine, then you can use `parallel --embed`. + + parallel --embed > newscript.sh + +Just edit the last lines of newscript.sh and copy it to the compute +server. + + += Test the installation = + +After this you should be able to do: + + parallel -j0 ping -nc 3 ::: qubes-os.org gnu.org freenetproject.org + +This will send 3 ping packets to 3 different hosts in parallel and print +the output when they complete. + +Watch the intro video for a quick introduction: +https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 + +Walk through the tutorial (man parallel_tutorial). You command line +will love you for it. + +When using programs that use GNU Parallel to process data for +publication please cite: + + Tange, O. (2024, February 22). GNU Parallel 20240222 ('Навальный'). + Zenodo. https://doi.org/10.5281/zenodo.10719803 + +Copyright (C) 2007-2022 Ole Tange, http://ole.tange.dk and Free +Software Foundation, Inc. + + += New versions = + +New versions will be released at: https://ftp.gnu.org/gnu/parallel/ + + += Dependencies = + +GNU Parallel should work with a normal full Perl installation. However, +if you system has split up Perl into multiple packages then these are +the important ones: + + opkg install perlbase-getopt perlbase-ipc procps-ng-ps perlbase-mime + +SPDX-FileCopyrightText: 2021-2022 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc. +SPDX-License-Identifier: GPL-3.0-or-later |