pax_global_header00006660000000000000000000000064132477460440014525gustar00rootroot0000000000000052 comment=c09c3a7df2464fc966c121a2a9a9d96f9fb0e98c nq-0.3.1/000077500000000000000000000000001324774604400121445ustar00rootroot00000000000000nq-0.3.1/.gitignore000066400000000000000000000000161324774604400141310ustar00rootroot00000000000000*~ fq nq ,*.* nq-0.3.1/COPYING000066400000000000000000000003331324774604400131760ustar00rootroot00000000000000nq is in the public domain. To the extent possible under law, Leah Neukirchen has waived all copyright and related or neighboring rights to this work. http://creativecommons.org/publicdomain/zero/1.0/ nq-0.3.1/Makefile000066400000000000000000000005441324774604400136070ustar00rootroot00000000000000ALL=nq fq tq CFLAGS=-g -Wall -O2 DESTDIR= PREFIX=/usr/local BINDIR=$(PREFIX)/bin MANDIR=$(PREFIX)/share/man all: $(ALL) clean: FRC rm -f nq fq check: FRC all prove -v ./tests install: FRC all mkdir -p $(DESTDIR)$(BINDIR) $(DESTDIR)$(MANDIR)/man1 install -m0755 $(ALL) $(DESTDIR)$(BINDIR) install -m0644 $(ALL:=.1) $(DESTDIR)$(MANDIR)/man1 FRC: nq-0.3.1/NEWS.md000066400000000000000000000011371324774604400132440ustar00rootroot00000000000000## 0.3.1 (2018-03-07) * Fix build on FreeBSD, OpenBSD and macOS. ## 0.3 (2018-03-06) * nq: add `-c` to clean job file when the process succeeded. * nq: avoid unnecessary quoting for the exec line. * Bugfix when `-q` was used with empty command lines. ## 0.2.2 (2017-12-21) * fq: fix when `$NQDIR` is set and inotify is used. (Thanks to Sebastian Reuße) * Support for NetBSD 7. ## 0.2.1 (2017-04-27) * fq: `-q` erroneously was on by default. ## 0.2 (2017-04-26) * fq: add `-n` to not wait * Support for platforms without O_DIRECTORY. * Support for SmartOS. ## 0.1 (2015-08-28) * Initial release nq-0.3.1/README.md000066400000000000000000000120541324774604400134250ustar00rootroot00000000000000## nq: queue utilities These small utilities allow creating very lightweight job queue systems which require no setup, maintenance, supervision, or any long-running processes. `nq` should run on any POSIX.1-2008 compliant system which also provides a working flock(2). Tested on Linux 2.6.37, Linux 4.1, OpenBSD 5.7, FreeBSD 10.1, NetBSD 7.0.2, Mac OS X 10.3 and SmartOS joyent_20160304T005100Z. The intended purpose is ad-hoc queuing of command lines (e.g. for building several targets of a Makefile, downloading multiple files one at a time, running benchmarks in several configurations, or simply as a glorified `nohup`), but as any good Unix tool, it can be abused for whatever you like. Job order is enforced by a timestamp `nq` gets immediately when started. Synchronization happens on file-system level. Timer resolution is milliseconds. No sub-second file system time stamps are required. Polling is not used. Exclusive execution is maintained strictly. Enforcing job order works like this: - every job has a flock(2)ed output file ala `,TIMESTAMP.PID` - every job starts only after all earlier flock(2)ed files are unlocked - Why flock(2)? Because it locks the file handle, which is shared across exec(2) with the child process (the actual job), and it will unlock when the file is closed (usually when the job terminates). You enqueue (get it?) new jobs using `nq CMDLINE...`. The job id is output (unless suppressed using `-q`) and `nq` detaches immediately, running the job in the background. STDOUT and STDERR are redirected into the log file. `nq` tries hard (but does not guarantee) to ensure the log file of the currently running job has +x bit set. Thus you can use `ls -F` to get a quick overview of the state of your queue. The "file extension" of the log file is actually the PID, so you can kill jobs easily. Before the job is started, it is the PID of `nq`, so you can cancel a queued job by killing it as well. Due to the initial `exec` line in the log files, you can resubmit a job by executing it as a shell command file, i.e. running `sh $jobid`. You can wait for jobs to finish using `nq -w`, possibly listing job ids you want to wait for; the default is all of them. Likewise, you can test if there are jobs which need to be waited upon using `-t`. By default, job ids are per-directory, but you can set `$NQDIR` to put them elsewhere. Creating `nq` wrappers setting `$NQDIR` to provide different queues for different purposes is encouraged. All these operations take worst-case quadratic time in the amount of lock files produced, so you should clean them regularly. ## Examples Build targets `clean`, `depends`, `all`, without occupying the terminal: % nq make clean % nq make depends % nq make all % fq ... look at output, can interrupt with C-c any time without stopping the build ... Simple download queue, accessible from multiple terminals: % mkdir -p /tmp/downloads % alias qget='NQDIR=/tmp/downloads nq wget' % alias qwait='NQDIR=/tmp/downloads fq -q' window1% qget http://mymirror/big1.iso window2% qget http://mymirror/big2.iso window3% qget http://mymirror/big3.iso % qwait ... wait for all downloads to finish ... As nohup replacement (The benchmark will run in background, every run gets a different output file, and the command line you ran is logged too.): % ssh remote remote% nq ./run-benchmark ,14f6f3034f8.17035 remote% ^D % ssh remote remote% fq ... see output, fq exits when job finished ... ## Assumptions `nq` will only work correctly when: - `$NQDIR` (respectively `.`) is writable. - `flock(2)` works in `$NQDIR` (respectively `.`). - `gettimeofday` behaves monotonic (using `CLOCK_MONOTONIC` would create confusing file names). Else job order can be confused and multiple tasks can run at once due to race conditions. - No other programs put files matching `,*` into `$NQDIR` (respectively `.`). ## nq helpers Two helper programs are provided: `fq` outputs the log of the currently running jobs, exiting when the jobs are done. If no job is running, the output of the last job is shown. `fq -a` shows the output of all jobs, `fq -q` only shows one line per job. `fq` uses `inotify` on Linux and falls back to polling for size change else. (`fq.sh` is a similar tool, not quite as robust, implemented as shell-script calling `tail`.) `tq` wraps `nq` and displays the `fq` output in a new tmux or screen window. (A pure shell implementation of `nq` is provided as `nq.sh`. It needs `flock` from util-linux, and only has a timer resolution of 1s. Lock files from `nq` and `nq.sh` should not be mixed.) ## Installation Use `make all` to build, `make install` to install relative to `PREFIX` (`/usr/local` by default). The `DESTDIR` convention is respected. You can also just copy the binaries into your `PATH`. You can use `make check` to run a simple test suite, if you have Perl's `prove` installed. ## Copyright nq is in the public domain. To the extent possible under law, Leah Neukirchen has waived all copyright and related or neighboring rights to this work. http://creativecommons.org/publicdomain/zero/1.0/ nq-0.3.1/_nq000066400000000000000000000007701324774604400126500ustar00rootroot00000000000000#compdef nq tq fq _nq_job() { compadd "$@" -- ${NQDIR:-.}/,*.*(:t) } _nq() { case "$service" in fq) _arguments -s -A : \ '-q[show only one line per job]' \ '-a[output for all jobs]' \ '*::job:_nq_job' ;; nq) _arguments -A : \ '-w[wait for jobs]:*:job:_nq_job' \ '-t[check jobs]:*:job:_nq_job' \ '(-):command name: _command_names -e' \ '*::arguments:_normal' ;; tq) _arguments : \ '(-):command name: _command_names -e' \ '*::arguments:_normal' ;; esac } _nq "$@" nq-0.3.1/fq.1000066400000000000000000000024141324774604400126350ustar00rootroot00000000000000.Dd August 25, 2015 .Dt FQ 1 .Os .Sh NAME .Nm fq .Nd job queue log viewer .Sh SYNOPSIS .Nm .Op Fl a .Op Fl n .Op Fl q .Op Ar job\ id ... .Sh DESCRIPTION .Nm is a simple utility for .Dq following the output of .Xr nq 1 jobs. .Pp Without arguments, the output of the currently running and queued as-of-now jobs is emitted; else the presented job ids are used. .Pp .Nm automatically terminates after the corresponding jobs are done. .Pp The options are as follows: .Bl -tag -width Ds .It Fl a Output all log files, even of already finished jobs. .It Fl n Don't wait for new output. Can be used to look at enqueued commands. .It Fl q Only print the first line of each job output (i.e. the .Li exec line). .El .Sh ENVIRONMENT .Bl -hang -width Ds .It Ev NQDIR Directory where lock files/job output resides, see .Xr nq 1 . .El .Sh EXIT STATUS .Ex -std .Sh INTERNALS On Linux, .Xr inotify 7 is used to monitor job output. On other operating systems, polling is used. .Sh SEE ALSO .Xr nq 1 , .Xr tq 1 .Sh AUTHORS .An Leah Neukirchen Aq Mt leah@vuxu.org .Sh LICENSE .Nm is in the public domain. .Pp To the extent possible under law, the creator of this work has waived all copyright and related or neighboring rights to this work. .Pp .Lk http://creativecommons.org/publicdomain/zero/1.0/ .\" .Sh BUGS nq-0.3.1/fq.c000066400000000000000000000077071324774604400127310ustar00rootroot00000000000000/* * fq [FILES...] - follow output of nq jobs, quitting when they are done * * To the extent possible under law, Leah Neukirchen * has waived all copyright and related or neighboring rights to this work. * http://creativecommons.org/publicdomain/zero/1.0/ */ #include #include #include #include #include #include #include #include #include #include #include #define DELAY 250000 #ifdef __linux__ #define USE_INOTIFY #endif #ifdef USE_INOTIFY #include char ibuf[8192]; #endif char buf[8192]; static int islocked(int fd) { if (flock(fd, LOCK_EX | LOCK_NB) == -1) { return (errno == EWOULDBLOCK); } else { flock(fd, LOCK_UN); return 0; } } static int alphabetic(const void *a, const void *b) { return strcmp(*(char **)a, *(char **)b); } int main(int argc, char *argv[]) { int i, fd, dirfd; off_t off, loff; ssize_t rd; int didsth = 0, seen_nl = 0; int opt = 0, aflag = 0, nflag = 0, qflag = 0; char *path; #ifdef USE_INOTIFY int ifd, wd; #endif close(0); while ((opt = getopt(argc, argv, "+anq")) != -1) { switch (opt) { case 'a': aflag = 1; break; case 'n': nflag = 1; break; case 'q': qflag = 1; break; default: fputs("usage: fq [-anq] [JOBID...]\n", stderr); exit(1); } } path = getenv("NQDIR"); if (!path) path = "."; #ifdef O_DIRECTORY dirfd = open(path, O_RDONLY | O_DIRECTORY); #else dirfd = open(path, O_RDONLY); #endif if (dirfd < 0) { perror("open dir"); exit(111); } if (optind == argc) { /* behave as if $NQDIR/,* was passed. */ DIR *dir; struct dirent *d; int len = 0; argc = 0; argv = 0; optind = 0; dir = fdopendir(dirfd); if (!dir) { perror("fdopendir"); exit(111); } while ((d = readdir(dir))) { if (d->d_name[0] != ',') continue; if (argc >= len) { len = 2*len + 1; argv = realloc(argv, len * sizeof (char *)); if (!argv) exit(222); } argv[argc] = strdup(d->d_name); if (!argv[argc]) exit(222); argc++; } qsort(argv, argc, sizeof (char *), alphabetic); } #ifdef USE_INOTIFY ifd = inotify_init(); if (ifd < 0) exit(111); #endif for (i = optind; i < argc; i++) { loff = 0; seen_nl = 0; fd = openat(dirfd, argv[i], O_RDONLY); if (fd < 0) continue; /* skip not running jobs, unless -a was passed, or we did not * output anything yet and are at the last argument. */ if (!aflag && !islocked(fd) && (didsth || i != argc - 1)) continue; write(1, "==> ", 4); write(1, argv[i], strlen(argv[i])); write(1, qflag ? " " : "\n", 1); didsth = 1; #ifdef USE_INOTIFY char fullpath[PATH_MAX]; snprintf(fullpath, sizeof fullpath, "%s/%s", path, argv[i]); wd = inotify_add_watch(ifd, fullpath, IN_MODIFY | IN_CLOSE_WRITE); if (wd == -1) { perror("inotify_add_watch"); exit(111); } #endif while (1) { off = lseek(fd, 0, SEEK_END); if (off < loff) loff = off; /* file truncated */ if (off == loff) { if (nflag && islocked(fd)) break; if (flock(fd, LOCK_EX | LOCK_NB) == -1 && errno == EWOULDBLOCK) { #ifdef USE_INOTIFY /* any inotify event is good */ read(ifd, ibuf, sizeof ibuf); #else /* poll for size change */ while (off == lseek(fd, 0, SEEK_END)) usleep(DELAY); #endif continue; } else { flock(fd, LOCK_UN); break; } } if (off - loff > sizeof buf) off = loff + sizeof buf; rd = pread(fd, &buf, off - loff, loff); if (qflag) { if (!seen_nl) { char *s; if ((s = memchr(buf, '\n', rd))) { write(1, buf, s+1-buf); seen_nl = 1; } else { write(1, buf, rd); } } } else { write(1, buf, rd); } loff += rd; } if (qflag && !seen_nl) write(1, "\n", 1); #ifdef USE_INOTIFY inotify_rm_watch(ifd, wd); #endif close(fd); } #ifdef USE_INOTIFY close(ifd); #endif return 0; } nq-0.3.1/fq.sh000077500000000000000000000004021324774604400131050ustar00rootroot00000000000000#!/bin/sh # fq - tail -F the queue outputs, quitting when the job finishes tailed=false for f in ${NQDIR:-.}/,*; do if ! nq -t $f; then tailed=true printf '==> %s\n' "$f" tail -F $f & p=$! nq -w $f kill $p fi done if ! $tailed; then cat $f fi nq-0.3.1/nq.1000066400000000000000000000117051324774604400126500ustar00rootroot00000000000000.Dd August 25, 2015 .Dt NQ 1 .Os .Sh NAME .Nm nq .Nd job queue utility .Sh SYNOPSIS .Nm .Op Fl c .Op Fl q .Ar command\ line ... .Nm .Fl t .Ar job\ id ... .Nm .Fl w .Ar job\ id ... .Sh DESCRIPTION The .Nm utility provides a very lightweight queuing system without requiring setup, maintenance, supervision or any long-running processes. .Pp Job order is enforced by a timestamp .Nm gets immediately when started. Synchronization happens on file-system level. Timer resolution is milliseconds. No sub-second file system time stamps are required. Polling is not used. Exclusive execution is maintained strictly. .Pp You enqueue(!) new jobs into the queue by running .Pp .Dl nq Ar command line ... .Pp The job id (a file name relative to .Ev NQDIR , which defaults to the current directory) is output (unless suppressed using .Fl q ) and .Nm detaches from the terminal immediately, running the job in the background. Standard output and standard error are redirected into the job id file. .Xr fq 1 can be used to conveniently watch the log files. .Pp The options are as follows: .Bl -tag -width Ds .It Fl c Clean up job id file when process exited with status 0. .It Fl q Suppress output of the job id after spawning new job. .It Fl t Enter .Em test mode : exit with status 0 when .Em all of the listed job ids are already done, else with status 1. .It Fl w Enter .Em waiting mode : wait in the foreground until .Em all listed job ids are done. .El .Sh ENVIRONMENT .Bl -hang -width Ds .It Ev NQDIR Directory where lock files/job output resides. Each .Ev NQDIR can be considered a separate queue. The current working directory is used when .Ev NQDIR is unset. .Ev NQDIR is created if needed. .It Ev NQJOBID The job id of the currently running job, exposed to the job itself. .El .Sh FILES .Nm owns all files in .Ev NQDIR (respectively .Pa \&. ) which start with .Dq Li \&, or .Dq Li \&., . These files are created according to the following scheme: .Pp .Dl ,hexadecimal-time-stamp.pid .Sh EXIT STATUS The .Nm utility exits 0 on success, and >0 if an error occurs; unless .Em test mode is used, in which case exit status 1 means there is a job running. .Pp On fatal errors, exit codes 111 and 222 are used. .Sh EXAMPLES Build .Xr make 1 targets .Ic clean , .Ic depends , .Ic all , without occupying the terminal: .Bd -literal -offset indent % nq make clean % nq make depends % nq make all % fq \&... look at output, can interrupt with C-c any time without stopping the build ... .Ed .Pp Simple download queue, accessible from multiple terminals: .Bd -literal -offset indent % alias qget='NQDIR=/tmp/downloads nq wget' % alias qwait='NQDIR=/tmp/downloads fq -q' window1% qget http://mymirror/big1.iso window2% qget http://mymirror/big2.iso window3% qget http://mymirror/big3.iso % qwait \&... wait for all downloads to finish ... .Ed .Pp As .Xr nohup 1 replacement (The benchmark will run in background, every run gets a different output file, and the command line you ran is logged too.): .Bd -literal -offset indent % ssh remote remote% nq ./run-benchmark ,14f6f3034f8.17035 remote% ^D % ssh remote remote% fq \&... see output, fq exits when job finished ... .Ed .Sh TRICKS The "file extension" of the log file is actually the PID of the job. .Nm runs all jobs in a separate process group, so you can kill an entire job process tree at once using .Xr kill 1 with a negative PID. Before the job is started, it is the PID of .Nm , so you can cancel a queued job by killing it as well. .Pp Thanks to the initial .Li exec line in the log files, you can resubmit a job by executing it as a shell command file, i.e. running .Pp .Dl sh Em job\ id .Pp Creating .Nm wrappers setting .Ev NQDIR to provide different queues for different purposes is encouraged. .Sh INTERNALS Enforcing job order works like this: .Bl -dash -compact .It every job has an output file locked using .Xr flock 2 and named according to .Sx FILES . .It every job starts only after all earlier flocked files are unlocked. .It the lock is released by the kernel after the job terminates. .El .Sh ASSUMPTIONS .Nm will only work correctly when: .Bl -dash .It .Ev NQDIR (respectively .Pa \&. ) is writable. .It .Xr flock 2 works correctly in .Ev NQDIR (respectively .Pa \&. ) . .It .Xr gettimeofday 2 behaves monotonic (using .Dv CLOCK_MONOTONIC would create confusing file names after reboot). .It No other programs put files matching .Li ,* into .Ev NQDIR (respectively .Pa \&. ) . .El .Sh SEE ALSO .Xr fq 1 , .Xr tq 1 . .Pp Alternatives to the .Nm system include .Xr batch 1 , .Xr qsub 1 , .Xr schedule 1 , .Xr srun 1 , and .Xr ts 1 . .\" .Sh STANDARDS .\" .Sh HISTORY .Sh AUTHORS .An Leah Neukirchen Aq Mt leah@vuxu.org .Sh CAVEATS All reliable queue status information is in main memory only, which makes restarting a job queue after a reboot difficult. .Sh LICENSE .Nm is in the public domain. .Pp To the extent possible under law, the creator of this work has waived all copyright and related or neighboring rights to this work. .Pp .Lk http://creativecommons.org/publicdomain/zero/1.0/ .\" .Sh BUGS nq-0.3.1/nq.c000066400000000000000000000154731324774604400127400ustar00rootroot00000000000000/* * nq CMD... - run CMD... in background and in order, saving output * -w ... wait for all jobs/listed jobs queued so far to finish * -t ... exit 0 if no (listed) job needs waiting * -q quiet, do not output job id * -c clean, don't keep output if job exited with status 0 * * - requires POSIX.1-2008 and having flock(2) * - enforcing order works like this: * - every job has a flock(2)ed output file ala ",TIMESTAMP.PID" * - every job starts only after all earlier flock(2)ed files finished * - the lock is released when job terminates * - no sub-second file system time stamps are required, jobs are started * with millisecond precision * - we try hard to make the currently running ,* file have +x bit * - you can re-queue jobs using "sh ,jobid" * * To the extent possible under law, Leah Neukirchen * has waived all copyright and related or neighboring rights to this work. * http://creativecommons.org/publicdomain/zero/1.0/ */ /* for FreeBSD. */ #define _WITH_DPRINTF #if defined(__sun) && defined(__SVR4) && !defined(HAVE_DPRINTF) #define NEED_DPRINTF #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef NEED_DPRINTF #include static int dprintf(int fd, const char *fmt, ...) { char buf[128]; // good enough for usage in nq va_list ap; int r; va_start(ap, fmt); r = vsnprintf(buf, sizeof buf, fmt, ap); va_end(ap); if (r >= 0 && r < sizeof buf) return write(fd, buf, r); return -1; } #endif static void swrite(int fd, char *str) { size_t l = strlen(str); if (write(fd, str, l) != l) { perror("write"); exit(222); } } static void write_execline(int fd, int argc, char *argv[]) { int i; char *s; swrite(fd, "exec"); for (i = 0; i < argc; i++) { if (!strpbrk(argv[i], "\001\002\003\004\005\006\007\010" "\011\012\013\014\015\016\017\020" "\021\022\023\024\025\026\027\030" "\031\032\033\034\035\036\037\040" "`^#*[]=|\\?${}()'\"<>&;\177")) { swrite(fd, " "); swrite(fd, argv[i]); } else { swrite(fd, " '"); for (s = argv[i]; *s; s++) { if (*s == '\'') swrite(fd, "'\\''"); else write(fd, s, 1); } swrite(fd, "'"); } } } int main(int argc, char *argv[]) { int64_t ms; int dirfd = 0, lockfd = 0; int opt = 0, cflag = 0, qflag = 0, tflag = 0, wflag = 0; int pipefd[2]; char lockfile[64]; pid_t child; struct timeval started; struct dirent *ent; DIR *dir; /* timestamp is milliseconds since epoch. */ gettimeofday(&started, NULL); ms = (int64_t)started.tv_sec*1000 + started.tv_usec/1000; while ((opt = getopt(argc, argv, "+chqtw")) != -1) { switch (opt) { case 'c': cflag = 1; break; case 'w': wflag = 1; break; case 't': tflag = 1; break; case 'q': qflag = 1; break; case 'h': default: goto usage; } } if (!tflag && !wflag && argc <= optind) { usage: swrite(2, "usage: nq [-c] [-q] [-w ... | -t ... | CMD...]\n"); exit(1); } char *path = getenv("NQDIR"); if (!path) path = "."; if (mkdir(path, 0777) < 0) { if (errno != EEXIST) { perror("mkdir"); exit(111); } } #ifdef O_DIRECTORY dirfd = open(path, O_RDONLY | O_DIRECTORY); #else dirfd = open(path, O_RDONLY); #endif if (dirfd < 0) { perror("dir open"); exit(111); } if (tflag || wflag) { snprintf(lockfile, sizeof lockfile, ".,%011" PRIx64 ".%d", ms, getpid()); goto wait; } if (pipe(pipefd) < 0) { perror("pipe"); exit(111); }; /* first fork, parent exits to run in background. */ child = fork(); if (child == -1) { perror("fork"); exit(111); } else if (child > 0) { char c; /* wait until child has backgrounded. */ close(pipefd[1]); read(pipefd[0], &c, 1); exit(0); } close(pipefd[0]); /* second fork, child later execs the job, parent collects status. */ child = fork(); if (child == -1) { perror("fork"); exit(111); } else if (child > 0) { int status; /* output expected lockfile name. */ snprintf(lockfile, sizeof lockfile, ",%011" PRIx64 ".%d", ms, child); if (!qflag) dprintf(1, "%s\n", lockfile); close(0); close(1); close(2); /* signal parent to exit. */ close(pipefd[1]); wait(&status); lockfd = openat(dirfd, lockfile, O_RDWR | O_APPEND); if (lockfd < 0) { perror("open"); exit(222); } fchmod(lockfd, 0600); if (WIFEXITED(status)) { dprintf(lockfd, "\n[exited with status %d.]\n", WEXITSTATUS(status)); if (cflag && WEXITSTATUS(status) == 0) unlinkat(dirfd, lockfile, 0); } else { dprintf(lockfd, "\n[killed by signal %d.]\n", WTERMSIG(status)); } exit(0); } close(pipefd[1]); /* create and lock lockfile. since this cannot be done in one step, use a different filename first. */ snprintf(lockfile, sizeof lockfile, ".,%011" PRIx64 ".%d", ms, getpid()); lockfd = openat(dirfd, lockfile, O_CREAT | O_EXCL | O_RDWR | O_APPEND, 0600); if (lockfd < 0) { perror("open"); exit(222); } if (flock(lockfd, LOCK_EX) < 0) { perror("flock"); exit(222); } /* drop leading '.' */ renameat(dirfd, lockfile, dirfd, lockfile+1); /* block until rename is committed */ fsync(dirfd); write_execline(lockfd, argc, argv); if (dup2(lockfd, 2) < 0 || dup2(lockfd, 1) < 0) { perror("dup2"); exit(222); } wait: if ((tflag || wflag) && argc - optind > 0) { /* wait for files passed as command line arguments. */ int i; for (i = optind; i < argc; i++) { int fd; if (strchr(argv[i], '/')) fd = open(argv[i], O_RDWR); else fd = openat(dirfd, argv[i], O_RDWR); if (fd < 0) continue; if (flock(fd, LOCK_EX | LOCK_NB) == -1 && errno == EWOULDBLOCK) { if (tflag) exit(1); flock(fd, LOCK_EX); /* sit it out. */ } fchmod(fd, 0600); close(fd); } } else { dir = fdopendir(dirfd); if (!dir) { perror("fdopendir"); exit(111); } again: while ((ent = readdir(dir))) { /* wait for all ,* files. */ if (ent->d_name[0] == ',' && strcmp(ent->d_name, lockfile+1) < 0) { int fd; fd = openat(dirfd, ent->d_name, O_RDWR); if (fd < 0) continue; if (flock(fd, LOCK_EX | LOCK_NB) == -1 && errno == EWOULDBLOCK) { if (tflag) exit(1); flock(fd, LOCK_EX); /* sit it out. */ close(fd); rewinddir(dir); goto again; } fchmod(fd, 0600); close(fd); } } closedir(dir); /* closes dirfd too. */ } if (tflag || wflag) exit(0); /* ready to run. */ swrite(lockfd, "\n\n"); fchmod(lockfd, 0700); close(lockfd); setenv("NQJOBID", lockfile+1, 1); setsid(); execvp(argv[optind], argv+optind); perror("execvp"); return 222; } nq-0.3.1/nq.sh000077500000000000000000000033341324774604400131240ustar00rootroot00000000000000#!/bin/sh # nq CMD... - run CMD... in background and in order, saving output to ,* files # # - needs POSIX sh + util-linux flock(1) (see nq.c for portable version) # - when run from tmux, display output in a new window (needs # GNU tail, C-c to abort the job.) # - we try hard to make the currently running ,* file have +x bit # - enforcing order works like this: # - every job has a flock(2)ed file # - every job starts only after all earlier flock(2)ed files finished # - the lock is released when job terminates # # To the extent possible under law, Leah Neukirchen # has waived all copyright and related or neighboring rights to this work. # http://creativecommons.org/publicdomain/zero/1.0/ if [ -z "$NQ" ]; then export NQ=$(date +%s) "$0" "$@" & c=$! ( # wait for job to finish flock -x .,$NQ.$c -c true flock -x ,$NQ.$c -c true chmod -x ,$NQ.$c ) & exit fi us=",$NQ.$$" exec 9>>.$us # first flock(2) the file, then make it known under the real name flock -x 9 mv .$us $us printf "## nq $*" 1>&9 if [ -n "$TMUX" ]; then tmux new-window -a -d -n '<' -c '#{pane_current_path}' \ "trap true INT QUIT TERM EXIT; tail -F --pid=$$ $us || kill $$; printf '\n[%d exited, ^D to exit.]\n' $$; cat >/dev/null" fi waiting=true while $waiting; do waiting=false # this must traverse in lexical (= numerical) order: # check all older locks are released for f in ,*; do # reached the current lock, good to go [ $f = $us ] && break if ! flock -x -n $f -c "chmod -x $f"; then # force retrying all locks again; # an earlier lock could just now have really appeared waiting=true flock -x $f -c true fi done done printf '\n' 1>&9 chmod +x $us exec "$@" 2>&1 1>&9 nq-0.3.1/tests000077500000000000000000000046541324774604400132450ustar00rootroot00000000000000#!/bin/sh : ${NQ:=../nq} : ${FQ:=../fq} set -e check() { msg=$1 shift if eval "$@" 2>/dev/null 1>&2; then printf 'ok - %s\n' "$msg" else printf 'not ok - %s\n' "$msg" false fi true } printf '1..36\n' rm -rf test.dir mkdir test.dir ( cd test.dir printf '# nq tests\n' check 'fails with no arguments' ! $NQ check 'succeeds enqueuing true' 'f=$($NQ true)' sleep 1 check 'generated a lockfile' test -f $f check 'lockfile contains exec line' grep -q exec.*nq.*true $f check 'lockfile contains status line' grep -q exited.*status.*0 $f check 'lockfile is not executable' ! test -x $f ) rm -rf test.dir mkdir test.dir ( cd test.dir printf '# queue tests\n' check 'enqueing true' f1=$($NQ true) check 'enqueing sleep 500' f2=$($NQ sleep 500) check 'first job is done already' $NQ -t $f1 check 'not all jobs are done already' ! $NQ -t check 'running job is executable' test -x $f2 check 'running job not done already' ! $NQ -t $f check 'can kill running job' kill ${f2##*.} sleep 1 check 'killed job is not executable anymore' ! test -x $f2 check 'killed job contains status line' grep -q killed.*signal.*15 $f2 ) rm -rf test.dir mkdir test.dir ( cd test.dir printf '# env tests\n' check 'enqueing env' f1=$($NQ env) $NQ -w check 'NQJOBID is set' grep -q NQJOBID=$f1 $f1 ) rm -rf test.dir mkdir test.dir ( cd test.dir printf '# killing tests\n' check 'spawning four jobs' 'f1=$($NQ sleep 100)' check 'spawning four jobs' 'f2=$($NQ sleep 1)' check 'spawning four jobs' 'f3=$($NQ sleep 100)' check 'spawning four jobs' 'f4=$($NQ sleep 1)' check 'killing first job' kill ${f1##*.} check 'killing third job' kill ${f3##*.} check 'second job is running' ! $NQ -t $f2 $NQ -w $f2 check 'fourth job is running' ! $NQ -t $f4 check 'all jobs are done' $NQ -w ) rm -rf test.dir mkdir test.dir ( cd test.dir printf '# fq tests\n' check 'spawning four jobs' 'f1=$($NQ sleep 100)' check 'spawning four jobs' 'f2=$($NQ echo two)' check 'spawning four jobs' 'f3=$($NQ sleep 300)' check 'spawning four jobs' 'f4=$($NQ sleep 400)' check 'fq tracks first job' '($FQ ,* & p=$!; sleep 1; kill $p) | sed 3q | grep -q sleep.*100' check 'killing first job' kill ${f1##*.} check 'killing fourth job' kill ${f4##*.} sleep 1 check 'fq tracks third job' '($FQ ,* & p=$!; sleep 1; kill $p) | sed 3q | grep -q sleep.*300' check 'killing third job' kill ${f3##*.} sleep 1 check 'fq outputs last job when no job running' '$FQ ,* | sed 3q | grep -q sleep.*400' ) rm -rf test.dir nq-0.3.1/tq000077500000000000000000000010501324774604400125120ustar00rootroot00000000000000#!/bin/sh # tq CMD... - tmux/screen wrapper for nq to display output in new window set -e s=$(nq "$@") p=${s##*.} printf '%s\n' "$s" if [ -n "$p" ]; then if [ -n "$TMUX" ]; then tmux new-window -a -d -n '<' -c '#{pane_current_path}' \ "trap true INT QUIT TERM EXIT; fq $s || kill $p; printf '[%d exited, ^D to exit.]\n' $p; cat >/dev/null" elif [ -n "$STY" ]; then screen -t '<' sh -c "trap true INT QUIT TERM EXIT; fq $s || kill $p printf '[%d exited, ^D to exit.]\n' $p; cat >/dev/null" screen -X other fi fi nq-0.3.1/tq.1000066400000000000000000000016751324774604400126630ustar00rootroot00000000000000.Dd August 25, 2015 .Dt TQ 1 .Os .Sh NAME .Nm tq .Nd job queue wrapper for tmux/screen .Sh SYNOPSIS .Nm .Ar command\ line ... .Sh DESCRIPTION .Nm is a tiny wrapper around the .Xr nq 1 job queue which automatically spawns a corresponding .Xr fq 1 watching process in a new .Xr tmux 1 or .Xr screen 1 window. .Pp You can cancel the .Xr nq 1 job by pressing .Ic C-c in the job output window. .Pp After the job has finished, the window will close on .Ic C-d . .Sh ENVIRONMENT .Bl -hang -width Ds .It Ev NQDIR Directory where lock files/job output resides, see .Xr nq 1 . .El .Sh EXIT STATUS .Ex -std .Sh SEE ALSO .Xr fq 1 , .Xr nq 1 , .Xr screen 1 , .Xr tmux 1 .Sh AUTHORS .An Leah Neukirchen Aq Mt leah@vuxu.org .Sh LICENSE .Nm is in the public domain. .Pp To the extent possible under law, the creator of this work has waived all copyright and related or neighboring rights to this work. .Pp .Lk http://creativecommons.org/publicdomain/zero/1.0/ .\" .Sh BUGS