2018-07-01 Pádraig Brady version 8.30 * NEWS: Record release date. 2018-06-30 Pádraig Brady tests: standardize perl usage in tests * tests/cp/fiemap-FMR.sh: Ensure perl is parameterized to $PERL, and ensure require_perl_ is used, so tests are skipped appropriately. * tests/cp/preserve-gid.sh: Likewise. * tests/du/long-from-unreadable.sh: Likewise. * tests/misc/env-S-script.sh: Likewise. * tests/misc/sort-benchmark-random.sh: Likewise. * tests/rm/deep-2.sh: Likewise. maint: copy: avoid new static analyzer warnings * src/copy.c (copy_internal): Use the lint protected src_mode, rather than accessing the src_sb again. Also unconditionally populate src_sb when !x->move_mode and in lint mode. Reported by Kamil Dudka with coverity and clang analyzer. maint: fix recent stale comments and spelling mistakes * doc/coreutils.texi: s/seperator/separator/. * tests/misc/env-S.pl: Likewise. * src/env.c: Fix stale comment. 2018-06-27 Pádraig Brady maint: disable overly agressive sc_gitignore_redundant * cfg.mk (sc_gitignore_redundant): Disabled for now as too aggressive flagging entries like /lib/arg-nonnull.h in a newly checked out repo. env: adjust diagnostics provided for shebang usage * src/env.c (main): Don't process '-' specially since that causes an issue on the openbsd getopt implementation where a lone '-' is now processed as an option, and anyway it doesn't particuarly help diagnosing common shebang usage issues. Also don't restrict the extra diagnostics for shebang usage to the case with 3 arguments, as further arguments can be passed to a script. * tests/misc/env-S.pl: Adjust accordingly. 2018-06-27 Assaf Gordon tests: accept getopt errors without single-quotes On OpenBSD 6.2, invalid single options produce error messages without single quotes: $ ./src/chroot -/ chroot: unknown option -- / As opposed to other systems: ./src/chroot: invalid option -- '/' Modify the grep search to accept this. * tests/misc/usage_vs_getopt.sh (checkprg): Change the grep pattern to accomodate no-single-quotes cases. 2018-06-27 Pádraig Brady tests: fix false failures when perl not available * tests/local.mk: Reference the stub that skips perl tests, with the correct path. tests: fix false failure with limited shebang lines * tests/misc/env-S-script.sh: Provide a wrapper to emulate shebang processing, but without length limits, which are 127 on Linux for example. maint: update gnulib to latest * gnulib: Update to latest, which incorporates a thread linking fix from Bruno Haible, which was seen on newer Ubuntu systems. 2018-06-27 Assaf Gordon tests: remove unused Data::Dumper perl module The module is not needed anymore (was used during development). Despite being a Perl core module, platforms like CentOS don't install it by default. Reported by Bruno Haible at https://lists.gnu.org/r/coreutils/2018-06/msg00093.html. * tests/misc/csplit-suppress-matched.pl: Remove Data::Dumper. 2018-06-24 Carlos Santos maint: fix -Werror=suggest-attribute=malloc in expr.c Add attribute 'malloc' to mpz_get_str to prevent the following on GCC 8.1.1 src/expr.c:117:1: error: function might be candidate for attribute 'malloc' if it is known to return normally [-Werror=suggest-attribute=malloc] mpz_get_str (char const *str, int base, mpz_t z) ^~~~~~~~~~~ cc1: all warnings being treated as errors * src/expr.c (mpz_get_str): Add _GL_ATTRIBUTE_MALLOC. 2018-06-24 Pádraig Brady maint: update gnulib to latest * gnulib: Update to latest. * .gitignore: Add new entries. * bootstrap.conf: Enable wchar-single, which will enable more efficient replacements of wcwidth and mbrtowc, as we indicate that the charset will no change between invocations of these functions. maint: sync longlong.h from gmp repo * src/longlong.h: Sync changes. No functional change. maint: avoid false positive in src/fs-magic-compare * src/local.mk (fs_normalize_perl_subst): `make src/fs-magic-compare` was reporting incorrectly that AFS was not being handled. Add a mapping to our KAFS identifier. * .gitignore: Add intermediate files from `make src/fs-magic-compare` 2018-06-23 Bernhard Voelker tests: initialize fail=0 to avoid "unary operator expected" errors With an uninitialized variable 'fail', the unquoted use like test $fail = 1 lead to the shell error "unary operator expected". The uninitialized 'fail' variable was a side effect of https://git.sv.gnu.org/cgit/gnulib.git/commit/?id=e91c0d4f9 which was pulled into coreutils-v8.26 with https://git.sv.gnu.org/cgit/coreutils.git/commit/?id=ef9650170 Coreutils test code relied and relies on 'fail' to be initialized, so initialize that variable here. * tests/local.mk (TESTS_ENVIRONMENT): Initialize fail=0. 2018-06-21 Jim Meyering maint: do not depend directly on gnulib's now-unused ftello module * bootstrap.conf (gnulib_modules): Remove ftello, since it is no longer used directly, since v8.9-11-geab97b307. 2018-06-20 Pádraig Brady tests: provide an option to relax the need for gdb * tests/rm/r-root.sh: gdb provides extra protection, but is not strictly necessary. So provide an option for maintainers to relax the requirements. rm: add --preserve-root=all to protect mounts * src/remove.c (rm_fts): With the --preserve-root=all extension, reject command line arguments that are mount points. * src/remove.h (rm_options): Add preserve_all_root to store config. * src/mv.c (rm_option_init): Init preserve_all_root to false. * src/rm.c (main): Init preserve_all_root as per option. (usage): Describe the new option. * src/remove.c (rm_fts): Lookup the parent device id, and reject the cli argument if a separate file system. * tests/rm/one-file-system.sh: Add a test case. * NEWS: Mention the new feature. 2018-06-20 Adam Borowski cp: add --reflink=never to force standard copy mode This mode is currently the default, but most if not all users of reflink-capable filesystems want --reflink=auto, which is often encapsulated into an alias. Adding --reflink=never allows overriding such an alias. * doc/coreutils.texi (cp invocation): Describe the new option. * src/cp.c: Support --reflink=never. * tests/cp/reflink-auto.sh: Add a test case. * NEWS: Mention the new feature. 2018-06-20 Assaf Gordon env: add -S/--split-string option Adopted from FreeBSD's env(1), useful for specifing multiple parameters on a shebang (#!) script line, e.g: #!/usr/bin/env -S perl -w -T Discussed in https://lists.gnu.org/r/coreutils/2018-04/msg00011.html * src/env.c (valid_escape_sequence,escape_char,scan_varname, extract_varname,validate_split_str,build_argv, parse_split_string): New functions. (main): Process new option and call parse_split_string. (usage): Mention new option. * tests/misc/env-S.pl: Test new option from the command line. * tests/misc/env-S-script.sh: Test new option from shebang scripts. * tests/local.mk (all_tests): Add new tests. * man/env.x (OPTIONS): Show a brief example of -S usage and point to the full documentation for more information. * doc/coreutils.texi (env invocation): Detail usage of -S/--split-string option. * NEWS: Mention new option. 2018-06-20 Assaf Gordon env: add -v/--debug option Prints verbose information about each step: $ env -v -uFOO -C /tmp BAR=BAZ date -u env: unset: FOO env: setenv: BAR=BAZ env: chdir: '/tmp' env: executing: date env: arg[0]= ‘date’ env: arg[1]= ‘-u’ Sun Apr 22 08:52:30 UTC 2018 Inspired by FreeBSD's env(1). * src/env.c (usage): Mention new option. (main): Print debug information if requested. * NEWS: Mention new option. * doc/coreutils.texi (env invocation): Mention -v/--debug. 2018-06-20 Assaf Gordon maint: refactor unsetenv call in env Keep unset envvars (-uFOO) in an array for later deletion, instead of reiterating over argv. Done in preparation for '-S string' feature. Related to '-u' discussion in https://lists.gnu.org/r/coreutils/2018-04/msg00013.html * src/env.c (append_unset_var,unset_envvars): New functions. (main): Use new functions. 2018-06-20 Kaxandra Labat ls: ignore case when coloring file extensions * src/ls.c (get_color_indicator): s/STREQ_LEN/c_strncasecmp/ * src/dircolors.hin: Remove a now redundant entry. * tests/ls/color-ext.sh: Add a new test. * tests/local.mk: Reference the new test. * NEWS: Mention the change in behavior. 2018-06-20 Pádraig Brady md5sum,b2sum,sha*sum: support -z,--zero option * doc/coreutils.texi (md5sum invocation): Describe the new option, and how it's not supported by --check, and how it disables escaping. * src/md5sum.c (delim): A new global to parmeterize the out delimiter. (main): Don't enable file name escaping with -z, and output '\0'. * tests/misc/md5sum-newline.pl: Add a test case. * NEWS: Mention the new feature. 2018-06-20 Pádraig Brady wc: optimize processing of ASCII in multi byte locales ===== Benchmark setup (on GNU/Linux) ==== $ yes áááááááááááááááááááá | head -n100000 > mbc.txt $ yes 12345678901234567890 | head -n100000 > num.txt ===== Before ==== $ time src/wc -Lm < mbc.txt real 0m0.186s $ time src/wc -m < mbc.txt real 0m0.186s $ time src/wc -Lm < num.txt real 0m0.055s $ time src/wc -m < num.txt real 0m0.056s ==== After ==== $ time src/wc -Lm < mbc.txt real 0m0.196s $ time src/wc -m < mbc.txt real 0m0.173s $ time src/wc -Lm < num.txt real 0m0.031s $ time src/wc -m < num.txt real 0m0.028s * src/wc.c (wc): Only call wide variant functions like iswprint() and wcwidth() for non is_basic() characters. I.E. non ISO C "basic character set" characters. This is especially significant on OSX where wcwidth() is very expensive (about 10x in tests). * NEWS: Mention the improvement. Suggested by Eric Fischer. 2018-06-14 Paul Eggert doc: port test.1 to doclifter * man/test.x: Use \& instead of quoting (Bug#31803). 2018-06-13 Paul Eggert doc: port man pages to doclifter Problem reported by Eric S. Raymond (Bug#31803). * man/test.x: Add SYNOPSIS section, since help2man understandably gets confused by the square brackets. * src/ln.c (usage): Omit parenthetical "(Nth form)" in usage, as it confuses doclifter. 2018-06-03 Pádraig Brady cp: preserve existing permissions with --no-preserve=mode This issue was introduced in commit v8.19-145-g24ebca6 * src/copy.c (copy_internal): With --no-preserve=mode, only reset permissions for newly created files. (copy_reg): Likewise. * NEWS: Mention the fix. * tests/cp/preserve-mode.sh: Add a test case. Fixes https://bugs.gnu.org/31675 2018-05-29 Pádraig Brady tests: fix periodic false failure in month alignment * tests/ls/abmon-align.sh: Base relative month adjustment from the middle of the month, to avoid failures due to months being repeated. Fixes https://bugs.gnu.org/31644 2018-05-25 Bjarni Ingi Gislason doc: formatting fixes in "du.x" and "rm.x" Avoid warnings from: groff -b -e -mandoc -T utf8 -rF0 -t -w w -z * man/du.x: Change ".BR" to ".B" if there is only one argument. Protect an end-of-sentence indicator (.?!) with '\&' if it does not mean an end of a sentence. Change '--' to '\-\-' if it indicates an option. * man/rm.x: Change '\=' to '='. 2018-05-17 Pádraig Brady cp: with --force; replace self referential symlinks * src/copy.c (copy_internal): Don't fail immediately upon getting ELOOP when running stat() on the destination, rather proceeding if -f specified, allowing the link to be removed. If the loop is not in the final component of the destination path, we still fail but at the subsequent unlink() stage. * doc/coreutils.texi (cp invocation): Adjust wording to say that --force doesn't work with dangling links, rather than all links that can't be traversed. * tests/cp/thru-dangling.sh: Add a test case. * NEWS: Mention the change in behavior. Discussed in https://bugs.gnu.org/31335 2018-05-15 Pádraig Brady cp: fix symlink checks when overwriting files Ensure this _does_ recreate the symlink Given "path1" and "path2" are on different devices. $ touch "path1/file" $ cd path2/; ln -s path1/file $ cp -dsf path1/file . Ensure this does _not_ overwrite file $ touch file $ ln -s file l1 $ cp -sf l1 file * src/copy.c (same_file_ok): Remove device ids from consideration, instead deferring to future EXDEV with --link or allowing the first case above to work. Also ensure that we do not exist this function too early, when the destination file is not a symlink, which protects against the second case. * tests/cp/cross-dev-symlink.sh: Add a test for the first case. * tests/cp/same-file.sh: Add a test for the second case above. * NEWS: Mention the bug fixes. * THANKS.in: Mention the reporters who also analyzed the code. Fixes https://bugs.gnu.org/31364 2018-05-15 Pádraig Brady cp: ensure --remove-destination doesn't traverse symlinks * src/cp.c (target_directory_operand): Allow through inaccessible arguments with -f or --remove. * doc/coreutils.texi (cp invocation): Clarify that -f doesn't directly impact the removal of non-traversable symlinks. * tests/cp/dir-rm-dest.sh: Test the new behavior. * tests/cp/thru-dangling.sh: Enforce -f behavior wrt symlinks. * NEWS: Mention the bug fix. Fixes https://bugs.gnu.org/31335 2018-05-14 Pádraig Brady maint: make chmod/chgrp/chown leak free under valgrind * src/chmod.c: Deallocate the mode change array in dev mode. * src/chown.c: Make chopt_free() actually deallocate, but only call in dev mode. * src/chgrp.c: Likewise. doc: improve formatting of nl --help * src/nl.c (usage): Better delineate the information. 2018-05-14 Paul Eggert who: simplify port to GCC 8 * src/who.c (make_id_equals_comment): Use simpler workaround for GCC bug 85602. Suggested by Martin Sebor in: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85602#c3 2018-05-03 Pádraig Brady build: make GCC 8 adjustments more portable * src/chown-core.h (chopt_free): Just define away this noop. * src/chown-core.c (chopt_free): Remove the empty implementation. 2018-05-03 Paul Eggert build: update gnulib submodule to latest maint: port to GCC 8 * src/chown-core.h (chopt_free, gid_to_name, uid_to_name): No longer const. * src/make-prime-list.c (xalloc): Add malloc attribute. * src/who.c (make_id_equals_comment): Work around GCC bug 85602 by using mempcpy rather than strncat. Although the old code was correct, strncat raises so many hackles that it’s not worth maintaining its use here. maint: remove strpbrk module * bootstrap.conf (gnulib_modules): Remove obsolete module strpbrk. build: update gnulib submodule to latest 2018-04-20 Pádraig Brady doc: retroactively adjust info about tail and closed output * NEWS: Expand on the 8.28 description of how tail more responsively reacts to closed output, and move from "Improvements" to "Changed behavior". * cfg.mk (old_NEWS_hash): Regenerate. Fixes https://bugs.gnu.org/31225 2018-04-18 Pádraig Brady doc: timeout --help: mention 0 DURATION disables timeout * src/timeout.c (usage): Mention that a duration of 0 disables the associated timeout, which is both concise info and useful functionality as timeouts are frequently configured. 2018-04-05 Eric Blake doc: retroactively document -e/-u addition in NEWS Prompted by https://bugs.gnu.org/31045 * NEWS: Update 8.8 blurb to mention other split additions. * cfg.mk (old_NEWS_hash): Regenerate. 2018-04-03 Paul Eggert doc: Clarify octal bits in permissions * doc/perm.texi (Numeric Modes): Briefly explain octal. Reorder description to make it more intuitive (Bug#29069). 2018-03-28 Tobias Stoeckmann cut: improve large file support on 32 bit Increase max range from SIZE_MAX to UINTMAX_MAX, which will allow cut to support line lengths up to the max file size on all systems. The inherent SIZE_MAX limitation in cut was removed with the enhancements in https://bugs.gnu.org/13127. Also numfmt gets similarly increased --field ranges due to shared code. * src/cut.c: s/size_t/uintmax_t/. * src/numfmt.c: Likewise. * src/set-fields.c: Likewise. * src/set-fields.h: Likewise. * tests/misc/cut-huge-range.sh: Adjust accordingly. * tests/misc/numfmt.pl: Likewise. * NEWS: Mention the improvement. 2018-03-27 Pádraig Brady tests: avoid a recent syntax-check failure * tests/ls/a-option.sh: s/framework_failure/&_/. 2018-03-27 Paul Eggert ls: -A now overrides -a Problem reported by Karl Berry (Bug#30963). * NEWS: Mention this. * src/ls.c (decode_switches): Implement this. * tests/ls/a-option.sh: New file. * tests/local.mk (all_tests): Add it. 2018-03-24 Roland Hieber doc: fix two typos in github templates * .github/ISSUE_TEMPLATE.txt: Fix typo "coreitils" in the URL to the bug tracker. * .github/PULL_REQUEST_TEMPLATE.txt: Likewise. 2018-03-16 Pádraig Brady ls: increase the allowed abmon width from 5 to 12 This will impact relatively few languages, and will make Arabic or Catalan etc. output unambiguous abbreviated month names. * src/ls.c (MAX_MON_WIDTH): Increase from 5 to 12. * NEWS: Mention the bug fix. * tests/ls/abmon-align.sh: Augment to check for ambiguous output. Fixes https://bugs.gnu.org/30814 2018-03-14 Brent Petit stat,tail: add support for the EXFS file system Enhanced XFS (EXFS) is a version of XFS maintained by HPE. EXFS uses a unique magic number to allow the use of community XFS, and EXFS filesystems at the same time. * src/stat.c (human_fstype): Add file system ID definition, and use "exfs" as the name. * NEWS: Mention the Improvement. 2018-03-06 Paul Eggert build: update gnulib submodule to latest 2018-03-05 Paul Eggert stat: work around IBM xlC bug Problem reported by John Wiersba (Bug#30718) * src/stat.c (human_time): Avoid giving an integer constant expression a name, as it runs afoul of a bug in IBM XL C/C++ for AIX 12.01.0000.0002. 2018-03-04 Bernhard Voelker maint: adjust email address of Keith Thompson in THANKS.in * THANKS.in (Keith Thompson): Update email address as requested by himself at https://lists.gnu.org/r/coreutils/2018-03/msg00004.html 2018-02-24 Pádraig Brady cp: set appropriate default permissions for special files This issue was introduced in commit v8.19-145-g24ebca6 * src/copy.c (copy_internal): When setting default permissions to use with --no-preserve=mode, only set executable bits for directories or sockets. * NEWS: Mention the fix. * tests/cp/preserve-mode.sh: Add a test case. Fixes https://bugs.gnu.org/30534 2018-01-21 Pádraig Brady doc: use consistent example format in manual * doc/coreutils.texi: Use @example consistently as we don't need the smaller or fixed width representation. This is especially true for the synopsis of commands. @smallexample is rendered left aligned for HTML which is awkward to read with the center aligned main content. 2018-01-10 Paul Eggert mv: clarify ‘mv -n A A’ change mv: fewer syscalls for ‘mv a b’ This builds on a previous patch for mv atomicity (Bug#29961). It merely improves performance; it does not fix bugs. * src/copy.h (struct cp_options): New members last_file, rename_errno. * src/copy.c (copy_internal): Support new rename_errno member for the copy options. Avoid calling stat when new members suggest it’s not needed. (cp_options_default): Initialize new members. * src/mv.c: Include renameat2.h. (main): With two arguments, first call ‘renamat2 (AT_FDCWD, "a", AT_FDCWD, "b", RENAME_NOREPLACE)’. Use its results to skip remaining processing if possible; for example, if it succeeds there is no need to stat either "a" or "b". Also, set x.last_file when it is the last file to rename. mv: improve -n atomicity Problem reported by Kamil Dudka (Bug#29961). * NEWS: Mention this. * src/copy.c: Include renameat2.h. (copy_internal): If mv, try renameat2 first thing, with RENAME_NOREPLACE. If this works, skip most of the remaining code. Also, fail quickly if it fails with EEXIST, and we are using -n. 2018-01-09 Michael Orlitzky doc: warn about following symlinks recursively in chown/chgrp In both chown and chgrp (which shares its code with chown), operating on symlinks recursively has a window of vulnerability where the destination user or group can change the target of the operation. Warn about combining the --dereference, --recursive, and -L flags. * doc/coreutils.texi (warnOptDerefWithRec): Add macro. (node chown invocation): Add it to --dereference and -L. (node chgrp invocation): Likewise. See also: CVE-2017-18018 2018-01-05 Paul Eggert cp: remove ASSIGN_BASENAME_STRDUPA * src/cp.c (do_copy): Just use ASSIGN_STRDUPA, as this simplifies the code and uses less memory. 2018-01-04 Paul Eggert mv: -n overrides -u * NEWS: Mention these fixes. * doc/coreutils.texi (cp invocation, mv invocation): Mention that -n is silent, and that it overrides -u. * src/cp.c, src/mv.c (main): -n overrides -u. 2018-01-03 Paul Eggert tr: add -A, for compatibility with AIX tr Problem reported by Michael (Bug#29946). * src/tr.c (main): Add undocumented -A option. 2018-01-03 Michael Orlitzky doc: clarify chown/chgrp --dereference defaults * doc/coreutils.texi: the documentation for the --dereference flag of chown/chgrp states that it is the default mode of operation. Document that this is only the case when operating non-recursively. 2018-01-02 Pádraig Brady tests: avoid false failure with xargs on AIX * tests/misc/shred-remove.sh: AIX xargs defaults to using '_' to indicate end of input, thus ignoring it. Rather than specifying -E to avoid this behavior, simplify by removing sed and xargs usage. 2018-01-01 Pádraig Brady maint: update all copyright year number ranges Run "make update-copyright" and then... * gnulib: Update to latest with copyright year adjusted. * tests/init.sh: Sync with gnulib to pick up copyright year. * bootstrap: Likewise. * tests/sample-test: Adjust to use the single most recent year. 2017-12-27 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.29 * NEWS: Record release date. 2017-12-23 Pádraig Brady tests: avoid false failure on AIX 7.2 * tests/tail-2/pipe-f.sh: Close stdout in a subshell to ensure the current shell isn't impacted. Subsequent piped commands like `echo foo | blah` were seen to fail due to the previous closing of stdout. Reported by Assaf Gordon. doc: describe recent build checks for 32 bit time_t * README: Document the new handling of 32 bit time_t, with examples of how to build in 64 bit mode on AIX. Also mention that GNU make is desired on AIX due to its mishandling of the "[" target. Suggested by Assaf Gordon. 2017-12-21 Pádraig Brady tests: fix recent portability issues on solaris 10 * tests/misc/ptx.pl: Escape the '^' character which is otherwise considered as a line continuation character. * tests/misc/shred-remove.sh: sed doesn't support \n. maint: remove reference to excluded changelog item * build-aux/git-log-fix: Remove old entry. 2017-12-20 Pádraig Brady maint: add doc/coverage to .gitignore * .gitignore: Ignore the generated coverage report. doc: remove older ChangeLog items * Makefile.am: Update the oldest documented version to 8.20 which is now about 5 years old. 2017-12-17 Bernhard Voelker doc: mention which privileges are needed to chmod POSIX specification for chmod(1): https://pubs.opengroup.org/onlinepubs/9699919799/utilities/chmod.html * doc/coreutils.texi (chmod invocation): Add a sentence about who can change the file mode bits of a file - (almost) a copy from what POSIX requires. Fixes https://bugs.gnu.org/29207. 2017-12-16 Pádraig Brady tests: fix recent regressions with dash * tests/misc/timeout.sh: dash outputs the "Killed" message to stderr rather than the terminal. * tests/misc/usage_vs_getopt.sh: dash doesn't yet support the POSIX proposed $'...' shell quoting syntax. build: avoid a signed overflow warning in ptx * src/ptx.c (fix_output_parameters): GCC 6.3.1 with ./configure --enable-single-binary would give: error: assuming signed overflow does not occur when simplifying conditional to constant [-Werror=strict-overflow] if (file_index > 0) So change the type of file_index to signed (size_t). 2017-12-10 Bernhard Voelker maint: adjust for the renamed nstrfime gnulib module * bootstrap.conf: s/strftime/nstrfrime/. 2017-12-10 Pádraig Brady build: update gnulib submodule to latest * gnulib: Update with various build/test fixes. tests: fix false failure in new dd/nocache_eof test * test/dd/nocache_eof.sh: Also handle fadvise64_64 which is used on 32 bit x86. Note strace internally maps fadvise64_64 to {arm,xtensa}_fadvise64_64. tail: fix tailing non seekable files on certain systems * src/tail.c (tail_bytes): On systems were blksize_t is unsigned and the same size or wider than off_t (android for example), our initialized (off_t) -1 would be promoted to unsigned before comparison, and thus fail to follow the appropriate path. * tests/tail-2/tail-c.sh: Add a test case. * NEWS: Mention the fix. This issue was introduced in commit v8.23-47-g2662702 Reported at https://github.com/termux/termux-app/issues/233 build: avoid build failure without sys/mtio.h * m4/jm-macros.m4: Check for the header. * src/dd.c: Avoid the workaround where the header is not available (on non glibc systems). * src/shred.c: Likewise. doc: reorganize ls -k and --time-style help * src/ls.c (usage): Clarify -k only applies to -s usage and directory 'total' lines. Move the description of TIME_STYLE out of the option section as it was awkward to read and write there within 80 columns. 2017-12-09 Pádraig Brady doc: clarify numeric setuid handling in chmod man page * man/chmod.x: Update the information to state one can clear the setuid and setgid bits for directories numerically using an additional leading '0' or a leading '='. That has been supported since v8.15-64-g8931cdb. Fixes https://bugs.gnu.org/29390 doc: shred: change 'truncate' to the more descriptive 'deallocate' * doc/coreutils.texi (shred invocation): s/truncate/deallocate/. * src/shred.c (usage): Likewise. Fixes https://bugs.gnu.org/29317 doc: clarify that cp --force may recreate files * doc/coreutils.texi (cp invocation): The language used to describe recreating the file was a little confusing as it mentioned opening a removed file. Fixes https://bugs.gnu.org/29315 2017-12-04 Kamil Dudka doc: fix default QUOTING_STYLE for %N format of stat(1) * doc/coreutils.texi (stat invocation): The default value of QUOTING_STYLE for the %N format of 'stat --printf' is 'shell-escape-always'. Fixes https://bugs.gnu.org/29563 Reported by Christian Groessler at https://bugzilla.redhat.com/1520399#c3 2017-12-02 Jean Delvare tests: make ls/block-size more readable * tests/ls/block-size.sh: The output of the test was hard to read. Add comments saying what we are testing to make it easier to understand. 2017-11-29 Bernhard Voelker Pádraig Brady tests: verify usage vs. getopt Verify that all options mentioned in usage are actually recognized by the program. * tests/misc/usage_vs_getopt.sh: Add test. * tests/local.mk (all_tests): Reference it. 2017-11-29 Pádraig Brady readlink: remove superfluous comma from usage output * src/readlink.c (usage): Remove ',' after --quiet option. 2017-11-29 Bernhard Voelker all: use consistent diagnostics for unknown long options Previously, e.g. cksum failed to output the offending unknown long option: $ cksum --unknown-opt cksum: invalid option -- '-' Try 'cksum --help' for more information. i.e., it tried to diagnose '-' as short option. Instead, it should diagnose the unknown long option: $ cksum --unknown-opt cksum: unrecognized option '--unknown-opt' Try 'cksum --help' for more information. * src/cksum.c (long_options): Add struct with null entry only. (main): Use it in the getopt_long call. * src/dd.c: Likewise. * src/hostid.c: Likewise. * src/hostname.c: Likewise. * src/link.c: Likewise. * src/logname.c: Likewise. * src/nohup.c: Likewise. * src/sleep.c: Likewise. * src/tsort.c: Likewise. * src/unlink.c: Likewise. * src/uptime.c: Likewise. * src/users.c: Likewise. * src/whoami.c: Likewise. * src/yes.c: Likewise. * NEWS (Improvements): Mention the fix. 2017-11-28 Pádraig Brady test: fix issues with tests/cp/preserve-mode.sh * tests/cp/preserve-mode.sh: This was the only use of awk, which may not be available on the system resulting in an ineffective test. Also the permissions bits for directories were not being checked at all. build: update gnulib submodule to latest * gnulib: Update with various build/test fixes. build: update gnulib submodule to latest * gnulib: Update including various build fixes. 2017-11-27 Bernhard Voelker timeout: also support short -v option * src/timeout.c (main): Add short option character 'v' to getopt_long call. * tests/misc/timeout.sh: Run the test both for the long and the short option. 2017-11-24 Pádraig Brady dd: support iflag=direct with arbitrary sized files * src/dd.c (iread): Handle read error with a non-aligned file offset in the O_DIRECT case. This is not an issue on XFS at least, but on EXT4 the final read will return EINVAL rather than the expected 0 to indicate EOF. * tests/dd/direct.sh: Test the iflag=direct case also. * NEWS: Mention the improvement. 2017-11-23 Pádraig Brady timeout: add --verbose to diagnose timeouts This is useful as handling in shell is complicated with the varying exit status in the --kill-after case. * src/timeout.c (main): Handle '-v' and store COMMAND for the diagnostic. (cleanup): Diagnose the signal name before sending. (usage): Document -v, --verbose. * doc/coreutils.texi (timeout invocation): Likewise. * tests/misc/timeout.sh: Add a test case. * NEWS: Mention the new feature Fixes https://bugs.gnu.org/21760 2017-11-18 Pádraig Brady tail: seek to the end of block devices * src/tail.c (tail_bytes): Try lseek(..., SEEK_END) when we can't determine the file size. * tests/tail-2/end-of-device.sh: Add a new root only test. * tests/local.mk: Reference the new test. * NEWS: Mention the improvement. Paul Eggert suggested using lseek() (rather than ioctl(BLKGETSIZE64)). Fixes https://bugs.gnu.org/29259 2017-11-14 Bernhard Voelker maint: include the module year2038 from gnulib * bootstrap.conf (gnulib_modules): Add 'year2038' to ensure that time_t is 64-bit (and thus works after 2038). Suggested by Bruno Haible in https://lists.gnu.org/r/bug-gnulib/2017-11/msg00022.html 2017-11-14 Bernhard Voelker maint: update gnulib to latest * gnulib: Update - mainly for the recent year2038 changes. * tests/init.sh: Update from gnulib/tests/init.sh. 2017-11-09 Assaf Gordon doc: add github issue/pull-request templates These templates instruct contributors not to use github, and instead use the upstream GNU development resources. Discussed in http://lists.gnu.org/archive/html/coreutils/2017-11/msg00007.html . * .github/ISSUE_TEMPLATE.txt, .github/PULL_REQUEST_TEMPLATE.txt: New files. 2017-11-07 Jim Meyering maint: make hook script reject "/archive/html" in lists.gnu.org URLS * scripts/git-hooks/commit-msg: Require the abbreviated "/r/" form in any log message URL. maint: shorten https://lists.gnu.org/archive/html/... links Each /archive/html/ part can be replace with /r/. Run this to induce the change: git grep -l archive/html|xargs perl -pi -e 's,/archive/html/,/r/,g' * TODO: Perform that substitution. * bootstrap: Likewise. * src/sort.c (sequential_sort): Likewise. * src/tail.c (tail_file): Likewise. * tests/misc/sort-merge-fdlimit.sh: Likewise. * tests/misc/stty-row-col.sh: Likewise. * tests/misc/unexpand.pl: Likewise. * tests/rm/readdir-bug.sh: Likewise. * tests/tail-2/inotify-rotate.sh: Likewise. 2017-11-06 Thomas Deutschmann tests: avoid false failure with inaccessible mount points * tests/ls/readdir-mountpoint-inode.sh: Skip the test if any mount points are inaccessible by the current user. Fixes https://bugs.gnu.org/29167 Reported at: https://bugs.gentoo.org/353164 2017-11-05 Bernhard Voelker doc: fix "Up" field of realpath usage examples Older versions of 'makeinfo' choke on a missing reference: ./doc/coreutils.texi:14177: `Realpath usage examples' has no Up field\ (perhaps incorrect sectioning?). makeinfo: Removing output file `doc/coreutils.info' due to errors; \ use --force to preserve. * doc/coreutils.texi (realpath invocation): Add a menu referencing the usage examples - introduced in v8.27-91-g7449f0d. 2017-11-05 Pádraig Brady maint: ensure https:// URLs are used in --help and man pages * configure.ac(AC_INIT): Specify the URL explicitly, so we're not dependent on unreleased autoconf. 2017-10-31 Assaf Gordon stat: output default formats for --terse in usage Suggested by L A Walsh in https://bugs.gnu.org/28763 . * src/stat.c (fmt_terse_fs): Define format for --terse -f here. (fmt_terse_regular): Define format for --terse here. (fmt_terse_selinux): Likewise for when SELinux is enabled. (default_format): Use the above constants. (usage): Output the formats for the terse modes. 2017-10-30 Pádraig Brady df: fix hang with fifo argument * src/df.c (main): stat() before open(), and avoid the optional open when given a fifo argument. * tests/df/unreadable.sh: Add a test case. * NEWS: Mention the fix. Fixes https://bugs.gnu.org/29038 2017-10-27 Jim Meyering build: ls.c: apply _GL_ATTRIBUTE_PURE to more functions * src/ls.c (DEFINE_SORT_FUNCTIONS): Apply _GL_ATTRIBUTE_PURE to each strcmp-derived function definition, since GCC8 with -Wsuggest-attribute=pure now warns it is needed. 2017-10-25 Vincent Lefevre doc: reference statfs(2) in the stat(1) man page * man/stat.x (SEE ALSO): Mention statfs(2) in addition to stat(2). Note statfs() is generally used rather than statvfs(), so we'll defer that reference to the SEE ALSO section of statfs(2). Fixes https://bugs.gnu.org/28989 2017-10-24 Pádraig Brady tests: avoid false failure when O_DIRECT isn't supported * tests/dd/nocache_eof.sh: Only run the O_DIRECT tests when 512 byte alignment is supported. Otherwise with older XFS on systems with > 1MiB pages, or on file systems not supporting O_DIRECT, there would have been false failures. * tests/dd/direct.sh: Clarify the skip message. 2017-10-24 Pádraig Brady dd: fix nocache regions passed to posix_fadvise() Previously with oflag=direct the call to invalidate_cache() was not passed to the kernel, as it was less than a page size, and a subsequent call was not made to invalidate the pending space. Similarly with oflag=nocache the pending space at EOF was not invalidated. Even though these amount to only a single page in the page cache it can be significant. For example on XFS before kernel patch v4.9-rc1-4-g0ee7a3f, O_DIRECT files would have been read inefficiently if any pages were cached, even if they were already synced to storage. * src/dd.c (i_nocache_eof, o_nocache_eof): New bools used to control when we want invalidate_cache(,0) to clear to EOF. (cache_round): Use IO_BUFSIZE (currently 132KiB) to minimize calls to the relatively expensive advise function, rather than page_size. This also makes it clear that while the kernel function operates on pages, this size is chosen for performance reasons. (invalidate_cache): Refactor to share more code between input and output paths. Use i_nocache_eof and o_nocache_eof rather than proxying off max_records. Ensure we invalidate full pages when clearing to EOF as the kernel will ignore any non complete pages. Fix the offset used for the output path. (dd_copy): Invalidate the cache of the input after the offset is updated, for consistency and so we don't try to invalidate before the start of the file. When we read EOF on input, set flags so that we invalidate to EOF. (main): Invalidate to EOF in more cases, by depending on the i_nocache_eof and o_nocache_eof flags. * doc/coreutils.texi (dd invocation): Clarify the alignment and persisted caveats on the example applying "nocache" to part of a file. * tests/dd/nocache_eof.sh: A new test. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix. Issue reported by Eric Bergen. 2017-10-24 Michael Stone doc: mention QUOTING_STYLE env var in ls man page * src/ls.c (usage): Mention QUOTING_STYLE with the --quoting-style option, and indicate it has lower precedence than that option. 2017-10-23 Pádraig Brady maint: apply suggested cleanup to recent stty.c change This should have been part of commit v8.28-17-gf926f7c * src/stty.c (check_argument): Align line continuation chars, and ensure the function macro is immune to usage with if/else. Suggested by Jim Meyering and Paul Eggert. b2sum: fix crash with --check and truncated input * src/md5sum.c (split_3): Ensure we don't walk off the end of the string. * tests/misc/b2sum.sh: Add test cases. Fixes https://bugs.gnu.org/28860 2017-10-23 Pádraig Brady stty: fix processing of options when -F is specified This was a latent issue that became significant with the addition of the -F option in FILEUTILS-3_16n-56-ge46a424 * src/stty.c (apply_settings): Refactor argument checking to a function macro. Augment the argument check to ignore NULLed out arguments (already processed -F). * NEWS: Mention the fix. * tests/misc/stty-invalid.sh: Add a test case. Fixes https://bugs.gnu.org/28859 2017-10-23 Pádraig Brady timeout: fix a small race that would ignore command exit This fixes a regression from commit v8.26-39-g2f69dba * src/timeout.c (block_cleanup_and_chld): Rename from block_cleanup to indicate we also block SIGCHLD to avoid the race where SIGCHLD fires between waitpid() polling and sigsuspend() waiting for a signal. * NEWS: Mention the fix. 2017-10-23 Thomas Jarosch timeout: fix regression when invoked with blocked SIGCHLD We inherit the signal mask from our parent process, therefore ensure SIGCHLD is not blocked. If SIGCHLD is blocked, sigsuspend() won't be interrupted when the child process exits and we hang until the timeout (SIGALRM). This fixes a regression from commit v8.26-39-g2f69dba * src/timeout.c (install_sigchld): Ensure SIGCHLD is unblocked. * NEWS: Mention the issue. 2017-10-01 Pádraig Brady build: reinstate distribution of man pages man pages change little between systems, so falling back to distributed pages make sense when cross compiling or lacking perl. * man/local.mk: Add all man pages to EXTRA_DIST so that they're distributed in the generated tarball. Use the dummy-man page generator if cross compiling. Set TZ to avoid a distcheck failure where man pages used a diffent month than those rebuilt (with a .timestamp). * man/dummy-man: Only fall back to generating a stub if copying an existing man page fails. * man/help2man: Sync portable TZ=UTC0 specification from upstream help2man. * NEWS: Mention the build-related change. Fixes https://bugs.gnu.org/28574 2017-10-01 Pádraig Brady maint: remove a duplicate entry from THANKS * .mailmap: Prefer Colin Watson's last used email address. 2017-09-24 Paul Eggert copy: revert recent patch for vulnerable dirs I plan to propose a better patch to catch vulnerable parent directories. * NEWS, doc/coreutils.texi (Target directory): Document this. * src/cp.c, src/install.c, src/ln.c, src/mv.c: Do not include targetdir.h. (target_directory_operand): Remove test for vulnerable parents. * src/cp.c (stat_target_operand): Remove. All uses removed. * src/local.mk (noinst_HEADERS): Remove src/targetdir.h. (src_ginstall_SOURCES, src_cp_SOURCES, src_ln_SOURCES) (src_mv_SOURCES): Remove src/targetdir.c. * src/targetdir.c, src/targetdir.h: Remove. * tests/mv/vulnerable-target.sh: Remove. * tests/local.mk (all_root_tests): Remove it. 2017-09-24 Pádraig Brady tests: fix test hang on case insenitive file systems * tests/split/filter.sh: Due to an invalid 'FILE = zero.in' construct trying to initialize a FILE variable, it would instead try to run the FILE command which is present on macOS 10.13 with APFS. We also remove a redundant duplicate test clause introduced during a rebase, and simplify the piped timeout command, to avoid requiring a subshell and associated quoting. * THANKS.in: Add the reporter Jack Howarth. Fixes https://bugs.gnu.org/28506 2017-09-20 Pádraig Brady tests: avoid a false failure in expr test with UTF8 * tests/misc/expr.pl: Skip the quote varying tests in the multi-byte locales as these tests aren't that interesting in those locales. Also ERR_SUBST is already defined for some tests so awkward to redefine to munge UTF8 quotes to ASCII. 2017-09-19 Assaf Gordon expr: add detailed syntax error messages Show offending argument instead of a generic 'syntax error' message. Suggested by Bernhard Voelker in https://bugs.gnu.org/28461#13 . * src/expr.c (syntax_error): Remove. (required_more_args): New function. (eval7, main): Replace syntax_error call with detailed die message. * tests/misc/expr.pl: Add tests for new messages. 2017-09-19 Pádraig Brady maint: fix new syntax-check failures from HTTPS adjustments * cfg.mk [old_NEWS_hash]: update with `make update-NEWS-hash`. [sc_long_lines]: Avoid flagging (long) URLs in NEWS. * src/sort.c: Tweak to a shorter line. * src/tail.c: Likewise. Introduced in v8.28-4-gbe87d61 maint: fix new syntax check failures from copy restrictions * doc/coreutils.texi: Remove doubled word. * src/targetdir.c: Explicitly mark exported function. * tests/local.mk: This is not a root only test. * tests/mv/vulnerable-target.sh: Use returns_. Introduced in v8.28-3-g44ccd1c shred: reinstate --remove file name length obfuscation This was unintentionally removed in v8.27-60-g2ae1460 * src/shred.c (wipename): Interate through all name lengths. * tests/misc/shred-remove.sh: Add test cases. * NEWS: Mention the bug fix. Fixes https://bugs.gnu.org/28507 2017-09-19 Paul Eggert maint: copy bootstrap from Gnulib all: prefer HTTPS in URLs copy: check for vulnerable target dirs * NEWS, doc/coreutils.texi (Target directory): Document this. * src/cp.c, src/install.c, src/ln.c, src/mv.c: Include targetdir.h. (target_directory_operand): Use the new targetdir_operand_type function to check for vulnerable target directories. * src/cp.c (stat_target_operand): New function. (target_directory_operand, do_copy): Use it. * src/local.mk (noinst_HEADERS): Add src/targetdir.h. (src_ginstall_SOURCES, src_cp_SOURCES, src_ln_SOURCES) (src_mv_SOURCES): Add src/targetdir.c. * src/targetdir.c, src/targetdir.h: New files. * tests/mv/vulnerable-target.sh: New test. * tests/local.mk (all_root_tests): Add it. 2017-09-13 Bernhard Voelker ptx: avoid infloop due to zero-length matches with -S regex * src/ptx.c (find_occurs_in_text): Die with an appropriate error diagnostic when the given regular expression returns a match of length 0. * tests/misc/ptx.pl (S-infloop): Add a test. * NEWS (Bug fixes): Mention the fix. Fixes https://bugs.gnu.org/28417 which was detected using Symbolic Execution techniques developed in the course of the SYMBIOSYS research project at COMSYS, RWTH Aachen University. 2017-09-01 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.28 * NEWS: Record release date. tests: fix false failure in recent ls --hyperlink test * tests/ls/hyperlink.sh: If the hostname or any part of the absolute path would be changed due to URL encoding, the test would fail. Therefore simplify to remove these components of the URL from consideration. 2017-08-31 Pádraig Brady maint: avoid a syntax-check failure * .gitignore: Remove lines indicated by sc_gitignore_redundant in a freshly checked out repo. 2017-08-30 Pádraig Brady tests: exclude the expensive gnulib fts-tests * gnulib: The only change in this gnulib update is the tagging of the fts-tests module as longrunning, which gnulib-tool currently implicitly excludes. This test was seen to take about 20s and 285MB. Reported by Assaf Gordon on space restricted VMs. tty: don't distinguish input errors * src/tty.c (main): Don't distinguish ENOTTY from other errors, because isatty() doesn't portably distinguish errors. Solaris returns ENOENT for all input errors for example. Musl also returns ENOENT, and ENODEV may be returned as disscussed at: http://openwall.com/lists/musl/2017/04/06/6 * tests/misc/tty.sh: Adjust accordingly. tests: avoid printf '0*d' construct unsupported by ash * tests/ln/sf-1.sh: Generate specific length with space padding which is supported. Reported by Assaf Gordon on Alpine Linux. 2017-08-30 Pádraig Brady tests: skip tests upon failure to set SELinux context On some setups the root:object_r:tmp_t context is invalid. This does indicate a limitation in the test framework, but for now we'll relax this to skipping the tests. The tests still run on a Fedora 25 system for example. * tests/cp/cp-a-selinux.sh: Upon chcon error, skip rather than ERROR. * tests/install/install-Z-selinux.sh: Likewise. * tests/misc/chcon.sh: Likewise. * tests/misc/runcon-no-reorder.sh: Likewise. * tests/misc/selinux.sh: Likewise. * tests/mkdir/restorecon.sh: Likewise. 2017-08-30 Kamil Dudka expr: fix a recently introduced memory leak * src/expr.c (eval6): Free memory allocated by mbs_logical_substr(). Introduced in v8.27-47-ga9f2be5. Detected by Coverity Analysis: Error: RESOURCE_LEAK: src/expr.c:851: leaked_storage: Variable "s" going out of scope leaks the storage it points to. 849| char *s = mbs_logical_substr (l->u.s, pos, len); 850| v = str_value (s); 851|-> } 852| freev (l); 853| freev (i1); 2017-08-30 Pádraig Brady build: fix build of renameat2 on Alpine Linux * gnulib: The only change included in this update it the added check for the presence of which is not present on Alpine Linux by default. tty: fix exit code with EINVAL * src/tty.c (main): All systems mention that isatty() man return EINVAL as well as (the POSIX compliant) ENOTTY. Also Centos 6 was seen to return EINVAL from ttyname(). * tests/misc/tty.sh: Fix a test issue where we assume standard input is always a valid tty. Reported by Assaf Gordon on OpenSolaris 5.10 and 5.11, and Centos 6.5 2017-08-30 Pádraig Brady runcon: revert "disable use of the TIOCSTI ioctl" This reverts commit v8.27-97-g8cb06d4 because the setsid() fallback was not implemented correctly and disabling the ioctl was not a complete solution to the security issue of the child being passed the tty of the parent. Given runcon is not really a sandbox command, the advice is to use `runcon ... setsid ...` to avoid this particular issue. 2017-08-29 Pádraig Brady stat: fix determination of max name length on BSD systems We only use one of statfs or statvfs for `stat -f` and on the BSDs we use statfs which doesn't have the f_namelen member. However on OpenBSD and later FreeBSD systems statfs does provide f_namemax, so use that. * NEWS: Mention the improvement for OpenBSD and FreeBSD. * m4/stat-prog.m4: Check for f_namemax in the statfs struct. * src/stat.c: Return '?' rather than '*' when we can't determine the max length of the file system. * tests/ln/sf-1.sh: This test was failing on all BSDs due to '*' being returned for the max length which caused the test to attempt to create 1Mi+1 names. The test now uses a short name when we can't determine the max name length to use. Reported by Assaf Gordon on various BSD based systems. 2017-08-29 Pádraig Brady stat,tail: support "AAFS" AppArmor file system * src/stat.c (human_fstype): This file system is used to manage AppArmor policy in the Linux kernel. all: update gnulib submodule to latest * bootstrap: Sync timestamp update. 2017-08-29 Pádraig Brady runcon: disable use of the TIOCSTI ioctl Similar to the issue with SELinux sandbox (CVE-2016-7545), children of runcon can inject arbitrary input to the terminal that would be run at the originating terminal privileges. The new libseccomp dependency is widely available and used on modern SELinux systems, but is not available by default on older systems like RHEL6 etc. * m4/jm-macros.m4: Check for libseccomp and warn if unavailable on selinux supporting systems. * src/local.mk: Link runcon with -lseccomp. * src/runcon.c (disable_tty_inject): A new function to disable use of the TIOCSTI using libseccomp, or with setsid() where libseccomp is unavailable. * tests/misc/runcon-no-inject.sh: A new test that uses python to make the TIOCSTI call, and ensure that doesn't succeed. * tests/local.mk: Reference the new test * NEWS: Mention the fix. Addresses http://bugs.gnu.org/24541 2017-08-29 Pádraig Brady ls: support --hyperlink to output file:// URIs Terminals such as iTerm2 and VTE based terminals (as of version 0.49.1), support hyperlinks when passed terminals codes as described at: https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda * src/ls.c (gobble_file): Allocate an absolute file name to output. (quote_name): Output the absolute name with the appropriate codes. (file_escape): A new function to encode file names as per rfc8089. (main): Handle the new option and call the file_escape_init() helper. Disable --dired when --hyperlink is specified. (print_dir): Get the absolute file name here too, so that the directory name can be linkified. * NEWS: Mention the new feature. * tests/ls/hyperlink.sh: Add a new test. * tests/local.mk: Reference the new test. * doc/coreutils.texi (ls invocation): Describe --hyperlink. 2017-08-29 Pádraig Brady doc: remove older ChangeLog items This saves about 0.5MB uncompressed from the tarball. * Makefile.am: Following on from v8.26-34-g2c64bc8 update the oldest documented version to 8.18 which is now about 5 years old. Also remove older ChangeLogs that were previously thought to be for changes not in the git history, but are adequately recorded upon review. * build-aux/ChangeLog-2007: Remove file. * lib/ChangeLog-2007: Likewise. * m4/ChangeLog-2007: Likewise. 2017-08-29 Colin Watson env: add --chdir option This is useful when chaining with other commands that run commands in a different context, while avoiding using the shell to cd, and thus having to consider shell quoting the chained command. * NEWS (New features): Document the new option. * doc/coreutils.texi (env invocation): Likewise. * src/env.c (usage): Likewise. (main): Implement the new option. * tests/misc/env.sh: Test the new option. 2017-08-29 Pádraig Brady tests: don't fail tests when failing to write files * tests/sample-test: Use framework_error_ rather than fail=1 * tests/chown/deref.sh: Likewise. * tests/chown/preserve-root.sh: Likewise. * tests/cp/src-base-dot.sh: Likewise. * tests/dd/unblock-sync.sh: Likewise. * tests/du/2g.sh: Likewise. * tests/du/inacc-dest.sh: Likewise. * tests/du/one-file-system.sh: Likewise. * tests/fmt/goal-option.sh: Likewise. * tests/ln/hard-backup.sh: Likewise. * tests/ls/color-dtype-dir.sh: Likewise. * tests/ls/m-option.sh: Likewise. * tests/ls/stat-dtype.sh: Likewise. * tests/ls/time-style-diag.sh: Likewise. * tests/ls/x-option.sh: Likewise. * tests/misc/chcon.sh: Likewise. * tests/misc/nohup.sh: Likewise. * tests/misc/od-N.sh: Likewise. * tests/misc/sort-compress.sh: Likewise. * tests/misc/tac-continue.sh: Likewise. * tests/misc/time-style.sh: Likewise. * tests/mv/backup-dir.sh: Likewise. * tests/mv/dir2dir.sh: Likewise. * tests/rm/dir-no-w.sh: Likewise. * tests/rm/dir-nonrecur.sh: Likewise. * tests/rm/inaccessible.sh: Likewise. * tests/rm/interactive-always.sh: Likewise. * tests/rm/interactive-once.sh: Likewise. * tests/rm/rm3.sh: Likewise. * tests/rm/v-slash.sh: Likewise. * tests/touch/relative.sh: Likewise. 2017-08-28 Josef Cejka Bernhard Voelker df: avoid stat() for dummy file systems with -l When systemd is configured to automount a remote file system - see 'man systemd.automount(5)', then the mount point is initially mounted by systemd with the file system type "autofs". When the resource is used later on, then the wanted file system is mounted over that mount point on demand. 'df -l' triggered systemd to mount the file system because it called stat() on the mount point. Instead of single-casing "autofs" targets, we can avoid stat()ing all dummy file systems (which includes "autofs"), because those are skipped later on in get_dev() anyway. *src/df.c (filter_mount_list): Also skip dummy file systems unless the -a option or a specific target are given. * NEWS: Mention the fix. Fixes http://bugzilla.suse.com/show_bug.cgi?id=1043059 2017-08-28 Assaf Gordon doc: add 'realpath usage examples' section * doc/coreutils.texi (Realpath usage examples): New section. 2017-08-28 Assaf Gordon doc: fix realpath index entry The 'readlink' node has '@findex realpath' in it. This results in info doc/coreutils.info realpath incorrectly jumping to the 'readlink' node (instead of the 'realpath' node). Change it to @cindex instead. * doc/coreutils.texi (readlink): Change '@findex realpath' to @cindex. 2017-08-28 Assaf Gordon realpath: improve usage description for --relative-{to,base} * src/realpath.c (usage): Explicitly say 'DIR' instead of 'FILE' for --relative-{to,base} parameters, to avoid giving the impression that regular files can be used as relative base. * doc/coreutils.texi (realpath): Same. 2017-08-24 Pádraig Brady ls: consistently quote symlink targets * src/ls.c (gobble_file): Disable the optimization to avoid quoting if the symlink target itself needs quoting. This was introduced with the quoting alignment adjustments in v8.25-106-g01971c0 * tests/ls/symlink-quote.sh: Add a test. * tests/local.mk: Reference the test. * NEWS: Mention the fix. 2017-08-24 Pádraig Brady tail: reinstate inotify use with FIFOs commit v8.27-44-g18f6b22 was too aggressive in only allowing inotify use with regular files. This will support responsive processing of `tail -f fifo | ...` * src/tail.c (any_non_regular): Adjust to allow FIFOs since inotify supports these well. * tests/tail-2/inotify-only-regular.sh: Adjust comment. 2017-08-19 Pádraig Brady maint: avoid a syntax check failure * src/sort.c: Don't include stdio--.h as fopen() is no longer used. tests: fix issues on alpine linux * tests/misc/seq-epipe.sh: Remove stale comment. * tests/misc/sort-debug-warn.sh: musl doesn't indicate a set_locale() failure with missing locales, so avoid a test portion in that case. * tests/misc/wc-files0.sh: Avoid a bug on older ash implementations. Addresses http://bugs.gnu.org/28054 2017-08-17 Paul Eggert ptx: fix some integer overflow bugs Problem reported by Lukas Zachar at: http://bugzilla.redhat.com/1482445 * src/ptx.c (line_width, gap_size, maximum_word_length) (reference_max_width, half_line_width, before_max_width) (keyafter_max_width, truncation_string_length, compare_words) (compare_occurs, search_table, find_occurs_in_text, print_spaces) (fix_output_parameters, define_all_fields): Use ptrdiff_t, not int, for object offsets and sizes. (WORD, OCCURS): Use ptrdiff_t, not short int. (WORD_TABLE, number_of_occurs, generate_all_output): Prefer ptrdiff_t to size_t where either will do. (total_line_count, file_line_count, OCCURS, fix_output_parameters) (define_all_fields): Use intmax_t, not int, for line counts. (DELTA): Remove. All uses changed. (OCCURS, find_occurs_in_text, fix_output_parameters): Use int, not size_t, for file indexes. (tail_truncation, before_truncation, keyafter_truncation) (head_truncation, search_table, define_all_fields) (generate_all_output): Use bool for booleans. (digest_word_file, find_occurs_in_text): Use x2nrealloc instead of checking for overflow by hand. (find_occurs_in_text, fix_output_parameters, define_all_fields): Omit unnecessary cast. (fix_output_parameters): Don’t assume integers fit in 11 digits. (fix_output_parameters, define_all_fields): Use sprintf return value rather than calling strlen. (define_all_fields): Do not rely on sprintf to generate a string that may contain more than INT_MAX bytes. (main): Use xstrtoimax, not xstrtoul. Use xnmalloc to catch integer overflow. nohup: simplify by using fcntl * src/nohup.c: Do not include cloexec.h. (main): Use fcntl rather than dup + set_cloexec_flag. sort: use pthread_sigmask, not sigprocmask POSIX says sigprocmask has unspecified behavior in a multithreaded program like ‘sort’. * src/sort.c (pthread_sigmask) [GNULIB_defined_pthread_functions]: New macro, for use when ‘sort’ is not multithreaded. (cs_enter, cs_leave): Use it. Pass address, not value, as this is typically a tad faster. All callers changed. sort: minor cleanups * src/sort.c (move_fd): Rename from move_fd_or_die, since it no longer can die. sort: file descriptor discipline Use O_CLOEXEC when creating file descriptors, so that subsidiary processes do not inherit file descriptors that they do not need. This is helpful for ‘sort’, as it is a multithreaded program that forks and execs. * bootstrap.conf (gnulib_modules): Add mkostemp, open, pipe2. * src/sort.c (create_temp_file): Open temporary file with O_CLOEXEC. (stream_open): Open the stream with O_CLOEXEC. (pipe_fork): Create the pipe with O_CLOEXEC. (check_output): Open the output file with O_CLOEXEC. (main): Use xfopen/xfclose to handle --files0-from, so that O_CLOEXEC is used properly. This is simpler anyway. * tests/misc/sort-files0-from.pl: Adjust to change in diagnostic wording. build: update gnulib submodule to latest 2017-08-13 Pádraig Brady kill: fix signal number to name lookup on AIX * src/operand2sig.c (operand2sig): AIX uses a different bit pattern in the returned status from the wait() functions and from shells. Therefore hardcode the selection of the lower bits of the number. * NEWS: Mention the fix. build: use the appropriate single file include option with xlc * configure.ac: Set USE_XLC_INCLUDE when __xlc__ is defined. * src/local.mk: Use it to select the appropriate include option. Reported by Michael Felt. tests: avoid false failures on AIX * tests/ln/sf-1.sh: Limit the symlink size to 1MiB to avoid memory exhaustion seen on NFS on AIX, giving: + printf '%0*d' 4294967296 0 + ./tests/ln/sf-1.sh: line 38: printf: warning: 0: Result too large * tests/id/setgid.sh: Skip the test when the adjusted gid would equal 4294967295, as that's reserved on AIX. Reported by Michael Felt. sort: handle musl locale differences in --debug reporting * src/sort.c (main): Don't assume hard_LC_COLLATE implies a successful setting of the locale as musl defaults to UTF8 when failing to set the specified locale. * tests/misc/sort-debug-warn.sh: Adjust for the now separated locale debug info and map the musl specific message back to the common case. Addresses https://bugs.gnu.org/28054 seq: produce consistent error messages upon write error * src/seq.c (io_error): Use the same error message as would be generated at exit time when closing the stdout stream. The inconsistency was added with commit v8.25-26-gc92585b. This was noticed due to an inconsistency in the expected error message generated by seq on musl libc. Addresses https://bugs.gnu.org/28054 tests: fix false failure with large printf formats * tests/misc/printf-surprise.sh: With musl libc the large printf format does succeed, outputting data. To avoid SIGPIPE being generated we ignore that signal and then handle the subsequent EPIPE error. Addresses https://bugs.gnu.org/28054 2017-08-12 Jim Meyering build: adjust warning options to work with latest GCC * configure.ac: Disable some new warnings to avoid false positives. Building with warnings enabled and latest gcc would evoke build failure without these changes. Disable the following in coreutils proper: -Wformat-overflow=2 -Wformat-truncation=2, and disable these for gnulib: -Wformat-truncation=2 -Wduplicated-branches gnulib: update to latest and adjust gl/modules/tempname.diff * gnulib: Update to latest. * gl/modules/tempname.diff: This patch failed to apply. Adjust it to reflect removal of the secure_getenv dependency. 2017-08-11 Jim Meyering chroot: fix typo in preceding change: didn't compile * src/chroot.c (usage): Add backslashes. 2017-08-10 Jim Meyering doc: correct technicality in chroot's --help output * src/chroot.c (usage): Use correct quoting in descriptive diagnostic. We would run `"$SHELL" -i`, not `${SHELL} -i`. 2017-08-08 Assaf Gordon doc: fix join example * doc/coreutils.texi (join invocation): Fix incorrect output in example. Reported by Phlosioneer in https://bugs.gnu.org/28014 . 2017-08-03 Paul Eggert build: update gnulib submodule to latest copy: more-accurate warning about destruction * src/copy.c (copy_internal): * tests/cp/backup-is-src.sh, tests/mv/backup-is-src.sh: Say "might destroy", not "would destroy". 2017-08-02 Pádraig Brady maint: avoid a syntax-check failure * src/shred.c (wipename): As per the comment, the arguments to error() are sufficiently quoted, so split the call over multiple lines to avoid the syntax-check. 2017-08-02 Paul Eggert build: update gnulib submodule to latest 2017-08-01 Paul Eggert copy: go back to failing 'cp --backup a~ a' Suggested by Kamil Dudka in: http://lists.gnu.org/archive/html/coreutils/2017-07/msg00072.html * NEWS: Document the changed nature of the fix. * doc/coreutils.texi, tests/cp/backup-is-src.sh: * tests/mv/backup-is-src.sh: Revert previous change. * src/copy.c (source_is_dst_backup): New function. (copy_internal): Use it. Fail instead of falling back on numbered backups when it looks like the backup will overwrite the source. Although this reintroduces a race, it's more compatible with previous behavior. 2017-07-30 Paul Eggert copy: sanity-check --suffix * src/cp.c, src/install.c, src/ln.c, src/mv.c (main): Use set_simple_backup_suffix, to sanity-check the user-supplied backup suffix. copy: make backup files more reliably * NEWS, doc/coreutils.texi (Backup options): Document the change. * bootstrap.conf (gnulib_modules): Add backup-rename. * src/copy.c (copy_internal): Silently switch to numbered backups if a simple backup might lose data. Use backup_file_rename to avoid races with numbered backups. * tests/cp/backup-is-src.sh, tests/mv/backup-is-src.sh: Adjust to match new behavior. shred: avoid rename race Use renameat2 to avoid a rename race condition, on recent-enough GNU/Linux. * bootstrap.conf (gnulib_modules): Add renameat2. * src/shred.c: Include renameat2.h. (wipename): Use renameat2 instead of rename. build: update gnulib submodule to latest 2017-07-24 Jim Meyering maint: fix grammar in a shred.c comment * src/shred.c: Remove spurious "to" in an old comment. 2017-07-23 Pádraig Brady maint: fix recent syntax-check failures * .gitignore: Add /lib/utime.h from the recent gnulib update. * src/nproc.c (usage): Adjust spacing to placate help2man. shred: remove redundant zeroing of freed memory * src/shred.c (dopass): shred used to read the input file, and so needed to ensure internal memory was cleared. This is no longer the case since SH-UTILS-1_16f-260-gf381610 so avoid this redundant clearing. (do_wipefd): Likewise. * NEWS: Remove the recent mention of this issue. maint: resync with blake2 upstream * src/blake2/blake2-impl.h: Don't use the equivalent explicit_bzero(). tests: avoid a false failure on AIX * tests/misc/sync.sh: Normalize the error messages when syncing a non read/write directory, as AIX gives the "Is a directory" error. Also ensure that sync(1) returns an error for this case on all systems. 2017-07-20 Paul Eggert shred: use explicit_bzero * NEWS: Document this. * bootstrap.conf (gnulib_modules): Add explicit_bzero. * gl/lib/randint.c (randint_free): * gl/lib/randread.c (randread_free): * src/blake2/blake2-impl.h (secure_zero_memory): * src/shred.c (dopass, do_wipefd): Prefer explicit_bzero to memset when erasing secrets. build: update gnulib submodule to latest 2017-07-10 Andreas Schwab nproc: fix indentation of usage output * src/nproc.c (usage): Align output. 2017-07-10 Jim Meyering groups: do not exit early Most programs take care to operate on all command-line-specified operands before exiting. That is an important feature that allows to identify all problems with the first run. However, groups would exit upon the first problematic user name. Bug introduced via commit v6.10-56-g167b8025ac. * src/groups.c (main): Do not exit immediately upon error. * tests/misc/groups-process-all.sh: New file. Test for this. * tests/local.mk (all_tests): Add it. * NEWS (Bug fixes): Mention this. 2017-07-08 Jim Meyering tests: groups-dash.sh: avoid false failure * tests/misc/groups-dash.sh: Avoid false failure on a system for which "none" is a valid user name. The first invocation would succeed, and the second would fail with "groups: ‘--’: no such user". Use a user name that cannot exist. doc: tweak wording * NEWS (Bug fixes): Tweak wording of the mv/cp-vs-symlink-ownership entry and the one about df. 2017-06-27 Assaf Gordon expr: add multibyte support Discussed in https://bugs.gnu.org/26779 . * NEWS: Mention the improvement. * bootstrap.conf: Add gnulib modules mbslen,mbschr. * src/expr.c (mbs_logical_substr): New function to return a substring based on logical character positions (instead of bytes). (mbs_logical_cspn): Similar to strcspn/mbscspn, but returns number of logical characters instead of byte offset. (mbs_offset_to_chars): New function to return number of logical characters fitting in a given byte offset. (docolon): Report matched logical characters instead of bytes. (eval6): For length/substr/index operations, use logical characters instead of bytes by calling the above new functions. * tests/misc/expr.pl: Repeat all tests with non-C locale to detect any regressions. * tests/misc/expr-multibyte.pl: New tests with multibyte input. * tests/local.mk: Add new test file. 2017-06-20 Jim Meyering maint: avoid spurious "make distcheck" failure When the generated file, doc/constants.texi, happens to be older than doc/coreutils.info, it will not be updated until/unless its generated contents change. This is due to way that rule is careful to update the file, to avoid provoking a pointless rerunning of makeinfo. Note that this does not happen when one first runs "make distclean", as recommended in README-release. However, I sometimes run it as a more-rigorous "make check", and shouldn't have to manually run "make distclean" first, in that case. Before this change, one could reproduce the failure by running `touch -dyesterday doc/constants.texi && make distcheck`. It would fail with "makeinfo: could not open ../../doc/coreutils.info-t for writing: Permission denied" * Makefile.am (dist-hook): Touch the two generated files, so that they cannot be out of date wrt doc/coreutils.texi. 2017-06-17 Pádraig Brady maint: use C99 for loop initial declarations where possible This results in a net reduction of about 120 lines. tail: only use inotify with regular files * src/tail.c (any_non_regular): A new function to check passed files. (main): Use the above to skip inotify if any non regular files passed like /dev/tty or /dev/ttyUSB0 etc. * tests/tail-2/inotify-only-regular.sh: A new test. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/21265 and http://bugs.gnu.org/27368 tail: with -f don't warn if doing a blocking read of a tty * src/tail.c: (main): Only issue the warning about -f being ineffective when we're not going into simple blocking mode. * tests/tail-2/follow-stdin.sh: Ensure the warning is output correctly. Fixes http://bugs.gnu.org/27368 2017-06-11 Pádraig Brady tail: exit promptly when output no longer writable This will support use cases like: tail -f file.log | grep -q trigger && process_immediately * src/tail.c (check_output_alive): A new function that uses select on fifos or pipes to detect if they're broken. (tail_forever): Call check_output_alive() periodically. (tail_forever_inotify): Merge the select() call from check_output_alive() into the select() originally present for the --pid case, and adjust accordingly. * tests/tail-2/pipe-f.sh: Add test cases. * NEWS: Mention the improvement. 2017-06-10 Jim Meyering maint: update to work with GCC7's -Werror=implicit-fallthrough=5 * src/system.h (FALLTHROUGH): Define. * src/cp.c (main): Use new FALLTHROUGH macro in place of comments. * src/basename.c (main): Likewise. * src/dircolors.c (append_quoted): Likewise. * src/echo.c (main): Likewise. * src/fold.c (main): Likewise. * src/join.c (main): Likewise. * src/kill.c (main): Likewise. * src/ls.c (get_funky_string, gobble_file): Likewise. * src/sort.c (parse_field_count, main): Likewise. * src/stat.c (print_it): Likewise. * src/tail.c (parse_obsolete_option): Likewise. * src/test.c (posixtest): Likewise. * src/wc.c (wc): Likewise. * src/who.c (main): Likewise. 2017-06-07 Pádraig Brady tail: with --pid, ensure all inotify events are processed * NEWS: Mention the bug fix. * src/tail.c (tail_forever_inotify): With --pid, avoid waiting for new events if there are still events to process. * tests/tail-2/inotify-dir-recreate.sh: Adjust to trigger. tests: fix issues with recently added tail test * tests/tail-2/inotify-dir-recreate.sh: Skip when inotify is not usable. Also remove a bash specific &> construct. 2017-06-02 Pádraig Brady copy: don't fail when unable to chown symlinks * src/copy.c (copy_internal): Honor the x->require_preserve flag for symlinks as we do for ordinary files, so we don't exit with failure upon failure to chown a symbolic link. * NEWS: Mention the bug fix. 2017-05-29 Sebastian Kisela doc: mention `setpriv --no-new-privs` feature in runcon info * doc/coreutils.texi (runcon invocation): Mention setpriv usage. Discussed at https://bugzilla.redhat.com/1360903 2017-05-17 Pádraig Brady mv: distinguish copy and rename operations with --verbose * src/copy.c (copy_internal): In x->move_mode distinguish whether we're copying, creating directory, or renaming. * tests/mv/backup-dir.sh: Adjust to new output. * tests/mv/mv-n.sh: Likewise. * tests/mv/mv-special-1.sh: Likewise. * NEWS: Mention the improvement. Fixes http://bugs.gnu.org/26971 2017-05-11 Prateek saxena uptime: remove inconsistent AM/PM from current time * src/uptime.c (main): 00-23 was always used for the hour component of the current time, so remove the AM/PM output (which was only present in some locales anyway). Also add seconds to the time to be more consistent with the usual procps-ng uptime implementation on GNU/Linux. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/26783 2017-05-03 Pádraig Brady maint: fix various typos in recent commits * NEWS: Grammar fixes. * HACKING: Likewise. 2017-05-03 Jaak Ristioja doc: Fixed typo in timeout man page * man/timeout.x: Correct spelling of "currently". Fixes http://bugs.gnu.org/26762 2017-04-29 Pádraig Brady doc: update the instructions for generating a coverage report * HACKING: Change from explicit instructions to using gnulib provided coverage testing targets. Also include instructions for adding root only tests to the report. Fixes http://bugs.gnu.org/26709 2017-04-27 Paul Eggert dd: simplify translator’s jobs * src/dd.c (print_xfer_stats): Format the SI units directly, without translating them, to simplify the translators’ jobs. See Bug#26621. 2017-04-26 Pádraig Brady date,touch: test and document large TZ security issue Add a test for CVE-2017-7476 which was fixed in gnulib at: http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=94e01571 * tests/misc/date-tz.sh: Add a new test which overwrites enough of the heap to trigger a segfault, even without ASAN enabled. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix. 2017-04-26 Pádraig Brady build: update gnulib submodule to latest * .gitignore: Add new entry as indicated by `make syntax-check`. 2017-04-24 Paul Eggert dd: status=progress outputs "6 s", not "6.00001 s" Problem reported by Benno Schulenberg (Bug#26621). * NEWS: Document this. * src/dd.c (print_xfer_stats): With status=progress, format times with %.0f rather than %g. Improve translator comments. 2017-04-22 Paul Eggert build: update gnulib submodule to latest maint: remove unused functions and constants These were found by clang. * gl/lib/rand-isaac.c (min): * gl/lib/randint.c (shift_right): * src/md5sum.c (algorithm): Remove; unused. date: adjust to gnulib parse-datetime changes * doc/coreutils.texi (Options for date): Capitalize a sentence. * tests/misc/date-debug.sh: Adjust --debug output to match recent changes to Gnulib’s parse-datetime module. build: update gnulib submodule to latest * gl/modules/tempname.diff: Update to match current Gnulib. 2017-04-17 Bogdan Drozdowski shred: fix invalid pattern generation for certain sizes * src/shred.c (fillpattern): Fix the "off by one" issue when testing whether we have enough space to copy the already written portion of the buffer to the remainder of the buffer. Specifically for buffer sizes that are (3*(2^x))+1, i.e. 7,13,... we both use an uninitialized byte and invoke undefined behavior in memcpy() operation on overlapping memory regions. * tests/misc/shred-passes.sh: Add an invocation that will trigger either valgrind UMR, or ASAN like: ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges #1 0x403065 in fillpattern src/shred.c:293 A direct test is awkward due to the random writes surrounding the problematic pattern writes. Fixes http://bugs.gnu.org/26545 2017-04-16 Bo Rydberg doc: fix awk example for getting penultimate field * doc/coreutils.texi (cut invocation): Add required brackets. Fixes http://bugs.gnu.org/26519 2017-04-05 Sebastian Kisela tail: revert to polling if a followed directory is replaced * src/tail.c (tail_forever_inotify): Add the IN_DELETE_SELF flag when creating watch for the parent directory. After the parent directory is removed, an event is caught and then we switch from inotify to polling mode. Till now, inotify has always frozen because it waited for an event from a watched dir, which has been already deleted and was not added again. * tests/tail-2/inotify-dir-recreate.sh: Add a test case. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/26363 Reported at https://bugzilla.redhat.com/1283760 2017-04-05 Pádraig Brady maint: fix syntax-check issues in previous tty commit * src/tty.c: Avoid EXIT_FAILURE to be more descriptive and to placate sc_some_programs_must_avoid_exit_failure. 2017-04-05 Paul Eggert tty: handle misconfigured namespaces On some platforms, isatty succeeds but ttyname fails. POSIX does not seem to allow this, but there it is. Problem reported by Christian Brauner (Bug#26371). While we’re at it, check for errors more carefully and return a new exit status 4 if stdin is closed or a similar error occurs. * doc/coreutils.texi (tty invocation): Document new behavior. * init.cfg (stderr_fileno_): Don't assume have_input_tty is not in the environment. * src/tty.c (TTY_STDIN_ERROR): New constant. (main): Exit with nonzero status if there is a usage error, like other coreutils programs. Check for error in getting stdin type. * tests/misc/tty.sh: New file. * tests/local.mk (all_tests): Add it. 2017-04-02 Pádraig Brady doc: refactor and update expand and unexpand --help * src/expand-common.c (emit_tab_list_info): A new function to output the extended info on --tab=LIST, including the new '+' and '/' prefixes. * src/expand-common.h: Declare the above. * src/expand.c (usage:): Call emit_tab_list_info and match alignment with that used in unexpand --help. * src/unexpand.c (usage): Likewise. 2017-04-02 Jacob Keller expand,unexpand: add support for incremental tab stops Support --tabs="1,+8" which is equivalent to --tabs="1,9,17,..." useful for viewing unified diff output with its 1 character gutter for example. * doc/coreutils.texi ({expand,unexpand} invocation): Document, using diff processing as the example. * src/expand-common.c (set_increment_size): Update the new increment_size global. (parse_tab_stops): Handle the new '+' prefix. (finalize_tab_stops): Verify both '+' and '/' prefixes are not used together. * tests/misc/expand.pl: Add test cases. * NEWS: Mention the new feature. 2017-03-30 Paul Eggert sort: update comment * src/sort.c: Update identifiers in comment. 2017-03-29 Chris Davies doc: clarify in dd man page that bs= overrides [io]bs= * src/dd.c (usage): Add the extra info. Reported in https://bugs.debian.org/859021 2017-03-27 Ludovic Courtès tests: avoid false ulimit failure on some systems * tests/misc/cut-huge-range.sh: On some systems returns_ may use more memory, so incorporate that in the determination of the ulimit value to use. Noticed on ARMv7 with bash-4.4.12, and x86_64 with bash-4.2.37. Fixes http://bugs.gnu.org/26253 2017-03-27 Pádraig Brady maint: avoid syntax check failure with wrapped returns_ * cfg.mk (sc_prohibit_env_returns): Allow wrapped calls to return_ of the form: `wrapper_ returns_ ...` which is needed with the following commit. 2017-03-27 Michael Heimpold split: add new --hex-suffixes option * doc/coreutils.texi (split invocation): Document the new option. * src/split.c (usage): Likewise. (main): Process the new option much like --numeric-suffixes, but with an adjusted alphabet. * tests/split/numeric.sh: Refactor to support --hex mode. * NEWS: Mention the new feature. 2017-03-27 Pádraig Brady md5sum,b2sum,sha*sum: don't erroneously trigger BSD reversed mode * src/md5sum.c (split_3): Verify hex digits internally before triggering the global bsd_reversed mode flag. (bsd_split_3): Likewise. * tests/misc/md5sum-bsd.sh: Add a test case. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/26263 2017-03-26 Philipp Thomas df: avoid querying excluded file systems * src/df.c (filter_mount_list): Avoid stat() on explicitly excluded file systems, which is especially significant in cases like `-x nfs` which may hang. * NEWS: Mention the bug fix. 2017-03-26 Pádraig Brady maint: avoid a static analysis warning in expand-common * src/expand-common.c (next_file): We're dependent on calling this function with NULL to initialize things appropriately. So enforce this with assert(), which avoids a warning from clang-anaylzer. split: process more efficiently when filters exit early * src/split.c (bytes_split): Don't write to an existing filter if it has exited. When filters exit early, skip input data if possible. Refactor out 2 redundant variables. * tests/split/filter.sh: Improve test coverage given the new more efficient processing. Also use a 10TB file to expand the file systems tested on. 2017-03-26 Pádraig Brady split: ensure input is processed when filters exit early commit v8.25-4-g62e7af0 introduced the issue as it broke out of the processing loop irrespective of the value of new_file_flag which was used to indicate a finite number of filters or not. For example, this ran forever (as it should): $ yes | split --filter="head -c1 >/dev/null" -b 1000 However this exited immediately due to EPIPE being propagated back through cwrite and the loop not considering new filters: $ yes | split --filter="head -c1 >/dev/null" -b 100000 Similarly processing would exit early for a bounded number of output files, resulting in empty data sent to all but the first: $ truncate -s10T big.in $ split --filter='head -c1 >$FILE' -n 2 big.in $ echo $(stat -c%s x??) 1 0 I was alerted to this code by clang-analyzer, which indicated dead assigments, which is often an indication of code that hasn't considered all cases. * src/split.c (bytes_split): Change the last condition in the processing loop to also consider the number of files before breaking out of the processing loop. * tests/split/filter.sh: Add a test case. * NEWS: Mention the bug fix. 2017-03-11 Pádraig Brady tests: avoid a false failure on OS X 10.5.8 * tests/misc/sort-debug-keys.sh: Disparate LC_CTYPE and LC_MESSAGES are not supported, with the result LC_MESSAGES=C is used throughout. Therefore just set LC_ALL in the test, and normalize the message variants with sed. Reported and tested by J Rogowsky. 2017-03-10 Pádraig Brady build: fix missing renameat() on OS X 10.5.8 * bootstrap.conf: Depend on renameat. Reported and tested by J Rogowsky. Fixes http://bugs.gnu.org/26044 2017-03-09 Paul Eggert tests: port to tzdb-2017a Problem reported by Bernhard Voelker in: http://lists.gnu.org/archive/html/coreutils/2017-03/msg00026.html * tests/misc/date-debug.sh: Port test to tzdb 2017a, and future-proof the America/Belize test. 2017-03-08 Pádraig Brady build: for factor use C in more cases for arm64 and ppc64 * src/longlong.h: Sync from gmp repo incorporating: Use asm-free umul_ppmm() on arm64 and ppc64. doc: rearrange a recent bug entry to an improvement in NEWS * NEWS: The stat,tail change was an improvement, not a bug fix. * cfg.mk [old_NEWS_hash]: update with `make update-NEWS-hash`. maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.27 * NEWS: Record release date. build: update gnulib submodule to latest Removes extraneous parse-datetime.c generated during build stat,tail: support "RDT" Linux kernel control file system * src/stat.c (human_fstype): This file system is the user interface for resource allocation in Intel's Resource Director Technology. doc: spelling fix for recent doc addition * doc/coreutils.texi (join invocation): s/preceeding/preceding/. 2017-03-07 Bernhard Voelker build: avoid redundant build of tr with --enable-single-binary * src/local.mk [check-duplicate-no-install]: Depend on the single-binary tr, or the system tr, as the edge case where these are not available only result in the sanity check being effectively ignored. 2017-03-07 Assaf Gordon build: fix 'install-html' target Switching to non-recursive makefiles broke the 'install-html' target: The gettext plumbing requires an 'install-html' target in po/Makefile. This was fixed in gettext v0.19.8.1-41-ge5a008a, but packages using older gettext need to manually patch po/Makefile.in.in. Reported (for 'sed') and suggested fix by Eric Blake in https://bugs.gnu.org/25690 . * bootstrap.conf (bootstrap_epilogue): Add 'install-{html,pdf,dvi,ps}' targets to po/Makefile.in.in (if needed). 2017-03-04 Pádraig Brady maint: bump makeinfo --version requirement to 6.1 * bootstrap.conf: s/4.13/6.1/ as versions previous to that generated invalid html with interspersed tags that were visible to the user. Version 6.1 is available for a year now, and is available in most distros. 2017-03-04 Bernhard Voelker doc: avoid makeinfo warning makeinfo issues the following: doc/coreutils.texi:6568: warning: @sc argument all uppercase,\ thus no effect. * doc/coreutils.texi (join invocation): Remove the @sc macro around the all uppercase "GNU". 2017-03-03 Pádraig Brady build: update gnulib submodule to latest Fixes a test-lock failure on MacOS timeout: handle multiple children on solaris * src/timeout.c (install_sigchld): A new function to install the SIGCHLD handler using sigaction() rather than signal(), because with the latter on solaris the signal handler is reset to default and thus sigsuspend() only returns for the first finished child. Reported by Assaf Gordon. tests: avoid a spurious failure on older debian * tests/misc/cut-huge-range.sh: Bump up the ulimit, to avoid a false failure due hitting the previously detected ulimit. Reported by Assaf Gordon. build: fix libstdbuf build on AIX 7 * src/libstdbuf.c: undef malloc so as libstdbuf is not linked with gnulib, and anyway the replacement is never needed since we never malloc(0). Reported by Assaf Gordon. 2017-03-01 Assaf Gordon doc: expand 'join' info section * doc/coreutils.texi (join invocation): Expand section to add examples and more details. Suggested by Dan Jacobson in https://bugs.gnu.org/25870 2017-03-01 Pádraig Brady doc: give a stronger security warning in md5/sha1 man pages * man/md5sum.x: Give a more direct warning againt the use of this hash algorithm for security purposes. * man/sha1sum.x: Likewise. Suggested by Jim Meyering. expand: avoid an extraneous warning on 32 bit * src/expand-common (parse-tab-stops): Exit earlier upon overflow so another warning isn't issued (on 32 bit) in add_tab_stop(). Flagged in https://hydra.nixos.org/build/49499970 2017-02-28 Pádraig Brady doc: indicate sha1 has the same limitations as md5 * doc/coreutils.texi (sha1sum invocation): Given that a SHA-1 preimage attack has occurred as documented at http://shattered.io/, document sha1sum as having the same limitations as md5sum. (md5sum): Parameterize the warning for use in both cases. * man/md5sum.x: Mention b2sum(1) as a more secure alternative. * man/sha1sum.x: Give the same warning as done for md5sum(1). maint: avoid a -Werror=null-dereference with GCC-6.3.1 * src/stty.c (sane_mode): Assert to inform the compiler we know the pointer will be valid. expand,unexpand: support specifying a trailing tab size * doc/coreutils.texi (expand invocation): Document the feature. (unexpand invocation): Likewise. * src/expand-common.c (extend_size): A new global to use when the last tab stop is prefixed by '/'. (set_extend_size): A new function to validate and set the new extend_size global. (parse_tab_stops): Call set_extend_size() for '/' prefixes. (finalize_tab_stops): Ensure a single specified '/' is treated like a standard tabsize, but also ensure that when '/' is specified with a single other entry that we process as a list rather than a tab size. (get_next_tab_stop): Use the tab size if set, for items after the user specified tab position list. * tests/misc/expand.pl: Add test cases * NEWS: Mention the new feature. Fixes http://bugs.gnu.org/25540 2017-02-26 Pádraig Brady nproc: support OMP_THREAD_LIMIT to set a max value This comes from the latest gnulib. Also handling of OMP_NUM_THREADS has been adjusted to support comma separated values indicating a nesting level, in which case the first value is taken. Also OMP_NUM_THREADS=0 is now ignored instead of being treated as 1, to match the behavior of libgomp. * NEWS: Mention the OMP_THREAD_LIMIT improvement, and OMP_NUM_THREADS now handling nested values. * doc/coreutils.texi (nproc invocation): Describe OMP_THREAD_LIMIT as a way to set the max value, with OMP_THREAD_LIMIT setting the min. * tests/misc/nproc-override.sh: A new test to exercise the updated gnulib code with all combinations of these OMP variables. * tests/local.mk: Reference the new test. 2017-02-26 Pádraig Brady build: update gnulib submodule to latest 2017-02-22 Pádraig Brady doc: add NEWS for an improvement to dd in the last release * NEWS: Mention the avoidance of the gotcha with specifying a hex constant like count=0x1000 etc. as that previously was silently interpreted as 0. 2017-02-22 Pádraig Brady cp: set SELinux context for --parents directories * src/copy.c (set_process_security_ctx, set_file_security_ctx): Export for use in cp.c. * src/copy.h: Likewise. * src/cp.c (make_dir_parents_private): Call the exported functions to set the security context for new and updated directories. * tests/cp/cp-a-selinux.sh: Add a test case. Fixes http://bugs.gnu.org/25378 2017-02-17 Pádraig Brady maint: tweaks so syntax tests pass for previous commit * .gitignore: placate sc_gitignore_redundant. * po/POTFILES.in: placate sc_po_check. 2017-02-16 Bernhard Voelker doc: avoid makeinfo warning Commit v8.26-38-g99deaff introduced this warning: MAKEINFO doc/coreutils.info ./doc/coreutils.texi:10268: \ warning: `.' or `,' must follow @xref, not `@'. * doc/coreutils.texi (readlink invocation): Add '.' after @xref macro. 2017-02-16 Paul Eggert maint: xsetmode renamed to xbinary-io * bootstrap.conf, src/base64.c, src/cat.c, src/cksum.c: * src/head.c, src/md5sum.c, src/od.c, src/split.c, src/sum.c: * src/tac.c, src/tail.c, src/tee.c, src/tr.c, src/wc.c: Adjust to renaming of the xsetmode module to xbinary-io, and of the xsetmode function to xset_binary_mode. build: update gnulib submodule to latest 2017-02-15 Paul Eggert maint: use xsetmode, not xfreopen This fixes a bug noted by Eric Blake. Code was using xfreopen to change files to binary mode, but this fails for stdout when in append mode. Such code should use xsetmode instead. This affects only the port on platforms like MS-Windows which distiguish text from binary I/O. * bootstrap.conf (gnulib_modules): Remove xfreopen and add xsetmode. Sort. * src/base64.c (main): * src/cat.c (main): * src/cksum.c (cksum): * src/head.c (head_file, main): * src/md5sum.c (digest_file): * src/od.c (open_next_file): * src/split.c (main): * src/sum.c (bsd_sum_file, sysv_sum_file): * src/tac.c (tac_file, main): * src/tail.c (tail_file): * src/tee.c (tee_files): * src/tr.c (main): * src/wc.c (wc_file): Use xsetmode, not xfreopen. build: update gnulib submodule to latest 2017-02-12 Pádraig Brady maint: tweaks so syntax tests pass for previous commit * src/force-link.h: Don't include headers already included by system.h * src/force-link.c: Likewise. Also include system.h and explicitly mark extern functions as such. 2017-02-11 Paul Eggert ln: replace destination links more atomically If the file B already exists, commands like 'ln -f A B' and 'cp -fl A B' no longer remove B before creating the new link. Instead, they arrange for the new link to replace B atomically. This should fix a race condition reported by Mike Crowe (Bug#25680). * NEWS, doc/coreutils.texi (cp invocation, ln invocation): Document this. * bootstrap.conf (gnulib_modules): Add symlinkat. * src/copy.c, src/ln.c: Include force-link.h. * src/copy.c (same_file_ok): It's also OK to remove a destination symlink when creating symbolic links, or when the source and destination are on the same file system and when creating hard links. * src/copy.c (create_hard_link, copy_internal): * src/ln.c (do_link): Rewrite using force_linkat and force_symlinkat, to close a window where the destination temporarily does not exist. * src/cp.c (main): Do not set x.unlink_dest_before_opening merely because we are in link-creation mode. * src/force-link.c, src/force-link.h: New files. * src/local.mk (copy_sources, src_ln_SOURCES): Add them. * tests/cp/same-file.sh: Adjust test case to match fixed behavior. 2017-02-09 Tobias Stoeckmann timeout: fix race possibly terminating wrong process The race is unlikely, as timeout(1) needs to receive a signal in the few operations between waitpid() returning and exit(). Also the system needs to have reallocated the just released pid in this time window. Previously we never disabled the signal handler that sent the termination signal to the "child" pid. However once waitpid() has reaped the child, the system is free to allocate that pid, so we must ensure we don't process any further signals. * build-aux/gen-lists-of-programs.sh: Build timeout(1) optionally... * configure.ac: ...predicated on sigsuspend() being available. * src/timeout.c (block_cleanup): A new function to ensure the cleanup() handler is disabled after waitpid has returned. (main): Use sigsuspend() to wait with cleanup() enabled but disabled once it returns, and thus disabled for the waitpid() call. (monitored_pid): Change to the more accurate pid_t. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/25624 2017-02-09 Pádraig Brady doc: note the relationship between realpath and readlink * doc/coreutils.texi (realpath invocation): Mention that realpath is the preferred command for canonicalization. (readlink invocation): Likewise. * man/readlink.x: Likewise. 2017-02-08 Janne Snabb tail: fix output of redundant headers when resuming * src/tail.c (check_fspec): Only enable printing of the file header if we've actually read some data and this is a new file. Also move printing of the file header to... (dump_remainder): ...here, to allow printing only when data read. * tests/tail-2/overlay-headers.sh: A new test for suspension and resumption of tail. * tests/local.mk: Reference the new test. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/23539 2017-02-08 Pádraig Brady tests: fix tail test race causing false failure * tests/tail-2/retry.sh: The replacement of the "missing" directory is not atomic, and therefore tail(1) can take a different path, especially if there is a delay between the rmdir(2) and creat(2). This is noticeable for example with `make coverage` because in that case the coverage files written by rmdir(1) on exit, induce a significant delay thus triggering the issue. tail: fix erroneous status about 'giving up' on file * src/tail.c (recheck): Set f->ignore before we use it to show the appropriate error. * tests/tail-2/retry.sh: Ensure the "giving up" message is not presented. 2017-02-08 Pádraig Brady doc: only distribute 5 years of ChangeLogs Remove old log files that have corresponding entries in the source code repository. This saves about 2.5MB uncompressed, 0.5M compressed. * Makefile.am (gen-ChangeLog): Adjust to taking all logs since a particular version (8.15 in this case). Also mention in the truncated log where to get older entries. (changelog_etc): Remove the no longer distributed files. * build-aux/git-log-fix: Remove now unused entries. * ChangeLog-200[5-8]: Delete. * doc/ChangeLog-2007: Likewise. * po/ChangeLog-2007: Likewise. * old/*: Likewise. 2017-02-06 Maxime de Roucy maint: fix HACKING instructions to run a single test * HACKING: s/make TEST=/make check TEST=/ The 'check' target was missing there since v8.20-57-geac397e. 2017-01-25 Pádraig Brady build: fix issue with HAVE_FALLOCATE on centos5 * src/copy.c (punch_hole): Work around an empty definition of HAVE_FALLOCATE which leads to a build error of: "error: #if with no expression" That was triggered by the inclusion of in commit v8.25-68-g89e1fef with kernel-headers-2.6.18. Reported by Nelson H. F. Beebe 2017-01-25 Manolis Ragkousis (tiny change) build: Properly expand cu_install_program when cross-compiling * src/local.mk (cu_install_program): Replace @INSTALL_PROGRAM@ with @INSTALL@ when cross-compiling; missed in commit 477a1e8e. Message-Id: <20170125163329.5690-1-manolis837@gmail.com> 2017-01-21 Mike Swanson dircolors: highlight windows archive format * src/dircolors.hin: Match *.{wim,swm,dwn,esd} 2017-01-21 Pádraig Brady maint: mention the recent date time zone bug fix * NEWS: Add the bug fix from commit v8.26-27-gb14be50 2017-01-21 Assaf Gordon maint: appease syntax-check failures due to recent updates * bootstrap: s/time stamp/timestamp/. * old/fileutils/NEWS: Likewise. * src/tail.c: Avoided a long line. 2017-01-20 Paul Eggert date: fix TZ= regression Problem reported by Paul Wise for Debian, in: https://bugs.debian.org/851934 This is fallout from the fix for GNU Bug#23035. * src/date.c (batch_convert): New args TZ and TZSTRING. All uses changed. (batch_convert, main): Adjust to parse_datetime2 API change. (main): Allocate time zone object. * tests/misc/date-debug.sh: Fix incorrect test case, caught by the fix. * tests/misc/date.pl: Test the fix. build: update gnulib submodule to latest 2017-01-15 Jim Meyering maint: update README-hacking, now that vc-dwim accepts --init * README-hacking: I've just released vc-dwim-1.8, so we can improve the documentation to reference its --initialize option here. 2017-01-14 Paul Eggert date: output "-00" for indeterminate time zone * NEWS: Document this behavior, which comes with recent Gnulib. * doc/coreutils.texi (Formatting file timestamps, du invocation) (Time conversion specifiers, Setting the time, Options for date): Mention when -00 is output for numeric time zones. Be more careful about Internet RFC numbers, ISO 8601, etc. date: new option spelling --rfc-email * NEWS: * doc/coreutils.texi (Time conversion specifiers) (Options for date, Examples of date): Document this. * src/date.c (rfc_email_format): Rename from rfc_2822_format. All uses changed. (usage, long_options): Support --rfc-email. maint: modernize URLs A lot of this is converting http: to https:. Also, gmane went away, so remove URLs that no longer work and are not easy to figure out what they were. Some of this stuff is so old that it no longer matters anyway. build: update gnulib submodule to latest 2017-01-10 Bernhard Voelker doc: move "File timestamps" to a separate chapter The above new section looked a bit odd as the only general documentation in between the utility chapters. * doc/coreutils.texi (File timestamps): Move to a separate chapter. 2017-01-10 Eric Blake maint: fix recent syntax check failures Commit 4f650aad was incomplete; it changed NEWS but not the hash, and introduced a grammar error. * cfg.mk (old_NEWS_hash): Update via 'make update-NEWS-hash'. * doc/coreutils.texi (File timestamps): Fix doubled word. 2017-01-09 Paul Eggert doc: cover file timestamps better Prompted by a bug report from Scott Deerwester (Bug#25407). * doc/coreutils.texi (File timestamps): New section. Revamp other sections to use this new section, and use more-consistent terminology. maint: standardize on "timestamp" as per POSIX 2017-01-08 Pádraig Brady stty: ensure no side effects from invalid options * src/stty.c (apply_settings): A new function refactored from main() that is used to both check and apply options. (main): Call apply_settings before we open the device, so all validation is done before interacting with a device. * NEWS: Mention the improvement. * tests/misc/stty.sh: Add a test case. 2017-01-05 Assaf Gordon tests: improve 'date --debug' tests Update tests following improvements to gnulib's parse-datetime.y module. See https://lists.gnu.org/archive/html/bug-gnulib/2017-01/msg00002.html * tests/misc/date-debug.sh: Add tests for each of the gnulib changes. 2017-01-05 Assaf Gordon build: update gnulib submodule to latest 2017-01-01 Pádraig Brady maint: update all copyright year number ranges Run "make update-copyright" and then... * gnulib: Update to latest with copyright year adjusted. * tests/init.sh: Sync with gnulib to pick up copyright year. * bootstrap: Likewise. * tests/sample-test: Adjust to use the single most recent year. 2016-12-28 Zooko doc: recommend b2sum as well as SHA2 b2sum is faster, easier to use safely, and more future-proof 2016-12-28 Pádraig Brady tests: avoid false fails on NFS due to EPERM * tests/chgrp/basic.sh: On some NFS setups a user is not allowed to set a group on a file even if a member of that group. Therefore skip this test on remote file systems. * tests/chgrp/default-no-deref.sh: Likewise. * tests/chgrp/no-x.sh: Likewise. * tests/chgrp/posix-H.sh: Likewise. * tests/chgrp/recurse.sh: Likewise. * tests/tail-2/inotify-rotate-resources.sh: Change to skipping on remote file systems in the standard way. 2016-12-27 Paul Eggert doc: Update POSIX part of README (Bug#25259) 2016-12-26 Pádraig Brady wc: with only --bytes, determine size more efficiently * src/wc.c (wc): Avoid reading the end of the file when the size is not a multiple of PAGE_SIZE, as the special case handling for files in /proc and /sys is only required when st_size is 0 or a multiple of PAGE_SIZE. * tests/misc/wc-proc.sh: Add a test case. 2016-12-20 Pádraig Brady maint: correct the version for the previous bug fix While st_size would have been incorrect for subsequent files since v7.1, it was only used since v8.24. * tests/misc/wc-files0.sh: s/7.1/8.24/ * NEWS: Likewise. Reported by Bernhard Voelker 2016-12-19 William R. Fraser wc: fix wrong byte counts when using --files-from0 * src/wc.c (main): Reset fstatus[0].failed between files when reusing the fstatus[0] entry in --files-from0 mode. This ensures a stat() is done for each file, avoiding incorrect counts and redundant reading. * NEWS: Mention the bug fix. * tests/misc/wc-files0.sh: Add a test case. Fixes http://bugs.gnu.org/23073 2016-12-18 Pádraig Brady tests: fix typos in previous commit * init.cfg (skip_if_mcstransd_is_running_): Fix typos _introduced my me_ in the previous commit. 2016-12-18 Nicolas Iooss tests: support non-MLS enabled SELinux systems When running "make check" on a Linux system running SELinux with a non-MLS policy, tests/mkdir/restorecon.sh test fails with: chcon: invalid context: root:object_r:tmp_t:s0: Invalid argument Indeed in such a configuration, contexts cannot have ":s0" suffix. * init.cfg (get_selinux_type): Refactor this function to here from various tests. Update to work with a non-MLS policy. (mls_enabled_): A new function to detect if MLS is enabled. (skip_if_mcstransd_is_running_): Update to not skip when MLS is not enabled. * tests/mkdir/restorecon.sh: Use a valid non-MLS context when needed. * tests/install/install-Z-selinux.sh: Likewise. * tests/cp/cp-a-selinux.sh: Likewise. * tests/misc/selinux.sh: Likewise. * tests/misc/chcon.sh: Skip if non-MLS as --range used throughout. Fixes http://bugs.gnu.org/22631 2016-12-08 Torbjörn Granlund factor: retry properly if Pollard rho gives a trivial factorization * src/factor.c (factor_using_pollard_rho): Handle trivial factor g = n. (factor_using_pollard_rho2): Handle trivial factor g1 = n1, g0 = n0. * tests/misc/factor.pl: Add a test case. Fixes http://bugs.gnu.org/25135 2016-12-08 Niels Möller factor: fix infinite loop in gcd2_odd * src/factor.c (gcd2_odd): Fix the case a1 == 0, a0 == 0. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/25135 2016-12-02 Pádraig Brady doc: fix --help for: od -t f[SIZE] * src/od.c (usage): SIZE is that of float, not integer. 2016-11-30 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.26 * NEWS: Record release date. tests: fix false failure with spaces in $PWD * tests/misc/ptx-overrun.sh: Quote appropriately to avoid this recently added issue, noticed by `make taint-distcheck`. maint: avoid "make distcheck" failure without excess .deps directories * Makefile.am (my-distcheck): Add the -r option to xargs so that rmdir doesn't return an error when there are no extraneous .deps dirs. 2016-11-29 Pádraig Brady tests: fix ERRORs and false FAILs on some platforms * tests/misc/ls-time.sh: Skip the test rather than ERROR when `touch -m -d ...` fails (Hurd). * tests/tail-2/follow-stdin.sh: Avoid false FAILs by ignoring the variances in sterror output. * tests/rm/rm-readdir-fail.sh: Likewise. Also avoid ERRORs on systems that don't define _D_EXACT_NAMELEN. 2016-11-28 Pádraig Brady head: fix processing of non-seekable input as seekable * src/head.c (elide_tail_bytes_file): Ensure we don't use st_size unless we've previously used seek() to determine the CURRENT_POS in the seekable file. This was seen to cause issue on FreeBSD 11 when the pipe buffer was filled with `yes | head --lines=-0`, in which case st_size was 64KiB while ST_BLKSIZE() was 4KiB. Reported by Assaf Gordon. 2016-11-28 Kamil Dudka install,mkdir: fix handling of -DZ and -pZ, respectively ... in the case where two or more directories nested in each other are created and each of them defaults to a different SELinux context. * src/install.c (make_ancestor): When calling defaultcon(), give it the same path that is given to mkdir(). The other path is not always valid wrt. current working directory. * src/mkdir.c (make_ancestor): Likewise. * NEWS: Mention the bug fix. Reported at https://bugzilla.redhat.com/1398913 2016-11-28 Pádraig Brady tac: fix mem corruption when failing to read non seekable inputs This was detected with ASAN, but can also be seen without ASAN with: $ tac - - <&- tac: standard input: read error: Bad file descriptor *** Error in `tac': malloc(): memory corruption: 0x... * src/tac.c (copy_to_temp): Don't close our output stream on (possibly transient) output error, or on input error. (temp_stream): clearerr() on the stream about to be reused, to ensure future stream use is not impacted by transient errors. * tests/misc/tac-2-nonseekable.sh: Add a test case. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/25041 2016-11-27 Pádraig Brady tail: fix uninitialized memory read when failing to read file Reproduced under UBSAN with `tail -f <&-` giving: tail.c:2220:18: runtime error: load of value 190, which is not a valid value for type ‘_Bool' * src/tail.c (tail_file): Ensure f->ignore is initialized in all cases where we can't tail the specified file. * tests/tail-2/follow-stdin.sh: Add a test case which checks stderr has no UBSAN warnings. Fixes http://bugs.gnu.org/25041 2016-11-26 Pádraig Brady doc: add NEWS entries for recent changes * NEWS: Mention in improvements about the workaround for the glibc issue with closed stdin, and the new supported file systems. * tests/misc/b2sum.sh: Spelling fix. 2016-11-26 Pádraig Brady stat,tail: sync with latest Linux file systems Update with the results from: kgit='https://git.kernel.org/cgit/linux/kernel/git' wget -q $kgit/torvalds/linux.git/plain/include/uapi/linux/magic.h \ -O src/fs-latest-magic.h make src/fs-magic-compare * src/stat.c (human_fstype): Add entries for: BALLOON_KVM, CGROUP2, DAXFS, ZSMALLOC. 2016-11-26 Pádraig Brady build: fix potential factor build failure on arm and risc * src/longlong.h: Sync from gmp repo incorporating: Protect umul_ppmm with do ... while (0) Replace obsolete ARC asm 'J' constraints with 'Cal' Provide umul_ppmm for riscv64 2016-11-26 Pádraig Brady shred,sort: ensure faster unaligned access to rand module glibc has changed the public define from _STRING_ARCH_unaligned to _STRING_INLINE_unaligned as per https://sourceware.org/bugzilla/show_bug.cgi?id=19462 * gl/lib/rand-isaac.c: Cater for both defines. * gl/lib/randread.c: Likewise. * src/system.h: Update commented out code. 2016-11-26 Paul Eggert shuf: test input-closed bug Problem reported by Alex Ryan (Bug#25029). * tests/misc/shuf.sh: Test for shuffling with stdin closed. build: update gnulib submodule to latest 2016-11-25 Paul Eggert numfmt: pacify Sun C 5.14 * src/numfmt.c (main): Don't implicitly coerce pointer in bool initializer. Although it's portable C99 code, it's confusing. 2016-11-25 Pádraig Brady tests: fix false failure on new ls test * tests/ls/quote-align.sh: Remove "total" line which can vary per file system depending on allocation. Reported by Assaf Gordon on OpenSolaris (5.11/5.10). 2016-11-25 Pádraig Brady pr: fix read from invalid memory with tabs in separator This was detected with: echo a > a; pr "-S$(printf "\t\t\t")" a -m a > /dev/null Resulting in ASAN triggering: ==================================================== ERROR: AddressSanitizer: global-buffer-overflow READ of size 1 at 0x00000041b622 thread T0 #0 0x40506a in print_sep_string ../src/pr.c:2241 #1 0x407ec4 in read_line ../src/pr.c:2493 #2 0x40985c in print_page ../src/pr.c:1802 #3 0x40985c in print_files ../src/pr.c:1618 #4 0x4036e0 in main ../src/pr.c:1136 * src/pr.c (init_parameters): Ensure we only override the specified separator when it's a single tab, thus matching the calculated separator length. * tests/pr/pr-tests.pl: Add a test case. * NEWS: Mention the fix. 2016-11-24 Pádraig Brady ptx: fix an invalid heap reference with short --width * src/ptx.c (fix_output_parameters): Ensure line_width doesn't go negative, which can happen when the --width is less than the --gap-size. * tests/misc/ptx-overrun.sh: Add a test case that triggers with ASAN. (Note the longer filename is needed to trigger). Fixes http://bugs.gnu.org/25011 2016-11-24 Bernhard Voelker doc: clarify that readlink's --quiet option is on by default * src/readlink.c (usage): Mark the --quiet/--silent option as active by default. * doc/coreutils.texi (readlink invocation): Likewise. 2016-11-23 Pádraig Brady tests: fix false failure with ASAN in rm-readdir-fail * tests/rm/rm-readdir-fail.sh: ASAN correctly indicated that fts was writing to freed memory. This was because we reused a single dirent in our readdir() test wrapper. Since fts was deallocating those dirents, we now get a new dirent for each call to our readdir wrapper. 2016-11-23 Pádraig Brady split: fix memory corruption during chunk extraction ASAN reported this error for: split -n2/3 /dev/null ERROR: AddressSanitizer: negative-size-param: (size=-1) #0 0x7f0d4c36951d in __asan_memmove (/lib64/libasan.so.2+0x8d51d) #1 0x404e06 in memmove /usr/include/bits/string3.h:59 #2 0x404e06 in bytes_chunk_extract src/split.c:988 #3 0x404e06 in main src/split.c:1626 Specifically there would be invalid memory access and subsequent processing if the chunk to be extracted was beyond the initial amount read from file (which is currently capped at 128KiB). This issue is not in a released version, only being introduced in commit v8.25-4-g62e7af0 * src/split.c (bytes_chunk_extract): The initial_read != SIZE_MAX should have been combined with && rather than ||, but also this condition is always true in this function so remove entirely. * tests/split/b-chunk.sh: Add a test case. Fixes http://bugs.gnu.org/25003 2016-11-23 Pádraig Brady tail: fix checking of remoteness when not using inotify In recent commit v8.25-93-g7fc7206 we used the f->remote flag which wasn't set in all cases. This was detected with ASAN giving this error when reading f->remote; runtime error: load of value 190, which is not a valid value for type '_Bool' * src/tail.c (fremote): Query the system even without inotify. (recheck): Always set f->fremote for valid files. 2016-11-23 Jim Meyering build: avoid racy failure of "make -jN install" Installing with -j2 or greater could result in a failure like /bin/sh: line 29: /P/bin/install: Permission denied when /P/bin/install specifies your PATH-selected install program. This would arise because we're using "install" to install all man/*.1 files, and that command would run concurrently with the one that installs "/P/bin/install" itself. We would run this command: "src/ginstall src/ginstall /P/bin/install", and it would result in intervals during which the destination file does not exist, is empty or incomplete and not executable. We addressed this problem long ago for installation of actual binaries by telling the installation rules to use our just-built bin/ginstall (only when not cross-compiling) rather than the PATH-resolved "install" program. This change is to do the same for those .1 files. * src/local.mk (INSTALL): Override automake's default of something like "INSTALL = /P/bin/install -c". (INSTALL_PROGRAM): Now that we set INSTALL, there is no longer any need to set this derived variable. Its default definition, "INSTALL_PROGRAM = ${INSTALL}" does what we require. Improved by Eric Blake. 2016-11-23 Bernhard Voelker maint: fix 'syntax-check' targets for VPATH builds * cfg.mk (sc_gitignore_missing): Add $(srcdir) to .gitignore filename. (sc_gitignore_redundant): Likewise. maint: add build-aux/ar-lib to .gitignore * .gitignore: Add entry for the above file which is created by './bootstrap'. 2016-11-23 Paul Eggert pr: fix integer overflow in buffer size calcs Problem reported by Marcel Böhme (Bug#24996). * configure.ac (WERROR_CFLAGS): Avoid -Wtype-limits. * src/pr.c (col_sep_string): Now a const pointer. All uses changed. (integer_overflow): New function. (separator_string, main, init_parameters, init_store_cols): Check for integer overflow. (align_column, read_line, print_stored): Avoid integer overflow. 2016-11-22 Bernhard Voelker comm: add --total option * src/comm.c (total_option): Add bool variable for the new option. (TOTAL_OPTION): Add enum value. (long_options): Add array element for the new option. (usage): Document the new option here. (compare_files): Count the lines in total[3], and output the summary at the end. (main): Accept the new option. * doc/coreutils.texi (comm invocation): Document it. * tests/misc/comm.pl: Test it. While at it, improve the test data to have 1 unique line in the first file, 2 unique lines in the second file, and 3 common lines. * NEWS (New Features): Mention the new option. Fixes http://bugs.gnu.org/24929 2016-11-22 Pádraig Brady all: update gnulib submodule to latest Also sync these copies with gnulib: * bootstrap: Sync missed part of AIX support. * tests/init.sh: Support compare_ on AIX. tests: fix false failure with FreeBSD 11 multibyte quoting * tests/misc/printf-quote.sh: FreeBSD 11 was seen to treat \u0378 as a printable character. Therefore change to using the \u0081 C1 control character. We use the UTF-8 representation because our printf implementation explicitly disallows \u0081 as input. Reported by Assaf Gordon 2016-11-22 Pádraig Brady tests: fix false fails due to passing env vars to returns_ On BSD /bin/sh it was seen that unexported env vars passed to returns_() would not be propagated to the wrapped command. * cfg.mk (sc_prohibit_env_returns): Add a syntax check to disallow. * tests/misc/csplit-io-err.sh: Rearrange to export vars in a subshell. * tests/rm/rm-readdir-fail.sh: Likewise. * tests/misc/nohup.sh: Export and unset vars around returns_. * tests/misc/printenv.sh: Likewise. Reported by Assaf Gordon 2016-11-22 Pádraig Brady tests: fix recent tail-2 test regressions * tests/tail-2/pipe-f.sh: Avoid issue with readable directories on BSD systems. * tests/tail-2/retry.sh: Likewise. Reported by Assaf Gordon 2016-11-22 Pádraig Brady ls: improve alignment of quoted names This provides better alignment when some names are quoted, which also provides better indication that quotes are not part of the name. * src/ls.c (align_variable_outer_quotes): A new variable set when ls is aligning columns (not using -m, non-zero -w), and has a variable quoting style (shell, shell-escape, c-maybe). (quote_name_buf): Writes to buffer rather than FILE, taking care to avoid data copying if possible. Refactored from... (quote_name): ...here. This now manages the buffer passed to quote_name_buf() and outputs the padding, colors and name in the appropriate order, while managing the --dired offsets. (get_color_indicator): A new function to return the color sequence, refactored from... (print_color_indicator): ...here. This now simply outputs. (print_dir): Refactor common parts to quote_name(). (clear_files): Reset the flag indicating at least one file is quoted in the current directory. (needs_quoting): A new function to indicate at the scan stage whether a name needs quoting. Called from... (gobble_file): ...here, until we find the first quoted file. (print_name_with_quoting): Mostly refactored to quote_name(). * tests/ls/quote-align.sh: A new test for various output formats. * tests/local.mk: Reference the new test. * NEWS: Mention the improvement. 2016-11-22 Pádraig Brady b2sum: a new checksum utility with md5sum like interface Note we don't support the --algorithm option of the b2sum command in the external BLAKE2 project, as that was deemed too confusing for users. "BLAKE2b" was chosen as the default algorithm to use, which is single threaded but performs well on 64 bit. * src/blake2: CC0 source copied from external project. * cfg.mk[VC_LIST_ALWAYS_EXCLUDE_REGEX]: Exclude blake2/ from syntax checks, make update-copyright, etc. * src/local.mk: Reference the sources for b2sum, and set the compilation flags. * doc/coreutils.texi (b2sum invocation): Reference the md5sum invocation node, and add descriptions of -l. * tests/misc/b2sum.sh: Add new test. * tests/local.mk: Reference new test. * AUTHORS: Add new binary. * README: Likewise. * build-aux/gen-lists-of-programs.sh: Likewise. * man/.gitignore: Likewise. * scripts/git-hooks/commit-msg: Likewise. * man/b2sum.x: New man page template. * man/local.mk: Reference new template. * src/.gitignore: Ignore new binaries. * src/blake2/.gitignore: Ignore new build atrifacts. * src/md5sum.c (usage): Describe the new -l option. * NEWS: Mention the new program. 2016-11-22 Jim Meyering maint: avoid "make distcheck" failure due to excess .deps directories * Makefile.am (my-distcheck): Remove all .deps directories before the recursive diff that searches for left-behind files. Otherwise, with automake master (some time after v1.15), "make distcheck" would fail due to those directories being left behind after "make distclean". 2016-11-19 Pádraig Brady cp: improve status message when omitting directories * src/copy.h (cp_options): Add a new flag for install(1). * src/copy.c (copy_internal): For cp, also output remediation advice which also indicates why directories aren't copied by default. The message is unchanged for install(1). * src/cp.c (cp_option_init): Init install_mode to false. * src/mv.c (cp_option_init): Likewise. * src/install.c (cp_option_init): Init install_mode to true. * tests/install/basic-1.sh: Add a test case. * tests/cp/link-deref.sh: Adjust test case. Fixes http://bugs.gnu.org/24958 2016-11-15 Pádraig Brady tests: fix non portable date in recent test * tests/misc/date.pl: Years < 1900 fail on solaris at least. Reported by Assaf Gordon tests: fix false failure on OS X * tests/misc/sort-h-thousands-sep.sh: The sv_SE locale may be available while sv_SE.utf8 is not. Therefore test and use the same locale, now definfed in a variable. Reported by Jim Meyering. 2016-11-15 Antonio Ospite dircolors: highlight Motion JPEG multimedia files * src/dircolors.hin: Add .mjpg and .mjpeg multimedia files. This resolves http://bugs.gnu.org/24436. 2016-11-13 Jim Meyering tests: use "returns_" rather than explicit comparison with "$?" The previous "returns_"-using change failed to convert many uses of "$?". Convert all but two of the remaining ones. * tests/ls/stat-vs-dirent.sh: Likewise. * tests/misc/head-write-error.sh: Likewise. * tests/misc/nice.sh: Likewise. * tests/misc/nohup.sh: Likewise. * tests/misc/stdbuf.sh: Likewise. * tests/misc/sync.sh: Likewise. * tests/tail-2/pid.sh: Likewise. * tests/tail-2/wait.sh: Likewise. Thanks to Bernhard Volker for spotting this. 2016-11-13 Pádraig Brady all: reduce usage of proper_name_utf8() This reduces a standard coreutils install size by about 160K. * src/cat.c: Change to proper_name() which removes about 18K text. * src/cp.c: Likewise. * src/df.c: Likewise. * src/du.c: Likewise. * src/getlimits.c: Likewise. * src/realpath.c: Likewise. * src/split.c: Likewise. * src/stdbuf.c: Likewise. * src/timeout.c: Likewise. * src/truncate.c: Likewise. * src/local.mk: Remove -llibiconv from the above programs. * cfg.mk (sc_check-AUTHORS): Adjust to use factor(1). * AUTHORS: Adjust to use ASCII to satisfy sc_check-AUTHORS. 2016-11-13 Bernhard Voelker maint: add 'returns_' to exclude list in sc_prohibit_and_fail_1 The previous commit v8.25-96-g22063c8 lets 'make syntax-check' fail, because the above check falsely detects a case where 'returns_' is already in use. * cfg.mk (sc_prohibit_and_fail_1): Add 'returns_' to exclude list. 2016-11-13 Jim Meyering tests: use "returns_" rather than explicit comparison with "$?" * tests/chmod/no-x.sh: Likewise. * tests/cp/no-deref-link1.sh: Likewise. * tests/cp/no-deref-link2.sh: Likewise. * tests/cp/no-deref-link3.sh: Likewise. * tests/du/move-dir-while-traversing.sh: Likewise. * tests/ls/infloop.sh: Likewise. * tests/ls/stat-failed.sh: Likewise. * tests/ls/time-style-diag.sh: Likewise. * tests/misc/chroot-fail.sh: Likewise. * tests/misc/env-null.sh: Likewise. * tests/misc/env.sh: Likewise. * tests/misc/nice-fail.sh: Likewise. * tests/misc/printenv.sh: Likewise. * tests/misc/timeout.sh: Likewise. * tests/tail-2/follow-name.sh: Likewise. 2016-11-10 Pádraig Brady tail: only retry file open if --retry specifed * src/tail.c (tail_file): On failure to open a file, set ignore=true when --retry is not specified. * tests/tail-2/assert-2.sh: Adjust to the new behavior. * tests/tail-2/retry.sh: Add a test case. Also change from `tail ... && fail=1` to the more robust `returns_ 1 ...` construct which detects segfaults etc. * NEWS: Document the fix. 2016-11-09 Pádraig Brady tail: ensure -f --retry 'missing' handles truncation * src/tail.c (tail_forever): The BLOCKING optimization is only enabled for non regular files (which can't be truncated), so ensure we don't enable that unless we've a valid file descriptor. * tests/tail-2/retry.sh: Add a test case. * NEWS: Mention the bug fix. tail: avoid outputting repeated data with remote files * src/tail.c (tail_forever): Only read up to st_size on network file systems to avoid the issue with a stale attribute cache returning a smaller st_size than we have already read(). The was seen with glusterfs at least and caused the complete file to be repeatedly output due to assuming the file was truncated in this case. * NEWS: Mention the fix. 2016-11-08 Pádraig Brady tail: terminate when following pipes and untailable non pipes * src/tail.c (ignore_pipe_or_fifo): Mark the descriptor as -1 for pipes so that any_live_files() detects correctly that the entry is no longer live. * tests/tail-2/pipe-f.sh: Add a test case. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/24903 which was detected using Symbolic Execution techniques developed in the course of the SYMBIOSYS research project at COMSYS, RWTH Aachen University. 2016-11-07 Pádraig Brady maint: misc tweaks to recent changes * doc/coreutils.texi(cut invocation): Give a more accurate description of cut field handling limitations. * tests/misc/stat-fmt.sh: Fix the test header to be more general. * tests/tail-2/retry.sh: Spellings. maint: refactor printing of backup suffix --help * src/system.h (emit_backup_suffix_note): A new function to output the backup suffix info. The strings are unchanged, so translations are not impacted. * src/cp.c (usage): Use the new function. * src/ln.c (usage): Likewise. * src/mv.c (usage): Likewise. * src/install.c (usage): Likewise. 2016-11-07 Rishabh Dave maint: simplify handling of backup --suffix in various tools * src/cp.c (main): Avoid the getenv("SIMPLE_BACKUP_SUFFIX") call, which is now done if needed in the gnulib backupfile module. Also avoid the redundant strdup, as we don't modify this suffix. * src/install.c (main): Likewise. * src/ln.c (main): Likewise. * src/mv.c (main): Likewise. Fixes http://bugs.gnu.org/23153 2016-11-07 Pádraig Brady date: add %q to output the quarter of the year * doc/coreutils.texi (date invocation): Document %q. * src/date.c (usage): Likewise. * tests/misc/date.pl: Add a test case. * cfg.mk (sc_strftime_check): Adjust to allow %q. * NEWS: Mention the new feature. 2016-11-07 Pádraig Brady all: update gnulib submodule to latest Incorporating these coreutils specific changes: backupfile: initialize default suffix within the implementation strftime,strptime: support %q to represent the quarter Also sync these copies with gnulib: * bootstrap: Now supports AIX. * gnulib: Update to the latest. * tests/init.sh: dash(1) is now disabled. 2016-11-05 Pádraig Brady doc: clarify in texinfo that cut doesn't reorder fields * doc/coreutils.texi (cut invocation): Clarify the more sophisticated field processing supported by awk(1). Fixes http://bugs.gnu.org/24881 2016-11-04 Pádraig Brady stat: make --format=%N honor the QUOTING_STYLE env var * doc/coreutils.texi (stat invocation): Describe the QUOTING_STYLE values now supported. * src/stat.c (getenv_quoting_style): A new function called from main, that sets the default quoting style for quotearg. (main): Call getenv_quoting_style() when %N specified. * tests/misc/stat-fmt.sh: Add a test case. * NEWS: Mention the improvement. Fixes http://bugs.gnu.org/23422 doc: improve the --help description of ls -N * src/ls.c (usage): The existing description was inaccurate as -N does not directly impact whether control characters are displayed or not. dd: warn about counts specified with confusing 0x prefix * src/dd.c (parse_integer): Suggest to use "00x" instead of "0x", which is significant for the "count", "seek", and "skip" operands. * tests/dd/misc.sh: Add a test case. Fixes http://bugs.gnu.org/24874 2016-10-26 Pádraig Brady md5sum,sha*sum: fix --ignore-missing with checksums starting with 00 * NEWS: Mention the fix. * src/md5sum.c (digest_file): Add a new MISSING parameter to return whether the file was missing, separately from the digest. * tests/misc/md5sum.pl: Add a test case. Fixes http://bugs.gnu.org/24795 2016-10-21 Bishop Bettini doc: fix typo in stdbuf man page * man/stdbuf.x: Correct spelling of immediately. Fixes http://bugs.gnu.org/24763 2016-10-16 Jim Meyering maint: we use die; remove now-unnecessary uses of abort * src/expr.c (integer_overflow): Remove an abort-after-die. * src/paste.c (write_error): Likewise. * src/sort.c (badfieldspec): Likewise. * src/tr.c (string2_extend): Likewise. Also remove a few break statements, each after an abort. 2016-10-16 Pádraig Brady all: use die() rather than error(EXIT_FAILURE) die() has the advantage of being apparent to the compiler that it doesn't return, which will avoid warnings in some cases, and possibly generate better code. * cfg.mk (sc_die_EXIT_FAILURE): A new syntax check rule to catch any new uses of error (CONSTANT, ...); 2016-10-15 Jim Meyering build: also distribute new file, src/die.h * src/local.mk (noinst_HEADERS): Add die.h. build: add die.h; avoid new warnings from GCC 7 * src/die.h (die): New file/function from grep. Note: we expect this file to migrate to gnulib. * src/csplit.c: Include die.h. (check_format_conv_type): Use die in place of error-nonzero;break; * src/install.c (strip): Likewise. * src/nl.c (proc_text): Likewise. This also suppresses a new warning from GCC 7's -Werror=strict-overflow. * src/tail.c (parse_options): Likewise. * src/basename.c (main): Adjust "fall through" comment so that GCC 7's -Wimplicit-fallthrough honors it. * src/cp.c (main): Add a "fall through" comment. * src/ls.c (gobble_file): Likewise. (get_funky_string): Adjust a "fall through" comment so it is recognized. * cfg.mk (exclude_file_name_regexp--sc_system_h_headers): Add die.h to this list of exempt src/*.h files. maint: pr: avoid new GCC 7 warning * src/pr.c (main): Avoid this warning from GCC 7: src/pr.c:1119:6: error: assuming signed overflow does not occur when \ simplifying conditional to constant [-Werror=strict-overflow] if (n_files == 0) gnulib: update to latest and adapt printf-quote.sh test * .gitignore: Add /lib/limits.h. * tests/misc/printf-quote.sh: Reflect that a single quote is now rendered as "'", rather than as ''\'''. 2016-10-04 Pádraig Brady rm: disallow --n alias for --no-preserve-root * src/rm.c (main): Ensure the full --no-preserve-root option is specified, rather than allowing --n etc. * tests/rm/r-root.sh: Add a test case. * NEWS: Mention the change in behavior. Improved by Jim Meyering. Fixes http://bugs.gnu.org/24604 2016-09-28 Pádraig Brady tail: -F now always processes initially untailable files which was not the case when inotify was not available. * src/tail.c (any_live_files): Simplify, since the IGNORE flag is now only set when a file should be ignored indefinitely. (recheck): Only output the "giving up on name" message when that's actually the case. Only set the IGNORE flag when ignoring a file indefinitely. (tail_file): Likewise. * tests/tail-2/retry.sh: Add a test case. Also run all existing test cases with and without inotify. NEWS: Mention the fix. THANKS.in: Add the reporter. Fixes http://bugs.gnu.org/24495 which was detected using Symbolic Execution techniques developed in the course of the SYMBIOSYS research project at COMSYS, RWTH Aachen University. 2016-09-22 Jim Meyering who: avoid new warning from upcoming gcc-7 * src/who.c (idle_string): This function would fail to compile with -Werror and today's built-from-git gcc due to this warning: src/who.c: In function 'print_user': src/who.c:201:36: error: may write format character ':' at offset 4 \ past the end of the destination [-Werror=format-length=] sprintf (idle_hhmm, "%02d:%02d", ^~~~~ The fix is to use an assertion to inform gcc of the existing invariant that guarantees the number of hours is less than 24. 2016-09-19 Paul Eggert ls: fix %%b format and precompute more The old code mishandled --time-spec='+%%b', as it misinterpreted the '%b' as being the month abbreviation. Also, it mishandled the extremely unlikely case of a month abbreviation containing '%'. The performance part of this patch sped up 'ls' by about 1% on my little benchmark of 'ls -lR' on the source directory in the en_US.UTF-8 locale (Fedora 24 x86-64). * NEWS: Document the bug fix. * src/ls.c (first_percent_b, abformat_init): New static functions. (ABFORMAT_SIZE): New constant. (use_abformat): New static var. (abmon, required_mon_width): Remove these static vars. (abmon_init): Now accepts a pointer to abmon, and returns a boolean. All callers changed. Reject month abbrs containing '%', as these would mess up strftime. Simplify mbsalign result checking, since (size_t) -1 exceeds ABFORMAT_SIZE. (abformat_init, align_nstrftime): Precompute all 24 formats at startup, rather than computing a format for each time stamp. (decode_switches): Call abformat_init instead of abmon_init. (align_nstrftime): Accept recentness bool instead of format. All callers changed. * tests/misc/time-style.sh: Test for format with '%%b'. 2016-09-11 George Burgess IV pinky,who: remove redeclarations of ttyname Given that `ttyname` is already conditionally declared in src/system.h, other declarations are redundant and problematic for example in ChromeOS which has a new FORTIFY implementation that, as an artifact of how it's implemented, causes the compiler to complain if certain standard library functions are redeclared without special compiler-specific attributes. * src/pinky.c: Remove declaration (which was unused anyway). * src/who.c: Remove declaration. * src/system.h (getlogin, getuid, geteuid, getgrgid, getpwuid, ttyname): Add the parameter type to the declarations to avoid warnings when these backup declarations are used. 2016-09-09 Jim Meyering gnulib: update to latest, for getprogname fixes Compiling in gnulib-tests/ would fail on some platforms due to a missing lib/assert.h. Reported by Assaf Gordon in https://lists.gnu.org/archive/html/coreutils/2016-09/msg00016.html 2016-09-09 Paul Eggert cp: use FICLONE instead of BTRFS_IOC_CLONE This doesn't affect the generated code on my system; it's merely a cleaner way to use the recently-introduced Linux-specific API. * m4/jm-macros.m4 (gl_CHECK_ALL_HEADERS): Check for linux/fs.h. * src/copy.c: Include if available. (FICLONE) [__linux__]: Define if not already defined. (clone_file): Use FICLONE instead of the older BTRFS_IOC_CLONE, as this ioctl is no longer btrfs-specific. 2016-09-07 Assaf Gordon date: add '--debug' option Usage example: date --debug -d 'now + 3 days' Print parsing and debugging information to stderr when using date's -d/--date option. See: http://lists.gnu.org/archive/html/coreutils/2016-01/msg00076.html * src/date.c (main): Add '--debug' option, enable debugging in gnulib's parse-datetime.y module. * tests/misc/date.pl: Repeat tests with '--debug' enable, ensure no regression. * tests/misc/date-debug.sh: Test output of '--debug' option. * tests/local.mk: Add above test. * NEWS: Mention new option. * doc/coreutils.texi: Likewise. 2016-09-07 Paul Eggert sort: port to strict C + valgrind * bootstrap.conf (gnulib_modules): Add flexmember. * src/sort.c: Include flexmember.h. (struct tempnode): Make the last member flexible. (create_temp_file): Port to strict C11/C99 rules for allocation alignment with flexible array members. build: update gnulib submodule to latest 2016-09-07 Pádraig Brady factor: reinstate immediate output with interactive use * src/factor.c (lbuf_putc): Only buffer more than a line when not using the tool interactively. * NEWS: Mention the bug fix. Fixes http://pad.lv/1620139 2016-09-07 Kamil Dudka ls: allow interruption when reading slow directories Postpone installation of signal handlers until they're needed. That is right before the first escape sequence is printed. * src/ls.c (signal_setup): A new function refactored from main() to set and restore signal handlers. (main): Move signal handler setup to put_indicator() so that the default signal handling is untouched as long as possible. Adjusted condition for restoring signal handlers to reflect the change. (put_indicator): Install signal handlers if called for the very first time. It uses the same code that was in main() prior to this commit. * NEWS: Mention the improvement. See https://bugzilla.redhat.com/1365933 Fixes http://bugs.gnu.org/24232 2016-09-01 Pádraig Brady doc: fix typo in date example * doc/coreutils.texi (date invocation): Add a missing quotation mark. Fixes http://bugs.gnu.org/24349 2016-08-31 Jim Meyering dircolors: recognize .zst and .tzst suffixes * src/dircolors.hin: Add .zst and .tzst suffixes for the Zstandard/zstd compression tools. See http://zStd.net and https://code.facebook.com/posts/1658392934479273/ 2016-08-30 Jim Meyering maint: tr: tweak/make-consistent a pair of loop indices * src/tr.c (unquote): Use indices of type "unsigned int", rather than a mix or "size_t" and "int", presuming that no command-line- specified string will have length longer than UINT_MAX. tr: move decls "down" 2016-08-29 Jim Meyering ptx: avoid new warning/error from upcoming gcc-7.x * src/ptx.c (fix_output_parameters): Switch to an unsigned type that matches the OCCURS.file_index type. This avoids the following error from gcc-7.0.0 20160829 (experimental): src/ptx.c:1220:14: error: assuming signed overflow does not occur \ when simplifying conditional to constant [-Werror=strict-overflow] if (file_index > 0) 2016-08-18 Pádraig Brady cp: with --parents --no-preserve=mode don't copy dir perms * src/cp.c (make_dir_parents_private): Use default permissions for created directories when --no-preserve=mode is specified. * tests/cp/cp-parents.sh: Add a test case. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/24251 doc: fix ambiguous multiplier info in dd man page * src/dd.c (usage): Add a comma which is significant for the rewrapping done in man pages. 2016-08-09 Assaf Gordon maint: refactor common expand(1) and unexpand(1) code * src/expand.c, src/unexpand.c: Move global variables from here... * src/expand-common.h, src/expand-common.c: ... to here. * src/expand.c, src/unexpand.c: (parse_tab_stops, validate_tab_stops, next_file): Move identical functions to new module. (add_tab_stop): Move to new module, including additional code from 'unexpand' (keeping max_column_width) which will have no effect in when used in 'expand'. Refactor common next-column calculation code into a new function 'get_next_tab_column'. * src/local.mk: (src_expand_SOURCES, src_unexpand_SOURCES): Add 'expand-common.c'; (noinst_HEADERS): Add 'expand-common.h'. * po/POTFILES.in: Add 'expand-common.c'. * tests/misc/expand.pl: Add more tests. * tests/misc/unexpand.pl: Likewise. * TODO: Move conclusions to above test after investigation. 2016-08-03 Philipp Thomas df: improve performance with many mount points Use hash table for seaching in filter_mount_list() and get_dev() This improves performance for 20K mount entries from: real 0m1.731s user 0m0.532s sys 0m1.188s to: real 0m1.066s user 0m0.028s sys 0m1.032s * src/df.c (devlist_table): Define hash table. (devlist_hash): Add hash function. (devlist_compare): Add hash comparison function. (devlist_for_dev): Add lookup function. (devlist_free): Add cleanup function. (filter_mount_list): Use the above hash table. While at it, rename the variable 'devlist' to 'seen_dev' for better readability. (me_for_dev): Use the above lookup function. NEWS: Mention the improvement. THANKS.in: Remove the committer; add original submitter Josef Cejka. 2016-07-28 Pádraig Brady doc: revert recent date(1) synopsis change * src/date.c (usage): This was thought to introduce other ambiguities, and was inconsistent with the format presented in the touch(1) docs. * doc/coreutils.texi (date invocation): Likewise. See http://bugs.gnu.org/24077 2016-07-27 Bernhard Voelker maint: pacify GCC 6 with -Wnull-dereference src/id.c:249:29: error: potential null pointer dereference \ [-Werror=null-dereference] pw_name = xstrdup (pwd->pw_name); ~~~^~~~~~~~~ src/whoami.c:89:11: error: potential null pointer dereference \ [-Werror=null-dereference] puts (pw->pw_name); ~~^~~~~~~~~ * src/id.c (main): Explicitly exit with EXIT_FAILURE after an eror to help gcc-6 to detect that the dereferenced pointer is valid. * src/whoami.c (main): Likewise. 2016-07-27 Bernhard Voelker build: update gnulib submodule to latest Mainly for * 77d36b5 obstack: pacify GCC 6 with -Wnull-dereference 2016-07-27 Jim Meyering maint: skip a check when en_US.UTF-8 collation rules are broken * cfg.mk (sc_THANKS_in_sorted): This check would fail on systems for which "." is not ignored. Add a quick sort-based check for that error, and skip the check on any broken system. 2016-07-27 Pádraig Brady doc: make date(1) synopsis consistent with output +FORMAT * src/date.c (usage): Make the characters used to summarize the input format, match the output +FORMAT characters. * doc/coreutils.texi (date invocation): Likewise. Fixes http://bugs.gnu.org/24077 2016-07-26 Bernhard Voelker maint: fix dependency of man/arch.1 The following is a reproducer for the wrong dependency: $ ./configure --enable-install-program=arch $ make $ rm -f src/arch man/arch.1 $ make man/arch.1 GEN man/arch.1 help2man: can't get `--help' info from man/arch.td/arch Try `--no-discard-stderr' if option outputs to stderr Makefile:14378: recipe for target 'man/arch.1' failed make: *** [man/arch.1] Error 127 * man/local.mk (man/arch.1): Change to depend on src/arch rather than src/uname: while the arch binary depends on uname.c and uname-arch.c, its man page depends on the arch binary. Reported downstream by Rodrigues Goldwyn in https://build.opensuse.org/request/show/415172 2016-07-18 Kamil Dudka sort: with -h, disallow thousands separator between number and unit * src/sort.c (traverse_raw_number): Accept thousands separator only if it is immediately followed by a digit. * tests/misc/sort-h-thousands-sep.sh: Cover the fix for this bug. Suggested by Pádraig Brady in http://bugs.gnu.org/24015 2016-07-18 Kamil Dudka sort: make -h work with -k and blank used as thousands separator * src/sort.c (traverse_raw_number): Allow to skip only one occurrence of thousands_sep to avoid finding the unit in the next column in case thousands_sep matches as blank and is used as column delimiter. * tests/misc/sort-h-thousands-sep.sh: Add regression test for this bug. * tests/local.mk: Reference the test. * NEWS: Mention the bug fix. Reported at https://bugzilla.redhat.com/1355780 Fixes http://bugs.gnu.org/24015 maint: sort.c: deduplicate code for traversing numbers * src/sort.c (traverse_raw_number): New function for traversing numbers. (find_unit_order): Use traverse_raw_number() instead of open-coding it. (debug_key): Likewise. 2016-07-08 Kamil Dudka install: with -Z, set default SELinux context for created directories * doc/coreutils.texi (install invocation): Update -Z documentation. * src/install.c (make_ancestor): Set default security context before calling mkdir() if the -Z option is given. (process_dir): Call restorecon() on the destination directory if the -Z option is given. (usage): Update -Z documentation. * tests/install/install-Z-selinux.sh: A new test for 'install -Z -D' and 'install -Z -d' based on tests/mkdir/restorecon.sh. * tests/local.mk: Reference the test. * NEWS: Mention the improvement. Reported at https://bugzilla.redhat.com/1339135 Fixes http://bugs.gnu.org/23868 2016-07-06 Pádraig Brady tests: avoid false failure on 32 bit in readdir() test * tests/rm/rm-readdir-fail.sh: Since we use the returned dirent from the readdir wrapper it must be the correct type and not just cast. Therefore setup so that we only have to define a wrapper for readdir(), which works appropriately on 32 and 64 bit. Issue reported by Bernhard Voelker, where rm was seen to invoke rmdir() on invalid file names. 2016-07-02 Assaf Gordon tests: improve recent seq EPIPE test Follow-up to commit c92585b1: improve epipe test script, with suggestions from Bernhard Voelker and Pádraig Brady. see: http://lists.gnu.org/archive/html/coreutils/2016-04/msg00067.html . * tests/misc/seq-epipe.sh: Avoid too many subshells, stricter validation of errors and output from seq, skip if trapping SIGPIPE is not supported. 2016-06-29 Pádraig Brady stty: fix sane setting of susp to ^z on Solaris * src/stty.c: Disable setting of "swtch" to ^z as that conflicts with and disables using ^z as "susp". * NEWS: Mention the bug fix. Reported and tested by Rich Burridge at: http://bugs.gnu.org/23866 2016-06-26 Peter Benie tests: verify that fts diagnoses readdir() failures * tests/rm/rm-readdir-fail.sh: A new test to simulate readdir() failing immediately or after returning a few entries, and verifying that rm does the appropriate thing. This was initially reported at: http://bugzilla.opensuse.org/show_bug.cgi?id=984910 where it was mentioned that readdir() may fail when an NFS server has a poor readdir cookie implementation. 2016-06-26 Pádraig Brady all: update gnulib submodule and tests/init.sh to latest * gnulib: Update to latest. * NEWS: Specifically mention the fts readdir() fix and reindent to standard indentation. * tests/init.sh: Update from gnulib. 2016-06-24 Paul Eggert maint: simplify partial writes etc. in 'yes' * src/yes.c: Include full-write.h. (main): Use full_write, not write, to simplify handling of partial writes. Don't bother using stdio to output data; just use full_write with a buffer as large as needed. Reuse operand strings if possible, and if the buffer would otherwise be large. 2016-06-23 Pádraig Brady yes: handle short writes * src/yes.c (main): Loop over the write buffer to handle the case where write may write less than requested. * NEWS: Mention the bug fix. Reported by Paul Eggert. yes: fix copy and paste issue with previous commit * src/yes.c (main): Output 'y' not '-'. * tests/misc/yes.sh: Add a test for default output. 2016-06-23 Paul Eggert maint: work even if argc == INT_MAX GCC 7 warned about undefined behavior in this unlikely case. Problem reported by Jim Meyering in: http://bugs.gnu.org/23825 * src/md5sum.c (main): * src/paste.c (main): * src/yes.c (main): Avoid undefined behavior when argc == INT_MAX. 2016-06-01 Pádraig Brady doc: clarify sort --key handling of default field separators * doc/coreutils.texi (sort invocation): Mention in the summary dicussion that --key is used to specify fields. Give a summary in the --key description, of the most common use case of specifying a field, and that by default those fields include the blank separators at the start of each field in the comparisons. 2016-05-31 Assaf Gordon sort: modify 'leading spaces' debug warning scenarios Print warning regardless of locale, avoid warning if key is zero width. Problem reported by Karl Berry in http://bugs.gnu.org/23665 . * src/sort.c: (key_warnings): change conditions for 'leading spaces' warning. * tests/misc/sort-debug-warn.sh: adjust tests accordingly. 2016-05-31 Assaf Gordon tests: sort-debug-warn: add progress information lines Easier troubleshooting of individual 'sort --debug' messages. Discussed in http://bugs.gnu.org/23665 . * tests/misc/sort-debug-warn.sh: add progress number before each sort invocation. 2016-05-15 Pádraig Brady dircolors: output colors for any $TERM matching *color* * src/dircolors.hin: Reduce the list by replacing all specific mentions of "color" with the pattern "*color*". This will also cater for other entries like "konsole-256color". Fixes http://bugs.gnu.org/23542 2016-05-14 KOBAYASHI Takashi nl: reset numbering for each section * NEWS: Mention the bug fix. * src/nl.c (proc_body, proc_footer): Reset numbering here too. * doc/coreutils.texi (nl invocation): Adjust for behavior change. * tests/misc/nl.sh: Add a test case. 2016-05-12 Pádraig Brady stat,tail: add support for the SMB2 remote file system * stc/stat.c (human_fstype): Add file system ID definition, and use "smb2" as the name. * NEWS (Improvements): Mention the change. Fixes http://bugs.gnu.org/23516 2016-05-04 Jim Meyering maint: avoid new warning from gcc (GCC) 7.0.0 20160503 (experimental) * src/id.c (main): When configured with --enable-gcc-warnings and using the very latest gcc built from git, building would fail with this: src/id.c:200:8: error: assuming signed overflow does not occur when \ simplifying conditional to constant [-Werror=strict-overflow] bool default_format = (just_user + just_group + just_group_list ^~~~~~~~~~~~~~ Rewrite to use bool-appropriate operators. 2016-04-26 Pádraig Brady tests: avoid false failure when df lists no root file system * tests/df/skip-duplicates.sh: Skip the test when there is no '/' entry listed by df, which was seen in certain chroot setups. 2016-04-24 Assaf Gordon seq: detect and report I/O errors immediately Ensure I/O errors are detected (and terminate seq), preventing seq from infloop (or running for long time with a large range) upon write errors or ignored SIGPIPE. Examples: seq 1 inf > /dev/full (seq_fast) seq 1.1 0.1 inf >/dev/full (print_numbers) * src/seq.c (io_error): A new function to diagnose appropriate stdio errors and exit the program with failure status. (seq_fast, print_numbers): Explicitly check for write errors and terminate the program with diagnostic. * tests/misc/seq-io-errors.sh: Test error detection with /dev/full. * tests/misc/seq-epipe.sh: Test error detection with broken pipes. * tests/local.mk: Add new tests. * NEWS: Mention the fix. 2016-04-22 Pádraig Brady stat,tail: add support for Windows Subsystem for Linux * src/stat.c (human_fstype): Add file system ID definition, and use "wslfs" as the name. * NEWS (Improvements): Mention the change. Fixes http://bugs.gnu.org/23273 2016-04-21 Pádraig Brady yes: avoid redundant diagnostics on write error * src/yes.c (main): For large inputs only write a single diagnostic for write errors. * tests/misc/yes.sh: Test when /dev/full is available. sleep,timeout: support overflowing floating point values * src/sleep.c (main): Allow ERANGE since we allow "inf" values. * src/timeout.c (parse_duration): Likewise. * tests/misc/sleep.sh: New file. Tests for sleep(1). * tests/misc/timeout-parameters.sh: Add case for newly allowed $LDBL_MAX. Also use returns_ throughout the file. Also avoid small timeout values which might give false failures under load. tests: avoid potential overflows with `expr` * cfg.mk (sc_prohibit_expr_unsigned): expr uses intmax_t internally when GMP is not available, so flag any uses of unsigned limits. * tests/misc/timeout-parameters.sh: Remove the overflow related to UINT_MAX as it's handled by the following case. Change the following case from 99... to $TIME_T_OFLOW to be more expressive in what it's actually testing. Directly check that commands succeed, rather than inspecting $? afterwards. * tests/dd/skip-seek-past-dev.sh: Guard against large device sizes. * tests/id/setgid.sh: Protect: Guard against large group IDs. * tests/misc/cut-huge-range.sh: Tweak comment to avoid syntax check. It's tempting to simplify to just skip the test if expr fails, but SIZE_MAX > INTMAX_MAX is the common case. 2016-04-20 Pádraig Brady tests: fix recent false failure due to a race * tests/cp/parent-perm-race.sh: This new race introduced in commit v8.25-5-g632eda5 is quite hard to hit, but is due to `ls > fifo` doing write()/close()/exit() once `cp` has open() the source fifo. Then the subsequent comparison of the destination file may fail due to the file being missing or empty. Previously `ls` generated output that was independent of `cp`. Now we must wait for `cp` to finish before inspecting the destination file that it wrote. 2016-04-18 Bernhard Voelker tests: let mkdir/p-v.sh fail when compare fails * tests/mkdir/p-v.sh: Set fail=1 when compare fails: the latter only sets a non-Zero return code; evaluating that is up to the caller. 2016-04-14 Bernhard Voelker seq: do not allow NaN arguments * src/seq.c (isnan): Define macro. (scan_arg): Add check if the given argument is NaN, and exit with a proper error diagnostic in such a case. (usage): Document it. * tests/misc/seq.pl: Add tests. * doc/coreutils.texi (seq invocation): Document the change. * NEWS (Changes in behavior): Mention the change. seq: do not allow 0 as increment value * src/seq.c (main): Exit with an error diagnostic when the given step value is Zero. (usage): Document it. * doc/coreutils.texi (seq invocation): Likewise. * tests/misc/seq.pl: Add tests. * NEWS (Changes in behavior): Mention the change. Reported by Маренков Евгений in: http://bugs.gnu.org/23110 2016-04-14 Yigal Korman stat,tail: add support for M1FS * src/stat.c (human_fstype): Add file system ID definition. * NEWS (Improvements): Mention the change, moving the previously added "prl_fs" change note from "Changes in behavior" to here. Fixes http://bugs.gnu.org/23283 2016-03-31 Pádraig Brady stat: report type for "prl_fs" file systems * src/stat.c (human_fstype): Identify the parallels file system. Also tag as remote so that tail(1) doesn't use inotify, which fails to detect changes made outside a VM. Fixes http://bugs.gnu.org/23143 2016-03-17 Paul Eggert date ls pr: fix time zone abbrs on SysV platforms The problematic code computed a struct tm in one time zone, and then printed it or converted it to a string in another. To be portable the same time zone needs to be used for both operations. On GNU platforms this is not an issue, but incorrect output can be generated on System V style platforms like AIX where time zone abbreviations are available only in the 'tzname' global variable. Problem reported by Assaf Gordon in: http://bugs.gnu.org/23035 * NEWS: Document the bug. * src/date.c (show_date): * src/ls.c (long_time_expected_width, print_long_format): * src/pr.c (init_header): * src/stat.c (human_time): Use localtime_rz instead of localtime, so that the time zone information is consistent for both localtime and time-formatting functions like fprintftime and nstrftime. For 'stat' this change is mostly just a code cleanup but it also causes stat to also print nanoseconds when printing time stamps that are out of localtime range, as this is more consistent with what other programs do. For programs other than 'stat' this fixes bugs with time zone formats that use %Z. * src/du.c, src/pr.c (localtz): New static var. (main): Initialize it. * src/du.c (show_date): New time zone argument, so that localtime and fprintftime use the same time zone information. All callers changed. * tests/misc/time-style.sh: New file. * tests/local.mk (all_tests): Add it. * tests/misc/date.pl: Test alphabetic time zone abbreviations. 2016-03-15 Jim Meyering maint: remove gl/lib/reg*.c.diff; no longer needed * gl/lib/regcomp.c.diff: Remove file, now that gnulib's regcomp.c compiles regex.c with -Wno-unused-parameter. * gl/lib/regex_internal.h.diff: Likewise. * gl/lib/regex_internal.c.diff: This file induced a change to ensure that the "Idx" type was unsigned and to remove a few "VAR < 0" comparisons. These days, it is probably fine to stay in sync with gnulib/glibc's copies of these files, so remove these patches, too. * gl/lib/regexec.c.diff: Likewise. Prompted by a report by Assaf Gordon and a suggestion from Paul Eggert. 2016-03-09 Eric Blake build: update gnulib submodule to latest Mainly for: *bdb72bc6 set-permissions: fix compilation on Cygwin * bootstrap: Sync with gnulib. * gl/lib/regcomp.c.diff: Regenerate against latest gnulib. 2016-03-06 Jim Meyering tests: avoid false-failure of split/filter.sh on XFS * tests/split/filter.sh: Use OFF_T_MAX-1 rather than OFF_T_MAX as the size of a test file, to avoid false failure on an XFS file system (or any file system permitting a file of size OFF_T_MAX). Reported as http://bugs.gnu.org/22931 2016-03-04 Eric Blake test: Document that -a and -o are undesirable POSIX recommends avoiding -a and -o, for good reason. src/test.c (usage): Mention that inherent ambiguities exist with binary -a and -o. Problem reported by Martin Gebert in: http://bugs.gnu.org/22909 2016-02-23 Paul Eggert all: be less strict about usage if POSIX 2008 sort, tail, and uniq now support traditional usage like 'sort +2' and 'tail +10' on systems conforming to POSIX 1003.1-2008 and later. * NEWS: Document this. * doc/coreutils.texi (Standards conformance, tail invocation) (sort invocation, uniq invocation, touch invocation): Document new behavior, or behavior's dependence on POSIX 1003.1-2001. * src/sort.c (struct keyfield.traditional_used): Rename from obsolete_used, since implementations are now allowed to support it. All uses changed. (main): Allow traditional usage if _POSIX2_VERSION is 200809. * src/tail.c (parse_obsolete_option): Distinguish between traditional usage (which POSIX 2008 and later allows) and obsolete (which it still does not). * src/uniq.c (strict_posix2): New function. (main): Allow traditional usage if _POSIX2_VERSION is 200809. * tests/misc/tail.pl: Test for new behavior. 2016-02-22 Paul Eggert doc: document sort -o F F danger Problem reported by Yanyan Jiang 蒋炎岩 in: http://bugs.gnu.org/22769 * doc/coreutils.texi (sort invocation): Mention possibility of 'sort -o F F' data loss during crashes. 2016-02-16 Daniel Lockyer date: free timezone variable to avoid valgrind warning * src/date.c (main): Free TZ variable to pacify valgrind, guarded by IF_LINT. 2016-02-15 Pádraig Brady split: adjust recent --number changes * src/split.c (lines_rr): Reinstate the conditional setting of the WROTE boolean, as otherwise split -n r/1 would consume all input when all --filter commands are stopped. There was a test in place to check for this, but it was incorrect as detailed below. (input_file_size): Immediately disallow --number with non seekable inputs, as such an invocation is not currently generally supported and will fail as the data overflows the internal buffer. * tests/split/l-chunk.sh: Adjust to again disallow -n /dev/zero. Also change all '&& fail=1' checks to use the 'returns_ 1' form. * tests/split/filter.sh: Change the no longer supported /dev/zero case to a regular $OFF_T_MAX file (supported on XFS for example). Also fix the timeout(1) commands so they're not subject to pipefail issues. 2016-02-15 Göran Uddeborg stty: fix translated --help so the gettext header is not printed * src/stty.c (usage): Remove an erroneous call to translate an empty string, added in commit v8.23-112-g564f84a, which results in the gettext header being printed for translated languages. * THANKS.in: Remove the now committer. * NEWS: Mention the bug fix. 2016-02-12 Paul Eggert tests: don't wait forever on GNU/Hurd * tests/cp/parent-perm-race.sh: Add timeouts so that the test does not wait forever on GNU/Hurd. This does not fix the underlying bug but at least lets the tests make progress. Problem reported by Nelson H.F. Beebe in: http://bugs.gnu.org/22624 Also, check that 'cp' actually copies the data; this is a better test anyway, and simplifies the test code. Also, avoid unlikely race if 'pid' is set in the environment. split: fix problems with /dev/zero Problem reported by Nelson H.F. Beebe in: http://bugs.gnu.org/22624 Other problems also fixed: basically, the code got confused because GNU/Linux reports that /dev/zero has size zero. * src/split.c (input_file_size): Now takes struct stat *, not just size. Always store the first buffer. All callers changed. Treat /dev/zero as an infinitely-large file, both on GNU/Linux where fstat and lseek say its size is zero, and on GNU/Hurd where they say the size is OFF_T_MAX. (cwrite): Return true on success. (bytes_split): Don't try to read past EOF, and stop if a write fails. (lines_rr): Omit stray check for ignorable errno. (main): Get file size only when n_units > 1, since that's the only time it is needed. Defer most of the work to input_file_size. * tests/split/l-chunk.sh: Adjust tests to match new behavior on oddball inputs. 2016-01-29 Vincent Lefevre doc: fix the --iso-8601 example in date --help * src/date.c (usage): The colon is used in the timezone offset since commit v8.24-64-g17bbf6c. * THANKS.in: Remove committer. Fixes http://bugs.gnu.org/22491 2016-01-24 Pádraig Brady build: avoid dynamic linking issue on Solaris sparc Solaris Studio 12 on sparc (not x86) will not remove unused functions, thus leaving a reference to an undefined program_name symbol from emit_try_help(). * src/system.h (emit_try_help): Change from an inline function to a macro, so that the inline function is not actually defined in libstdbuf.c. Fixes http://bugs.gnu.org/22430 2016-01-20 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.25 * NEWS: Record release date. 2016-01-19 Pádraig Brady tests: avoid false failure in tail inotify test * tests/tail-2/wait.sh: Restrict inotify specific test portion to local file systems and also not with ---disable-inotify specified. Failure noticed on NFS. tests: avoid false failure due to remove() ignoring u-w on NFS * tests/rm/rm1.sh: Also remove the group write bit which was required on one NFS setup at least. Note u-w was enough to deny file creation, g-w was also required to deny file removal. * tests/rm/cycle.sh: Likewise. * tests/mv/perm-1.sh: Likewise. tests: avoid false failure in preserve-slink-time.sh on GPFS * tests/cp/preserve-slink-time.sh: Add a delay between the ln and the cp so that there is enough difference between the timestamps so GPFS won't discard the update. Reported by Assaf Gordon. tests: avoid false failure in shred-passes.sh * tests/misc/shred-passes.sh: Specify an exact amount to shred, to avoid running out of simulated random data on file systems with a large st_blksize like GPFS for example. Reported by Assaf Gordon. 2016-01-19 Pádraig Brady tail: disable inotify with only non existent files tests/tail-2/F-headers.sh and test/tail-2/retry.sh fail on on remote file systems due to tail going into inotify mode due to not being able to determine the remoteness of the non existent files. * src/tail.c (any_non_remote_file): A new function used to disable inotify when there are no open files, as we can't determine remoteness in that case. * NEWS: Mention the bug fix. 2016-01-18 Pádraig Brady all: don't unconditionally use leaf optimization for NFS * gnulib: Update to latest where the only change is to not unconditionally enable leaf optimization for fts on NFS, as it was seen to abort() with some NFS servers as per: https://bugzilla.redhat.com/1299169 This affects utilities that traverse directories like cp, rm, chmod etc. * NEWS: Adjust the improvement message to leave only XFS. 2016-01-18 Assaf Gordon tests: adjust memory limits in head-c.sh * tests/misc/head-c.sh: Further increase the limit to 8MB over the base test command. 5.1MB was seen as minimum extra needed on FreeBSD 9.3 x86_64. 2016-01-18 Pádraig Brady tests: adjust delays to avoid false positives * tests/rm/dangling-symlink.sh: The effective delay was only 1.5s. So delay before the operation, and increase the iteration count by 1 to increase the delay to 6.3s. This failure was noticed once on a FreeBSD 10.2 x86_64 virtual machine. * tests/dd/stats.sh: Remove quotes on sleep argument to ensure arguments are accumulated appropriately. * tests/du/move-dir-while-traversing.sh: sleep before operation, to increase the effective delay from 1.6s to 3.2s. * tests/tail-2/flush-initial.sh: Likewise. * tests/tail-2/pipe-f2.sh: Sleep first so that the effective delay matches the commented value. test: skip test rather than fail with Solaris 10 sed * tests/rm/r-root.sh: This test was seen to ERROR due to sed input not having a terminating '\n'. Therefore just skip with diagnostics upon failure to adjust the error output with sed. Reported by Assaf Gordon. 2016-01-16 Pádraig Brady tests: fix false failure with spaces in absolute $PWD * tests/install/create-leading.sh: Ensure $PWD is quoted. Reported by `make dist-check`. 2016-01-16 Pádraig Brady build: update gnulib submodule to latest Mainly for: * dce117a sig2str: list all signals on FreeBSD >= 7 * cff4b38 acl-permissions: port to USE_ACL==0 platforms 2016-01-16 Pádraig Brady tests: show mount list on failure for df tests * init.cfg (dump_mount_list_): A new function to output the system mount list. * tests/df/df-symlink.sh: Call dump_mount_list_ upon failure. * tests/df/over-mount-device.sh: Likewise. * tests/df/problematic-chars.sh: Likewise. * tests/df/skip-rootfs.sh: Likewise. * tests/df/total-verify.sh: Likewise. * tests/df/unreadable.sh: Likewise. tests: avoid false failure with BTRFS subvolumes * tests/df/df-symlink.sh: Only check the symmetry of the source <-> target lookup, when the source is only mounted once, which isn't the case if '.' is a BTRFS subvolume for example. Reported by Assaf Gordon on a default OpenSUSE 42.1 install. mv: consistently warn about multiply specified source dirs * src/copy.c (copy internal): Remember directories irrespective of their link count, because on some file systems like BTRFS, directories always have a link count of 1. 2016-01-15 Pádraig Brady tests: remove a non portable localtime test * tests/misc/date.pl: This test was dependent on perl unconditionally calling the system localtime() as date(1) does. However perl has its own configure checks in place to avoid buggy localtime implementations. Therefore comment out this test as a FIXME for now. This test was seen to fail on FreeBSD 10 where localtime() returns nonsense values instead of failing for arguments >= 2^56. tests: adjust memory limits in head-c.sh * tests/misc/head-c.sh: Further increase the limit, to account for the fact that head(1) needs at least 1.576MB over the base test command on FreeBSD 10. 4MB should be enough headroom while still providing over allocation protection. Reported by Assaf Gordon. tests: simplify invalid signal determination for kill -l * src/operand2sig.c (operand2sig): Add a detailed comment explaining why we validate even very large shell exit status values. * tests/misc/kill.sh: Add a test case for the ksh scheme. Simplify the INVALID signal number determination which also avoids a false failure on systems like FreeBSD 10 with incomplete signal list (caused by inaccurate NSIG). tests: support older fallocate number formats * tests/cp/fiemap-extents.sh: Support RHEL6 fallocate which doesn't support IEC suffixes like "MiB" on numbers. Also add some extra framework_failure_ protections. Reported by Assaf Gordon. tests: avoid coredumps when determining memory limits * init.cfg (get_min_ulimit_v_): Refactor ulimit call to... (ulimit_supported_): ...here, and add calls to avoid coredumps. 2016-01-13 Pádraig Brady sort: with --debug, flag setlocale() failures on OpenBSD Locale categories are not equivalent on OpenBSD, and LC_COLLATE only supports "C" for example. Now LC_ALL is supported to set multiple other categories on OpenBSD, so setlocale(LC_ALL, "") returns a string indicating which categories were updated and which ignored. Therefore... * src/sort.c (main): ...Call setlocale(LC_COLLATE, "") to explicitly check whether a specified LC_ALL or LC_COLLATE environment variable value is supported for the LC_COLLATE category. Also use !! to explicitly convert to bool to support c89 systems where bool is an int, and thus would get values > 1. Reported by Assaf Gordon. 2016-01-13 Pádraig Brady maint: remove temporary doc/constants.texi-t during build * doc/local.mk (constants.texi): Remove temp file. maint: fix old spelling mistakes in NEWS * NEWS: s/descendent/descendant/; s/indiscriminite/indiscriminate/. * cfg.mk (old_NEWS_hash): Update via `make update-NEWS-hash`. doc: suggest dd "sync" flag to maximize "nocache" effectiveness doc/coreutils.texi (dd invocation): Add oflag=sync to the streaming example. Also reference the "direct" flag. Mention this is only a request to the system. * src/dd.c (usage): Mention the "sync" flag along with "nocache". Also mention that it's only a request to drop the cache. * THANKS.in: Add reporter Francois Rigault. doc: mention the '0#' printf flags with stat %a * src/stat.c (usage): Mention the '#' and '0' flags are useful with %a. * doc/coreutils.texi (stat invocation): Likewise. Also give an example printing unambiguous octal output. Reported at http://bugs.debian.org/810539 2016-01-13 Pádraig Brady mv: fix data loss with repeated source dir and same destination commit v8.23-31-g90aa291 failed to consider this case, where the previous rename has failed, thus causing the following to remove the specified directory: mv dir dir dir * src/copy.c (copy_internal): Assume this rename attempt has succeeded, as a previous failure will already have been handled, and we don't want to remove the source directory in this case. * tests/cp/duplicate-sources.sh: Consolidate this test file to... * tests/mv/dup-source.sh: ...here. Add test cases for same source and dest. * tests/local.mk: Remove the consolidated test. * NEWS: Mention the bug fix. Reported at https://bugzilla.redhat.com/1297464 2016-01-13 Pádraig Brady tests: include part of wc-files0.sh skipped by mistake * tests/misc/wc-files0.sh: $fail is always set to 0, so correct the test to include that part of the test case. wc: avoid ambiguous output with '\n' in file names * src/wc.c (write_counts): Shell escape the file name if it contains '\n' so only a single line per file is output. * tests/misc/wc-files0.sh: Add a test case. * NEWS: Mention the improvement. 2016-01-13 Assaf Gordon numfmt: add the -z,--zero-terminated option * doc/coreutils.texi (numfmt invocation): Reference the description. * src/numfmt.c: Parameterize '\n' references. * tests/misc/numfmt.pl: Add tests for character and field processing. * NEWS: Mention the new feature. 2016-01-13 Pádraig Brady paste: add the -z,--zero-terminated option * doc/coreutils.texi (paste invocation): Reference -z description. * src/paste.c (main): Parameterize the use of '\n'. * tests/misc/paste.pl: Add test cases. * NEWS: Mention the new feature. comm: support NUL --output-delimiter for consistency * src/comm.c (main): Track the output delimiter length, so that it can be adjusted to 1 for the NUL delimiter. Also rename the global variable from "delimiter" to "col_sep" so its use is more obvious, and to distinguish from the recently added "delim" global variable. * tests/misc/comm.pl: Adjust accordingly. comm: add the -z,--zero-terminated option * doc/coreutils.texi (comm invocation): Reference option description. * src/comm.c (main): Use readlinebuffer_delim() to support a parameterized delimiter. * tests/misc/comm.pl: Add test cases. * NEWS: Mention the new feature. tac: support an empty (NUL) --separator * doc/coreutils.texi (tac invocation): Mention the NUL delineation with an empty --separator. * src/tac.c (main): Allow an empty separator when -r not specified. * tests/misc/tac.pl: Add test cases. * NEWS: Mention the new feature. Fixes http://bugs.gnu.org/8103 cut: add the -z,--zero-terminated option * doc/coreutils.texi (cut invocation): Reference the description. * src/cut.c: Parameterize '\n' references. * tests/misc/cut.pl: Add tests for character and field processing. * NEWS: Mention the new feature. 2016-01-13 Richard Russon head,tail: add the -z,--zero-terminated option * doc/coreutils.texi: Reference the option description. * src/head.c: Parameterize the delimiter character. * src/tail.c: Likewise. * tests/misc/head.pl: Add test case. * tests/misc/tail.pl: Likewise. * NEWS: Mention the new feature. 2016-01-13 Pádraig Brady join,sort,uniq: with -z, treat '\n' as a field separator * NEWS: Mention the change in behavior. * doc/coreutils.texi (newlineFieldSeparator): A new description, referenced from ({join,sort,uniq} invocation). * src/system.h (field_sep): A new inline function to determine if a character is a field separator. * src/join.c (usage): s/whitespace/blank/ to be more accurate wrt which characters are field separators. (xfields): s/isblank/field_sep/. * src/sort.c (inittables): Likewise. * src/uniq.c (find_field): Likewise. * tests/misc/join.pl: Adjust -z test, and add a test/example for processing the whole record with field processing. * tests/misc/sort.pl: Add -z test cases, including case with '\n'. * tests/misc/uniq.pl: Add -z -f test case with \n. 2016-01-09 Bernhard Voelker tests: fix races wrt output files of background processes At least the false positive in tail-2/follow-stdin.sh could be seen on a 4-core i5 system with -j8. Fix similar cases. too. * tests/tail-2/follow-stdin.sh: Empty the 'out' file in each iteration. Otherwise, under heavy system load, 'check_tail_output' would see the expected output from the previous round before tail would have the chance to come up in the background. While at it, move the creation of the 'exp' file out of the loop. * tests/dd/stats.sh: Empty the output file of the background process here, too. * tests/misc/cat-buf.sh: Likewise. * tests/misc/stdbuf.sh: Likewise. * tests/tail-2/wait.sh: Likewise. 2016-01-06 Pádraig Brady stty: support "flusho" local setting This is used to discard terminal output on FreeBSD based systems. * src/stty.c (mode_info): Add FLUSHO where supported. (usage): Document for "local" and "combination" settings. * doc/coreutils.texi (local stty settings): Document, mentioning that this option is supported but ignored on GNU/Linux. 2016-01-06 Pádraig Brady stty: support [-]drain setting to control waiting for pending Tx Instead of commit v8.24-132-g5171bef which only provides control to disable this behavior (with -I), provide the symmetrical "[-]drain" special setting. * src/stty.c (main): Parse the [-]drain setting instead of -I, and treat like a global option. (usage): Adjust accordingly. * tests/misc/stty.sh: Test "drain" with and without options. * NEWS: Mention the new feature. 2016-01-04 Pádraig Brady stty: support -I, --immediate to not wait for pending transmission In some cases an initial drain may block indefinitely as discussed at: http://lists.gnu.org/archive/html/coreutils/2016-01/msg00007.html * src/stty.c (main): Use TCSANOW rather than TCSADRAIN if -I specified. (usage): Document the new option. * doc/coreutils.texi (stty invocation): Likewise. * tests/misc/stty.sh: Ensure -I is supported. * NEWS: Mention the new feature. 2016-01-03 Pádraig Brady doc: avoid unwanted display of the 'coreutils' command info * doc/coreutils.texi (coreutils): Remove this direntry so that `info coreutils` will display the table of contents, rather than the unlikely intended coreutils Multi-call invocation. This also supports lookups like `info coreutils readlink`, but note that only works with the standard info viewer, and not with pinfo for example. doc: mention in more places that -D will create --target-directory * src/install.c (usage): Mention this commonly required functionality in the -D option description. * doc/coreutils.texi (install invocation): Likewise for the --target-directory description. install: only attempt to create a target dir once * src/install.c (main): As an optimization, when calling install_file_in_dir() for each file, only attempt to create the target directory once, as this is invariant over the loop. install: fix relative copies to absolute directory with -D * src/install.c (mkancesdirs_safe_wd): Unconditionally restore the current working directory when possibly called multiple times (from install_file_in_dir()). * tests/install/create-leading.sh: Add a test case. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/21497 2016-01-02 Bernhard Voelker maint: fix sc_long_lines to warn about 100+ maximum line length again Since commit v8.23-60-g414a8cf, the above check failed to find sources where the length of the longest line is a 3 or more digit number. * cfg.mk (sc_long_lines): Remove the '\' escape character before '{' and '}', because this would mean literal '{' and '}' characters in an extended regular expression in sed(1). 2016-01-01 Pádraig Brady maint: avoid a sc_long_lines failure in recent commit * src/dd.c (alloc_ibuf): Shorten line length. 2016-01-01 Pádraig Brady maint: update all copyright year number ranges Run "make update-copyright" and then... * gnulib: Update to latest with copyright year adjusted. * tests/init.sh: Sync with gnulib to pick up copyright year. * bootstrap: Likewise. * tests/sample-test: Adjust to use the single most recent year. 2015-12-31 Paul Eggert build: update gnulib submodule to latest dd: append spaces to shorter status=progress line Problem noted by Pádraig Brady in: http://bugs.gnu.org/22277#8 Also, make the output a bit more precise while we're at it. * NEWS: Document this. * src/dd.c (previous_time): Remove, replacing with ... (next_time): New var. All uses changed. This avoids some rounding errors, and should be a bit faster. (newline_pending): Remove, replacing with ... (progress_len): New var. All uses changed. This lets us keep track of how many trailing spaces to append. (print_xfer_stats): Get the time first thing, so that it's closer to being correct. Count the bytes output, and append trailing spaces if needed. Add remarks to translators about translation lengths. dd: summarize in --human-readable format too Problem reported by Linda Walsh in: http://bugs.gnu.org/17505 * NEWS: Document this. * doc/coreutils.texi (dd invocation): Use a simpler script. Adjust output example to match new behavior. * src/dd.c (human_size): Remove. All uses changed to use human_readable and ... (human_opts): ... this new constant. (abbreviation_lacks_prefix): New function. (print_xfer_stats): Use it. Output both --si and --human-readable summaries, but only if they have prefixes. * tests/dd/reblock.sh, tests/dd/stats.sh: Test new behavior. 2015-12-22 Pádraig Brady build: update to latest gnulib Mainly for these changes: - freadptr: fix to work with ungetc on all uClibc configs - fts: enable leaf optimization for XFS - fts: ensure leaf optimization used for NFS - strftime-tests: avoid false failure on OS X - intprops-tests: avoid warnings (causing CI failures) * NEWS: Update with items from above that are significant from the previous coreutils release. 2015-12-22 Pádraig Brady doc: describe test operator precedence and associativity * doc/coreutils.texi (Connectives for test): Add notes on precedence and associativity. Also mention the portability caveats with these operators. * cfg.mk: Avoid sc_prohibit_test_minus_ao for coreutils.texi. Fixes http://bugs.gnu.org/22216 2015-12-17 Pádraig Brady tests: avoid root test false failure without libxattr * tests/cp/cp-mv-enotsup-xattr.sh: Exclude xattr copy verification when libxattr is not available. 2015-12-17 Bernhard Voelker build: rely on umask for setting permissions in tarball Commit v8.24-116-g896006b changed the file permissions for the tarball using ug+rw. Better to let the umask decide whether the file should be group-writeable or not. * Makefile.am (dist-hook): Change chmod call from 'ug+rw' to '+rw'. 2015-12-16 Pádraig Brady tests: avoid intermittent ulimit -v failures * init.cfg (get_min_ulimit_v_): Increase the determined virtual memory limit for the command by a page size to avoid alignment issues causing false failures for subsequent runs. * tests/misc/cut-huge-range.sh: Be defensive and match the cut invocations under test, more closely with the form used to determine the VM memory limit. This was noticed on i686 linux at: http://hydra.nixos.org/build/28990456 2015-12-16 Pádraig Brady build: ensure all files are writable in the tarball * Makefile.am (dist-hook): Ensure all files are writable so that extracted tarballs are easy to remove, without getting prompts about removing read-only files. Reported by Benno Schulenberg. 2015-12-15 Pádraig Brady maint: ensure emit_mandatory_arg_note() used if needed * cfg.mk (sc_emit_mandatory_arg_note): Add new syntax check. maint: include 'env' commands in returns_ syntax check * cfg.mk (sc_prohibit_and_fail_1): Remove 'env' from the exclusion list. * tests/misc/env.sh: Adjust accordingly. * tests/misc/help-version.sh: Likewise. tests: improve kill tests * tests/misc/kill.sh: - verify kill -[lt] support multiple parameters. - verify kill -[lt] fail with (some) invalid parameters. - Use returns_1 ... || fail=1, rather than ... && fail=1. 2015-12-14 Pádraig Brady sort: fix --debug marking for -b -k1.x We were erroneously skipping blanks in the marked comparison _after_ the key start offset was applied. * src/sort.c (debug_keys): Don't skip starting blanks if already handled by begfield(). * tests/misc/sort-debug-keys.sh: Add a test case. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/22155 stat: report file system type for "overlayfs" * src/stat.c (human_fstype): Add "overlayfs", and tag it as "remote" to ensure that tail continues to use the more conservative polling mode. * README-release: Remove a stale comment about updating fremote() in tail.c. Also give a link to *_SUPER_MAGIC definitions not in the standard linux/magic.h location. * NEWS: Mention that this file system is recognized. 2015-12-14 Pádraig Brady kill: add undocumented -L for compatibility procps, bash >= 4.4, and util-linux >= 2.23 all have the -L option to output a table. * src/kill.c (main): Have -L synonymous with -t. Luckily there is no signal that begins with 'L'. * tests/misc/kill.sh: Update the test. 2015-12-14 Pádraig Brady tests: add tests for the kill command * tests/misc/kill.sh: Add first tests for kill. * tests/local.mk: Reference the new test. maint: fix incorrect spellings etc * NEWS: Use a standard version specification. * doc/coreutils.texi: Fix incorrect grammar. * src/df.c: Update a stale comment. * src/copy.c: Fix incorrect spellings. * src/factor.c: Likewise. * src/ls.c: Likewise. * src/pr.c: Likewise. * src/relpath.c: Likewise. * src/shred.c: Likewise. * src/sort.c: Likewise. * src/split.c: Likewise. * src/stdbuf.c: Likewise. * tests/misc/seq-precision.sh: Likewise. 2015-12-13 Pádraig Brady maint: remove form feed characters from sources These are no longer useful and can impact viewing patches in some mail programs for example. * src/cksum.c: Remove ^L characters. * src/comm.c: Likewise. * src/kill.c Likewise. * src/ln.c Likewise. * src/nl.c Likewise. * src/pr.c Likewise. * src/ptx.c Likewise. * src/split.c Likewise. * src/tr.c Likewise. * src/tsort.c Likewise. * src/uniq.c Likewise. * src/wc.c Likewise. * cfg.mk (sc_prohibit-form-feed): A new syntax check. 2015-12-07 Eric Blake sort: promote '--debug' * src/sort.c (usage): Mention --debug in KEYDEF paragraph. 2015-12-07 Paul Eggert doc: promote 'sort --debug' * README, doc/coreutils.texi (Introduction, sort invocation): Suggest 'sort --debug' more prominently. 2015-12-04 Pádraig Brady doc: improve the description of sort --random-sort * src/sort.c (usage): Avoid mentioning the implementation, rather focusing on the result of the operation. 2015-12-03 Paul Eggert build: port to Studio C on Solaris 12 Reported by Rich Burridge in: http://bugs.gnu.org/22087 * configure.ac (HAVE_UT_HOST, HAVE_C_LINE, stdbuf): Pacify picky compilers that complain about unreachable statements. 2015-12-01 Bernhard Voelker doc: clarify behaviour of mv for existing target directories * doc/coreutils.texi (mv invocation): Add a sentence to clarify that mv(1) will only replace empty directories in the target, and fail with a diagnostic otherwise. Discussed at: http://lists.gnu.org/archive/html/coreutils/2015-12/msg00000.html 2015-11-30 Bernhard Voelker doc: avoid confusing K argument in the man pages of head and tail The character 'K' representing numeric user input can be confusing, especially in the context where it can be suffixed by the multipliers K, M, G, etc. Use NUM instead. * doc/coreutils.texi (head invocation): Replace @var{k} by @var{num}. Also change @var{count} in the paragraph about the obsolete option syntax. (tail invocation): Likewise. Furthermore, adjust the option order to alphabetical sorting. * src/head.c (usage): s/K/NUM/ in as placeholder in the -c and -n options. Furthermore, print the actual DEFAULT_NUMBER instead of the hard-coded 10. * src/tail.c (usage): s/K/NUM/ in as placeholder in the -c and -n options. Add "[+]" to clarify that NUM may optionally be preceded by this sign. Remove the redundant explanation of the optional '+' prefix for NUM, as that is now clear from the option description above. Suggested by Dan Jacobson in http://bugs.gnu.org/22042 2015-11-28 Bernhard Voelker doc: reference each other in the man pages of head and tail * man/head.x: Add "SEE ALSO" section referencing tail(1). * man/tail.x: Add "SEE ALSO" section referencing head(1). Suggested by Dan Jacobson in http://bugs.gnu.org/22041 2015-11-27 Pádraig Brady build: support reproducible builds with consistent man pages Use a consistent date in the generated man pages even if SOURCE_DATE_EPOCH is not set. * Makefile.am: Generate .timestamp in the tarball. * man/local.mk: Pass .timestamp if available to help2man. * man/dummy-man: Don't bother with the year in the stub. Reported in http://bugs.debian.org/806321 2015-11-27 Pádraig Brady doc: update to latest help2man Sync with version 1.47.3 which provides these significant changes: - support for reproducible builds by using $SOURCE_DATE_EPOCH. - Adjust spacing of italic text at roman/italic boundaries. 2015-11-25 Pádraig Brady tests: for compare_(), use cmp -s where available * tests/init.sh (compare_): Only fall back to cmp without the POSIX defined -s option, where this is not available. build: avoid makeinfo unless required * doc/local.mk (constants.texi): Avoid calling makeinfo unless the constants derived from tail and shred have changed. This avoids a dependence on makeinfo from patched sources, and avoids an expensive makeinfo call for developers. doc: clarify descriptions for tr --squeeze-repeats * doc/coreutils.texi (uniq invocation): Fix the description of problematic input to say "blank lines" rather than "two or more blank lines". (tr invocation): Clarify that -s only works with SET1 when not translating (when SET2 not specified). Also explicitly state in examples where blank lines are deleted. Also add "deleting" to the menu item. * src/tr.c (usage): Improve the -s summary to say it always operates on the last specified SET. 2015-11-24 Pádraig Brady doc: provide an example for prefixing all tail output lines * doc/coreutils.texi (tail invocation): Provide an example using awk to convert tail ==> file <== headers to file: prefixes on each line. Suggested by Stephen Shirley. 2015-11-23 Dmitry Monakhov copy: fix copying of extents beyond the apparent file size fallocate can allocate extents beyond EOF via FALLOC_FL_KEEP_SIZE. Where there is a gap (hole) between the extents, and EOF is within that gap, the final hole wasn't reproduced, resulting in silent data corruption in the copied file (size too small). * src/copy.c (extent_copy): Ensure we don't process extents beyond the apparent file size, since processing and allocating those is not currently supported. * tests/cp/fiemap-extents.sh: Renamed from tests/cp/fiemap-empty.sh and re-enable parts checking the extents at and beyond EOF. * tests/local.mk: Reference the renamed test. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/21790 2015-11-23 Pádraig Brady md5sum,sha*sum: ensure --ignore-missing fails when no file verified * src/md5sum.c (digest_check): Update a matched_checksums bool upon matched checksum, and fail (loudly unless --status is specified) if there were no matches. Also change properly_formatted_lines to a bool while at it since we don't need to track the plurality. * tests/misc/md5sum.pl: Add a test case. Suggested by Jim Meyering. 2015-11-23 Luther Thompson md5sum,sha*sum: add --ignore-missing for checking a subset of files * doc/coreutils.texi (md5sum invocation): Document the new option. * src/md5sum.c (digest_file): Return an empty digest to indicate a missing file. (digest_check): Don't fail or output status given an empty checksum. (usage): Document the new option. (main): Process and validate the new option. * tests/misc/md5sum.pl: Add new test cases. * NEWS: Mention the new feature. Fixes http://bugs.gnu.org/15604 2015-11-23 Pádraig Brady maint: allow 'sha*sum:' tag in commit summaries To support "md5sum,sha*sum:" Suggested by Bernhard Voelker. 2015-11-21 Pádraig Brady doc: give a tee example for combining process substitution outputs This can be useful if you want to further process data from process substitutions. For example: datagen | tee >(md5sum --tag) > >(sha256sum --tag) | sort * doc/coreutils.texi (tee invocation): Mention that -p is useful with pipes that may not consume all data. Add an example, similar to the one above. * THANKS.in: Add Jirka Hladky. 2015-11-21 Pádraig Brady tests: avoid false failure on older NFS implementations * tests/ls/stat-failed.sh: Skip the test if 'd' is returned as the type, and document where this was seen. Also flag failure to write small temp files during the test as an error rather than a failure. Fixes http://bugs.gnu.org/21130 2015-11-18 Pádraig Brady build: fix VPATH build with --disable-dependency-tracking * src/local.mk: Run `mkdir -p src` in all our explicit rules, as in a VPATH build the src/ dir is only created at configure time as a side effect of dependency tracking generation. 2015-11-17 Jim Meyering scripts: update versions of gettext and libtool maint: remove unmaintained file, c99-to-c89.diff * src/c99-to-c89.diff: Remove file. * src/local.mk (EXTRA_DIST): Remove it from this list. * README (Pre-C99 build failure): Update section. 2015-11-12 Assaf Gordon csplit: check and report fwrite errors with errno discussed in: http://lists.gnu.org/archive/html/coreutils/2015-10/msg00091.html * src/csplit.c: (save_line_to_file): check fwrite failures, report and exit immediately instead of deferring to 'close_output'. * tests/misc/csplit-io-err.sh: test fwrite failure using LD_PRELOAD. * tests/local.mk: add new test. 2015-11-11 Pádraig Brady stat: improve support for new Linux pseudo FS and ACFS * src/stat.c (human_fstype): Add new file system ID definitions. * NEWS: Mention the improvement. 2015-11-10 Pádraig Brady tests: ensure programs are built before testing programs may not be built due to missing system dependencies, or any program can be excluded at configure time with --enable-no-install-program. So ensure we're not testing the system versions in these cases. * init.cfg (print_ver_): Call require_built_ first. * tests/misc/tty-eof.pl: Skip programs not built. * tests/Coreutils.pm (run_tests): Likewise. * tests/misc/ls-misc.pl: Use 'env test' rather than abs path. * tests/misc/test-diag.pl: Likewise. * tests/local.mk: Adjust include order for dependencies. * tests/misc/arch.sh: Remove redundant calls to require_built_. * tests/misc/chroot-fail.sh: Likewise. * tests/misc/groups-dash.sh: Likewise. * tests/misc/groups-version.sh: Likewise. * tests/misc/stdbuf.sh: Likewise. * tests/cp/acl.sh: Remove problematic call to print_ver_ [gs]etfacl. * tests/mv/acl.sh: Likewise. * cfg.mk (sc_env_test_dependencies): A new syntax check to enforce specifying dependencies with print_ver_ for programs specified through the env command. * du/bigtime.sh: Add new print_ver_ dependencies. * du/max-depth.sh: Likewise. * dd/ascii.sh: Likewise. * tests/ls/capability.sh: Likewise. * tests/ls/root-rel-symlink-color.sh: Likewise. * tests/misc/chroot-fail.sh: Likewise. * tests/misc/readlink-fp-loop.sh: Likewise. * tests/misc/sort-debug-keys.sh: Likewise. * tests/readlink/can-e.sh: Likewise. * tests/readlink/can-f.sh: Likewise. * tests/readlink/can-m.sh: Likewise. * tests/tail-2/inotify-race.sh: Likewise. * tests/tail-2/inotify-race2.sh: Likewise. * tests/touch/no-create-missing.sh: Likewise. * tests/touch/no-dereference.sh: Likewise. * tests/misc/printenv.sh: Tweak to avoid syntax check trigger. * tests/misc/help-version.sh: Likewise. * tests/misc/yes.sh: Likewise. * tests/misc/printf-quote.sh: Use previously unused $prog. * configure.ac (EXTRA_MANS): Add $gl_no_install_prog to the list so that check-x-vs-1 syntax check is satisfied. 2015-11-10 Pádraig Brady maint: use standard spacing in shebang line in tests It's better to be consistent even though spacing is insignificant: http://www.in-ulm.de/~mascheck/various/shebang/#blankrequired 2015-11-10 Pádraig Brady tests: cleanup trapping of signal names A side effect of this cleanup is we no longer depend on our own kill command being built. * init.cfg (require_trap_signame_): A new function to verify that the shell supports specifying traps by signal name. (require_kill_group_): A new function to ensure the shell supports sending a signal to a group. * tests/du/move-dir-while-traversing.sh: Ensure trap supports signal names. * tests/misc/stty-invalid.sh: Likewise. * tests/misc/stty-pairs.sh: Likewise. * tests/misc/stty-row-col.sh: Likewise. * tests/misc/stty.sh: Likewise. * tests/misc/sort-compress.sh: Likewise. Also simplify trap call. * tests/install/trap.sh: Likewise. * tests/misc/timeout.sh: Likewise. * tests/dd/stats.sh: Likewise. Also use default kill command. * tests/misc/timeout-group.sh: Likewise. 2015-11-06 Pádraig Brady tests: fix dirent d_type support verification * init.cfg (require_dirent_d_type_): Don't use df -x to exclude XFS, since this depends on a correct mtab which is brittle and often not correct within chroots. * tests/d_type-check: Check also the d_type of files, which excludes XFS appropriately. Specify all argument and return types to avoid truncated pointers being passed, which skipped the test due to crashes on x86_64 at least. Simplify the C library lookup by reusing the interpreter's. chroot issue reported at https://bugzilla.redhat.com/1263341 2015-11-04 Pádraig Brady maint: add a syntax check to avoid unstyled quoting * src/paste.c (main): Use our styled wrapper for quotearg_colon(). * cfg.mk (sc_prohibit-quotearg): A new syntax check to avoid future uses of unstyled quotearg to one of the internal slots, and thus destined for diagnostic output. paste: avoid confusing backslash quoting in diagnostic * src/paste.c (main): Setting the quoting style to "escape" went against the intent of the comment about presenting doubled backslashes to the user. Instead use "c-maybe" which is the only mode which avoids doubled backslashes, and provides protection against arbitrary control characters. * tests/misc/paste.pl: Adjust accordingly. ls: default to --quoting=shell-escape for output to terminal * src/ls.c (decode_switches): Set "shell-escape" if isatty(). * doc/coreutils.texi (ls invocation): Update the defaults description. * NEWS: Mention the change in behavior. It should not have backwards compat issues, but mentioning here just in case. printf: support the %q format to quote for shell * src/printf.c (usage): Mention the new format. (print_formatted): Handle the quoting by calling out to the quotearg module with "shell-escape" mode. * doc/coreutils.texi (printf invocation): Document %q. * tests/misc/printf-quote.sh: New test. * tests/local.mk: Reference new test. * NEWS: Mention the new feature. 2015-11-04 Pádraig Brady ls: avoid redundant processing when already escaping This is mainly noticeable when the multi-byte code within ls.c is triggered by multi-byte quotes. $ seq 200000 | xargs touch $ time ls-old -U --quoting=locale --hide-control-chars >/dev/null real 0m0.483s $ time ls-new -U --quoting=locale --hide-control-chars >/dev/null real 0m0.430s * src/ls.c (quote_name): Avoid rescanning the output looking for unprintable chars when we know the quoting mode already escapes them. * tests/misc/ls-misc.pl: Add tests for all quoting modes, with and without -q, to verify this assumption. 2015-11-04 Pádraig Brady ls: document and test new shell-escape quoting * doc/coreutils.texi (ls invocation): Describe the new 'shell-escape' and 'shell-escape-always' quoting options. * src/ls.c (usage): Mention the new quoting options. * tests/misc/ls-misc.pl: Add a test for 'shell-escape' test: use consistent quoting * src/test.c (test_syntax_error): Reuse verror() rather than open coding the error output format. (term): Don't hardcode '' quoting. (main): Likewise. 2015-11-04 Pádraig Brady all: avoid quoting file names when possible Quote file names using the "shell-escape" or "shell-escape-always" methods, which quote as appropriate for most shells, and better support copy and paste of presented names. The "always" variant is used when the file name is embedded in an error message with surrounding spaces. * cfg.mk (sc_error_shell_quotes): A new syntax check rule to suggest quotef() where appropriate. (sc_error_shell_always_quotes): Likewise for quoteaf(). * src/system.h (quotef): A new define to apply shell quoting when needed. I.E. when shell character or ':' is present. (quoteaf): Likewise, but always quote. * src/*.c: Use quotef() and quoteaf() rather than quote() where appropriate. * tests/: Adjust accordingly. 2015-11-04 Pádraig Brady build: update gnulib submodule to latest Includes support for "shell-escape" from quotearg 2015-11-03 Pádraig Brady md5sum: ensure a single status line per file * src/md5sum.c: Use the same file name escaping method used when generating and checking checksums. I.E. ensure a single line per file by starting the line with '\' for any file name containing '\n' and replacing those with "\\n". * NEWS: Move the item from changes in behavior to improvements, since this is no longer a backwards incompat change when processing stdout status messages. * tests/misc/md5sum.pl: Remove quotes from expected status output. * tests/misc/sha1sum.pl: Likewise. 2015-11-03 Pádraig Brady all: replace most uses of quotearg_colon() with quote() Related to commit v8.24-61-g6796698 this provides more consistent quoting, as quotearg_colon() defaults to "literal" quoting by default, while quote() provides appropriate quoting for diagnostics by default. * gl/modules/randread: Depend on quote module rather than quotearg. * gl/lib/randread.c: Used quote() not quotearg_colon(). * src/: Likewise. * src/shred.c: Likewise. Also avoid unnecessary quoting introducing overhead when wiping names. * cfg.mk: Relax the matching expression to allow "qname" variables as used in shred.c to satisfy the check. * tests/: Adjust accordingly. 2015-11-02 Pádraig Brady doc: fix texinfo for short options taking a parameter * doc/coreutils.texi: (tail invocation): Add missing -s, along with the existing --sleep-interval description. (csplit invocation): s/--suffix/--suffix-format/. (head invocation): Use same variable (COUNT) for -n and --head-count. (seq invocation): Add opindex items for all options. (ptx invocation): Likewise. Fix typo s/--flac-truncation/--flag-truncation/. (touch invocation): State explicitly that -d takes a parameter, which also indicates that an '=' is not to be used for the short option syntax. (ls invocation): Likewise for the -w option. Fixes http://bugs.gnu.org/21809 2015-10-28 Pádraig Brady maint: allow 'all:' tag in commit summaries To tag changes that are user visible and affect all (or many) commands. doc: clarify iso-8601 formats used by ls and du * doc/coreutils.texi (du invocation): Remove 'like' from mentions of ISO-8601 as the components are individually conformant. (ls invocation): Likewise, except for --time-style=iso for recent files, where the MM-DD component is not a valid ISO-8601 timestamp. 2015-10-27 Pádraig Brady date: use extended format timezone for --iso-8601 * src/date.c (main): Use %:z rather than %z with --iso-8601 as the standard states to consistently use extended format. Note either format can be parsed by date. * tests/misc/date.pl: Adjust accordingly. * doc/coreutils.texi (du invocation): Clarify that "iso" time styles are only similar to ISO-8601. (ls invocation): Likewise. (date invocation): Adjust the comment stating that only --rfc-3339 output can be parsed by date(1). * NEWS: Mention the change in behavior. Reported at http://bugs.debian.org/799479 doc: reference related commands from users(1) and groups(1) * man/users.x: See also getent, who * man/groups.x: See also getent Addresses http://bugs.gnu.org/21735 copy,dd: simplify and optimize NUL bytes detection * src/factor.c: Move LIKELY() definition to... * src/system.h: ...here. (is_nul): Reimplement with a version that doesn't require a sentinel after the buffer, and which calls down to (the system optimized) memcmp. Performance analyzed at http://rusty.ozlabs.org/?p=560 * src/dd.c (alloc_obuf): Simplify the is_nul() call by not needing to write the sentinel. * src/copy.c (sparse_copy): Likewise. (copy_reg): Simplify the buffer allocation by avoiding consideration of the sentinel in the buffer size calculation. 2015-10-27 Pádraig Brady all: quote string arguments in error messages These strings are often file names or other user specified parameters, which can give confusing errors in the presence of unexpected characters for example. * cfg.mk (sc_error_quotes): A new syntax check rule. * src/*.c: Wrap error() string arguments with quote(). * tests/: Adjust accordingly. * NEWS: Mention the improvement. 2015-10-27 Pádraig Brady md5sum: quote all printed file names This is especially significant when using --check with files generated on a windows system, where the \r characters produce corrupted and confusing error messages. This also ensures status messages are output on a single line. * src/md5sum.c: Use quote() for printed file names. * tests/misc/md5sum.pl: Adjust accordingly. * NEWS: Mention the change in behavior. Fixes http://bugs.gnu.org/21757 2015-10-21 Pádraig Brady ls: fix off by one error when determining max display columns * src/ls.c (main): Account for the first column not including a separator when calculating max_idx. * tests/ls/w-option.sh: Add a test case. * NEWS: Mention the bug fix. 2015-10-20 Pádraig Brady ls: allow -w0 to mean no limit on line length * src/ls.c (print_with_separator): Renamed from print_with_commas, and parameterized to accept the separator to print. Also fix an edge case where '\n' not printed when the POS variable overflows SIZE_MAX. (print_current_files): Degenerate -x and -C to using the cheaper print_with_separator() in the -w0 case. * doc/coreutils.texi (ls invocation): Document the new feature. * tests/ls/w-option.sh: A new test. * tests/local.mk: Reference the new test. * NEWS: Mention the improvement. Fixes http://bugs.gnu.org/21325 2015-10-19 Pádraig Brady ls: detect terminal color support using glob patterns * src/ls.c (know_term_type): Corresponding to commit v8.24-48-gc249a5a, use fnmatch to inspect the dircolors database. Noticed due to failing tests/ls/color-{dtype-dir,term}.sh tests. tests: adjust recent change to csplit VM limit * tests/misc/csplit-heap.sh: A little more memory is required for the full run case. Noticed with --enable-single-binary. 2015-10-19 Pádraig Brady maint: fixes to support improved sc_tight_scope The gnulib provided sc_tight_scope target was ineffective, as it was checking against an invalid blank regular expression, and thus ignoring any extern function issues. This is now fixed up in gnulib, and so we need to fix our scoping issues before the next gnulib update. * cfg.mk: Setup and document the tight_scope config variables appropriately. * src/selinux.h: Since declared in *_SOURCES, use the two line form for the extern function declarations. * src/set-fields.h: Add the extern declarations, and since declared in noinst_HEADERS use the single line form. 2015-10-19 Pádraig Brady tests: avoid failure when auto selecting factor tests * tests/factor/run.sh: If this template is found through `grep -El "print_ver_.* factor"` for example, then just skip it. 2015-10-19 Pádraig Brady factor: remove unreachable SQUFOF code at compile time It was a little confusing as to whether the SQUFOF algorithm was enabled, and in fact there were no options available to enable it. Therefore clarify the 3 configurable behaviors for the code to 3 defines at the top of the program, and only include the SQUFOF code if enabled at compile time. $ size src/factor-before text data bss 93997 1412 2504 $ size src/factor-after text data bss 87885 1404 2504 * src/factor.c: Only include the SQUFOF factor code when enabled via the USE_SQUFOF define. * doc/coreutils.texi (factor invocation): Update note about factor limits, as we can factor 128 bit numbers without GMP. 2015-10-18 Pádraig Brady doc: reference shuf(1) from the sort (-R) man page * man/sort.x: Cross reference with shuf(1). * src/sort.c (usage): Mention shuf(1) with -R option. Suggested in http://bugs.debian.org/641166 2015-10-18 Bernhard Voelker maint: avoid duplicate test runs * tests/local.mk (all_tests): Remove the tests 'tests/id/setgid.sh' and 'tests/mkdir/smack-root.sh' because they are mentioned in the 'all_root_tests' list; these tests are skipped anyway during a non-root run because flagged with 'require_root_'. 2015-10-16 Paul Eggert doc: remove obsolete performance comment sha512sum can be faster than sha256sum. E.g., ‘dd if=/dev/zero bs=1024k count=1024 | time sha256sum’ reports 8.16 user CPU seconds on my host, whereas sha512sum consumes 5.45 seconds (Fedora x86-64 on an AMD Phenom II X4 910e). Although sha512sum is still considerably slower on x86, a good chunk of uses are on 64-bit hosts and anyway there’s little point to scaring people away from sha512sum nowadays. * doc/coreutils.texi (sha2 utilities): Remove obsolete comment. 2015-10-16 Jim Meyering maint: avoid uniq.c warning from bleeding-edge gcc's -Wstrict-overflow * src/uniq.c (main): Make the type of "nfiles" unsigned, to avoid a brand new warning from a gcc I built from today's sources (gcc version 6.0.0 20151015 (experimental) (GCC)): src/uniq.c:523:14: error: assuming signed overflow does not occur \ when simplifying conditional to constant [-Werror=strict-overflow] if (nfiles == 2) ^ 2015-10-16 Pádraig Brady dircolors: support globbing of TERM entries * src/dircolors.c (dc_parse_stream): Support globbing of TERM entries, to allow entries like "TERM *256color*" for example. * src/dircolors.hin: Reduce the internal list with globbing. * tests/misc/dircolors.pl: New test cases. * NEWS: Mention the improvement. 2015-10-15 Pádraig Brady tests: avoid false failure in rm/r-root.sh with gdb warnings * tests/rm/r-root.sh: Skip the test if there are gdb warnings that will impact further stderr checks. For example some buggy gdb versions may report "Got object file from memory but can't read symbols: File truncated". Also fix an incorrect stderr check from the previous change. Reported by Bernhard Voelker. tests: avoid false failure in rm/r-root.sh under load * tests/rm/r-root.sh: Use gdb rather than timeout(1) as the last resort protection against unlinkat() calls. The timeout of 2s was susceptible to false positives under load, and gdb is stronger protection in any case. We remove the "expensive" tag on this test also since it should be robust. Reported by Jim Meyering. tests: avoid false failure in a tail test under load * tests/tail-2/follow-stdin.sh: Use the standard tail testing framework to avoid the race seen under very high load, and also test the non inotify case. Reported by Jim Meyering 2015-10-14 Pádraig Brady tests: avoid false failures with default ACLs To reproduce: setfacl -dm group::rx . setfacl -dm other::rx . make check * init.cfg (require_no_default_acl_): A new function to skip when default ACLs are detected, or if the getfacl utility is not available then skip if any non LSM ACLs detected. * tests/cp/existing-perm-race.sh: Call require_no_default_acl_. * tests/mkdir/parents.sh: Likewise. * tests/mkdir/perm.sh: Likewise. 2015-10-11 Pádraig Brady tail: no longer warn about unrecognized file systems * src/tail.c (fremote): No longer prompt the user to email with the unrecognized file system constant, since we have process in place to sync periodically with the latest Linux constants, and the fall back polling mode is still fully functional. 2015-10-10 Pádraig Brady tests: avoid false failure when restorecon is ineffective * tests/cp/cp-a-selinux.sh: Ensure we skip the portion of the test depending on restorecon to be effective. I.E. also skip when restorecon warns, as it doesn't exit with error status when matchpathcon fails to find a match for a file. This is the case in /tmp on Fedora for example, in which case the new destination that cp creates will have the default security context of the process, rather than the explicit context we set on the source file. Details at: http://bugzilla.redhat.com/1247641 tests: make a long running test responsive to Ctrl-C * tests/misc/sort-compress-hang.sh: Use --foreground with the timeout(1) command (noting the caveats), to run the sort command in the foreground program group, and thus be responsive to Ctrl-C. This very_expensive_ test takes over a minute on a i3-2310M, with RAM backed /tmp. 2015-10-07 Pádraig Brady tests: adjust recent changes to virtual memory limits * tests/dd/no-allocate.sh: Account for timeout(1) when determining the required mem, as timeout has additional shared libs. This avoids the need for the hardcoded 4M addition to the limit. * tests/misc/head-c.sh: Increase the base limit, to account for the fact that head(1) will allocate some additional mem in this case. * tests/misc/cut-huge-range.sh: Remove mention of specific limits. * tests/misc/printf-surprise.sh: Likewise. Reported by Dmitry V. Levin 2015-10-06 Pádraig Brady csplit: remove erroneous mention of -m in --help * src/csplit.c (usage): -m is not accepted, only --suppress-matched. * tests/misc/csplit-suppress-matched.pl: Spelling fix. Reported by Ondrej Oprala 2015-10-03 Paul Eggert tee: simplify argv handling * src/tee.c (tee_files): Last arg is now char ** instead of char const **, as that is a bit simpler. All callers changed. Modify files[-1], not files[nfiles], as that is a bit faster and simpler. Latter problem pointed out by Rainer Deyke in: http://bugs.gnu.org/21611 build: update gnulib submodule to latest build: Pacify GCC 5 on 32-bit hosts This pacifies GCC 5 in a better way, without disabling diagnostics. * src/df.c (main): Tell compiler that optind is positive. * src/shred.c (known): New function. (dopass): Go back to off_t for file sizes. Avoid integer overflow if we run off the end of the file. Tell compiler that a write cannot write more bytes than requested. 2015-10-02 Pádraig Brady tail: handle kernel dentry unlink race Avoid the intermittent loss of "... has become inaccessible" messages. That would cause tests/tail-2/assert.sh to fail sometimes, mainly on uniprocessor systems. * src/tail.c (tail_forever_inotify): Also monitor IN_DELETE events on the directory, to avoid a dentry unlink()..open() race, where the open() on the deleted file was seen to succeed after an, unlink() and a subsequent IN_ATTRIB, was sent to tail. Note an IN_ATTRIB is sent on the monitored file to indicate the change in number of links, and we can't just use a decrease in the number of links to determine the file being unlinked, due to the possibility of the file having multiple links. Reported by Assaf Gordon and Ludovic Courtès. Fixes http://bugs.gnu.org/21460 2015-10-02 Pádraig Brady build: avoid -Wstrict-overflow warnings with GCC 5.1 on 32 bit * src/shred.c (dopass): With -O2, GCC 5.1 gives "assuming signed overflow does not occur when simplifying conditional to constant" warnings, in regard to the signed (off_t) variables. Therefore use unsigned (uintmax_t) instead, and a separate boolean to cater for the special meaning of the negative part of the integer range. Noticed at http://hydra.nixos.org/build/24983447 build: update gnulib submodule to latest Includes a change to xalloc.h to avoid -Wstrict-overflow warnings with GCC 5.1 on 32 bit with optimization enabled. A subsequent commit will fix similar issues in shred. 2015-10-02 Dario Giovannetti dircolors: add xterm-termite entry * src/dircolors.hin: Add "xterm-termite" as this VTE based terminal emulator is quite different from xterm, despite the name. For example "Termite supports italic text and it won't work if TERM is set to xterm. Even the backspace key won't work properly anymore for applications relying on terminfo". Reported also by Lukas Sabota and Sven-Hendrik Haase. 2015-09-22 Dave Chiluk df: prioritize mounts nearer the device root In the presence of bind mounts of a device, the 4th "mount root" field from /proc/self/mountinfo is now considered, so as to prefer mount points closer to the root of the device. Note on older systems with an /etc/mtab file, the source device was listed as the originating directory, and so this was not an issue. Details at http://pad.lv/1432871 * src/df.c (filter_mount_list): When deduplicating mount entries, only prefer sources nearer or at the root of the device, when the target is nearer the root of the device. * NEWS: Mention the change in behavior. 2015-09-22 Pádraig Brady build: update gnulib submodule to latest This includes a change to propagate the 4th "mount root" field from /proc/self/mountinfo from the mountlist module, which is needed in a subsequent commit in df. * gl/lib/regcomp.c.diff: Regenerate against latest gnulib. * gl/lib/regex_internal.c.diff: Likewise. * gl/lib/regex_internal.h.diff: Likewise. * cfg.mk: Exclude diffs from trailing whitespace check, which is generally correct, and now needed. 2015-09-22 Bernhard Voelker maint: use adaptive approach for `ulimit -v` based tests When configured with either 'symlinks' or 'shebangs' as value for the --enable-single-binary option, tests based on `ulimit -v` are skipped. The reason is that the multicall 'coreutils' binary requires much more memory due to shared libraries being loaded, and the size of the 'date' binary (~290KiB) compared to the multicall binary (~5MiB), of course. Finally, in the case of 'shebangs', the starting shell requires more memory, too Instead of using hard-coded values for the memory limit, use an adaptive approach: first determine the amount of memory for a similar, yet more trivial invocation of the command, and then do the real test run using that limit (plus some buffer in some cases). * init.cfg (require_ulimit_v_): Remove function. (get_min_ulimit_v_): Add function to determine the minimum memory limit required for a given command in an adaptive way. * cfg.mk (sc_prohibit_test_ulimit_without_require_): Change the name of the above function in the syntax-check rule. * tests/cp/link-heap.sh: Use the above function to determine the minimum memory required to run a command simpler than in the real test run. Use that limit plus a buffer there. While at it, change to list of commands in the subshell to fail also if the beginning `ulimit -v` fails. * tests/dd/no-allocate.sh: Likewise. * tests/misc/csplit-heap.sh: Likewise. * tests/misc/cut-huge-range.sh: Likewise. * tests/misc/head-c.sh: Likewise. * tests/misc/printf-surprise.sh: Likewise. * tests/split/line-bytes.sh: Likewise. * tests/rm/many-dir-entries-vs-OOM.sh: Likewise - doing it separately for each program under test. 2015-09-22 Pádraig Brady maint: avoid deprecation warning with * src/runcon.c (main): As per the compile time warning from libselinux-2.4-3, lookup the class with string_to_security_class(), rather than using defines from flask.h. 2015-09-21 Pádraig Brady sort,numfmt: with --debug, diagnose failure to set locale * src/sort.c (main): With --debug, warn upon setlocale() failure, which can happen due to incorrectly specified environment variables, or due to memory exhaustion (simulated with ulimit -v), etc. * tests/misc/sort-debug-warn.sh: Add a test case. See also http://savannah.gnu.org/bugs/11004 2015-09-18 Bernhard Voelker du: avoid to stat all mount points if possible du calls stat for each mount point at startup. This would block or even make du fail if stat for an unrelated mount point hangs. The result is not needed in the normal case anyway and therefore should be avoided. Issue introduced in commit v8.19-2-gcf7e1b5. * src/du.c (fill_mount_table): Move function up as it's not used ... (mount_point_in_fts_cycle): ... here, i.e., the DI_MNT set is initialized and filled only iff FTS has detected a directory cycle. (main): Remove the initialization and filling of the DI_MNT set here, and free the DI_MNT set only if it was used. 2015-09-17 Paul Eggert shred: don’t document -NUMBER option The -NUMBER option was removed from ‘shred’ in 1999, but the manual wasn’t updated to match. Problem reported by Nick Rose in: http://bugs.gnu.org/21502 * doc/coreutils.texi (shred invocation): Remove documentation for -NUMBER option. 2015-09-11 Assaf Gordon numfmt: use new set-fields module to parse --field numfmt --field=LIST can accept the same options as cut. * bootstrap.conf: remove xlist, linked-list * src/local.mk: link numfmt with set-fields * src/numfmt.c: use set-fields.c instead of custom field parsing code. (include_field): adapt to new code. * tests/misc/numfmt.pl: add new tests, adapt current tests to new error message wording from set-fields.c 2015-09-11 Assaf Gordon cut: refactor into set-fields module Extract the functionality of parsing --field=LIST into a separate module, to be used by other programs. * src/cut.c: move field parsing code from here ... * src/set-fields.{c,h}: ... to here. (set_fields): generalize by supporting multiple parsing/reporting options. (struct range_pair): rename to field_range_pair. * src/local.mk: link cut with set-field. * po/POTFILES.in: add set-field.c * tests/misc/cut.pl: update wording of error messages 2015-09-08 Pádraig Brady maint: fix heap manipulations in previous commit * src/sort.c (main): Ensure we don't free() and invalid pointer when reading implicit stdin. Also avoid "definitely lost" valgrind warnings in the --files0-from case. 2015-09-04 Pádraig Brady maint: avoid "definitely lost" valgrind warnings Since commit v8.23-19-g8defcee, main() will return, rather than call exit(), this inducing "definitely lost" warnings in valgrind's leak checker. That precludes using the following to flag memory leaks: valgrind --leak-check=full --error-exitcode=1 \ --errors-for-leak-kinds=definite * src/pr.c (main): In dev builds, explicitly free memory allocated. * src/sort.c (main): Likewise. * src/tail.c (main): Likewise. * src/tsort.c (tsort): Likewise. 2015-09-02 Pádraig Brady tests: test numfmt stdin behavior * tests/misc/tty-eof.pl: Add numfmt to the list of programs that accept input on stdin. ls,ptx: restrict quotearg use to file name output * src/ls.c (getenv_quoting_style, decode_switches, parse_ls_color): Use quote() rather than quotearg(), as the latter defaults to outputting the input unquoted. * src/ptx.c (main): Likewise. base64: no longer support hex or oct --wrap params * src/base64.c (main): Support decimal numbers with leading zeros, by disabling the auto detection of octal and hex. It's not envisaged that base conversion is needed for --wrap parameters, and in the edge case it is, $((0x0)) shell constructs can be used. * tests/misc/base64.pl: Adjust accordingly. * NEWS: Mention the change in behavior. base64: use stricter validation on wrap column * src/base64.c (main): Use the higher level xnumtoumax() rather than xstrtoumax(), which is simpler and improves validation of input. Also pass the _empty_ rather than NULL string as the suffixes parameter so that invalid trailing characters are not allowed. For example -w08 is now flagged as an error, rather than being interpreted as 0. A subsequent commit will further improve verification of numbers with leading zeros by dropping backwards compatibility wrt auto parsing oct and hex numbers. * tests/misc/base64.pl: Add tests for invalid wrap values. 2015-09-02 Pádraig Brady base32: A new program similar to base64 Suggested in https://bugzilla.redhat.com/1250113 * AUTHORS: Add base32. * THANKS.in: Add suggester. * README: Reference the new program. * NEWS: Mention the new program. * src/.gitignore: Ignore the new binary. * bootstrap.conf: Reference the gnulib base32 module. * build-aux/gen-lists-of-programs.sh: Add base32. * man/base32.x: A new template. * man/.gitignore: Ignore the new man page. * man/local.mk: Reference the new man page. * doc/coreutils.texi (base32 invocation): Document the new command. * src/local.mk: Adjust to build base32 based on base64.c. * src/base64.c: Parameterize to use the correct headers, functions and buffer sizes, depending on which binary is being built. * tests/misc/base64.pl: Adjust to test both base32 and base64. * tests/misc/tty-eof.pl: Add base32 as a program that accepts input on stdin without any options specified. * scripts/git-hooks/commit-msg: Add base32 to the template. 2015-09-01 Pádraig Brady build: update gnulib submodule to latest This includes a tweak to support building the gnulib base32 module with -Wsuggest-attribute=const 2015-08-31 Pádraig Brady doc: clarify in --help/man where short options take no param * src/shred.c (usage): For -u, separate the decscription of the short and long option, to clarify that the short option takes no parameter. * src/split.c (usage): Likewise for -d. * src/tee.c (usage): Likewise for -p. * src/uniq.c (usage): Likewise for -D. Suggested by Stephane Chazelas 2015-08-30 Bernhard Voelker tests: avoid FP of ls/stat-free-color.sh with newer glibc Since glibc-2.22, specifically commit [0], the opendir() implementation implicitly makes an additional stat call thus leading to a FP. Seen on openSUSE:Tumbleweed since snapshot 20150821. [0] https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=46f894d8c60a * tests/ls/stat-free-color.sh: Change the test to verify that ls(1) needs the same number of stat-like calls for a single, empty directory argument as for one with a few directory entries (sub-directory, regular file, symlink, etc.). 2015-08-27 Pádraig Brady doc: clarify where ambiguous if short options take no param * doc/coreutils.texi (split invocation): Clarify that -d takes no param. (uniq invocation): Likewise for -D. (shred invocation): Likewise for -u. (tee invocation): Likewise for -p. 2015-08-22 Paul Eggert ls: allow -w18446744073709551616 Problem reported by Beco in: http://bugs.gnu.org/21325 * src/ls.c (set_line_length): New function. (decode_switches): Use it to decode COLUMNS and -w. 2015-08-18 Bernhard Voelker maint: add syntax check to ensure larger man/*.x files have a Copyright * cfg.mk (sc_man_check_x_copyright): Add rule to ensure that non-trivial .x files in the 'man/' subdirectory, i.e., files exceeding a line count of 20 or a byte count of 1000, contain a proper Copyright notice. 2015-07-23 Paul Eggert build: fprintftime/nstrftime API changes * bootstrap.conf (gnulib_modules): Add time_rz, since the main source code now uses timezone_t. * src/date.c (batch_convert, main, show_date): * src/ls.c (align_nstrftime, long_time_expected_width) (print_long_format): * src/stat.c (human_time): Use timezone_t rather than boolean to specify which time zone is wanted. * src/ls.c (localtz): New static var. (main): Initialize it. build: update gnulib submodule to latest 2015-07-21 Pádraig Brady doc: discourage use of uname -i and -p options * src/uname.c (usage): State that the non POSIX -i and -p options are non-portable. * doc/coreutils.texi (uname invocation): Mention the discrepancies even across GNU/Linux distros, and that the results should be used as informational only, rather than impacting any logic decisions. Fixes http://bugs.gnu.org/13001 2015-07-15 Pádraig Brady doc: improve man page for realpath -m * src/realpath.c (usage): Mention 'directory' in the --help output, so that ENOTDIR errors may be more easily investigated, by inspecting the man page. Reported at http://pad.lv/1474519 2015-07-15 Peter Bray tests: avoid test warning with perl >= 5.22 * doc/local.mk (sc-lower-case-var): Escape a literal left curly bracket, needed with perl >= 5.22 Fixes http://bugs.gnu.org/21060 sync: fix build with separate $(LIB_FDATASYNC) * src/local.mk: Link with $(LIB_FDATASYNC) needed on some Solaris 10 systems for example. Fixes http://bugs.gnu.org/21059 2015-07-09 Pádraig Brady dircolors: add tmux entries * src/dircolors.hin: tmux entries were added to ncurses in: http://ncurses.scripts.mit.edu/?p=ncurses.git;a=blobdiff;\ f=misc/terminfo.src;h=ce9bdc3b;hp=7e576ef1;hb=be512fa0;hpb=ee1bcda2 2015-07-06 Pádraig Brady shred: fix pattern selection for certain iteration counts This was detected in about 25% of runs with gcc -fsanitize=address ERROR: AddressSanitizer: global-buffer-overflow on address ... READ of size 4 at 0x000000416628 thread T0 #0 0x40479f in genpattern src/shred.c:782 #1 0x4050d9 in do_wipefd src/shred.c:921 #2 0x406203 in wipefile src/shred.c:1175 #3 0x406b84 in main src/shred.c:1316 #4 0x7f3454a1ef9f in __libc_start_main (/lib64/libc.so.6+0x1ff9f) #5 0x4025d8 (/tmp/coreutils-8.23/src/shred+0x4025d8) 0x000000416628 is located 56 bytes to the left of global variable '*.LC49' from 'src/shred.c' (0x416660) of size 17 0x000000416628 is located 12 bytes to the right of global variable 'patterns' from 'src/shred.c' (0x416540) of size 220 SUMMARY: AddressSanitizer: global-buffer-overflow src/shred.c:782 * src/shred.c (gen_patterns): Restrict pattern selection to the K available, which regressed due to v5.92-1462-g65533e1. * tests/misc/shred-passes.sh: Add a deterministic test case. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/20998 2015-07-03 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.24 * NEWS: Record release date. maint: fix distclean-check failure due to THANKS * Makefile.am: Remove our dependence on src/sort which induces awkward dependencies for `make dist` since THANKS will be rebuilt once src/sort is newer. Instead we remove the problematic -f option to sort which actually doesn't change the order given our current input. maint: avoid false syntax-check failure in distcheck * cfk.mk (sc_tests_executable): Restrict the check to git files, so we don't flag any gnulib files added to test/ during `make distcheck`. tests: update gnulib submodule and tests/init.sh to latest * gnulib: Update to latest, with fixes to tests edge cases. * tests/init.sh: Update from gnulib. 2015-07-03 Pádraig Brady tests: avoid side effects of $SHELL environment variable Since non interactive shells don't generally set $SHELL, its value is propagated through the tests and may cause issues; for example if $SHELL implicitly adjusts $PATH when run. Instead we set $SHELL to that determined by the posix-shell module, and use that consistently for all test sub scripts, including those created thorugh the `split --filter` command. * tests/local.mk: Explicitly set $SHELL to $(PREFERABLY_POSIX_SHELL) which defaults to $CONFIG_SHELL and thus usually /bin/sh. * tests/envvar-check: Remove bash environment variables with side effects, in case /bin/bash was selected for $SHELL. * tests/misc/help-version.sh: Remove redundant initialization of $SHELL. * tests/install/strip-program.sh: Use $SHELL for sub script. * tests/misc/sort-compress-hang.sh: Likewise. * tests/misc/sort-compress-proc.sh: Likewise. * tests/misc/sort-compress.sh: Likewise. * tests/misc/timeout-group.sh: Likewise. * tests/rm/fail-eperm.xpl: Remove redundant elision of bash env vars. * tests/misc/pwd-long.sh: Likewise. 2015-07-02 Pádraig Brady tests: avoid false failures on OpenBSD 5.7 * tests/du/threshold.sh: Homogenize getopt error messages. * tests/misc/numfmt.pl: Likewise. * tests/mv/i-3.sh: Skip on *BSD not just FreeBSD. 2015-07-01 Pádraig Brady wc: fix reading of /proc files on aarch64 tests/misc/wc-proc.sh fails when the page size is 64K * src/wc.c (wc): The lseek adjustment should be based on st_blksize, rather than on the internal buffer size. This is significant on aarch64 where st_blksize in /proc is the 64K (the page size) and thus larger than the internal buffer. * src/split.c (main): Even though the similar processing is done on the internal buffer size, that's based on st_blksize and so fine in this regard. Add an assert to enforce this. Avoid this path for the undocumented ---io-blksize option. 2015-07-01 Pádraig Brady build: update gnulib submodule to latest Mainly with build fixes for FreeBSD and OS X. numfmt: increase precision on 32 bit FreeBSD * m4/jm-macros.m4 (HAVE_FPSETPREC): Define if needed. * src/numfmt.c (main): Call fpsetprec() if needed. Fixes large-15 and large-16 test failures on 32 bit FreeBSD. tests: avoid false failure on FreeBSD systems * tests/misc/stty.sh: FreeBSD returns ENOTTY for the TIOCEXT ioctl, so just avoid this option for now. 2015-07-01 Pádraig Brady factor: ensure atomic output through pipes The new tests/misc/factor-parallel.sh test was seen to fail on FreeBSD (derived) systems, which was due to split(1) --filter reading partial lines through pipes, as factor(1) was writing a little over PIPE_BUF each time. * src/factor.c (lbuf): A new structure to internally buffer lines. (lbuf_alloc): A new function to allocate enough at program start. (lbuf_putint): A new function to buffer a uintmax_t. (lbuf_flush): A new function to write directly to standard output. (lbuf_putc): A new function to buffer a character and if enough lines are buffered, then output complete lines <= PIPE_BUF, and continue to buffer the rest. (main): Call the internal buffer allocator, and register the final flush from the internal buffer at program exit. 2015-07-01 Pádraig Brady tests: fix false failure on slower systems * tests/dd/stats.sh: Wait 20s for dd to write 250MB through a fifo, rather than 10s for 500MB. The failure was seen often on a lightly loaded SPARC-Enterprise-T5220 running Solaris 10. 2015-06-29 Jim Meyering maint: stdbuf.c: avoid the OS X putenv function * bootstrap.conf (gnulib_modules): Add setenv, to make this module dependency explicit; setenv is also used by split. * src/stdbuf.c (set_LD_PRELOAD) [__APPLE__]: Use the OS X setenv function, rather than putenv, per that documentation: https://developer.apple.com/\ library/mac/documentation/Darwin/Reference/ManPages/man3/putenv.3.html 2015-06-29 Jim Meyering build: numfmt.c: avoid a shadowing warning * src/numfmt.c (parse_field_arg): Rename parameter s/optarg/arg/, to avoid shadowing getopt's global variable. Otherwise, building on OS X, with --enable-gcc-warnings, I saw this: In file included from src/numfmt.c:19:0: src/numfmt.c: In function 'parse_field_arg': ./lib/config.h:3109:25: error: declaration of 'rpl_optarg' shadows\ a global declaration [-Werror=shadow] 2015-06-29 Assaf Gordon numfmt: fix printf argument order * src/numfmt.c (double_to_human): Fix the argument order passed to snprintf, which happened to work on amd64 with its separate va_arg storage area for floats¹, but would fail tests for example on i686. ¹ https://blog.nelhage.com/2010/10/amd64-and-va_arg/ 2015-06-27 Pádraig Brady maint: clarify df's use of device IDs from /proc/self/mountinfo * src/df.c (filter_mount_list): Clarify why we still stat even though devices IDs may already be available. Note using /proc/self/mountinfo is still an advantage to get filtered items with accurate device patchs in chroots and with bind mounts. I.E. on older setups with static /etc/mtab, df will now bypass that to get the more accuracte and dynamic info. 2015-06-26 Pádraig Brady chroot: quote argument in error diagnostic * src/chroot.c (main): Quote the passed argument, to avoid confusing error messages. 2015-06-25 Pádraig Brady maint: clarify integer operations in recent commit * src/factor.c (print_uintmaxes): Comment that the value of n_out doesn't matter on error, and add an explicit cast to avoid any future warnings. Suggested by Jim Meyering RE commit v8.23-229-g4d2d6c5 tests: avoid false failure when running as root * tests/misc/sync.sh: Ensure dir is unreadable before including the permission check. 2015-06-25 Pádraig Brady tests: avoid false failures with LD_PRELOAD=libasan.so.2 The LD_PRELOAD checks by -fsanitize=address are overly strict: https://groups.google.com/forum/#!topic/address-sanitizer/jEvOJgkDqQk A workaround is to first export LD_PRELOAD=libasan.so.2 The tests below are adjusted so that workaround is not discarded. * tests/cp/no-ctx.sh: Append to $LD_PRELOAD. * tests/df/no-mtab-status.sh: Likewise. * tests/df/skip-duplicates.sh: Likewise. * tests/ls/getxattr-speedup.sh: Likewise. * tests/rm/r-root.sh: Likewise. * tests/cp/nfs-removal-race.sh: Likewise. Also check that LD_PRELOAD is effective to aid future maintainability and avoid false failure if libasan.so.2 is not preloaded. 2015-06-25 Pádraig Brady build: update to latest gnulib Mainly for -fsanitize=address and -fsanitize=undefined fixes 2015-06-24 Pádraig Brady build: allow build to complete with -fsanitize=address * src/make-prime-list.c (main): When building with the above option, avoid this build stopping error: "LeakSanitizer: detected memory leaks" maint: avoid undefined behavior in qsort call GCC 5.1.1 -fsanitize=undefined with glibc 2.21 is returning: "runtime error: null pointer passed as argument 1, which is declared to never be null" * src/ptx.c (sort_found_occurs): Avoid the call with no entries. factor: avoid interspersed lines for parallel runs * src/factor.c (n_out): A new global variable to track how much data has been written to stdout. (print_factors_single): Use n_out to determine whether to flush the current (and previous) lines. * tests/misc/factor-parallel.sh: Add a new test. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix. seq: handle exponents more consistently src/seq.c (scan_arg): Set precision and width _after_ exponentiation. For example, this will make '1.1e1 12' and '11 1.2e1' equivalent. One can still set the precision by specifying extra precision on the start value, or more naturally with a precision on a step value. * tests/misc/seq-precision.sh: Add new cases. seq: use consistent output format with hex integers * src/seq.c (scan_arg): Set precision to 0 for hex constants (while avoiding hex floats). This will use then use the fast path for these arguments. Note we also set the precision of inf to 0 here, which ensures we use consistent precision on output where possible. * tests/misc/seq-precision.sh: Add corresponding test cases. seq: support inf last item more generally/efficiently * src/seq.c (main): Call seq_fast for infinite last value. This implicitly avoids format conversion on the 999999 -> 1000000 transition. * src/seq.c (seq_fast): Generalize the buffer handling, and adjust to handle the "inf" last value specifics. * tests/misc/seq-precision.sh: A new test. * tests/local.mk: Reference the new test. 2015-06-22 Pádraig Brady doc: list numfmt in the main menu with "Numeric operations" * doc/coreutils.texi (main menu): Add numfmt. numfmt: don't hardcode floating point limits * src/numfmt.c (MAX_UNSCALED_DIGITS): Set this to LDBL_DIG rather than hardcoding at 18 for better portability. * tests/misc/numfmt.pl: Restrict limit tests to supported platforms. 2015-06-21 Pádraig Brady numfmt: handle leading zeros correctly * src/numfmt.c (simple_strtod_int): Don't count leading zeros as significant digits. Also have leading zeros as optional for floating point numbers. * tests/misc/numfmt.pl: Add test cases. * NEWS: Mention the fix. 2015-06-21 Pádraig Brady numfmt: avoid integer overflow when rounding Due to existing limits this is usually triggered with an increased precision. We also add further restrictions to the output of increased precision numbers. * src/numfmt.c (simple_round): Avoid intmax_t overflow. (simple_strtod_int): Count digits consistently for precision loss and overflow detection. (prepare_padded_number): Include the precision when excluding numbers to output, since the precision determines the ultimate values used in the rounding scheme in double_to_human(). * tests/misc/numfmt.pl: Add previously failing test cases. * NEWS: Mention the fix. 2015-06-20 Pádraig Brady numfmt: support user specified output precision * src/numfmt.c (usage): Update the --format description to indicate precision is allowed. (parse_format_string): Parse a precision specification like the standard printf does. (double_to_human): Honor the precision in --to mode. * tests/misc/numfmt.pl: New tests. * doc/coreutils.texi (numfmt invocation): Mention the new feature. * NEWS: Likewise. 2015-06-19 Dylan Cali numfmt: implement support for field ranges * src/numfmt.c: Replace field handling code with logic that understands field range specifiers. Instead of processing a single field and printing line prefix/suffix around it, process each field in the line checking whether it has been included for conversion. If so convert and print, otherwise just print the unaltered field. (extract_fields): Removed. (skip_fields): Removed. (process_line): Gutted and heavily reworked. (process_suffixed_number): FIELD is now passed as an arg instead of using a global. (parse_field_arg): New function that parses field range specifiers. (next_field): New function that returns pointers to the next field in a line. (process_field): New function that wraps the field conversion logic (include_field): New function that checks whether a field should be converted (compare_field): New function used for field value comparisons in a gl_list. (free_field): New function used for freeing field values in a gl_list. Global variable FIELD removed. New global variable all_fields indicates whether all fields should be processed. New global variable all_fields_after stores the first field of a N- style range. New global variable all_fields_before stores the last field of a -M style range. New global variable field_list stores explicitly specified fields to process (N N,M or N-M style specifiers). (usage): Document newly supported field range specifiers. * bootstrap.conf: Include xlist and linked-list modules. numfmt now uses the gl_linked_list implementation to store the field ranges. * tests/misc/numfmt.pl: Add tests for 'cut style' field ranges. Adjust existing tests as partial output can occur before an error Remove test for the 'invalid' field -5.. this is now a valid range. * gnulib: update to avoid compiler warnings in linked-list. * NEWS: Mention the new feature. 2015-06-19 Pádraig Brady doc: use correct units in df | numfmt example * src/numfmt.c (usage): Don't scale output from df so that numfmt outputs the correct values. numfmt: handle suffixes consistently with --{from,to}-unit * src/numfmt.c (unit_to_umax): Support SI (power of 10) suffixes with the --from-unit and --to-unit options. Treat suffixes like is done with --from=auto, which for example will change the meaning of --to-unit=G to that of --to-unit=Gi. The suffix support was previously undocumented and it's better to avoid the traditional coreutils suffix handling in numfmt by default. * doc/coreutils.texi: Document the new behavior. Also fix a typo mentioning {from,to}=units=. * tests/misc/numfmt.pl: Adjust accordingly. * NEWS: Mention the change in behavior. 2015-06-17 Pádraig Brady maint: remove stale online manual items at release * gnulib: Update to get the new gnu-web-doc-update with --mirror option. * README-release: Use the --mirror option in the instructions. Also clarify and update various release steps. 2015-06-09 Pádraig Brady tail: display consistent diagnostics upon file replacement * src/tail.c (recheck): Display diagnostices for replaced files even with reused inodes which is a common case. * tests/tail-2/F-vs-missing.sh: Use correct diagnostic in comment. * tests/tail-2/F-vs-rename.sh: Likewise. tail: display file headers correctly with inotify * src/tail.c (tail_forever_inotify): Use the fspec pointer to distinguish previously output files, rather than a descriptor from the inotify event. That event descriptor was that of the parent directory when files were created or renamed etc. (check_fspec): Adjust for the new comparison. Also show the header when the file is truncated, since we show data in this case also. * tests/tail-2/F-headers.sh: A new test case. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix. 2015-06-07 Pádraig Brady maint: sync .gitignore items with gnulib entries * .gitignore: Add entries for potentially generated headers. Also remove a couple of items already present in lib/.gitignore. * cfg.mk (sc_gitignore_missing): A new syntax check rule to identify missing .gitignore entries. (sc_gitignore_redundant): A new syntax check rule to identify redundant .gitignore entries. Reported by Tomas Nordin. 2015-06-06 Pádraig Brady tests: fix false failure in recent test adjustment * configure.ac: Comment on why we link rather than run the test, and remove the moot __ELF__ check since we never ran it anyway, and the new CFLAGS and LDFLAGS are a more direct test of support. * tests/misc/wc-parallel.sh: Fix a syntax error in the previous change. * tests/misc/md5sum-parallel.sh: Use better error checking, consistent with that used in wc-parallel.sh. 2015-06-06 Paul Eggert build: port to AIX Problems reported by Michael Felt, and and part of this fix taken from code suggested by Pádraig Brady in: http://bugs.gnu.org/20733#112 * configure.ac (stdbuf_supported): Check for warnings, and for -fPIC and -shared, for AIX. * src/stat.c (STRUCT_STATVFS): Define to struct statvfs64 if STATFS is statvfs64. * src/sync.c (sync_arg) [_AIX]: Open in write mode, since AIX fsync doesn't work on read-only file descriptors. * tests/misc/wc-parallel.sh: Skip test if xargs -P does not work. build: update gnulib submodule to latest 2015-06-05 Paul Eggert build: update gnulib submodule to latest 2015-06-04 Paul Eggert build: port single_binary_prog to POSIX shell Problem reported privately by Michael Felt. * Makefile.am (install-exec-hook): * src/local.mk (src/coreutils_symlinks, src/coreutils_shebangs) (clean-local): Port to POSIX shell, which doesn't allow 'for i in ; do ...'. 2015-06-02 Pádraig Brady build: add a dependency on the gnulib tempname module * bootstrap.conf: Add "tempname" which is needed by mktemp(1). The explicit dependency supports running gnulib-tool with the --conditional-dependencies option, used to minimize built modules. Note on a Fedora 22 system, that results in avoiding redundant builds of: areadlinkat.o asnprintf.o fd-hook.o fseterr.o printf-args.o printf-parse.o sockets.o vasnprintf.o. However --conditional-dependencies is not enabled, since it currently precludes the inclusion of gnulib tests. 2015-06-01 Pádraig Brady build: update to latest gnulib Mainly with build fixes for Mac OS X. tests: fix race in tail test without inotify * tests/tail-2/wait.sh: Without inotify, skip a portion of the test that checks that -F never outputs from a tailed descriptor after the followed name is recreated, because tail_forever() doesn't guarantee that. Noticed at http://hydra.nixos.org/build/22766288 tests: fix false failure on loaded systems * tests/misc/uniq-perf.sh: Use our standard 10s timeout, which is sufficient to trigger the failure and also avoids a false failure on slow/loaded systems. Noticed at http://hydra.nixos.org/build/22766288 2015-05-29 Pádraig Brady maint: avoid new coverity warnings * src/sync.c (sync_arg): Initialise variable to avoid unitialized access if assert is disabled. * src/head.c (elide_tail_bytes_file): Support this function with ---presume-input-pipe and larger files, which regressed with commit v8.23-47-g2662702. (elide_tail_lines_file): Likewise. * src/dd.c (dd_copy): Explicitly don't try to ftruncate() upon failure to lseek() (the existing check against st_size was already protecting that). * src/factor.c (factor_using_squfof): Assert (only when linting due to performance) to avoid the implication of divide by zero. * src/od.c (read_block): Remove dead code. * src/tac.c (tac_seekable): Likewise. * src/ls.c (gobble_file): Likewise. 2015-05-27 Pádraig Brady build: remove workarounds for unsupported gettext versions Now that we depend on gettext >= 0.19.2 remove the workaround for issues in autopoint 0.18.3. Note the scheme currently used in newer gettext (autopoint) to avoid these issues requires autoconf >= 2.69, therefore we update this requirement also. Note the gettext version dependence from gnulib comes from gnulib using gettext macros, and coreutils indirectly depends on the gettext module due to: http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=41dca647 bootstrap will then update m4/po.m4 and thus require a supportng gettext version. * bootstrap: Remove moot warning (resyncing with gnulib). * configure.ac (AC_PREREQ): Change to 2.69 (now 3 years old). 2015-05-27 Pádraig Brady tests: ln/hard-to-sym FAIL rather than ERROR when ln fails * tests/ln/hard-to-sym.sh: Only call framework_failure_ when ln returns success. copy: prefer our hardlink to symlink emulation on OS X 10.10 * src/copy.c (CAN_HARDLINK_SYMLINKS): Don't enable use of linkat() on Darwin 14, as the gnulib fallback emulation there doesn't preserve ownership and timestamps etc. This fixes a test failure in tests/cp/link-symlink.sh * tests/cp/link-deref.sh: Adjust accordingly. 2015-05-25 Pádraig Brady maint: remove sys/types.h include order check * src/system.h: This was inadvertently ineffective due to a typo in commit v8.9-10-ge1aaf89 (Jan 2011), but has not caused any issues, so remove. 2015-05-22 Pádraig Brady mkdir: fix -pZ with existing parent directories When the parent directory exists and has a different default context to the final directory, the context was incorrectly left as that of the parent directory. * src/mkdir.c (process_dir): Because defaultcon() is called for existing ancestors (as it must be to avoid races), then we must unconditionally call restorecon() on the last component due to the already documented caveat with make_dir_parents(). Alternatively you could temp disable o->set_security_context around make_dir_parents(), but that would be subject to races. * tests (tests/mkdir/restorecon.sh): Add a TODO for improvement. Reference mknod and mkfifo with print_ver_. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/20616 2015-05-22 Assaf Gordon tests: df-output: accept multiple spaces in header * tests/df/df-output.sh: Allow for multiple spaces in the header line of 'df', resulting from alignment with disk sizes >= 10TB. 2015-05-19 Pádraig Brady tests: fix non POSIX constructs causing failures with dash * tests/cp/no-ctx.sh: Scope of `var=val func` is inconsistent across shells, so avoid that construct with functions. * tests/df/no-mtab-status.sh: Likewise. * tests/tail-2/inotify-race.sh: `read` needs an argument. * tests/tail-2/inotify-race2.sh: Likewise. 2015-05-17 Pádraig Brady build: fix 'dist' and 'syntax-check' targets in VPATH build * cfg.mk: Various syntax-check adjustments so that it's not assumed the $builddir is the base distribution directory. * Makefile.am: Likewise for the 'dist' target. build: avoid issues with case insensitive file systems * cfg.mk (sc_case_insensitive_file_names): A new syntax-check rule. * tests/tail-2/descriptor-vs-rename.sh: Rename from tests/tail-2/f-vs-rename.sh * tests/local.mk: Reference the renamed test. Reported by Jim Meyering. 2015-05-14 Bernhard Voelker tests: fix check for local file system in inotify-rotate-resources.sh * tests/tail-2/inotify-rotate-resources.sh: s/(is_local_dir)/\1_/ 2015-05-14 Pádraig Brady tests: fix async allocation race on BTRFS * tests/dd/sparse.sh: Sync files before checking allocations, which may be done asynchronously on NFS and BTRFS at least. Also mark this test as very expensive on remote file systems. * tests/du/2g.sh: Likewise, also use fallocate if available to efficiently allocate the large file, otherwise skip on remote file systems. * tests/tail-2/inotify-rotate-resources.sh: Use the more standard is_local_dir_() to check remoteness. * tests/cp/fiemap-empty.sh: Comment on the sync issue for this currerntly unused test. Fixes http://bugs.gnu.org/20570 2015-05-13 Pádraig Brady timeout: with --foreground don't send SIGCONT * src/timeout.c (cleanup): Don't send SIGCONT to the monitored program when --foreground is specified, as it's generally not needed for foreground programs, and can cause intermittent signal delivery issues with monitors like GDB for example. * doc/coreutils.texi (timeout invocation): Mention that SIGCONT is not sent with --foreground. * NEWS: Mention the behavior change. 2015-05-13 Pádraig Brady split: auto set suffix len for --numeric-suffixes= doc: clarify the operation of wc -L * src/wc.c (usage): State that it calculates display width. * doc/coreutils.texi (wc invocation): Detail the distinct items used to determine the display width. 2015-05-11 Pádraig Brady tail: consistently output all data for truncated files Generally if logs are truncated, they're truncated to 0 length, so output all existing data when our heuristic determines truncation. Note with inotify, truncate() and write() are often determined independently and so all data would be written if that was the case. * src/tail.c (check_fspec): Reset file offset to 0 upon truncation. (tail_forever): Likewise. (recheck): Add a FIXME for the related issue where tail may lose data due to tail discounting older log files too early. * tests/tail-2/truncate.sh: A new test. * tests/local.mk: Reference the new test. * NEWS: Mention the fix. 2015-05-11 Pádraig Brady tail: fix inotify startup races The previous fixes to races in the various tail tests, identified actual races in the tail inotify implementation. With --follow=descriptor, if the tailed file was replaced before the inotify watch was added, then any subsequent changes were ignored. Similarly in --follow=name mode, all changes to a new name were effectively ignored if that name was created after the original open() but before the inotify_add_watch(). * src/tail.c (tail_forever_inotify): Fix 3 cases. 1. With -f, don't stop tailing when file removed before watch. 2. With -f, watch right file when file replaced before watch. 3. With -F, inspect correct file when replaced before watch. Existing tests identify these when tail compiled with TAIL_TEST_SLEEP. * tests/tail-2/inotify-rotate-resources.sh: This test also identifies the issue with --follow=name when TAIL_TEST_SLEEP is used. Adjust so the test is immune to such races, and also fail quicker on remote file systems. * tests/tail-2/inotify-race2.sh: A new test using GDB, based on inotify-race.sh, which tests the -F race without needed recompilation with sleeps. * tests/local.mk: Reference the new test. * NEWS: Mention the bug. 2015-05-11 Pádraig Brady tests: cleanup background processes upon interruption Reap background processes so that: - Stray processes aren't left on the system - Files aren't held open causing deletion issues on NFS - Partitions used to run the tests from can be unmounted * tests/tail-2/F-vs-missing.sh: Add the `kill && wait` of the background $pid(s) to cleanup_(). * tests/tail-2/F-vs-rename.sh: Likewise. * tests/tail-2/f-vs-rename.sh: Likewise. * tests/tail-2/append-only.sh: Likewise. * tests/tail-2/assert-2.sh: Likewise. * tests/tail-2/assert.sh: Likewise. * tests/tail-2/flush-initial.sh: Likewise. * tests/tail-2/inotify-hash-abuse.sh: Likewise. * tests/tail-2/inotify-hash-abuse2.sh: Likewise. * tests/tail-2/inotify-race.sh: Likewise. * tests/tail-2/inotify-rotate-resources.sh: Likewise. * tests/tail-2/inotify-rotate.sh: Likewise. * tests/tail-2/pid.sh: Likewise. * tests/tail-2/pipe-f2.sh: Likewise. * tests/tail-2/retry.sh: Likewise. * tests/tail-2/symlink.sh: Likewise. * tests/tail-2/tail-n0f.sh: Likewise. * tests/tail-2/wait.sh: Likewise. * tests/cp/existing-perm-race.sh: Likewise. * tests/cp/file-perm-race.sh: Likewise. * tests/cp/parent-perm-race.sh: Likewise. * tests/cp/sparse-to-pipe.sh: Likewise. * tests/dd/stats.sh: Likewise. * tests/du/move-dir-while-traversing.sh: Likewise. * tests/misc/cat-buf.sh: Likewise. * tests/misc/help-version.sh: Likewise. * tests/misc/printf-surprise.sh: Likewise. * tests/misc/sort-compress-proc.sh: Likewise. * tests/misc/sort-spinlock-abuse.sh: Likewise. * tests/misc/stdbuf.sh: Likewise. * tests/misc/tac-continue.sh: Likewise. * tests/misc/timeout-group.sh: Likewise. * tests/mv/i-3.sh: Likewise. * tests/rm/dangling-symlink.sh: Likewise. * tests/rm/isatty.sh: Likewise. * cfg.mk (sc_prohibit_test_background_without_cleanup_): A new syntax-check to ensure cleanup_() is defined when background tasks are created in a test. 2015-05-11 Pádraig Brady tests: avoid hung processes due to gdb SIGCONT handling * tests/tail-2/inotify-race.sh: Add a `wait` to ensure that we reap all background gdb and tail processes. That resulted in the test hanging intermittently and upon investigation was due to gdb intermittently failing to terminate the child process due to receiving a SIGCONT signal. Therefore we avoid using timeout(1) which sends that signal, and instead rely on tail's inbuilt --pid monitoring on a background sleep process. Given this new implementation, the VERY_EXPENSIVE guard was removed. Related issues with this test hanging were previously discussed at: https://lists.gnu.org/archive/html/bug-coreutils/2009-12/msg00025.html tests: fix races in and standardize the tail tests * tests/tail-2/F-vs-missing.sh: Use standard "fastpoll" options (-s.1 --max-unchanged-stats=1) to speedup the non-inotify case. Add the non-inotify case to the test. `wait` on the background tail process to terminate which should avoid the need for the non standard `retry_delay_ cleanup ...` on NFS. * tests/tail-2/F-vs-rename.sh: Remove 'out' at the start of the loop, to avoid a race in checking its contents. Also ensure 'a' & 'b' files are present before the tail process starts. Use the standard "fastpoll" options as above. * tests/tail-2/f-vs-rename.sh: Likewise. * tests/tail-2/append-only.sh: Use more standard variable names. * tests/tail-2/flush-initial.sh: Use "fastpoll" options for non-inotify platforms. Also `wait` on the background tail to avoid stray processes and file cleanup issues on NFS. * tests/tail-2/inotify-hash-abuse.sh: Always run non-inotify case. Use "fastpoll" options. Use a more standard retry_delay_ instead of a hardcoded sleep loop. Add a `wait` on the background tail. * tests/tail-2/inotify-hash-abuse2.sh: Likewise. * tests/tail-2/inotify-rotate-resources.sh: Wait just on the specific tail $pid needed. * tests/tail-2/inotify-rotate.sh: Use "fastpoll" options. * tests/tail-2/pid.sh: Use standard variable names. Add a `wait` on the background tails. * tests/tail-2/pipe-f2.sh: Likewise. * tests/tail-2/tail-n0f.sh: Likewise. * tests/tail-2/retry.sh: Use "fastpoll" options. * tests/tail-2/symlink.sh: Likewise. * tests/tail-2/wait.sh: Likewise. Speedup by using sub second parameters to timeout(1). Improve the part ensuring that -F never follows a renamed file. * tests/tail-2/infloop-1.sh: Remove invalid test. tail(1) was not being passed the --pid=$yes_pid option, retry_delay_ wasn't used to avoid races, and yes could write huge files before being killed. * tests/local.mk: Remove the invalid test reference. * tests/tail-2/assert-2.sh: Rewrite using retry_delay_(). Since no longer hardcoding large delays, remove the VERY_EXPENSIVE tag. * tests/tail-2/assert.sh: Likewise. 2015-05-10 Paul Eggert build: rewrite is_ENOTSUP without an #if directive * src/system.h (is_ENOTSUP): Avoid in-function #if directive. 2015-05-09 Jim Meyering build: avoid a warning form gcc's new -Wlogical-op Without this change, very recent gcc (e.g., version 6.0.0 20150509) would print the following when configured with --enable-gcc-warnings: src/copy.c:165:30: error: logical 'or' of equal expressions \ [-Werror=logical-op] && (errno == EOPNOTSUPP || errno == ENOTSUP || errno == ENOSYS)) ^ * src/system.h (is_ENOTSUP): New function. * src/copy.c (punch_hole): Use it. * src/ls.c (errno_unsupported, gobble_file): Use it. 2015-04-30 Pádraig Brady doc: standardize messages about the '-' stdin FILE * src/system.h (emit_stdin_note): A new function, refactoring the usage note about the '-' FILE implying stdin. * src/base64.c (usage): Use the new function to emit the note in a standard location and with standard separation. * src/cat.c (usage): Likewise. * src/csplit.c (usage): Likewise. * src/cut.c (usage): Likewise. * src/expand.c (usage): Likewise. * src/fmt.c (usage): Likewise. * src/head.c (usage): Likewise. * src/md5sum.c (usage): Likewise. * src/nl.c (usage): Likewise. * src/od.c (usage): Likewise. * src/paste.c (usage): Likewise. * src/pr.c (usage): Likewise. * src/ptx.c (usage): Likewise. * src/shred.c (usage): Likewise. * src/shuf.c (usage): Likewise. * src/sort.c (usage): Likewise. * src/sum.c (usage): Likewise. * src/tac.c (usage): Likewise. * src/tail.c (usage): Likewise. * src/tsort.c (usage): Likewise. * src/unexpand.c (usage): Likewise. * src/wc.c (usage): Likewise. * src/join.c (usage): Adjust the separation used for the message referring to FILE1 or FILE2 as stdin. * src/comm.c (usage): Add a message using the same wording (translation) as used in join. * src/split.c (usage): Reword to using FILE rather than INPUT, allowing use of emit_stdin_note(). Also remove the mention of "fixed-size" pieces as this isn't now always the case. Fixes http://pad.lv/1450179 tests: don't skip df tests with /proc/self/mountinfo * tests/df/no-mtab-status.sh: getmntent is no longer called when /proc/self/mountinfo is present, thus causing the test to be skipped. Therefore wrap fopen() to ignore mountinfo, and use the test genmntent table instead. * tests/df/skip-duplicates.sh: Likewise. 2015-04-28 Yunlian Jiang build: fix SINGLE_BINARY build when printf is a macro * src/coreutils.c (usage): include coreutils.h outside the printf call, because if it's a macro you will get the error: embedding a #include directive within macro arguments is not supported 2015-04-28 Pádraig Brady maint: avoid a new -Werror=return-type warning in yes.c * src/yes.c (main): Simplify the logic so that the compiler can see this function always returns a value. This was seen with GCC 5.0 in SINGLE_BINARY mode. tests: run ls capability coloring test irrespective of $LS_COLORS * tests/ls/no-cap.sh: Ensure the test isn't skipped even if capability coloring is disabled in the current $LS_COLORS. Also just enable/disable capability coloring to avoid the dircolors(1) overhead. 2015-04-27 Pádraig Brady maint: use gnulib styling with the online manual The equivalent styling added in v8.23-155-g3b98ee7, is now applied to gnulib using projects by default. build: rely on gnulib to determine printf routines are safe gnulib now only checks that the printf routines never crash, which is all coreutils currrently requires, and so we revert commit v8.23-81-gf57bfbb to let gnulib decide whether to replace the system printf routines. maint: fix printf format for signed integers With GCC 5 and the newly added warnings from gnulib, ensure the correct signed integer is passed for the printf format, to avoid -Werror=format= failures. build: update gnulib submodule to latest Fix file-has-acl build failure on RHEL/Centos 6. Fix GCC 5 warnings with printf and in fts.c. build: reduce gettext dependency to 0.19.2 * bootstrap.conf: 0.19.2 is available on openSUSE-13.2, Debian-8.0, and Ubuntu-14.10. Given there were issues with earlier 0.19 gettext releases, set this as the new minimum. * configure.ac (AM_GNU_GETTEXT_VERSION): Likewise. Reported by Bernhard Voelker 2015-04-26 Paul Eggert ls: on GNU/Linux, remove dependency on libacl * src/local.mk (src_ls_LDADD): Change from LIB_ACL to LIB_HAS_ACL. build: update gnulib submodule to latest * bootstrap.conf (gnulib_modules): Add file-has-acl. (buildreq): Bump autopoint and gettext to 0.19.4. * configure.ac (AM_GNU_GETTEXT_VERSION): Bump to 0.19.4. * gl/lib/tempname.c.diff, gl/lib/tempname.h.diff: Merge recent gnulib changes. 2015-04-23 Michael Witten doc: fix grammar issue in truncate info * doc/coreutils.texi (truncate invocation): The word 'their' is incorrect; 'each file' is the antecedent, and is singular, so 'its' is the correct pronoun. 2015-04-22 Pádraig Brady build: fix potential factor build failure on arm and mips * src/longlong.h: Sync with the latest longlong.h from libgmp to: - Use __builtin_c[lt]zl on arm64. - Fix sparc64 vis3 build failure due to missing __clz_tab. - Avoid a clang build issue on mips. - Support thumb2 arm 32 bit system. * src/cfg.mk (sc_ensure_comma_after_id_est): Exclude longlong.h to ease merges. 2015-04-22 Pádraig Brady maint: avoid -Werror=strict-overflow warnings with GCC 5 All warnings were of the form: "assuming signed overflow does not occur when simplifying conditional to constant [-Werror=strict-overflow]" * src/dd.c (cache_round): Use an appropriately sized unsigned type, to avoid possibility of undefined signed overflow. * src/mknod.c (main): Likewise. * src/pr.c (pad_down): Likewise. * src/wc.c (main): Likewise. * src/tail.c (main): Assert that argc >= 0 thus allowing the compiler to assume without implication that argc - optind is positive. 2015-04-17 Pádraig Brady dircolors: add 'MISSING' to the default database * src/dircolors.hin: Add the MISSING entry, to indicate this as a possibility in new templates output from dircolors, and also to ease comparison with existing databases that generally do define a MISSING entry. 2015-04-13 Andreas Gruenbacher doc: clarify how cp behaves with default ACLs * doc/coreutils.texi (cp invocation): Mention that when copying files without preserving permissions, the umask or a default ACL affect the mode of new files. * THANKS.in: Remove committer. Related to http://bugs.gnu.org/8527 2015-04-13 Pádraig Brady df: fix --local hanging with inaccessible remote mounts * src/df.c (filter_mount_list): With -l, avoid stating remote mounts. * init.cfg: Avoid test hangs with inaccessible remote mounts. * tests/df/no-mtab-status.sh: Skip with inaccessible remote mounts. * tests/df/skip-rootfs.sh: Likewise. * tests/df/total-verify.sh: Likewise. * NEWS: Mention the bug fix. Reported at http://bugzilla.redhat.com/1199679 2015-04-08 Mitchel Humpherys doc: correct pluralization for mkfifo and mknod * doc/coreutils.texi: `mkfifo' and `mknod' use the optContext macro which adds a description for the SELinux security context in addition to the single option already described in each case. The result in both cases is two options being introduced as `option' (singular). Fix this by introducing them as `options' (plural). 2015-04-03 Pádraig Brady df: fix use of uninitialized variable reported by valgrind Conditional jump or move depends on uninitialised value(s) at 0x40380C: get_field_values (df.c:840) by 0x403E16: get_dev (df.c:994) by 0x404D65: get_all_entries (df.c:1364) by 0x405926: main (df.c:1714) * src/df.c (get_dev): Initialize the fsu.fsu_bavail_top_bit_set member, when adding placeholder entries. (main): Avoid a "definitely lost" memory leak warning from valgrind, reported by Bernhard Voelker. 2015-03-31 Pádraig Brady doc: clarify that ls --sort=time is newest first * src/ls.c (usage): Add punctuation to avoid ambiguity in the description of the --time option. Mention that both the -u and --sort=time default order is newest first. 2015-03-31 Stephane Chazelas tail: fix -f to follow changes after a rename * src/tail.c (tail_forever_inotify): Only monitor write()s and truncate()s to files in --follow=descriptor mode, thus avoiding the bug where we removed the watch on renamed files. Also adjust the inotify event processing code that is now significant only in --follow=name mode. * tests/tail-2/F-vs-rename.sh: Improve this existing test by running in both polling and inotify modes. * tests/tail-2/f-vs-rename.sh: A new test based on the existing one. * tests/local.mk: Reference the new test. * NEWS: Mention the bug. Fixes http://bugs.gnu.org/19760 2015-03-28 Pádraig Brady doc: move numfmt info to the 'Numeric operations' section * doc/coreutils.texi: Move numfmt info to this section, as numfmt functionality aligns more with seq and factor, than fmt and pr etc. 2015-03-27 Paul Eggert nohup: clarify stdin redirection Problem reported by Isaac Schwabacher in: http://bugs.gnu.org/20214 * doc/coreutils.texi (nohup invocation): Clarify that when nohup's stdin gets redirected, it's unreadable. * doc/coreutils.texi (nohup invocation): * src/nohup.c (usage): Don't promise /dev/null. 2015-03-27 Assaf Gordon tests: fix false test failure with df on Debian/kFreeBSD * tests/fs/skip-duplicates.sh: On this platform .mnt_opts is significant so define to empty to avoid a NULL deref in read_file_system_list(). Fixes http://bugs.gnu.org/20210 2015-03-26 Christoph Anton Mitterer doc: disambiguate the ls --color description * src/ls.c (usage): Avoid the implication that the default ls behavior is to --color=always. Reported in http://bugs.debian.org/781208 2015-03-26 Pádraig Brady doc: clarify the date standard output formats * src/date.c (usage): Use FMT rather than TIMESPEC as the parameter, since it's simpler to understand and can be better aligned. Give an example for the --iso-8601 output format. Adjust the example used for the 3 standard formats to be unambiguous with respect to day/mon ordering and use of leading zeros in the time. Reorder the options descriptions slightly, so that the 3 standards options are together. Indent the multi-line descriptions so that grouping is obvious. Remove a redundant description of the --rfc-3339 format, which is obvious in the existing example. Separate these 3 standards options to their own translatable string to simplify translation. Change 'date and time' to 'date/time' in the --iso-8601 description to be consistent with --rfc-3339 and to help avoid the implication that the time is always output or even output by default. Fixes http://bugs.gnu.org/20203 2015-03-24 Pádraig Brady maint: apply basic styling to online manual * README-release: Reference http://www.gnu.org/s/coreutils/manual.css to apply basic styling to the online coreutils manual, consistent with the Emacs documentation. doc: clarify the uniq -D man page description * src/uniq.c (usage): The description was very confusing in the man page due to the stripped newlines. Add punctuation for clarification. wc: use a more adaptive wc -l implementation * src/wc.c (wc): Allow any block to select the count implementation, rather than just using the first 10 lines. This also simplifies the code from 3 loops to 2. 2015-03-23 Dan Jacobson doc: clarify default order for ls --sort=size * src/ls.c (usage): Mention that default order is largest first. Fixes http://bugs.gnu.org/20172 2015-03-20 Pádraig Brady maint: really fix wildcard quoting in sc_tests_executable * cfg.mk (sc_tests_executable): The previous commit avoided the globbing, but also passed on the quoted wildcards to find(1). We could use eval to handle the quoting, though that's a bit awkward and dangerous, so instead explicitly disable globbing for the whole make target subshell. Note noglob is not available on solaris, where we fall back to set -f. Note also that zsh uses set -F for this, but that's moot here. Also correct the find(1) expression to include the -o between each wildcard. 2015-03-19 Pádraig Brady maint: fix wildcard quoting in sc_tests_executable * cfg.mk (sc_tests_executable): If there are files with $TEST_EXTENSIONS in the current directory, then the lack of quoting of the $test_extensions_rx contents could result in globbing and an inconsequential run. find(1) produces warnings only with more than one expansion. 2015-03-19 Kristoffer Brånemyr wc: speedup counting of short lines Using a test file generated with: yes | head -n100M > 2x100M.txt before> time wc -l 2x100M.txt real 0.842s user 0.810s sys 0.033s after> time wc -l 2x100M.txt real 0.142s user 0.111s sys 0.031s * src/wc.c (wc): Split the loop that deals with -l into 3. The first is used at the start of the input to determine if the average line length is < 15, and if so the second loop is used to look for '\n' internally to wc. For longer lines, memchr is used as before to take advantage of system specific optimizations which any outweigh function call overhead. Note the first 2 loops could be combined, though in testing, GCC 4.9.2 at least, wasn't sophisticated enough to separate the loops based on the "check_len" invariant. Note also __builtin_memchr() isn't significant here as GCC currently only applies constant folding with that. * NEWS: Mention the improvement. 2015-03-10 Giuseppe Scrivano yes: improve efficiency when all args aren't buffered * src/yes.c (main): Even when the internal buffer isn't large enough, output what we've buffered already, and interate over the rest. This improves the performance in the edge case where there are many small arguments that overflow the buffer. * tests/misc/yes.sh: Add a test case for the many small arguments case. 2015-03-09 Pádraig Brady yes: output data more efficiently yes(1) may be used to generate repeating patterns of text for test inputs etc., so adjust to be more efficient. Profiling the case where yes(1) is outputting small items through stdio (which was the default case), shows the overhead of continuously processing small items in main() and in stdio: $ yes >/dev/null & perf top -p $! 31.02% yes [.] main 27.36% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 14.51% libc-2.20.so [.] fputs_unlocked 13.50% libc-2.20.so [.] strlen 10.66% libc-2.20.so [.] __GI___mempcpy 1.98% yes [.] fputs_unlocked@plta Sending more data per stdio call improves the situation, but still, there is significant stdio overhead due to memory copies, and the repeated string length checking: $ yes "`echo {1..1000}`" >/dev/null & perf top -p $! 42.26% libc-2.20.so [.] __GI___mempcpy 17.38% libc-2.20.so [.] strlen 5.21% [kernel] [k] __srcu_read_lock 4.58% [kernel] [k] __srcu_read_unlock 4.27% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 2.50% libc-2.20.so [.] __GI___libc_write 2.45% [kernel] [k] system_call 2.40% [kernel] [k] system_call_after_swapgs 2.27% [kernel] [k] vfs_write 2.09% libc-2.20.so [.] _IO_do_write@@GLIBC_2.2.5 2.01% [kernel] [k] fsnotify 1.95% libc-2.20.so [.] _IO_file_write@@GLIBC_2.2.5 1.44% yes [.] main We can avoid all stdio overhead by building up the buffer _once_ and outputting that, and the profile below shows the bottleneck moved to the kernel: $ src/yes >/dev/null & perf top -p $! 15.42% [kernel] [k] __srcu_read_lock 12.98% [kernel] [k] __srcu_read_unlock 9.41% libc-2.20.so [.] __GI___libc_write 9.11% [kernel] [k] vfs_write 8.35% [kernel] [k] fsnotify 8.02% [kernel] [k] system_call 5.84% [kernel] [k] system_call_after_swapgs 4.54% [kernel] [k] __fget_light 3.98% [kernel] [k] sys_write 3.65% [kernel] [k] selinux_file_permission 3.44% [kernel] [k] rw_verify_area 2.94% [kernel] [k] __fsnotify_parent 2.76% [kernel] [k] security_file_permission 2.39% yes [.] main 2.17% [kernel] [k] __fdget_pos 2.13% [kernel] [k] sysret_check 0.81% [kernel] [k] write_null 0.36% yes [.] write@plt Note this change also ensures that yes(1) will only write complete lines for lines shorter than BUFSIZ. * src/yes.c (main): Build up a BUFSIZ buffer of lines, and output that, rather than having stdio process each item. * tests/misc/yes.sh: Add a new test for various buffer sizes. * tests/local.mk: Reference the new test. Fixes http://bugs.gnu.org/20029 2015-03-06 Pádraig Brady build: fix make dependencies for test.1 In certain parallel build situations this would give the error: help2man: can't get `--help' info from man/test.td/[ Makefile:14189: recipe for target 'man/test.1' failed * man/local.mk (test.1): Depend on `[` rather than `test`, as `test --help` outputs nothing. Also move dir.1 and vdir.1 back to the main list, as they're no more exceptions than sha1sum etc. 2015-03-05 Bernhard Voelker maint: be less strict about executable permissions of tests With "umask 0027" or even "umask 0077", the git clone of coreutils does not have the executable bit set for 'other' (or 'group). Therefore, "make syntax-check" would fail. * cfg.mk (sc_tests_executable): Change the -perm argument of find(1) to only print the names of the files which are not executable by the user, rather than insisting on ugo+x (octal 111). 2015-03-05 Stéphane Aulery doc: mention persistence caveats in sync man page * man/sync.x ([BUGS]): Mention the caveats and reference the system call man pages for more details. Fixes http://bugs.gnu.org/19995 Debian bug #507085 reported by jidanni@jidanni.org 2015-03-04 Pádraig Brady tee: generalize the --write-error option to --output-error Adjust commit v8.23-140-gfdd6ebf to add the --output-error option instead of --write-error, and treat open() errors like write() errors. * doc/coreutils.texi (tee invocation): s/write-error/output-error/. * src/tee.c (main): Exit on open() error if appropriate. * tests/misc/tee.sh: Add a case to test open() errors. * NEWS: Adjust for the more general output error behavior. Suggested by Bernhard Voelker. 2015-03-04 Jarosław Gruca maint: update stale comment about ls color sequences * src/ls.c (color_indicator[C_END]): Comment with the correct sequence, which was used since commit v6.10-61-g483297d Fixes http://bugs.gnu.org/19992 2015-02-27 Shane M Seymour tail,stat: improve support for the IBRIX file system Note that IBRIX used to have a different magic number 0x013111A7 instead of the current 0x013111A8. However, the former is no longer used and the version of IBRIX it was used in is really ancient, so it's extremely unlikely anyone is still using it. Therefore, just add the newer magic number. Mark IBRIX as a 'remote' file system type as inotify support had never been officially tested with it. * src/stat.c (human_fstype): Add file system ID definition. * NEWS: Mention the improvement. Fixes http://bugs.gnu.org/19951 2015-02-24 Pádraig Brady tee: add --write-error to control handling of closed pipes tee is very often used with pipes and this gives better control when writing to them. There are 3 classes of file descriptors that tee can write to: files(1), pipes(2), and early close pipes(3). Handling write errors to 1 & 2 is supported at present with the caveat that failure writing to any pipe will terminate tee immediately. Handling write errors to type 3 is not currently supported. To improve the supported combinations we add these options: --write-error=warn Warn if error writing any output including pipes. Allows continued writing to still open files/pipes. Exit status is failure if any output had error. --write-error=warn-nopipe, -p Warn if error writing any output except pipes. Allows continued writing to still open files/pipes. Exit status is failure if any non pipe output had error. --write-error=exit Exit if error writing any output including pipes. --write-error=exit-nopipe Exit if error writing any output except pipes. Use the "nopipe" variants when files are of types 1 and 3, otherwise use the standard variants with types 1 and 2. A caveat with the above scheme is that a combination of pipe types (2 & 3) is not supported robustly. I.e. if you use the "nopipe" variants when using both type 2 and 3 pipes, then any "real" errors on type 2 pipes will not be diagnosed. Note also a general issue with type 3 pipes that are not on tee's stdout, is that shell constructs don't allow to distinguish early close from real failures. For example `tee >(head -n1) | grep -m1 ..` can't distinguish between an error or an early close in "head" pipe, while the fail on the grep part of the pipe is distinguished independently from the resulting pipe errors. This is a general issue with the >() construct, rather than with tee itself. * NEWS: Mention the new feature. * doc/coreutils.texi (tee invocation): Describe the new option. * src/tee.c (usage): Likewise. (main): With --write-error ignore SIGPIPE, and handle the various exit, diagnostics combinations. * tests/misc/tee.sh: Tess all the new options. Fixes http://bugs.gnu.org/11540 2015-02-23 Bernhard Voelker tee: close "-" file This is a cleanup to the previous commit v8.23-138-g7ceaf1d. * src/tee.c (tee_files): Do not exempt the "-" file from being closed, as this is no longer stdout but a normal file. 2015-02-20 Bernhard Voelker tee: treat '-' operand as file name as mandated by POSIX Since v5.2.1-1247-g8dafbe5, tee(1) treated '-' as stdout while POSIX explicitly requires to treat this as a file name. Revert this change, as the interleaved output - due to sending another copy of input to stdout - is not considered to be useful. Discussed in http://lists.gnu.org/archive/html/coreutils/2015-02/msg00085.html * src/tee.c (tee_files): Remove the special handling for "-" operands. (usage): Remove the corresponding sentence. * doc/coreutils.texi (common options): Remove the "tee -" example. (tee invocation): Document that tee(1) now treats "-" as a file name. * tests/misc/tee.sh: Add a test case for "tee -". While at it, re-indent the above multi-argument processing case and extend that to 13 operands, as POSIX mandates that, too. * tests/misc/tee-dash.sh: Remove now-obsolete test. * tests/local.mk (all_tests): Remove the above test. * NEWS (Changes in behavior): Mention the change. 2015-02-18 Pádraig Brady tee: exit early if no more writable outputs * src/tee.c (main): Don't continue reading if we can't output anywhere. * tests/misc/tee.sh: Ensure we exit when no more outputs. * NEWS: Mention the change in behavior. tests: support stderr verification with returns_() * tests/init.sh (returns_): Disable tracing for this wrapper function, so that stderr of the wrapped command is unchanged, allowing for verification of the contents. 2015-02-16 Bernhard Voelker maint: prefer STREQ_LEN and STRPREFIX over strncmp in all cases * cfg.mk (sc_prohibit_strncmp): Improve the search pattern: use _sc_search_regexp to find all invocations of strncmp except when used on a macro definition line; just match the function name with an opening parenthesis. Before, the expression missed places where the comparison against 0 was in a subsequent line. * src/system.h (STRNCMP_LIT): Shorten 'literal' to 'lit' to move the whole definition of the macro into one line - thus making sc_prohibit_strncmp pass. (STRPREFIX): Add space before parenthesis. * src/du.c (main): Prefer STREQ_LEN over strncmp. * src/pinky.c (scan_entries): Likewise. * src/tac.c (tac_seekable): Likewise. * src/who.c (scan_entries): Likewise. 2015-02-16 Pádraig Brady stty: fix setting of 'extproc' on BSD This setting is unusual on BSD as it's read normally in the local flags returned by tcgetattr(), but can only be set with an ioctl. Setting with tcsetattr() is ignored. * src/stty.c (NO_SETATTR): A new flag to indicate the setting is read and displayed like a normal termios flag, but is set in some other manner. (main): Skip tcsetattr() for this setting when this flag is set. Also fixup the exiting 'extproc' processing to handle the '-extproc' case correctly. (sane_mode): Skip setting '-extproc' for 'sane' to avoid the error. This isn't ideal but matches the operation of the BSD native stty. 2015-02-12 Bernhard Voelker maint: enhance '.mailmap' mappings for THANKS generation * .mailmap (jeff.liu@oracle.com): There are 3 different names in the 'git log' output for this email address; choose "Jeff Liu" as canonical form. (Алексей Шилин): Convert name to latin1 ("Aleksej Shilin") to improve the sort order of the generated 'THANKS' file. 2015-02-11 Bernhard Voelker maint: sort THANKS using our own gear At least 'sort' on openSUSE/Fedora have a bug in the case-folding code of their I18N downstream patch which leads to wrong sort results, e.g. "Dániel" coming after "Dylan". * Makefile.am (THANKS): Sort the final contributor list using our own sort implementation - as others may result in a different order; add a FIXME comment to remove this again once common platforms have a functional 'sort -f'. Add '-k1,1' for a better sort order. While at it, save a grep and perl call to prepare the list from 'THANKS.in' by doing all in the first perl call. 2015-02-11 Pádraig Brady maint: remove redundant diff filtering from sc_long_lines * cfg.mk (sc_long_lines): diff files are now completely excluded, so no need to filter portions of them. Reported by Bernhard Voelker. 2015-02-10 Pádraig Brady build: update to latest gnulib This includes a change to require --with-libmount to be used with configure, due to the many libmount dependencies. * bootstrap: Sync with gnulib to exit early on gnulib-tool error. * gl/lib/tempname.c.diff: Adjust for gnulib changes. * gl/lib/tempname.h.diff: Likewise. * gl/modules/tempname: Likewise. * doc/.gitignore: Add new gendocs_template_min gnulib script. 2015-02-10 Pádraig Brady maint: exclude diff files from long lines check * cfg.mk: Add .diff files to the exclusion expression for sc_long_lines, since the gnulib code might be >= 80 chars. Note 80 char lines trigger due to the added +/- diff marks. Also normalize the $$ used in the other sc_long_lines exclusion expressions. maint: avoid arbitrary memory access with buggy localtime() * src/ls.c (align_nstrftime): Be defensive and validate the tm_mon index before using to access the abmon array. This was _not_ seen to be an issue any system. See https://bugzilla.redhat.com/1190454 doc: give an example of using a seed for random operations * doc/coreutils.texi (Random sources): Give an example using openssl, generating a reproducible arbitrary amount of randomly distributed data, given a seed value. 2015-02-09 Bernhard Voelker maint: make sc_dd_O_FLAGS more robust The construct "diff ... || diff=1 || diff=" does not set the variable in all cases. This could be triggered with: $ env make diff=1 make sc_dd_O_FLAGS dd_O_FLAGS maint.mk: ./src/dd.c has inconsistent O_ flag lists cfg.mk:59: recipe for target 'sc_dd_O_FLAGS' failed make: *** [sc_dd_O_FLAGS] Error 1 * cfg.mk (sc_dd_O_FLAGS): Remember $? of the diff command directly and check its value later rather than using the above mentioned mapping. 2015-02-09 Bernhard Voelker maint: make sc_THANKS_in_duplicates more robust The previous commit v8.23-124-g7b1ca5f made the above syntax-check rule fail, because that took the whole content of THANKS.in for comparison. * cfg.mk (sc_THANKS_in_duplicates): Strip off the header (all before the first empty line) and the footer (all past the next empty line) from 'THANKS.in' for the check. 2015-02-08 Bernhard Voelker maint: sort contributors in THANKS.in * THANKS.in: Document the preferred sort order as a comment at the top of the file. Change "Марк Коренберг" to latin1 ("Mark Korenberg"). Sort all entries. * cfg.mk (sc_THANKS_in_sorted): Add rule to ensure that 'THANKS.in' remains sorted. 2015-02-06 Pádraig Brady maint: document a caveat in the tail inotify implementation * src/tail.c (main): Document another caveat with the inotify implementation wrt multiple hardlinked files. 2015-02-06 Pádraig Brady tail: return inotify resources where possible Each user has a maximum number of inotify watches, so handle the cases where we exhaust these resources. * src/tail.c (tail_forever_inotify): Ensure we inotify_rm_watch() the watch for an inode, when replacing with a new watch for a name. Return all used inotify resources when reverting to polling. Revert to polling upon first indication of inotify resource exhaustion. Revert to polling on any inotify resource exhaustion. Diagnose resource exhaustion correctly in all cases. Avoid redundant reinsertion in the hash for unchanged watches (where only attributes of the file are changed). * tests/tail-2/retry.sh: Avoid false failure when reverting to polling. * tests/tail-2/inotify-rotate.sh: Likewise. * tests/tail-2/symlink.sh: Likewise. * tests/tail-2/inotify-rotate-resources.sh: New test to check that we're calling inotify_rm_watch() for replaced files. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix. * THANKS.in: Thanks for reporting and problem identification. 2015-02-06 Pádraig Brady tests: fix recent regression in tail inotify test * tests/tail-2/inotify-rotate.sh (cleanup_fail_): Set fail=1 so that failures are identified. Regression in v8.23-63-g111a2b9 Also use print_ver_ rather than open coding --verbose support. Also check for more than a single 'b' which seems brittle. 2015-02-05 Yury Usishchev build: ensure make-prime-list doesn't access out of bounds memory The -fsanitize=address run associated with v8.22-75-gf940fec failed to check make-prime-list, as src/primes.h is not regenerated with `make clean`. Running with -fsanitize=address indicates a read 1 byte beyond the allocated buffer. $ rm src/make-prime-list.o $ make AM_CFLAGS=-fsanitize=address src/make-prime-list $ src/make-prime-list 5000 ================================================================= ==13913==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61e00000fa43 at pc 0x4016f5 bp 0x7fff9d9840e0 sp 0x7fff9d9840d0 READ of size 1 at 0x61e00000fa43 thread T0 #0 0x4016f4 in main src/make-prime-list.c:214 #1 0x7f98892c5fdf in __libc_start_main (/lib64/libc.so.6+0x1ffdf) #2 0x401774 (src/make-prime-list+0x401774) 0x61e00000fa43 is located 0 bytes to the right of 2499-byte region [0x61e00000f080,0x61e00000fa43) allocated by thread T0 here: #0 0x7f98896ba7b7 in malloc (/lib64/libasan.so.1+0x577b7) #1 0x400f3f in xalloc src/make-prime-list.c:163 #2 0x400f3f in main src/make-prime-list.c:198 SUMMARY: AddressSanitizer: heap-buffer-overflow src/make-prime-list.c:214 main Shadow bytes around the buggy address: 0x0c3c7fff9ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3c7fff9f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3c7fff9f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3c7fff9f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c3c7fff9f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c3c7fff9f40: 00 00 00 00 00 00 00 00[03]fa fa fa fa fa fa fa 0x0c3c7fff9f50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3c7fff9f60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3c7fff9f70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3c7fff9f80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c3c7fff9f90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa ... ==13913==ABORTING * src/make-prime-list.c (main): Bounds check the incremented index, before using to access the buffer. Fixes http://bugs.gnu.org/19784 2015-02-03 Pádraig Brady build: fix invalid gnulib patch * gl/lib/tempname.c.diff: Fix recent breakage so it applies again. Invalid patch was noticed at http://hydra.nixos.org/eval/1172233 * cfg.mk: Exempt diff files from these "id_est" syntax checks. (sc_ensure_gl_diffs_apply): A new syntax check, to ensure all patches under gl/ apply cleanly. Note we use --fuzz=0 to check patches apply cleanly for safety, due to the patch(1) issue detailed in commit v8.21-117-g46f7e05 * gl/lib/regcomp.c.diff: Rediffed. * gl/lib/regex_internal.c.diff: Likewise. * gl/lib/regex_internal.h.diff: Likewise. * gl/lib/regexec.c.diff: Likewise. * gl/lib/tempname.h.diff: Likewise. 2015-02-02 Bernhard Voelker tests: avoid a FP on aarch64 when comparing env/printenv output The following test fails on aarch64 on openSUSE's OpenBuildService due to glibc's execvp reversing the pointers of 'environ', i.e., the output of "env|tac" equals "env env" on that platform. * tests/misc/printenv.sh: Use 'env env' to work around the behavior on that platform. While at it, fix the grep pattern which suppressed all environment variables starting with an underscore "_" instead of "$_" (and "$LD_PRELOAD") only. 2015-02-02 Pádraig Brady dircolors: sync with Xiph file extensions * src/dircolors.hin: Remove deprecated anx, axa, axv. Add opus. Suggested by Scott Teal. 2015-01-31 Bernhard Voelker doc,maint: fix use of "i.e." in documentation and comments To align with all other places (and correct grammar), change all upper-case "I.E." to "I.e.". Furthermore, ensure that "i.e." is followed by a comma. Finally, ensure to use a double-space before "I.e.," at the beginning of a sentence. The following was used to change all offending uses (apart from old ChangeLog files): $ git grep -liF 'i.e.' \ | xargs sed -i \ -e 's/I\.E\./I.e./g' \ -e 's/\. \(I\.e\.\)/. \1/g' \ -e 's/\([Ii]\.e\.\)\( \)/\1,\2/g' \ -e 's/\([Ii]\.e\.\)$/\1,/g' * cfg.mk (sc_prohibit_uppercase_id_est): Add new rule. (sc_ensure_double_space_after_dot_before_id_est): Likewise. (sc_ensure_comma_after_id_est): Likewise. (old_NEWS_hash): Refresh hash via "make update-NEWS-hash". * NEWS: Change use of "id est" abbreviation via the above command. * README: Likewise. * README-prereq: Likewise. * doc/coreutils.texi: Likewise. * gl/lib/rand-isaac.c: Likewise. * gl/lib/tempname.c.diff: Likewise. * man/stdbuf.x: Likewise. * src/cat.c: Likewise. * src/copy.c: Likewise. * src/copy.h: Likewise. * src/cp.c: Likewise. * src/cut.c: Likewise. * src/dd.c: Likewise. * src/df.c: Likewise. * src/fiemap.h: Likewise. * src/longlong.h: Likewise. * src/ls.c: Likewise. * src/numfmt.c: Likewise. * src/pr.c: Likewise. * src/shred.c: Likewise. * src/shuf.c: Likewise. * src/split.c: Likewise. * tests/Coreutils.pm: Likewise. * tests/df/df-symlink.sh: Likewise. * tests/df/skip-rootfs.sh: Likewise. * tests/init.sh: Likewise. * tests/ls/color-norm.sh: Likewise. * tests/misc/basename.pl: Likewise. * tests/misc/ls-misc.pl: Likewise. * tests/misc/md5sum-bsd.sh: Likewise. * tests/misc/shred-exact.sh: Likewise. * tests/misc/sort.pl: Likewise. * tests/misc/stdbuf.sh: Likewise. * tests/misc/tac-continue.sh: Likewise. * tests/rm/r-root.sh: Likewise. * tests/tail-2/symlink.sh: Likewise. 2015-01-29 Pádraig Brady doc: clarify the output format for the *sum utilities * src/md5sum.c (usage): Detail the reasons for the default double space between checksum and file name. * doc/coreutils.texi (md5sum invocation): Likewise. Explicitly mention the 3 formats that --check supports. Fixes http://bugs.gnu.org/19725 2015-01-28 Giuseppe Scrivano sync: support syncing specified arguments * m4/jm-macros.m4 (coreutils_MACROS): Check for syncfs(). * man/sync.x: Add references to syncfs, fsync and fdatasync. * doc/coreutils.texi (sync invocation): Document the new feature. * src/sync.c: Include "quote.h". (AUTHORS): Include myself. (MODE_FILE, MODE_DATA, MODE_FILE_SYSTEM, MODE_SYNC): New enum values. (long_options): Define. (sync_arg): New function. (usage): Describe that arguments are now accepted. (main): Add arguments parsing and add support for fsync(2), fdatasync(2) and syncfs(2). * tests/misc/sync.sh: New (and only) test for sync. * tests/local.mk: Reference the new test. * AUTHORS: Add myself to sync's authors. * NEWS: Mention the new feature. 2015-01-25 Jim Meyering scripts: autotools-install: update * scripts/autotools-install: Increase automake's version number to 1.15 and add Stefano Lattarini's new GPG key ID. Increase gettext's version to 0.19.4 and add Daiki Ueno's GPG key ID. Also move VERSION definition "up" so that it is once again automatically updated via the emacs hook snippet at the end of the file. 2015-01-23 Pádraig Brady stty: only print supported options for combined options * src/stty.c (usage): Don't reference unsupported options, in the combined options descriptions. * doc/coreutils.texi (stty invocation): Adjust for the new order of the 'sane' and 'raw' combined options. Also add -iutf8 to the 'sane' list. stty: document the 'status' character where supported * src/stty.c (usage): On systems that support this setting (BSD), display 'status' in the list of adjustable special characters. * doc/coreutils.texi (stty invocation): Mention the option, and that it's not currently supported on Linux. 2015-01-23 Pádraig Brady stty: document the 'discard' character setting The equivalent of this is 'flush', but that was never documented as an option (though was output with stty -a). Therefore use the more descriptive name, also generally used on BSD systems. Note even though this setting seems ineffective on Linux, supporting the setting is useful to allow terminal programs to receive the default ^O character code. * doc/coreutils.texi (stty invocation): Document the 'discard' option. * src/stty.c (struct control_info): Add 'discard'; same as 'flush'. (display_all): Show 'discard' rather than 'flush' char. (display_changed): Likewise. (usage): Document the 'discard' option. 2015-01-23 Pádraig Brady stty: add support for extproc/LINEMODE Add support for the "extproc" option which is well described at: http://lists.gnu.org/archive/html/bug-readline/2011-01/msg00004.html * src/stty.c (usage): Describe the extproc option if either the Linux EXTPROC local option is defined, or the equivalent BSD TIOCEXT ioctl is defined. (main): Make the separate ioctl call for extproc on BSD. * doc/coreutils.texi (stty invocation): Describe the option, and reference the related RFC 1116. * NEWS: Mention the new feature. 2015-01-19 Bernhard Voelker tests: use returns_ function for new split test * tests/split/record-sep.sh: Use the recently added returns_ function to simplify the shell syntax in this test. Also remove the redirection of stdout/stderr to /dev/null as this eases analyzing errors. 2015-01-19 Assaf Gordon split: new -t option to select record separator * src/split.c (eolchar): A new variable to hold the separator character (unibyte for now). This is reference throughout rather than hardcoding '\n'. (usage): Describe the new --separator option, and mention records along with lines so there is no ambiguity that all options treat lines and records equivalently. (main): Have -t update eolchar, or default to '\n'. * tests/split/record-sep.sh: New test case. * tests/local.mk: Reference the new test. * doc/coreutils.texi (split invocation): Document the new option. Adjust --lines, --line-bytes, --number=[lr]/... to mention they pertain to records if --separator is specified. * NEWS: Mention the new feature. 2015-01-19 Pádraig Brady maint: fix typo in THANKS.in in previous commit * THANKS.in: s/Stehpen/Stephen/ 2015-01-19 Pádraig Brady doc: clarify that du operands are interdependent Following on from http://bugs.gnu.org/17546 make it more obvious that du may elide specified operands to avoid double counting in the set. * src/du.c (usage): Specify that du operates on the set of operands, rather than each independently. * doc/coreutils.texi (du invocation): Likewise. Also state that the number of entries printed may change due to the order specified. Currently, deeper items specified earlier will result in them being displayed, but don't mention that implementation detail in the documentation. * THANKS.in: Add reporter. Reported by Stephen Shirley 2015-01-14 Bernhard Voelker tests: use compare-vs-/dev/null instead of 'test -s' When some program produces unexpected output, that use of compare-vs-/dev/null will ensure that the surprising output is printed in the test's output. With "test -s err" only, one would have to instrument and rerun in order to see the offending output. * cfg.mk (sc_prohibit_and_fail_1): Exempt 'compare' from this check. * tests/dd/misc.sh: Change "tests -s ... || fail=1" to "compare /dev/null ... && fail=1". * tests/misc/nice.sh: Likewise. * tests/rm/read-only.sh: Likewise. * tests/tail-2/inotify-race.sh: Likewise. * tests/touch/no-dereference.sh: Likewise. Suggested by Jim Meyering in http://lists.gnu.org/archive/html/coreutils/2015-01/msg00042.html 2015-01-14 Pádraig Brady tests: add extra protection against unexpected exits Many tests use `program ... && fail=1` to ensure expected error situations are indicated. However that would mask an unexpected exit (like a crash). Therefore explicitly check the expected exit code. Note where error messages are also verified, the extra protection is not added. * tests/init.sh (returns_): A new helper function to check the return code of a command, and used throughout the tests. * cfg.mk (sc_prohibit_and_fail_1): Add a syntax check to avoid new instances of this issue. 2015-01-11 Pádraig Brady tests: avoid skipping some df tests with libmount * tests/df/no-mtab-status.sh: Provide libmount placeholders, to avoid skipping the test when libmount is in use. * tests/df/skip-duplicates.sh: Likewise. * tests/df/skip-rootfs.sh: Comment that the test is moot when libmount (/proc/self/mountinfo) is being used. 2015-01-10 Pádraig Brady maint: clean up some test issues identified with shellcheck * tests/cp/cp-a-selinux.sh: Comment why unused variables are assigned. Fix misspellings noticed while adjusting. * tests/cp/fiemap-perf.sh: Fix quoting. * tests/misc/shuf.sh: Avoid useless use of cat. * tests/misc/printf-surprise.sh: Likewise. 2015-01-08 Daiki Ueno maint: adjustments related to previous shuf crash fix * tests/misc/shuf.sh: Improve the test so it detects crashes in more cases. * NEWS: Mention the previous fix. 2015-01-08 Assaf Gordon maint: skip long-lines syntax-check if unsupported * cfg.mk(sc_long_lines): Skip if required wc and sed options are not supported, which is currently the case on OS X. 2015-01-05 Paul Eggert shuf: do not mishandle 'shuf -i0-0 1' Problem reported by Daiki Ueno in: http://bugs.gnu.org/19520 * src/shuf.c (main): Avoid core dump if !input_range. * tests/misc/shuf.sh: Test for this bug. 2015-01-01 Bernhard Voelker maint: update further copyright year number ranges * bootstrap: Update copyright year manually (missing in previous gnulib update). * tests/init.sh: Likewise. The entries in the exemption list are processed by "grep -vEf ./.x-update-copyright", and therefore evaluated as an extended regular expression (ERE). Thus, the "bootstrap" entry also matched for bootstrap.conf which we want to be updated. * .x-update-copyright: Change all entries to EREs, i.e. including the caret ^ and dollar sign $ meta-characters matching the beginning and the end of a line. * bootstrap.conf: Update copyright year by "make update-copyright". Finally, the only one showing up with the following command should be the COPYING file: $ git grep 'Copyright .* Free Software' | grep -v '2015 Free Software' 2015-01-01 Pádraig Brady doc: fix man page formatting for split CHUNKS options * src/split.c (usage): Indent the info on CHUNKS so that help2man can match it and align appropriately in its own section. Fixes http://bugs.gnu.org/19228 2014-12-31 Pádraig Brady build: update to latest gnulib Pick up an errno adjustment in xstrtol() that fixes a spurious test failure on Darwin 14.0.0. Also update copyright year to 2015 avoiding a syntax-check failure. 2014-12-31 Pádraig Brady maint: update all copyright year number ranges Run "make update-copyright" and then... * tests/sample-test: Adjust to use the single most recent year. * tests/du/bind-mount-dir-cycle-v2.sh: Fix case in copyright message, so that year is updated automatically in future. 2014-12-29 Pádraig Brady tests: fix possible 8 minute running time of inotify-rotate.sh Commit v8.23-63-g111a2b9 removed the expensive tag on this test, as it runs quickly on systems with inotify. However without that it would take about 8 minutes for the test to complete all iterations. * tests/tail-2/inotify-rotate.sh: Tag as expensive without inotify. Also adjust the polling parameters used on systems without inotify so that the test completes within about 15 seconds. 2014-12-29 Pádraig Brady build: update to latest gnulib Included in this are gnulib changes 3ea43e02 2768ceb7 which make the device IDs from /proc/self/mountinfo available to df. This can be leveraged by a subsequent change to df to present a more accurate list of file systems. * bootstrap: Merge from gnulib. * src/ls.c (dev_ino_pop): s/obstack_blank/obstack_blank_fast/ as this API/ABI has changed, giving memory exhausted errors if negative (large positive) numbers are passed to obstack_blank(). * tests/df/skip-duplicates.sh: Adjust as the new gnulib code requires a non NULL mnt_opts even when mnt_type is not "none". 2014-12-29 Pádraig Brady maint: remove duplicate names from THANKS * .mailmap: Adjust so that there is only a single entry per name in the generated THANKS. 2014-12-26 Pádraig Brady doc: update the URL referencing SI prefixes Identified at https://www.gnu.org/software/gnun/linc/linc.html * doc/coreutils.texi (Block size): Fix the stale link. 2014-12-24 Pádraig Brady tests: fix error message check on some systems http://hydra.nixos.org/build/18129583 identified (on OS X) an incorrect test assumption in the previous commit. * gl/lib/xdectoint.c (__xnumtoint): Suppress the EINVAL error message as it's redundant in this context. * tests/misc/tail.pl: Suppress _optionally_ appended strerror messages. * tests/fmt/base.pl: Likewise. * tests/pr/pr-tests.pl: Likewise. * tests/split/l-chunk.sh: Likewise. 2014-12-19 Pádraig Brady diagnose too-large numbers better Following on from commit v8.23-82-gaddae94, consistently diagnose numbers that are too large, so as to distinguish from other errors, and make the limits obvious. * gl/modules/xdectoint: A new module implementing xdecto[iu]max(), which handles the common case of parsing a bounded integer and exiting with a diagnostic on error. * gl/lib/xdectoimax.c: The signed variant. * gl/lib/xdectoint.c: The parameterized implementation. * gl/lib/xdectoint.h: The interface. * gl/lib/xdectoumax.c: The unsigned variant. * bootstrap.conf: Reference the new module. * cfg.mk (exclude_file_name_regexp--sc_require_config_h_first): Exclude the parameterized templates. * src/csplit.c: Output EOVERFLOW or ERANGE errors if appropriate. * src/fmt.c: Likewise. * src/fold.c: Likewise. * src/head.c: Likewise. * src/ls.c: Likewise. * src/nl.c: Likewise. * src/nproc.c: Likewise. * src/shred.c: Likewise. * src/shuf.c: Likewise. * src/stdbuf.c: Likewise. * src/stty.c: Likewise. * src/tail.c: Likewise. * src/truncate.c: Likewise. * src/split.c: Likewise. * src/pr.c: Likewise. * tests/pr/pr-tests.pl: Adjust to avoid matching errno diagnostic. * tests/fmt/base.pl: Likewise. * tests/split/l-chunk.sh: Likewise. * tests/misc/shred-negative.sh: Likewise. * tests/misc/tail.pl: Likewise. Also remove the redundant existing ERR_SUBST from test err-6. * tests/ls/hex-option.sh: Check HEX/OCT options. * tests/misc/shred-size.sh: Likewise. * tests/misc/stty-row-col.sh: Likewise. 2014-12-18 KO Myung-Hun build: add $(EXEEXT) suffix to man page make targets * man/local.mk: Add $(EXEEXT) suffix to the executables, which is significant on OS/2 for example. build: don't call OS/2 routines on all systems * src/system.h: Add a missing __OS2__ ifdef guard. Also adjust spacing around () to avoid a syntax-check failure. 2014-12-18 KO Myung-Hun build: expand a response file and a wildcard on OS/2 OS/2 traditional shells(cmd) do not expand a response file(@file) or a wildcard. Expand them in each utility itself. * src/system.h (initialize_main): Define on OS/2. Expand a response file and a wildcard. 2014-12-16 Bernhard Voelker maint: fix how to request changes for THANKS.in * THANKS.in: Change the comment at the top to send change requests regarding this file to the main mailing list rather than referring to cp's --help output for the mailing list's address - which does not include that information anymore. 2014-12-16 Paul Eggert dd: fix typo in previous change Reported by Bernhard Voelker in: http://lists.gnu.org/archive/html/bug-gnulib/2014-12/msg00191.html * src/dd.c (scanargs): s/IN/OUT/. 2014-12-15 Paul Eggert dd: diagnose too-large numbers better Reported by Isabella Parakiss in: http://lists.gnu.org/archive/html/bug-gnulib/2014-12/msg00184.html * src/dd.c (parse_integer): Return strtol_error code, not bool. All callers changed. (scanargs): Improve quality of diagnostic when a number is too large. 2014-12-10 Pádraig Brady build: use the system printf routines in most cases glibc <= 2.5 would crash when passed invalid long double values, therefore internal gnulib routines were used, essentially only by od, to output such invalid values. Later glibc versions don't crash, as per https://sourceware.org/bugzilla/show_bug.cgi?id=4586 and subsequently od was adjusted to use the system printf routines through the use of the ftoastr module with commit v8.7-22-ga71c22f. Consequently our testing of this feature was moot, and use of the gnulib printf replacement for printf(1), od(1) and error(3) etc. was redundant. * configure.ac (gl_printf_safe): Unset so that we don't check that "nan" is output for these long double values. * tests/misc/od-float.sh: Adjust all existing checks to fail if od exits with failure status (like crashing for example). Add a new case for one of the problematic invalid long double values for x86_64. We only check that od exits successfully at present, which may change if https://sourceware.org/bugzilla/show_bug.cgi?id=17661 is resolved. 2014-12-03 Pádraig Brady maint: avoid -fsanitize=undefined warning in rand-isaac * gl/lib/rand-isaac.c (isaac_refill): readisaac() purposefully passes unaligned pointers to avoid memory copies. This is only done on platforms where this is defined, so avoid the associated runtime warning generated with -fsanitize=undefined, which is: lib/rand-isaac.c:125:182: runtime error: store to misaligned address 0x63100003d7fd for type 'isaac_word', which requires 8 byte alignment 0x63100003d7fd: note: pointer points here 47 ce ed a4 be be be 00 00 00 00 00 00 00 00 ... ^ 2014-12-01 Pádraig Brady maint: avoid signed overflow warning with -O3 Prompted by the implicit -O3 added by american-fuzzy-lop, seen with GCC 4.9.2 on x86_64. src/pr.c: In function 'print_files.part.5': src/pr.c:1781:6: error: assuming signed overflow does not occur when simplifying conditional to constant [-Werror=strict-overflow] if (cols_ready_to_print () == 0) This happens because cols_ready_to_print() is inlined thus reducing the comparison to the N variable in print_page(). Now this can't overflow due to the protection when parsing the specified column, but use an unsigned type to avoid the apparent signed overflow. * src/pr.c (cols_ready_to_print): Increment an unsigned type to avoid the subsequent signed overflow warning. 2014-12-01 Boris Ranto du: handle sub-bind-mount cycles gracefully This patch fixes the handling of sub-bind-mount cycles which are incorrectly detected as the file system errors. If you bind mount the directory 'a' to its subdirectory 'a/b/c' and then run 'du a/b' you will get the circular dependency warning even though nothing is wrong with the file system. This happens because the first directory that is traversed twice in this case is not a bind mount but a child of bind mount. The solution is to traverse all the directories in the cycle that fts detected and check whether they are not a (bind) mount. * src/du.c (mount_point_in_fts_cycle): New function that checks whether any of the directories in the cycle that fts detected is a mount point. * src/du.c (process_file): Update the function to use the new function that looks up all the directories in the fts cycle instead of only the last one. * tests/du/bind-mount-dir-cycle-v2.sh: New test case that exhibits the described behavior. * tests/local.mk: Reference the new root test. * NEWS: Mention the bug fix. 2014-11-29 Pádraig Brady tests: avoid hardlink to symlink tests where not supported These checks weren't correctly avoided in commit v8.23-66-g222d7ac * tests/cp/same-file.sh: Avoid all hardlink to symlink tests on platforms where that's not supported. Identified by http://hydra.nixos.org/build/17636446 2014-11-28 Pádraig Brady build: fix missing casts from recent change * src/dd.c (alloc_[io]buf): I committed a stale patch that omitted the casts needed on 32 bit. Identified by http://hydra.nixos.org/build/17610188 2014-11-27 Pádraig Brady rm: fix prompted number of arguments to remove on some platforms "zu" was output on solaris 8 for example rather than the number, since coreutils-8.22. * cfg.mk: Disallow %z, since we don't currently use the gnulib fprintf module, so any usage with it is non portable. Also our usage with error() currently works only through an ancillary dependency on the vfprintf gnulib module. * src/rm.c (main): Use %PRIuMAX rather than %zu for portability. * src/dd.c (alloc_[io]buf): Likewise for consistency. * src/od.c (main): Likewise. * src/split.c (set_suffix_length): Likewise. * NEWS: Mention the rm bug fix. Reported in http://bugs.gnu.org/19184 2014-11-26 Pádraig Brady tests: fix portability issue in dd/ascii test Solaris 8 was seen to issue this error: "printf: `&': illegal format character" * test/dd/ascii.sh: Use the coreutils printf in this test rather than the system one, to avoid portability issues. 2014-11-25 Paul Eggert build: port new rule for coreutils.h to old Bash Reported by Ted Carr in: http://bugs.gnu.org/19184 * src/local.mk (src/coreutils.h): Don't assume single_binary_progs is nonempty. 2014-11-24 Tobias Stoeckmann paste: fix possible truncated output with large files If '\n' was present at the size_t boundary of a file, then that and subsequent data would be discarded. * src/paste.c (paste_parallel): Avoid the overflow issue by changing the flag to a boolean rather than a count. * NEWS: Mention the bug fix. 2014-11-24 Pádraig Brady df: only suppress remote mounts of separate exports with --total * src/df.c (filter_mount_list): Separate remote locations are generally explicitly mounted, so list each even if they share the same remote device and thus storage. However with --total keep the suppression to give a more accurate value for the total storage available. (usage): Expand on the new implications of --total and move it in the options list according to alphabetic order. doc/coreutils.texi (df invocation): Mention that --total impacts on deduplication of remote file systems and also move location according to alphabetic order. * tests/df/skip-duplicates.sh: Add remote test cases. * NEWS: Mention the change in behavior. Reported in http://bugs.debian.org/737399 Reported in http://bugzilla.redhat.com/920806 Reported in http://bugzilla.opensuse.org/866010 Reported in http://bugzilla.opensuse.org/901905 2014-11-23 Pádraig Brady df: ensure -a shows all remote file system entries commit v8.22-125-g9d736f8 printed placeholder "-" values for device names that didn't match the preferred device name for a particular mount point. However that was seen to erroneously suppress values for aliased host names or exports, common with remote file systems. * src/df.c (me_for_dev): Rename from devname_for_dev() so that we can determine the remoteness as well as the name for the preferred mount entry. (get_dev): Don't output place holder values when both current and preferred mount entries are remote. Reported in http://bugs.debian.org/737399 2014-11-21 Pádraig Brady tests: add a case verifying mv on case insensitive file systems * NEWS: Update the recent entry to also mention the avoidance of incorrectly unlinking a multi-hardlinked "source" file when presented with source and dest that only differ in case. * src/copy.c (same_file_ok): Mention the case issue with same_name(). * tests/mv/hardlink-case.sh: Test the issue on HFS+. * tests/local.mk: Reference the new test case. * tests/mv/vfat: Remove an old related but unused test case. doc: mention how to avoid newlines impacting ls -1 * src/ls.c (usage): Mention the -b and -q options in the -1 description. * doc/coreutils.texi (ls invocation): Likewise. 2014-11-21 Pádraig Brady tests: chcon: avoid false failure with newer selinux file_t is now mapped to unlabeled_t as per: http://danwalsh.livejournal.com/68189.html Therefore use the latter to ensure we match correctly. This is needed on >= Fedora 21 for example, while it also works on earlier releases. 2014-11-20 Boris Ranto mv: fail when moving a file to a hardlink We may run into a race condition if we treat hard links to the same file as distinct files. If we do 'mv a b' and 'mv b a' in parallel, both a and b can disappear from the file system. The reason is that in this case the unlink on src is called and the system calls can end up being run in the order where unlink(a) and unlink(b) are the last two system calls. Therefore exit with an error code so that we avoid the potential data loss. * src/copy.c (same_file_ok): Don't set unlink_src that was used by mv, and return false for two hardlinks to a file in move_mode. *src/copy.c (copy_internal): No longer honor the unlink_src option, used only by mv. NEWS: Mention the change in behavior. * tests/cp/same-file.sh: Augment to cover the `cp -a hlsl1 sl1` case. * tests/mv/hard-verbose.sh: Remove no longer needed test. * tests/local.mk: Remove the reference to hard-verbose.sh. * tests/mv/hard-4.sh: Adjust so we fail in this case. * tests/mv/i-4.sh: Likewise. * tests/mv/symlink-onto-hardlink-to-self.sh: Likewise. 2014-11-17 Pádraig Brady maint: run strftime syntax check on newer systems * cfg.mk (sc_strftime_check): Adjust regex to handle newer glibc info formatting with different indentation and quoting. 2014-10-29 Pádraig Brady maint: tests: fix comments about retry_delay_ * tests/tail-2/F-vs-missing.sh: Comment with the correct total delay. * tests/tail-2/F-vs-rename.shi: Likewise. * tests/tail-2/flush-initial.sh: Likewise. * tests/tail-2/inotify-hash-abuse.sh: Likewise. * tests/tail-2/pipe-f2.sh: Likewise. * tests/misc/chroot-fail.sh: Initialize can_chroot_root in all cases. tests: make inotify-rotate more robust and efficient * tests/tail-2/inotify-rotate.sh: Use retry_delay_ to employ an exponential backoff with a total delay of up to 25.5s. The 15s delay was seen to trigger a false failure in http://hydra.nixos.org/build/16546517 Also remove the .1s sleep in each of the 50 iterations to reduce the running time of the test and thus the expensive_ tag on this test was removed. Also ensure that we use the standard exit procedure upon failure to avoid any erroneous diagnostics due to persistent files on NFS. 2014-10-29 Bernhard Voelker tests: make du/move-dir-while-traversing more robust * tests/du/move-dir-while-traversing.sh: Catch failure of retry_delay_ when waiting for the watcher to get ready. 2014-10-27 Pádraig Brady doc: mention that df -a includes duplicate file systems * src/df.c (usage): Mention that duplicate file systems are shown with this option, not just dummy file systems. * doc/coreutils.texi (df invocation): For the --all option, expand on the class of normally suppressed mount entries that it includes. Reported in http://bugs.debian.org/737399 2014-10-23 Pádraig Brady maint: improve sc_long_lines syntax-check speed sc_long_lines was the slowest syntax check before$ time make sc_long_lines long_lines real 0m2.740s after $ time make sc_long_lines long_lines real 0m0.677s * src/cfg.mk (sc_dd_max_sym_length): s/--max-line-length/-L/ for compat with BSDs. (sc_long_lines): Prefilter with wc -L to only identify lines in files that have lines longer than 80 characters. 2014-10-22 Mike Frysinger tests: d_type-check: don't hardcode the C library name * tests/d_type-check: The hardcoded name doesn't hold true for all Linux/glibc platforms, let alone Linux/non-glibc. Use ctypes.util.find_library() instead to search for the library. 2014-10-16 Bernhard Voelker tests: avoid false failure when comparing /proc files At least the MHz number in /proc/cpuinfo may change, thus leading to a false positive failure when comparing the expected against the actual output file. Use an invariant file instead: /proc/version. * tests/misc/head-c.sh: s/cpuinfo/version/ 2014-10-16 Pádraig Brady tests: fix test hang with unstable inodes in /proc * cp/proc-zero-len.sh: Search the 'err' file for the error to ignore, not stdin. 2014-10-15 Pádraig Brady chroot: call chroot() unconditionally to handle bind mounted "/" * src/chroot.c (is_root): Adjust to compare canonicalized paths rather than inodes, to handle (return false in) the case where we have a tree that is constructed by first bind mounting "/" (thus having the same inode). (main): Unconditionally call chroot() because it's safer and of minimal performance benefit to avoid in this case. This will cause inconsistency with some platforms not allowing `chroot / true` for non root users. * tests/misc/chroot-fail.sh: Adjust appropriately. * NEWS: Mention the bug fixes. Fixes http://bugs.gnu.org/18736 2014-10-14 Pádraig Brady copy: avoid an extraneous error when reporting errors * src/copy.c (copy_reg): If sparse_copy() failed, then an erroneous error about failing to extend the file would be reported. cp: read sparse files more efficiently with non regular destination * src.copy.c (copy_reg): Use fiemap to read sparse files, even if the output is not to a regular file. * NEWS: Mention the improvement. 2014-10-14 Pádraig Brady cp: avoid speculative preallocation with --sparse=always With --sparse=always use fallocate(...PUNCH_HOLE...) to avoid any permanent allocation due to speculative preallocation employed by file systems such as XFS. * m4/jm-macros.m4: Check for and fallocate(). * src/copy.c (punch_hole): A new function to try and punch a hole at the specified offset if supported. (create_hole): Call punch_hole() after requesting a hole. (extent_copy): Likewise. * NEWS: Mention the improvement. 2014-10-14 Pádraig Brady copy: detect smaller holes than the copy buffer size Previously cp would not detect runs of NULs that were smaller than the buffer size used for I/O (currently 128KiB). * src/copy.c (copy_reg): Use an independent hole_size, set to st_blksize, to increase the chances of detecting a representable hole, in a run of NULs read from the input. (create_hole): A new function refactored from sparse_copy() and extent_copy() so we have a single place to handle holes. (sparse_copy): Adjust to loop over the larger input buffer in chunks of the passed hole size. Also adjust to only call lseek once per hole, rather than at least once per input buffer. * tests/cp/sparse.sh: Add test cases for various sparse chunk sizes. * NEWS: Mention the improvement. 2014-10-09 Wieland Hoffmann doc: clarify that timeout limits are not 2038 seconds * man/timeout.x: The 2038 that the sentence is referring to is the year 2038, not 2038 seconds (the default unit used for timeouts). 2014-10-08 Pádraig Brady maint: avoid new signed overflow warning on 32 bit Prompted by http://hydra.nixos.org/build/15682577 with GCC 4.8.3 on i686 src/tac.c:557:6: error: assuming signed overflow does not occur when simplifying conditional to constant [-Werror=strict-overflow] if (bytes_copied < 0) This happens because copy_to_temp() is inlined in tac_nonseekable(), thus reducing the comparison to the bytes_copied variable in copy_to_temp. Now this can't overflow on either 32 or 64 bit due to the protection of the preceding fwrite(). We could use a guard like "if (bytes_copied <= OFF_T_MAX - bytes_read)" to avoid the warning, but rather than a runtime branch, just use an unsigned type to avoid apparent signed overflow on systems where the accumulation is not promoted to unsigned (32 bit size_t, 64 bit off_t). * src/tac.c (copy_to_temp): Increment an unsigned type to avoid the subsequent signed overflow warning. 2014-10-07 Jim Meyering tests: split/b-chunk.sh: avoid spurious fail on non-Linux * tests/split/b-chunk.sh: Skip each file that does not exist. Some systems lack /proc/version or /sys/kernel/profiling 2014-10-07 Pádraig Brady maint: avoid syntax-check failures in previous patch * tests/misc/od-j.sh: Non standard comparison order. * tests/split/b-chunk.sh: Confusing input file name. * tests/tail-2/tail-c.sh: Redundant require ulimit. 2014-10-07 Paul Eggert wc: don't miscount /sys and similar file systems Fix similar problems in head, od, split, tac, and tail. Reported by George Shuklin in: http://bugs.gnu.org/18621 * NEWS: Document this. * src/head.c (elseek): Move up. (elide_tail_bytes_pipe, elide_tail_lines_pipe): New arg CURRENT_POS. All uses changed. (elide_tail_bytes_file, elide_tail_lines_file): New arg ST and remove arg SIZE. All uses changed. * src/head.c (elide_tail_bytes_file): * src/od.c (skip): Avoid optimization for /sys files, where st_size is bogus and st_size == st_blksize. Don't report error at EOF when not optimizing. * src/head.c, src/od.c, src/tail.c: Include "stat-size.h". * src/split.c (input_file_size): New function. (bytes_split, lines_chunk_split, bytes_chunk_extract): New arg INITIAL_READ. All uses changed. Use it to double-check st_size. * src/tac.c (tac_seekable): New arg FILE_POS. All uses changed. (copy_to_temp): Return size of temp file. All uses changed. * src/tac.c (tac_seekable): * src/tail.c (tail_bytes): * src/wc.c (wc): Don't trust st_size; double-check by reading. * src/wc.c (wc): New arg CURRENT_POS. All uses changed. * tests/local.mk (all_tests): Add tests/misc/wc-proc.sh, tests/misc/od-j.sh, tests/tail-2/tail-c.sh. * tests/misc/head-c.sh: * tests/misc/tac-2-nonseekable.sh: * tests/split/b-chunk.sh: Add tests for problems with /proc and /sys files. * tests/misc/od-j.sh, tests/misc/wc-proc.sh, tests/tail-2/tail-c.sh: New files. 2014-10-04 Bernhard Voelker doc: document stat's output with the --terse option * doc/coreutils.texi (stat invocation): Add a paragraph documenting stat's output format when the --terse option is specified, both in normal and in --file-system mode. Reported by Dan Jacobson in http://bugs.gnu.org/18624 2014-10-02 Nick Alcock tests: fix false failure for test referencing libdl * init.cfg (gcc_shared_): -ldl has to be positioned after the object files that may rely upon it. This fixes tests/cp/nfs-removal-race.sh which references dlsym() from libdl. 2014-10-01 Bernhard Voelker maint: avoid double semicolon syntax check failure A syntax-check recently added to gnulib would trigger a failure (once gnulib gets updated here) for a statement introduced with commit v8.23-43-gaf2a4ed: src/dd.c:806: char const *time_fmt = _(", %g s, %s/s\n");; maint.mk: Double semicolon detected make: *** [sc_prohibit_double_semicolon] Error 1 * src/dd.c (print_xfer_stats): s/;;/;/ 2014-09-30 Federico Simoncelli dd: new status=progress level to print stats periodically * src/dd.c: Report the transfer progress every second when the new status=progress level is used. Adjust the handling and description of the status= option so that they're treated as mutually exclusive levels, rather than flags with implicit precedence. * doc/coreutils.texi (dd invocation): Document the new progress status level. Reference the new level in the description of SIGUSR1. * tests/dd/stats.sh: Add new test for status=progress. * tests/dd/misc.sh: Change so status=none only takes precedence if it's the last level specified. * NEWS: Mention the feature. 2014-09-30 Pádraig Brady dd: use more robust SIGUSR1 handling * src/dd.c (ifd_reopen): A new wrapper to ensure we don't exit upon receiving a SIGUSR1 in a blocking open() on a fifo for example. (iftruncate): Likewise for ftruncate(). (iread): Process signals also after a short read. (install_signal_handlers): Install SIGINFO/SIGUSR1 handler even if set to SIG_IGN, as this is what the parent can easily set from a shell script that can send SIGUSR1 without the possiblity of inadvertently killing the dd process. * doc/coreutils.texi (dd invocation): Improve the example to show robust usage wrt signal races and short reads. * tests/dd/stats.sh: A new test for various signal races. * tests/local.mk: Reference the new test. * NEWS: Mention the fix. 2014-09-24 Paul Eggert test: check for Fedora 20 sort key bug Problem reported by Göran Uddeborg in: http://bugs.gnu.org/18540 * tests/misc/sort.pl: New test 23. 2014-09-24 David Sterba mv: use reflink=auto mode by default On some filesystems (BTRFS), moving a file within the filesystem may cross subvolume boundaries and we can use a lightweight reflink copy, similar to what cp(1) can do, which is faster than a full file copy. This is enabled by default because it's only an optimization for the fall back copy and does not break user expectations or usability. * src/mv.c (cp_option_init): Set the reflink mode to AUTO. * NEWS: Mention the improvement. 2014-09-23 Pádraig Brady stty: only list supported options in --help and man pages * src/stty.c (usage): Exclude unsupported options from --help, which for example impacts the "dsusp" and "cdtrdsr" options on Linux. Fixes http://bugs.gnu.org/18506 2014-09-23 Michal Nazarewicz doc: fix use of "e.g." in stdbuf help message "E.g." stands for latin "exempli gratia" which is typically read as "for example". "E.g." does not stand for the word "example". As such, "for e.g." might be read as "for for example". Fix this usage by simply replacing "e.g." with "example". 2014-09-22 Bernhard Voelker build: pass program name to help2man after other options * man/local.mk (.x.1): Move the program name argument down after the last option argument when calling $(run_help2man). While the other way would be accepted for the GNU help2man program, it is not for the 'dummy-man' script (called as a fallback on systems lacking perl). The wrong order was introduced in commit v8.21-119-gb3578fc while adding the --info-page option. build: fix argument count check in dummy-man again * man/dummy-man: Fix argument count check, now only permitting exactly 1 argument, the program name. Reported by Andreas Schwab 2014-09-22 Alban Bedel build: fix an inverted test breaking dummy-man * man/dummy-man: Fix argument count check, allowing dummy-man to run (on systems without perl). Fixes http://bugs.gnu.org/18531 2014-09-19 Pádraig Brady build: fix dependency issues with man page generation * .gitignore: Remove reference to no longer generated make file. * configure.ac: Don't bother generating placeholder make file. * man/local.mk: Hardcode the man page deps list for normal builds to be compatible with all make implementations and configure options. Note in SINGLE_BINARY mode, all man pages will be generated on any change to the coreutils binary, but development will generally not be done in this mode, so this shouldn't be an issue. Fixes http://bugs.gnu.org/18055 2014-09-19 Jim Meyering maint: don't trigger gcc-5's new -Wlogical-not-parentheses warning * src/dircolors.c (main): Parenthesize !VAR as LHS to "<", to avoid triggering gcc's new -Wlogical-not-parentheses warning. 2014-09-19 Pádraig Brady cp: fix handling of -H with multiply specified source dirs Following on from commit v5.92-729-g130dd06, also avoid the erroneous directory hardlink warning with -H. * src/copy.c (copy_internal): Also handle the -H case for command line arguments. * tests/cp/duplicate-sources.sh: Augment the test case. * NEWS: Augment the news entry. 2014-09-19 Pádraig Brady cp: issue correct warning and ignore duplicate source dirs * src/copy.c (copy_internal): Handle the case where we have the same destination directory as already encountered, which can only be due to the corresponding source directory being specified multiple times. * tests/cp/duplicate-sources.sh: Add a test for the new multiply specified directory case, and the existing multiply specified file case. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix. doc: output correct --help references with --program-prefix * src/system.h (emit_ancillary_info): Take the invariant PROGRAM_NAME as a parameter, so that consistent references are made to online docs and texinfo nodes, when a --program-prefix is in place. Note the man pages don't need this fix as they're generated before the program prefix is used. * NEWS: Mention the improvements in references to online documentation. doc: ensure the correct texinfo nodes are referenced in --help * src/system.h (emit_ancillary_info): For commands that don't have a 1:1 mapping with the texinfo node names, provide a mapping to the correct node. * doc/coreutils.texi: Add some extra cross references noticed while checking this. Fixes http://bugs.debian.org/762092 2014-09-16 Bernhard Voelker tests: fix false du failure on newer XFS On XFS, when creating the ~2G test file 'big' in a for-loop by appending 20M each time, the file ends up using ~4G - visible in 'st_blocks'. The unused space would be reclaimed later. This feature is called "speculative preallocation" which aims at avoiding fragmentation. According to the XFS FAQ [1], there are two particular aspects of XFS speculative preallocation that are triggering this: 1. "Applications that repeatedly trigger preallocation and reclaim cycles [after file close] can cause fragmentation. Therefore, this pattern is detected and causes the preallocation to persist beyond the lifecycle of the file descriptor." 2. "Preallocation sizes grow as files grow larger." [1] http://xfs.org/index.php/XFS_FAQ Avoid one of the above by only doing a single close (reclaim cycle). * tests/du/2g.sh: Similar to the fix for a dd test (see commit v8.22-65-g7c03fe2), avoid speculative preallocation by creating the 'big' file in one go instead of appending to it in the loop. Remove debugging statements as the output with 'set -x' is sufficient nowadays. 2014-09-11 Paul Eggert cat: allow copying empty files to themselves Problem reported by Vincent Lefevre in: http://bugs.gnu.org/18449 * src/cat.c (main): Allow copying an empty file to itself. * tests/misc/cat-self.sh: New test. * tests/local.mk (all_tests): Add it. 2014-09-11 Pádraig Brady doc: reference online info pages directly from man pages * src/system.h (emit_ancillary_info): Add a direct reference to the corresponding online info documentation. Corresponding redirects were put in place on www.gnu.org to allow for concise links. * help2man: Adjust to add the "online help" link (and subsequent translation bugs link) to a "REPORTING BUGS" section. Also add the concise links for further information in --help to the "SEE ALSO" section, and dispense with the more verbose default for that. 2014-09-10 Pádraig Brady doc: adjust reference to info nodes in man pages old form: coreutils '$cmd invocation' new form: '(coreutils) $cmd invocation' The old form erroneously referenced the node for the 'coreutils' multi-call program. Now that problematic node name was renamed in commit v8.23-18-g72e470b, but the newer less ambiguous form also has the advantage of working with the pinfo viewer for example. Full discussion at http://bugs.gnu.org/18428 * man/local.mk: Adjust man page references to texinfo nodes. * src/system.h: Adjust --help references to texinfo nodes. 2014-09-10 Pádraig Brady maint: include libstdbuf.c in extraneous headers check * cfg.mk (sc_system_h_headers): Don't exclude libstdbuf.c. * src/libstdbuf.c: Remove headers already included in system.h. build: adjust previous transformations on libstdbuf name * src/local.mk (transform): commit v8.23-22-g6f9b018 discarded all transformations on the libstdbuf.so name. Be more conservative and only exclude the $(program_transform_name) portion for libstdbuf. build: avoid name transformations on libstdbuf * src/local.mk (transform): Skip the transformation for libstdbuf since that should not be subject to name clashes, and we need to reference the name directly in LD_PRELOAD etc. * configure.ac: Add a comment on the coupling of pkglibexec_PROGRAMS to $(transform). Issue reported at https://trac.macports.org/ticket/44922 Improved by Nick Bowler 2014-09-09 Pádraig Brady maint: fix syntax-check issues in recent commit Avoid 2 new syntax-check failures introduced in commit v8.23-19-g8defcee * cfg.mk (sc_some_programs_must_avoid_exit_failure): s/exit/return/. * src/whoami.c (main): Reinstate translation marker for diagnostic. 2014-09-08 Paul Eggert maint: avoid file-scope names of the form _[a-z]* The C standard says this isn't portable, if you include standard include files. * build-aux/gen-single-binary.sh: * src/coreutils-arch.c (single_binary_main_arch) (single_binary_main_uname): * src/coreutils-dir.c (single_binary_main_ls) (_single_binary_main_dir): * src/coreutils-vdir.c (single_binary_main_ls) (_single_binary_main_vdir): * src/coreutils.c (SINGLE_BINARY_PROGRAM): Remove leading _ from single_binary prefix. * src/numfmt.c (round_style): Rename from _round. All uses changed. (inval_style): Rename from _invalid. All uses changed. maint: prefer 'return status;' to 'exit (status);' in 'main' * build-aux/gen-single-binary.sh: Don't use ATTRIBUTE_NORETURN for main functions. * src/base64.c, src/basename.c, src/cat.c, src/chcon.c, src/chgrp.c: * src/chmod.c, src/chown.c, src/chroot.c, src/cksum.c, src/comm.c: * src/cp.c, src/csplit.c, src/cut.c, src/date.c, src/dd.c, src/df.c: * src/dircolors.c, src/dirname.c, src/du.c, src/echo.c, src/env.c: * src/expand.c, src/expr.c, src/factor.c, src/fmt.c, src/fold.c: * src/getlimits.c, src/groups.c, src/head.c, src/hostid.c: * src/hostname.c, src/id.c, src/install.c, src/join.c, src/kill.c: * src/link.c, src/ln.c, src/logname.c, src/ls.c, src/make-prime-list.c: * src/md5sum.c, src/mkdir.c, src/mkfifo.c, src/mknod.c, src/mktemp.c: * src/mv.c, src/nice.c, src/nl.c, src/nohup.c, src/nproc.c: * src/numfmt.c, src/od.c, src/paste.c, src/pathchk.c, src/pinky.c: * src/pr.c, src/printenv.c, src/printf.c, src/ptx.c, src/pwd.c: * src/readlink.c, src/realpath.c, src/rm.c, src/rmdir.c, src/runcon.c: * src/seq.c, src/shred.c, src/shuf.c, src/sleep.c, src/sort.c: * src/split.c, src/stat.c, src/stdbuf.c, src/stty.c, src/sum.c: * src/sync.c, src/tac.c, src/tail.c, src/tee.c, src/timeout.c: * src/touch.c, src/tr.c, src/true.c, src/truncate.c, src/tsort.c: * src/tty.c, src/uname.c, src/unexpand.c, src/uniq.c, src/unlink.c: * src/uptime.c, src/users.c, src/wc.c, src/who.c, src/whoami.c: In 'main' functions, Prefer 'return status;' to 'exit (status);'. * src/coreutils-arch.c (_single_binary_main_uname) (_single_binary_main_arch): * src/coreutils-dir.c, src/coreutils-vdir.c (_single_binary_main_ls) (_single_binary_main_dir, _single_binary_main_vdir): Omit ATTRIBUTE_NORETURN. Return a value. * src/coreutils.c (SINGLE_BINARY_PROGRAM): Omit ATTRIBUTE_NORETURN. (launch_program): Now static. * src/dd.c (finish_up): New function. (quit, main): Use it. * src/getlimits.c (main): Return a proper exit status. * src/test.c (test_main_return): New macro. (main): Use it. * src/logname.c, src/nohup.c, src/whoami.c: Use 'error' to simplify exit status in 'main' function. * src/yes.c (main): Use 'return' rather than 'error' to exit, so that GCC doesn't suggest ATTRIBUTE_NORETURN. doc: rename "coreutils invocation" to "Multi-call invocation" This supports longstanding shell commands like 'info coreutils "touch invocation"'. Problem reported by Vincent Lefevre via Bob Proulx in: http://bugs.gnu.org/18428 * doc/coreutils.texi (Multi-call invocation): Rename from "coreutils invocation". doc: mention which commands are optional * doc/coreutils.texi (coreutils invocation, df invocation) (stty invocation, whoami invocation, nproc invocation) (arch invocation, hostname invocation, hostid invocation) (uptime invocation, chroot invocation, nice invocation) (stdbuf invocation): Document that the command is installed optionally. 2014-09-06 Bernhard Voelker doc: fix typos * doc/coreutils.texi: Fix normal typos: s/pseudorandom/pseudo-random/; s/behaviour/behavior/; s/linux-based/Linux-based/; s/nonnegative/non-negative/. Fix IEC's long name: s/Electronical/Electrotechnical/. Wrap 'getopt' into the @code{} macro. Fix a grammatical error (from myself): s/splitted/split/. 2014-09-03 Paul Eggert doc: spell "indeterminate" correctly * doc/coreutils.texi (timeout invocation): Fix misspelling. Reported by Yureruchihirosan via OKANO Takayoshi in: http://bugs.gnu.org/18394 2014-08-22 Pádraig Brady maint: ensure fiemap extents flags are compared correctly * src/extent-scan.c (extent_scan_read): Following on from the flags size adjustment in commit v8.23-13-g1505b37, verify that the internal representation of the flags is never truncated which could happen in the unlikely case on 32 bit if the kernel flags ever expanded to 64 bits which is theoretically possible given the reserved space. 2014-08-22 Paul Eggert maint: avoid int64_t and similar types unless they're needed C11 doesn't require them, even POSIX doesn't strictly require the 64-bit versions, and it makes the code a bit clearer if they're used only when needed. * src/copy.c (write_zeros, extent_copy): * src/extent-scan.h (struct extent_info.ext_length): Use off_t, not uint64_t, for a value derived from a file offset. * src/extent-scan.h (struct extent_info.ext_flags) Prefer plain unsigned int to uint32_t where either will do. (struct extent_scan.ei_count): Use size_t, not uint32_t, for a value bounded by SIZE_MAX. * src/factor.c (MAGIC64, MAGIC63, MAGIC65): Remove unnecessary casts to uint64_t. 2014-08-21 Yurij Goncharuk maint: refactor ls QUOTING_STYLE env var handling * src/ls.c (main): As per the FIXME comment, move the QUOTING_STYLE handling to a separate function. 2014-08-21 Fridolin Pokorny doc: clarify that duplicate NFS mounts are skipped by df * doc/coreutils.texi (df invocation): Add a sentence that eliding duplicate entries for the same file system is not limited to bind mounts, but also happens for remote file systems like NFS. 2014-08-19 Pádraig Brady df: improve mount point selection with inaccurate mount list v8.23 has a test failure on Fedora rawhide build servers in tests/df/skip-duplicate.sh. This was due to no '/' entry being output by df. That was due to an inaccurate /proc/mounts on the build environment as stat(/mnt/point) identified all these /proc/mounts entries as having the same device id: / rootfs / /dev/md1 /dev devtmpfs /run tmpfs /boot /dev/md0 /proc/filesystems /dev/md1 Since the device name on the right changes for a given id, that causes the entries to be continually replaced, thus resulting in no '/' entry. I'm guessing this is due to the mock environment bind mounting unneeded or sensitive items to a dummy file on the host / (/dev/md1) though have not looked into those details. So rather than relying on an accurate /proc/mounts, the attached patch takes a more conservative replacement approach and only swaps a new device entry when the mount point matches. That should handle all practical cases while also avoiding this situation. * src/df.c (filter_mount_list): Only replace entries with different device names when the mount point also matches. 2014-08-11 Rasmus Borup Hansen cp: remove redundant possibly expensive heap deallocation If the hash structures grow sufficiently large so that the system is actively swapping, then the deallocation can take a significant amount of time. Details at: http://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html * src/cp.c (main): Only call hash deallocation routines when in lint checking mode. * THANKS.in: Remove as now in the git author list. 2014-08-04 Reuben Thomas doc: indicate that FILE arguments are optional with rm -f * src/rm.c (usage): s/FILE/[FILE]/. Fixes http://bugs.gnu.org/18187 2014-08-03 Anders Jonsson numfmt: fix misspelling in --debug message * src/numfmt.c (parse_format_string): s/overridding/overriding/. Fixes http://bugs.gnu.org/18050 2014-08-01 Paul Eggert tests: fix typo in tail-2/inotify-race Reported by Andreas Schwab in: http://bugs.gnu.org/18057 * tests/tail-2/inotify-race.sh (break_src): Use abs_top_srcdir, not abs_top_builddir. 2014-08-01 Bernhard Voelker chroot: perform chdir("/") again unless new --skip-chdir is specified Since commit v8.22-94-g99960ee, chroot(1) skips the chroot(2) syscall for "/" arguments (and synonyms). The problem is that it also skips the following chdir("/") call in that case. The latter breaks existing scripts which expect "/" to be the working directory inside the chroot. While the first part of the change - i.e., skipping chroot("/") - is okay for consistency with systems where it might succeed for a non-root user, the second part might be malicious, e.g. cd /home/user && chroot '/' bin/foo In the "best" case, chroot(1) could not execute 'bin/foo' with ENOENT, but in the worst case, chroot(1) would execute '/home/user/bin/foo' in the case that exists - instead of '/bin/foo'. Revert that second part of the patch, i.e., perform the chdir("/) in the common case again - unless the new --skip-chdir option is specified. Restrict this new option to the case of "/" arguments. * src/chroot.c (SKIP_CHDIR): Add enum. (long_opts): Add entry for the new --skip-chdir option. (usage): Add --skip-chdir option, and while at it, move the other to options into alphabetical order. (main): Accept the above new option, allowing it only in the case when NEWROOT is the old "/". Move down the chdir() call after the if-clause to ensure it is run in any case - unless --skip-chdir is specified. Add a 'newroot' variable for the new root directory as it is used in a couple of places now. * tests/misc/chroot-fail.sh: Invert the last tests which check the working directory of the execvp()ed program when a "/"-like argument was passed: now expect it to be "/" - unless --skip-chdir is given. * doc/coreutils.texi (chroot invocation): Document the new option. Document that chroot(1) usually calls chdir("/") unless the new --skip-chdir option is specified. Sort options. * NEWS (Changes in behavior): Mention the fix. (New features): Mention the new option. * init.cfg (nonroot_has_perm_): Add chroot's new --skip-chdir option. * tests/cp/preserve-gid.sh (t1): Likewise. * tests/cp/special-bits.sh: Likewise. * tests/id/setgid.sh: Likewise. * tests/misc/truncate-owned-by-other.sh: Likewise. * tests/mv/sticky-to-xpart.sh: Likewise. * tests/rm/fail-2eperm.sh: Likewise. * tests/rm/no-give-up.sh: Likewise. * tests/touch/now-owned-by-other.sh: Likewise. Reported by Andreas Schwab in http://bugs.gnu.org/18062 2014-07-31 Pádraig Brady build: avoid building stdbuf on cygwin * configure.ac: Don't add stdbuf to the list of programs to build if EXEEXT is set, as that is not handled in configure.ac for libstdbuf.so yet (see bin_PRGRAMS handling in configure.ac). Also the LD_PRELOAD mechanism will need to be adjusted to support cygwin in any case, so avoid stdbuf completely in this case for now. Problem reported by Eric Blake. 2014-07-29 Eric Blake doc: clarify that floating point parses "inf" * doc/coreutils.texi (Floating point): Document handling of "inf", "infinity", "NaN", and so on. 2014-07-19 Paul Eggert maint: fix message translation glitches Problem reported by Sebastian Rasmussen in: http://bugs.gnu.org/18054 * gl/lib/randread.c (randread_error): Don't put multiple string literals inside _(...), as xgettext doesn't support that. * src/chroot.c (main): In diagnostics, don't bother to distinguish between setting the number of supplemental group IDs to a zero or to a nonzero value, as the underlying system call is the same either way. This also makes the string easier to translate correctly. 2014-07-18 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.23 * NEWS: Record release date. build: port new bootstrap script to POSIX shells * build-aux/gen-single-binary.sh: Avoid bash only constructs. Reported by Assaf Gordon tests: fix false failure with spaces in the test directory * tests/misc/env.sh: Skip if we can't execute the generated shebang, which would be the case if there are spaces in the directory hierarchy. This is triggered by `make distcheck` 2014-07-18 Pádraig Brady build: don't distribute generated coreutils.h This issue was identified by the manifest comparisons done by `make distcheck` * src/local.mk (noinst_HEADERS): Remove coreutils.h from this always distributed list. (nodist_src_coreutils_SOURCES): Add coreutils.h as its contents are determined at configure time, so pointless to distribute. (src_coreutils_SOURCES): Define explicitly so that the corresponding nodist_ variable is honored. (DISTCLEANFILES): Add coreutils.h to this rather than CLEANFILES, as its contents are determined at configure time. 2014-07-18 Pádraig Brady tests: fix false failure in cp --preserve=context test With libselinux-2.2.1-6.fc20.x86_64, kernel-3.12.6-300.fc20.x86_64 `cp --preserve=context src dst` was seen to succeed when src and dst where on the same fixed context file system, as lsetfilecon() returned success in this case when the context wasn't being changed. * tests/cp/cp-a-selinux.sh: Copy from a different file system to most likely have a different context that will test context setting logic correctly. 2014-07-17 Pádraig Brady maint: avoid a syntax-check warning in previous commit * cfg.mk: Avoid warning for @AMDEP_TRUE@ usage in man/local.mk 2014-07-17 Paul Eggert build: port to non-GNU make This fixes a problem with native Solaris 'make', which does not grok '-include' lines (a GNU extension to POSIX 'make'). * configure.ac (man/dynamic-deps.mk): Create it, with an old time stamp, if doing dynamic dependency tracking. * man/local.mk (DISTCLEANFILES): Put man/dynamic-deps.mk here, rather than in CLEANFILES. (man/dynamic-deps.mk): Don't create it read-only, so that we can easily touch it later. Include it with '@AMDEP_TRUE@@am__include@', not with '-include', as '-include' does not work with native Solaris 'make'. 2014-07-16 Assaf Gordon numfmt: fix isblank() usage for some unibyte locales * src/numfmt.c (simple_strtod_int): Replace isdigit() with c_isdigit() to avoid locale concerns and -Wchar-subscripts warnings on cygwin. Remove the now redundant locale guard. (simple_strtod_human): Cast characters to unsigned so that the promoted int value passed to isblank() is positive, allowing it to work correctly for all characters in unibyte locales. Previously character 0xA0, i.e. non-breaking space, would be misclassified for example. (process_suffixed_number): Likewise. (skip_fields): Likewise. Both issues were triggered by the -Wchar-subscripts warning on GCC 4.8.3 on cygwin, due to the is*() implementations used there, but the issue is present on all platforms defaulting to signed chars. * NEWS: Mention the bug fix. Reported by Eric Blake 2014-07-13 Pádraig Brady tests: fix portabilitiy issue in new od-endian test * tests/misc/od-endian.sh: '\n' is not generally supported in the replacement, so use the more portable "\\$NL". tests: fix portability issue in new dd/ascii test * tests/dd/ascii.sh: Quote so that '\\' is passed to printf rather than a single '\', as that's not portable. bash, dash, zsh and external solaris printf were seen to support printf '\%03o' 1 2 while ksh, freebsh shell and external GNU printf need the more portable printf '\\%03o' 1 2 Note we don't use env here to call the coreutils printf implementation, as there are many printf calls, so relying on portable shell implementations will be faster. tests: avoid hang when files created with wrong permissions * tests/split/b-chunk.sh: Never show the rm prompt which would hang the test suite. This was seen when split erroneously created files with no persmissions, which was triggered by this compiler bug in clang 3.4: http://llvm.org/bugs/show_bug.cgi?id=18346 maint: avoid clang -Wint-to-pointer-cast warning * src/chroot.c: Explicitly cast int to pointer type. build: update gnulib submodule to latest * gnulib: Sync recent cleanups and the fix for missing df entries in the presence of bind mounts: http://bugs.gnu.org/17833 * NEWS: Detail the df bug fix. 2014-07-13 Pádraig Brady sort: avoid undefined operation with destroying locked mutex This didn't seem to cause any invalid operation on GNU/Linux at least, but depending on the implementation, mutex deadlocks could occur. For example this might be the cause of lockups seen on Solaris: http://lists.gnu.org/archive/html/coreutils/2013-03/msg00048.html This was identified with valgrind 3.9.0 with this setup: seq 200000 > file.sort valgrind --tool=drd src/sort file.sort -o file.sort With that, valgrind would _intermittently_ report the following: Destroying locked mutex: mutex 0x5419548, recursion count 1, owner 2. at 0x4C2E3F0: pthread_mutex_destroy(in vgpreload_drd-amd64-linux.so) by 0x409FA2: sortlines (sort.c:3649) by 0x409E26: sortlines (sort.c:3621) by 0x40AA9E: sort (sort.c:3955) by 0x40C5D9: main (sort.c:4739) mutex 0x5419548 was first observed at: at 0x4C2DE82: pthread_mutex_init(in vgpreload_drd-amd64-linux.so) by 0x409266: init_node (sort.c:3276) by 0x4092F4: init_node (sort.c:3286) by 0x4090DD: merge_tree_init (sort.c:3234) by 0x40AA5A: sort (sort.c:3951) by 0x40C5D9: main (sort.c:4739) Thread 2: The object at address 0x5419548 is not a mutex. at 0x4C2F4A4: pthread_mutex_unlock(in vgpreload_drd-amd64-linux.so) by 0x4093CA: unlock_node (sort.c:3323) by 0x409C85: merge_loop (sort.c:3531) by 0x409F8F: sortlines (sort.c:3644) by 0x409CE3: sortlines_thread (sort.c:3574) by 0x4E44F32: start_thread (in /usr/lib64/libpthread-2.18.so) by 0x514EEAC: clone (in /usr/lib64/libc-2.18.so) * src/sort.c (sortlines): Move pthread_mutex_destroy() out to merge_tree_destroy(), so that we don't overlap mutex destruction with threads still operating on the nodes. (sort): Call the destructors only with "lint" defined, as the memory used will be deallocated implicitly at process end. * NEWS: Mention the bug fix. 2014-07-13 Shayan Pooya sort: fix two threading issues reported by valgrind Neither issue impacts on the correct operation of sort. The issues were detected by both valgrind 3.8.1 and 3.9.0 using: seq 200000 > file.sort valgrind --tool=drd src/sort file.sort -o file.sort For tool usage and error details see: http://valgrind.org/docs/manual/drd-manual.html * src/sort.c (queue_insert): Unlock mutex _after_ signalling the associated condition variable. Valgrind flags this with: "Probably a race condition: condition variable 0xffeffffb0 has been signaled but the associated mutex 0xffeffff88 is not locked by the signalling thread." The explanation at the above URL is: "Sending a signal to a condition variable while no lock is held on the mutex associated with the condition variable. This is a common programming error which can cause subtle race conditions and unpredictable behavior." This should at least give more defined scheduling behavior. (merge_tree_destroy): Make symmetrical with merge_tree_init() thus destroying the correct mutex. Valgrind flags this with: "The object at address 0x5476cf8 is not a mutex." 2014-07-13 Pádraig Brady maint: avoid an inconsequential mem leak * src/df.c (get_disk): Avoid an inconsequential mem leak spotted by coverity. Also s/duplicities/duplicates/. 2014-07-13 Jim Meyering build: adjust new rule not to depend on bash-4.x * man/local.mk (man/dynamic-deps.mk): Use the same code to derive FOO from man/FOO.1 as in the .x.1 rule below. Using the more concise "name=$${man:4: -2}" is not portable enough. 2014-07-13 Alex Deymo build: support building all tools in a single binary Add the --enable-single-binary option to the configure file. When enabled, this option builds a single binary file containing the selected tools. Which tool gets executed depends on the value of argv[0] which can be set implicitly through symlinks to the single program. This setup reduces significantly the size of a complete coreutils install, since code from lib/libcoreutils.a is not duplicated in every one of the more than 100 binaries. Runtime overhead is increased due to more dynamic libraries being loaded, and extra initialization being performed for all utils. Also initially a larger binary is loaded from storage, though this is usually alleviated due to caching and lazy mmaping of unused blocks, and in fact the single binary should have better caching characteristics. Comparing the size of the individual versus single binary on x86_64: $ cd src $ size coreutils $ size -t $(../build-aux/gen-lists-of-programs.sh --list-progs | grep -Ev '(coreutils|libstdbuf)') | tail -n1 text data bss dec hex filename 1097416 5388 88432 1191236 122d44 src/coreutils 4901010 124964 163768 5189742 4f306e (TOTALS) Storage requirements are reduced similarly: $ cd src $ du -h coreutils $ du -ch $(../build-aux/gen-lists-of-programs.sh --list-progs | grep -Ev '(coreutils|libstdbuf)') | tail -n1 1.2M coreutils 5.3M total When installing, the makefile will create either symlinks or shebangs based on the --enable-single-binary setting, for each configured tool. In this way, all the tools are still callable individually, but they are all implemented by the same "coreutils" binary installed on the same directory. * .gitignore: Add new generated files. * Makefile.am: New rules to generate build-aux/gen-single-binary.sh and install symlinks. * NEWS: Mention the new feature. * README: Add "coreutils" to the list of utils. * bootstrap.conf: Regenerate src/single-binary.mk * build-aux/gen-lists-of-programs.sh: New --list-progs option. * build-aux/gen-single-binary.sh: Regenerate * configure.ac: New --enable-single-binary option and other variables. Disallow --enable-single-binary=symlinks with --program-prefix et. al. * man/coreutils.x: Manpage hook. * man/local.mk: Add manpage hook and fix dependencies. * src/coreutils.c: Multicall implementation. * src/local.mk: New rules for the single binary option. * tests/local.mk: Add $single_binary_progs to support require_built_() from init.cfg * tests/misc/env.sh: Avoid the use of symlink to echo. * tests/misc/help-version.sh: Add exception for coreutils. * tests/install/basic-1.sh: Really avoid using ginstall strip functionality if there is an issue with the independent strip command. * src/kill.c: Changes to call exit() in main. * src/readlink.c: Likewise. * src/shuf.c: Likewise. * src/timeout.c: Likewise. * src/truncate.c: Likewise. 2014-07-03 Bernhard Voelker maint: simplify parsing df's output in shell scripts Avoid complicated and error-prone parsing of df's output via sed(1), cut(1), etc., and instead use df's more modern --output option. * src/ioblksize.h (in a comment): Simplify the extraction of the device name of the mounted file system from df's output. * tests/dd/skip-seek-past-dev.sh: Likewise. * tests/du/2g.sh: Likewise for the 'avail' column here. Also avoid the deprecated use of "tail -NUM". * tests/misc/stat-mount.sh: While at it, remove the determination of the mount point of "." via df(1) plus sed(1) as it is unused since commit v8.5-159-gf57cb37 anyway. Instead, improve this test by verifying that the output of "stat -c%m ." at least starts with a slash '/'. 2014-07-02 Pádraig Brady pwd: revert default mode back to -P Revert commit v8.22-131-g3e89d5b as even though POSIX states that the default mode should be -L, common practice for stand-alone pwd implementations is to default to -P. * src/pwd.c (usage): Retain mention of the default mode of operation. Suggested by Bob Proulx 2014-07-02 Pádraig Brady tests: avoid false failure with spaces in mount point paths * tests/ls/readdir-mountpoint-inode.sh: Quote appropriately to process mount points with spaces in the path. Previously items like these would usually be skipped, though if the path also contained a '-' for example, that would stat stdin, thus producing a wrong inode and a false failure. Fixes http://bugs.gnu.org/17863 tests: avoid errors on systems without getmntent * tests/df/no-mtab-status.sh: Skip if getmntent() not available. * tests/df/skip-duplicates.sh: Likewise. Fixes http://bugs.gnu.org/17863 2014-07-02 Jim Meyering tests: use "env builtin" in place of "$abs_top_builddir/src/builtin" This works for any program that might also be a shell built-in. It is more concise. In addition, it makes output more reproducible: some diagnostics include argv[0], which will now be just the program name, rather than the full absolute name of the executable. * tests/misc/sort-compress.sh: Use env kill, rather than absolute name. * tests/install/trap.sh: Likewise. * tests/misc/timeout.sh: Likewise. * tests/touch/no-dereference.sh: Do the same for two uses of "test". * tests/touch/no-create-missing.sh: Likewise. 2014-07-02 Alex Deymo maint: fix context_t init warning on systems without selinux On systems without libselinux, context_t is defined as an int type, but defined as a pointer type by SELinux. * src/chcon.c: Init with 0 rather than NULL to avoid warning. 2014-07-01 Jim Meyering tests: adapt tests to change in pwd's default * tests/misc/pwd-option.sh (base): Initialize with -P, now that -L is the default, to accommodate an initial working directory with a symlink component. * tests/misc/readlink-fp-loop.sh: Use $(env pwd -P) to get the absolute working directory. Using "env" ensures we do not invoke any shell built-in, and PATH ensures we invoke the one from coreutils. * tests/readlink/can-e.sh: Likewise. * tests/readlink/can-f.sh: Likewise. * tests/readlink/can-m.sh: Likewise. 2014-07-01 Pádraig Brady maint: fix const correctness warnings with security_context_t The security_context_t type was always an artificial separation from a standard char* string, and various libselinux using code assumed both were synonymous. In addition, prior to libselinux 2.3 function declarations were incorrect wrt constness of this type. Here we replace security_context_t with char*, and also provide a wrapper function to cater for the const issue on older libselinux. * src/system.h (se_const): A new function to avoid and identify the const issue on older libselinux. * src/copy.c: s/security_context_t/char */. * src/cp.c: Likewise. * src/id.c: Likewise. * src/install.c: Likewise. * src/ls.c: Likewise. * src/mkdir.c: Likewise. * src/mkfifo.c: Likewise. * src/mknod.c: Likewise. * src/runcon.c: Likewise. * src/selinux.c: Likewise. * tests/cp/no-ctx.sh: Likewise. * src/chcon.c: Likesize. 2014-07-01 Namhyung Kim chcon: avoid redundant context allocations Since context is verified by security_check_context() it can be used in change_file_context() without converting to context_t every time. * src/chcon.c (change_file_context): Use specified_context directly. 2014-07-01 Namhyung Kim chcon: use security_check_context() for context validation context_new() and _free() are used for checking validity of a specified context. libselinux provides security_check_context for this purpose so use it. Note that context_new() can fail for a valid context - e.g. ENOMEM. * src/chcon.c (main): Use security_check_context(). 2014-06-30 Pádraig Brady pwd: assume -L mode by default, as per POSIX * src/pwd.c (main): Adjust default mode to be "logical" and independent of the POSIXLY_CORRECT env var. (usage): Mention the default mode of operation. * doc/coreutils.texi (pwd invocation): Adjust accordingly. * tests/misc/pwd-option.sh: Likewise. * NEWS: Mention the change in behavior. 2014-06-26 Pádraig Brady tests: use predetermined NON_ROOT_GID * HACKING: GID is more useful in tests than group name, so rename input param from NON_ROOT_GROUP to NON_ROOT_GID to make it obvious that only a group ID is now acceptable, thus allowing GID lookups to be avoided throughout the tests. * init.cfg (require_root_): Likewise. * tests/misc/truncate-owned-by-other.sh: Avoid looking up the GID. * tests/touch/now-owned-by-other.sh: Likewise. * tests/misc/chroot-credentials.sh: Likewise. Also fix an instance of comparison against NON_ROOT_GROUP which would have given a false failure if a non numeric value was passed in. * tests/id/setgid.sh: Use previously looked up gid as a more accurate base for the subsequent adjustment, and move the uid lookup within chroot, rather than having the overhead of a separate `id` invocation. maint: mention when the du cycle warning bug was introduced * NEWS: Mention bug introduced in v8.1 (with commit v8.0-88-g8ba5d1a). 2014-06-26 Petr Stodůlka id: output the effective group for the process * src/id.c (print_full_info): When no user is specified, output the effective group for the _process_, rather than the default group from the system database, which may be different. * tests/id/setgid.sh: Add a case for `id` as well as `id -G`. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/7320 Reported at http://bugzilla.redhat.com/1016163 2014-06-25 Pádraig Brady df: report correct device in presence of eclipsed mounts * src/df.c (last_device_for_mount): A new function to identify the last device mounted for a mount point. (get_disk): Use the above to discard mount entries for a device, where a later mount entry uses a different device name than that of the user specified device. * tests/df/over-mount-device.sh: A new root test. * tests/local.mk: Reference the new test. * NEWS: Reword for all these related recent fixes. Discussed at: http://bugs.gnu.org/16539#69 2014-06-24 Pádraig Brady df: look for accessible mount points for specified devices * src/df.c (get_disk): Include whether we can access the mount dir, in the mount entry selection criteria. This handles the case where a device is (bind) mounted multiple times with the shortest mount path not being accessible, while some of the other mount points are. Discussed at: http://bugs.gnu.org/16539#63 2014-06-24 Pádraig Brady df: output placeholder values for inaccessible mount points A system provided mount entry may be unavailable due to TOCTOU race, or if another device has been over-mounted at that position, or due to access permissions. In all these cases output "-" placeholder values rather than either producing an error, or in the over-mount case outputting values for the wrong device. * src/df.c (device_list): A new global list now updated by filter_mount_list(). (filter_mount_list): Adjust to take a parameter as to whether update the global mount list, or only the mount <-> device ID mapping. (get_dev): Use the device ID mapping to ensure we're not outputting stats for the wrong device. Also output placeholder values when we can't access a system specified mount point. (get_all_entries): Set the DEVICE_ONLY param for filter_mount_list(). (devname_for_dev): A new function to search the mount <-> dev mapping. * test/df/skip-duplicates.sh: Adjust accordingly. * NEWS: Mention the bug fixes. Discussed at: http://bugs.gnu.org/16539 2014-06-24 Pádraig Brady du: ignore directory cycles due to bind mounts * src/du.c (process_file): Treat cycles due to bind mounts like cycles due to following symlinks. * tests/du/bind-mount-dir-cycle.sh: Adjust accordingly. * NEWS: Mention the change in behavior. Reported at http://bugzilla.redhat.com/836557 doc: state in ls --help, that -h affects --size format * src/ls.c (usage): State that --human changes --size format. Fixes http://bugs.gnu.org/17838 2014-06-19 Pádraig Brady doc: clarify chgrp restrictions * doc/coreutils.texi (chown invocation): Mention the system dependent restrictions on setting groups. (chgrp invocation): Likewise. Reference the 'chown' superset. * man/chgrp.x: Cross reference chown(1) which is the superset interface, and also chown(2) which gives details of the platform restrictions. Fixes http://bugs.gnu.org/17495 2014-06-18 Pádraig Brady df: use all of the last device details provided * src/df.c (filter_mount_list): Recent commit v8.22-108-g25a2c94 failed to copy file system type along with the updated device name. Therefore simply replace the existing mount entry with the current one with all the latest device details. Note the name, even if not shorter in this entry, will be replaced with a shorter name in a subsequent mount entry. * tests/df/skip-duplicates.sh: Add a test case. seq: fix incorrect output with start or end of -0 * src/seq.c (main): Avoid seq_fast() with a start or end of -0. * tests/misc/seq.pl: Add test cases. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/17800 2014-06-13 Pádraig Brady tail: fix --follow to use polling mode for VXFS Veritas File System can run in single instance or clustered mode, so mark as remote to avoid using inotify for the latter case. * src/stat.c (human_fstype): Tag VXFS as remote, to use polling for the clustered variant (VXCFS). * NEWS: Mention the bug fix. Reported by Ondřej Vašík in http://bugzilla.redhat.com/1104244 Fixes http://bugs.gnu.org/17770 2014-06-11 Pádraig Brady maint: suppress a theoretical buffer overflow warning * src/numfmt.c (double_to_human): The printf format is built up in a stack buffer which is big enough to hold any of the possible formats. However the size parameter passed to snprintf was too big by 1 when GROUP was true. So decrease the buffer available to snprintf to avoid this theoretical in practise but valid coverity warning. tests: fix version printing for ginstall in one test * tests/install/d-slashdot.sh: 'ginstall' is the binary name in the build directory, not 'install'. 2014-06-11 Bernhard Voelker install: allow options -D and -t to be used together * src/install.c (install_file_in_file_parents): Factor out the creation of any parent directories into ... (mkancesdirs_safe_wd): ... this new function. (install_file_in_dir): Add the parameter 'mkdir_and_install', and call the above new function if it evaluates to true. (main): During parsing of the -t option, move the check whether the target_directory exists down after the option parsing loop, and do not complain about stat(optarg,...) failing if -D was given. Pass 'mkdir_and_install' to install_file_in_dir(). * doc/coreutils.texi (install invocation): Remove the (false) restriction that -D would be ignored together with -t. Instead, clarify install's new bahavior. Fix the node's reference in the top-level @direntry for consistency. * src/install/basic-1.sh: Add tests for the now-allowed combination of the -D and -t options. * NEWS: Mention the improvement. 2014-06-11 Pádraig Brady build: update gnulib submodule and tests/init.sh to latest * gnulib: Update to latest. * tests/init.sh: Update from gnulib. 2014-06-10 Pádraig Brady build: update GCC warnings * configure.ac: Remove the -Wsuggest-attribute=pure enablement on GCC >= 4.7, as that was moot since gnulib was already enabling that warning in its default set. The false positive was seen with 4.6.2, but confirmed not present in 4.6.3, so that's sufficiently old to just leave this enabled unconditionally. Remove the -Wsuggest-attribute={const,noreturn} enablement, as gnulib already has those in the default set. Enable the -Wlogical-op warning for GCC >= 4.8.0 as that is confirmed OK with coreutils at least, due to fixing: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=43772 Remove the -Wno-logical-op override since the main -Wlogical-op flag is now sufficient to control this warning as of GCC 4.6.3 at least. 2014-06-03 Pádraig Brady tests: consolidate tests for true and false * src/true.c (main): Add a comment about the possibility of true returning EXIT_FAILURE due to write failure. * tests/misc/false-status.sh: Fix so we're testing the tool and not the shell builtin. Add a case for true(1). * tests/misc/help-version.sh: Skip /dev/full test for true as well as false since the exit status is tested separately. Also remove the iterations for different LC_MESSAGES, as this was only applied for false(1). Translations are not honored in the test dir and so would need separate handling in any case. 2014-06-02 Ben Walton maint: avoid compiler warnings with some assert() implementations * src/df.c (get_dev): asssert() on Solaris 10 is not marked as __noreturn__ and thus the compiler may think V is uninitialized later on in the function. * THANKS.in: Remove the now committer. build: be more specific about .git repo before enabling warnings * configure.ac: When looking for a .git checkout, exclude repos that contain a .tarball-version file as these are probably releases that are imported into git for patch management. 2014-06-01 Pádraig Brady cut: restore special case handling of -f with -d$'\n' commits v8.20-98-g51ce0bf and v8.20-99-gd302aed changed cut(1) to process each line independently and thus promptly output each line without buffering. As part of those changes we removed the special handling of --delimiter=$'\n' --fields=... which could be used to select arbitrary (ranges of) lines, so as to simplify and optimize the implementation while also matching the behavior of different cut(1) implementations. However that GNU behavior was in place for a long time, and could be useful in certain cases like making a separated list like `seq 10 | cut -f1- -d$'\n' --output-delimiter=,` although other tools like head(1) and paste(1) are more suited to this operation. This patch reinstates that functionality but restricts the "line behind" buffering behavior to only the -d$'\n' case. We also fix the following related edge case to be more consistent: before> printf "\n" | cut -s -d$'\n' -f1- | wc -l 2 before> printf "\n" | cut -d$'\n' -f1- | wc -l 1 after > printf "\n" | cut -s -d$'\n' -f1- | wc -l 1 after > printf "\n" | cut -d$'\n' -f1- | wc -l 1 * src/cut.c (cut_fields): Adjust as discussed above. * tests/misc/cut.pl: Likewise. * NEWS: Mention the change in behavior both for v8.21 and this effective revert. * cfg.mk (old_NEWS_hash): Adjust for originally omitted v8.21 entry. * src/paste.c: s/delimeter/delimiter/ comment typo fix. 2014-06-01 Pádraig Brady stat: avoid redundant stat() calls * src/stat.c (find_bind_mount): NAME is invariant in the loop, so only stat(NAME) outside the loop. 2014-05-29 Pádraig Brady df: use the last device name provided by the system The device name reported for a particular mount entry may no longer be valid if the mount point was subsequently mounted on a different device. Therefore honor the order of the mount list returned by the system and use the last reported device name. * src/df.c (filter_mount_list): When discarding the current mount entry, ensure that a new device name is not also discarded. * tests/df/skip-duplicates.sh: Add a test case. Also fix a false failure in the edge case of a system with only a single file system. * NEWS: Mention the fix. 2014-05-26 Pádraig Brady stdbuf: support OS X * src/stdbuf.c (set_LD_PRELOAD): Adjust to use Mac OS X specific environment variables on __APPLE__ platforms. Fixes http://bugs.gnu.org/17590 2014-05-26 Jim Meyering build: uname: avoid shadowing warning * src/uname.c (main) [__APPLE__]: Rename inner "s" to "cs", to avoid gcc shadowing warning. 2014-05-26 Pádraig Brady doc: clarify --zero-terminated option * src/join.c (usage): Reword to avoid implication that the NUL byte is only generated as the output delimeter. * src/sort.c (usage): Likewise. * src/shuf.c (usage): Likewise. Also since we're changing the translation string take the opportunity to separate out the description to a separate string to reduce translation overhead. * src/uniq.c (usage): Likewise. * src/stty.c (usage): s/null/NUL/ for consistency. * src/basename.c (usage): Reword for accuracy/consistency. * src/dirname.c (usage): Likewise. * src/du.c (usage): Likewise. * src/env.c (usage): Likewise. * src/printenv.c (usage): Likewise. * src/readlink.c (usage): Likewise. * src/realpath.c (usage): Likewise. * doc/coreutils.texi: Consolidate/share the descriptions of --null, --zero and --zero-terminated. 2014-05-25 Jim Meyering build: libstdbuf.so: avoid new OS X link failure * src/local.mk (src_libstdbuf_so_LDADD): Add $(LIBINTL), to avoid link failure on OS X. cat,cp,split: use a larger buffer for copying * src/ioblksize.h (IO_BUFSIZE): Double the size, to 128KB. Add four more columns to the blksize-vs-bandwidth table 2014-05-24 Paul Eggert doc: use nicer quotes * doc/coreutils.texi: Add "@documentencoding UTF-8". 2014-05-22 Pádraig Brady stat,tail: improve support for LogFS and ConfigFS * src/stat.c (human_fstype): Add new file system ID definitions. * NEWS: Mention the improvement. 2014-05-22 Pádraig Brady maint: enforce consistent width and case of file system constants * src/stat.c (human_fstype): Adjust a couple of existing constants to be a consistent width and capitalization so that the src/fs-magic-compare target works without reporting false positives. * cfg.mk (sc_fs-magic-compare): A new syntax check to enforce this. Improved by: Jim Meyering 2014-05-21 Pádraig Brady stdbuf: support compilers other than __GNUC__ __SUNPRO_C >= 0x590 /*12.0*/ supports __attribute((constructor)) required by stdbuf, so use a more direct check for this. Note ensure that --libexecdir is set to the appropriate install location for libstdbuf.so so that stdbuf works when installed on the system like it does when running tests in the build directory. * configure.ac (stdbuf_supported): Use a test prog to determine support. * src/libstdbuf.c (stdbuf): Define appropriately for non GCC compilers, and provide early feedback (compilation warning) if trying to compile libstdbuf without the necessary support. * src/stdbuf.c (set_LD_PRELOAD): Add a note on having stdbuf look for libstdbuf.so in the default lib search path. * cfg.mk (sc_prohibit-gl-attributes): Adjust so we can exclude libstdbuf.so from prohibiting '__attribute', since we want this form to avoid silently eliding this required attribute on non GCC. Reported and tested by Rich Burridge. 2014-05-21 Pádraig Brady tests: use chroot --user rather than internal setuidgid tool * init.cfg (require_root_): Adjust to use chroot, and make `require_built_ chroot` implicit when chroot used in the test. * po/POTFILES.in: Remove reference to setuidgid tool. * src/.gitignore: Likewise. * src/local.mk: Likewise. * src/setuidgid.c: Remove. * tests/cp/preserve-gid.sh: s/setuidgid/chroot --user/. * tests/cp/special-bits.sh: Likewise. * tests/id/setgid.sh: Likewise. * tests/misc/truncate-owned-by-other.sh * tests/mv/sticky-to-xpart.sh: Likewise. * tests/rm/fail-2eperm.sh: Likewise. * tests/rm/no-give-up.sh: Likewise. * tests/touch/now-owned-by-other.sh: Likewise. * tests/misc/chroot-fail.sh: Skip if chroot not built. chroot: make changing root check more robust * src/chroot.c (is_root): A new helper function to determine if the passed argument is the root directory based on inode comparison. (main): Use the new helper rather than comparing strings. * tests/misc/chroot-fail.sh: Add cases for alternative root paths. chroot: exit immediately upon failure * src/chroot.c (main): Consistently exit with failure status immediately upon hitting a terminal issue, rather than diagnosing multiple issues lest users think previous failing actions are optional. 2014-05-21 Pádraig Brady chroot: with --userspec clear root's supplemental groups It's dangerous and confusing to leave root's supplemental groups in place when specifying other users with --userspec. In the edge case that that is desired one can explicitly specify --groups. Also we implicitly set the system defined supplemental groups for a user. The existing mechanism where supplemental groups needed to be explicitly specified is confusing and not general when the lookup needs to be done within the chroot. Also we extend the --groups syntax slightly to allow clearing the set of supplementary groups using --groups=''. * src/chroot.c (setgroups): On systems without supplemental groups, clearing then is a noop and so should return success. (main): Lookup the primary GID with getpwuid() when just a numeric uid is specified, and also infer the USERNAME from this call, needed when we're later looking up the supplemental groups for a user. Support clearing supplemental groups, either implicitly for unknown users, or explicitly when --groups='' is specified. * tests/misc/chroot-credentials.sh: Various new test cases * doc/coreutils.texi (chroot invocation): Adjust for the new behavior. * NEWS: Mention the change in behavior. 2014-05-16 Pádraig Brady chroot: don't chdir() if not changing root This allows chroot to be used as a light weight tool to change user identification for a command, while not changing the current working directory. It also makes `chroot / true` consistently succeed on all platforms for non root users. * src/chroot.c (main): If the same root is specified. i.e. '/' then don't change the current working directory, and avoid the overhead of the other redundant calls. * tests/misc/chroot-fail.sh: Remove failure guard previously needed on some systems. Also add an explicit case to ensure we don't change directory. * NEWS: Mention the change in behavior. 2014-05-13 Pádraig Brady maint: avoid clang -Wtautological-constant-out-of-range-compare warning * src/df.c (decode_output_arg): Use only enum constants to avoid clang "warning: comparison of constant -1 with expression of type 'display_field_t' is always false" 2014-05-13 Pádraig Brady df: ignore non file system entries in /proc/mounts Linux with network namespaces contains entries in /proc/mounts like: proc net:[4026532464] proc rw,nosuid,nodev,noexec,relatime 0 0 resulting in a failure to stat 'net:[...]', inducing a warning and an exit with failure status. * src/df.c (get_dev): Ignore all relative mount points. * tests/df/skip-duplicates.sh: Add an entry to test relative dirs. 2014-05-13 Pádraig Brady df: fix handling of symlinks in mount list The symlink handling in commit v8.21-172-g33660b4 was incomplete in the case where there were symlinks in the mount list itself. For example, in the case where /dev/mapper/fedora-home was in the mount list and that in turn was a symlink to /dev/dm-2, we have: before> df --out=source /dev/mapper/fedora-home devtmpfs after > df --out=source /dev/mapper/fedora-home /dev/mapper/fedora-home * src/df.c (get_disk): Compare canonicalized device names from the mount list. Note we still display the non canonicalized name, even if longer, as we assume that is the most representative. * tests/df/df-symlink.sh: This could theoretically fail on some systems depending on the content of the mount list, but adjust to fail on any system where symlinks are present in the mount list for the current dir. 2014-05-13 Pádraig Brady df: also deduplicate virtual file systems * src/df.c (filter_mountlist): Remove the constraint that a '/' needs to be in the device name for a mount entry to be considered for deduplication. Virtual file systems also have storage associated with them (like tmpfs for example), and thus need to be deduplicated since they will be shown in the default df output and subject to --total processing also. * test/df/skip-duplicates.sh: Add a test to ensure we deduplicate all entries, even for virtual file systems. Also avoid possible length operations on many remote file systems in the initial check of df operation. Also avoid the assumption that "/root" is on the same file system as "/". * NEWS: Mention the change in behavior. tests: fix spurious failure with leading spaces in file names * tests/ls/stat-vs-dirent.sh: This test lists all parent directories, and would spuriously fail if any of those had a file name with a leading space as the first entry. There is only ever a single space between the right aligned inode number and the file name, so process accordingly. 2014-05-12 Assaf Gordon build: avoid bootstrap error with gettext 0.18.3.1 * bootstrap: Create critical bootstrap files for autopoint, before gnulib re-generates them, avoiding the issue. See: http://lists.gnu.org/archive/html/coreutils/2013-11/msg00038.html http://savannah.gnu.org/bugs/?40083 https://pad.lv/1311895 2014-05-10 Jim Meyering shred: don't infloop upon negative size * src/shred.c (main): With the preceding change, shred -s-2 FILE would write 64KB blocks forever -- or until disk full. This change makes shred reject a negative size. * tests/misc/shred-negative.sh: New file. * tests/local.mk (all_tests): Add it. 2014-05-10 Paul Eggert shred: fix overflow checking of command-line options * src/shred.c (main): Limit -n (number of passes) value to ULONG_MAX, not to UINT32_MAX, since the vars are unsigned long. Limit the -s (file size) value to OFF_T_MAX. 2014-05-07 Bernhard Voelker tests: improve new ascii test of dd * tests/dd/ascii.sh: Avoid unnecessary subshells. Catch dd's exit code. Remove testing artifact. In the case of a comparison failure, show the differences in octal format in addition to "binary files differ". Simplify the creation of the 'in' file. 2014-05-06 Paul Eggert dd: fix conv=ascii, conv=ebcdic, conv=ibm to match POSIX Problem reported by Don Baggett in . * NEWS: * doc/coreutils.texi (dd invocation): Document this. * src/dd.c (conversions): conv=ascii implies conv=unblock. conv=ebcdic and conv=ibm imply conv=block. (ascii_to_ebcdic, ebcdic_to_ascii): Correct to match POSIX 1003.1-2013. * tests/dd/ascii.sh: New file. * tests/local.mk (all_tests): Add it. 2014-05-06 Pádraig Brady tests: improve diagnostics when asserting empty files * tests/chmod/c-option.sh: Use `compare /dev/null ... || fail=1` rather than `test -s ... && fail=1`, so that the file contents are output, thus improving diagnostics for failing tests. * tests/cp/acl.sh: Likewise. * tests/cp/cp-a-selinux.sh: Likewise. * tests/cp/cp-mv-enotsup-xattr.sh: Likewise. * tests/cp/reflink-perm.sh: Likewise. * tests/dd/misc.sh: Likewise. * tests/misc/env-null.sh: Likewise. * tests/misc/env.sh: Likewise. * tests/misc/nice.sh: Likewise. * tests/misc/nohup.sh: Likewise. * tests/misc/printenv.sh: Likewise. * tests/misc/xattr.sh: Likewise. * tests/mv/update.sh: Likewise. * tests/rm/deep-2.sh: Likewise. * tests/rm/read-only.sh: Likewise. * tests/split/r-chunk.sh: Likewise. * tests/tail-2/follow-stdin.sh: Likewise. * tests/tail-2/inotify-race.sh: Likewise. * tests/tail-2/wait.sh: Likewise. * tests/touch/no-dereference.sh: Likewise. * cfg.mk (sc_prohibit_test_empty:): New syntax-check. * tests/cp/proc-zero-len.sh: Adjust to avoid false syntax-check failure. * tests/cp/proc-zero-len.sh: Likewise. * tests/mv/part-symlink.sh: Likewise. * tests/tail-2/infloop-1.sh: Likewise. 2014-05-06 Pádraig Brady mv,cp: preserve symlink xattrs when copying across file systems * src/copy.c (copy_internal): Include the copy_attr() call for symlinks. This should not dereference symlinks, since llistxattr() is used in attr_copy_file() in libattr, and so should copy all but the filtered extended attributes. Note we don't just move the copy_attr() call before the set_owner() call, as that would break capabilities for non symlinks. * tests/cp/cp-mv-enotsup-xattr.sh: Add a test case. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/16131 2014-05-05 Jarkko Sakkinen tests: initial SMACK tests * init.cfg (require_smack_): New function. * local.mk: Referenced new tests. * tests/id/smack.sh: SMACK tests (new file). * tests/mkdir/smack-no-root.sh: SMACK tests (new file). * tests/mkdir/smack-root.sh: SMACK tests (new file). 2014-05-04 Jim Meyering maint: autotools-install: update tool version numbers to latest * scripts/autotools-install (tarballs): Update to latest. 2014-05-03 Jim Meyering maint: don't let envvar setting break ChangeLog generation * Makefile.am (gen-ChangeLog): Clear amend_git_log when we don't set it, so that an envvar setting cannot cause trouble. 2014-05-01 Pádraig Brady numfmt: improve processing throughput by 800% The devmsg() calls that took quote_n() arguments, didn't normally output anything, but still incurred the overhead of those quote_n() calls. * src/numfmt.c (devmsg): Move the inline function with _internal_ enablement check to... * src/system.h: ...here as a variadic macro, with the enablement check at the outer level. * src/factor.c: As per numfmt.c but there is no performance change in this case. * NEWS: Mention the significant performance improvement. 2014-05-01 Pádraig Brady numfmt: support zero padding using --format="%010f" * src/numfmt.c (setup_padding_buffer): Simplify the code by not explicitly dealing with heap exhaustion. (parse_format_string): Likewise. Handle multiple grouping modifiers as does the standard printf. Handle the new leading zero --format modifier. (double_to_human): Use more defensive coding against overwriting stack buffers. Honor the leading zeros width. (usage): Mention the leading zero --format modifier. (main): Allow --padding in combo with a --format (width), as the number of leading zeros are useful independent of the main field width. * doc/coreutils.texi (numfmt invocation): Likewise. * tests/misc/numfmt.pl: Add new test cases. * NEWS: Mention the improvement. 2014-04-29 Pádraig Brady doc: clarify the pr --page-width descriptions * doc/coreutils.texi (pr invocation): Clarify that -w or -W will be rounded down so that each column has the same width. Adjust the wording for -W, to avoid the implication that the width of -S is insignificant to the page width. * src/pr.c (usage): Add a period to avoid ambiguity in the man page output. 2014-04-29 Pádraig Brady ptx: fix whitespace trimming with multiple files This issue was identified by running the test suite with http://code.google.com/p/address-sanitizer/ which is included in GCC 4.8 and enabled with -fsanitize=address This was checked on Fedora 20 with GCC 4.8 as follows: $ yum install libasan # http://bugzilla.redhat.com/991003 $ rm -f src/ptx.o $ make check AM_CFLAGS='-fsanitize=address' SUBDIRS=. VERBOSE=yes $ failure identified in tests/test-suite.log To see this particular failure triggered with multiple files: $ src/ptx <(echo a) <(echo a) 2>&1 | asan_symbolize.py -d ================================================================= ==32178==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000e74f at pc 0x435442 bp 0x7fffe8a1b290 sp 0x7fffe8a1b288 READ of size 1 at 0x60200000e74f thread T0 #0 0x435441 in define_all_fields coreutils/src/ptx.c:1425 #1 0x7fa206d31d64 in __libc_start_main ??:? #2 0x42f77c in _start ??:? 0x60200000e74f is located 1 bytes to the left of 3-byte region [0x60200000e750,0x60200000e753) allocated by thread T0 here: #0 0x421809 in realloc ??:? #1 0x439b4e in fread_file coreutils/lib/read-file.c:97 Shadow bytes around the buggy address: 0x0c047fff9c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c047fff9ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c047fff9cb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c047fff9cc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c047fff9cd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fd fd =>0x0c047fff9ce0: fa fa 03 fa fa fa fd fd fa[fa]03 fa fa fa 00 00 0x0c047fff9cf0: fa fa 04 fa fa fa 04 fa fa fa fd fa fa fa fd fa 0x0c047fff9d00: fa fa 00 fa fa fa fd fa fa fa 00 fa fa fa 00 fa 0x0c047fff9d10: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa 0x0c047fff9d20: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa 0x0c047fff9d30: fa fa fd fa fa fa 00 fa fa fa 00 fa fa fa 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 ASan internal: fe ==32178==ABORTING The initial report and high level analysis were from Jim Meyering... "The underlying problem is that swallow_file_in_memory() is setting the contents of the global text_buffer for the first file, then updating it (clobbering old value) for the second file. Yet, some pointers to the initial buffer have been squirreled away and later, one of them (keyafter) is presumed to point into the new "text_buffer", which it does not. The subsequent SKIP_WHITE_BACKWARDS use backs up "cursor" and goes out of bounds." * src/ptx.c (text_buffers): Maintain references for the limits of each buffer corresponding to each file, rather than just the last processed. (struct OCCURS): Add a member to map back to the corresponding file. Note normally this could be computed from the "reference" member rather than needing the extra storage, however this is not possible when in --references mode. (find_occurs_in_text): Reference the array rather than a single entry. (define_all_fields): Likewise. Also avoid computing the file index since this is now stored directly. (main): Update text_buffers[] array rather than a single text_buffer. * tests/misc/ptx-overrun.sh: Even though this issue is already triggered with AddressSanitizer, add a new case to demonstrate the whitespace trimming issue, and to trigger without AddressSanitizer. Fixes https://bugs.gnu.org/16171 2014-04-24 Pádraig Brady maint: make ChangeLog generation more robust * Makefile.am (gen-ChangeLog): Sync changes from GNU hello, to ensure exit status is propagated, and to support an optional git-log-fix file. 2014-04-24 Edgars Irmejs maint: remove trailing comma from enum to make gl/ c89 compat * gl/lib/fadvise.c (fadvice_t): This might go to gnulib some day so remove the trailing comma. Fixes http://bugs.gnu.org/17329 2014-04-22 Assaf Gordon doc: clarify meaning of stat's mtime and ctime formats * src/stat.c (usage): s/modification time/data modification time/; s/change time/status change time/ * doc/coreutils.texi: Ditto. 2014-04-18 Pádraig Brady tests: make ls tests independent of COLORTERM env Since the recent commit v8.22-68-g08783f1, ls coloring is now dependent on the COLORTERM environment variable. * tests/envvar-check: Unset COLORTERM from test environment. * tests/ls/color-dtype-dir.sh: Ensure coloring is used. * tests/misc/ls-misc.pl: Likewise. Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/10397646 2014-04-17 Pádraig Brady doc: disambiguate chmod man page reference to (a) users * man/chmod.x: Don't rely on the bold markup for 'a' to distinguish it as that's not done in all cases. 2014-04-17 David Michael dircolors: add hurd and mach-gnu-color terminal types mach-color was replaced by mach-gnu-color in Hurd in 2012. mach-color is left for compatibility and corresponding definitions for mach-color are still found in ncurses. * src/dircolors.hin: Add hurd and mach-gnu-color. 2014-04-16 Guilherme de Almeida Suckevicz ls: don't output colors with unknown TERM env variable --colors controls whether to output colors depending on whether we're connected to a terminal or not, while this change gives control over which terminals we output colors to. * NEWS: Mention the change in behavior. * src/ls.c (known_term_type): A new function to search the static list from dircolors.h (parse_ls_colors): Honor the TERM when both LS_COLORS and COLORTERM are non empty. * tests/ls/color-term.sh: A new test. * tests/local.mk: Reference the new test. Fixes http://bugs.gnu.org/15992 2014-04-16 Benno Schulenberg doc: improve remove prompt translator comment * src/remove.c (prompt): Explain where the difficulty with translating these two strings resides, and suggest an alternative: the one that Paul Eggert first proposed back in 2002, which seems fully resistant. 2014-04-16 Bernhard Voelker maint: df: avoid sizeof(char**) static analysis warning * src/df.c (alloc_table_row): Use the size of char** to enlarge the table. Spotted by Coverity. 2014-04-11 Pádraig Brady tests: fix false dd conv=sparse failure on newer XFS * tests/dd/sparse.sh: When testing that a hole is created, use an existing sparse destination file, so that we're not write extending the file size, and thus avoiding speculative preallocation which can result in smaller holes than requested. Workaround suggested by Brian Foster 2014-04-10 Pádraig Brady maint: avoid static analysis warnings in ln.c * src/ln.c (do_link): It's not obvious that record_file() is a noop in the symlink case (in that case dest_set is NULL and so ignored). So to make it obvious, and to avoid false positives seen in coverity, add the explicit condition here. 2014-04-09 Pádraig Brady maint: document timeout --preserve-status in v8.21 NEWS * NEWS: Add the omitted new feature. * cfk.mk (old_NEWS_hash): Adjust accordingly. doc: clarify in --help that -Z doesn't take an argument * NEWS: Fix a confusing old entry. * cfg.mk (old_NEWS_hash): Adjust accordingly. * src/cp.c (usage): Separate the -Z and --context descriptions. * src/install.c: Likewise. * src/mkdir.c: Likewise. * src/mkfifo.c: Likewise. * src/mknod.c: Likewise. Fixes http://bugs.gnu.org/17220 2014-04-04 Pádraig Brady maint: various cleanups * tests/misc/numfmt.pl: Fix comment misspelling. * src/cut.c: Likewise. * src/tsort.c (detect_loop): Replace an fprintf() with error(). shred: overwrite inode storage used by some file systems * doc/coreutils.texi (shred invocation): Mention some reasons why clearing slack space might be useful. * src/shred.c (do_wipefd): Add initial writes for each pass for small regular files in case the storage for those is in the inode, and thus a larger write up to a block size would bypass that. Move the direct I/O control to... (dopass): ... here so we can avoid enabling it for these small initial writes. It's better to retry direct I/O for each pass anyway to handle the case where direct I/O is disabled for only the last portion of a file when the size is not a multiple of the block size. Note we don't avoid the sync for the initial write as it will be small but more importantly could be on a different part of the disk and so worth doing independently to ensure the write is not discarded. * tests/misc/shred-exact.sh: Check some more direct I/O cases. * NEWS: Mention the improvements. The inode storage issue was mentioned by Paul Eggert. 2014-04-03 Pádraig Brady shred: avoid a data pass on empty files * src/shred.c (do_wipefd): Don't increase the size written for an empty file up to a full block. Also increase the size to OFF_T_MAX in the edge case where we do overflow. * NEWS: Mention the shred improvements from recent changes. * tests/misc/shred-passes.sh: Adjust as we no longer write a BLKSIZE of data for empty files. 2014-04-03 Paul Eggert cp: don't reserve a device number * src/copy.c (copy_internal): Replace dev_t arg DEVICE with struct stat pointer arg PARENT. All callers changed. This removes an unwarranted assumption that dev_t values of 0 cannot occur in file systems. See: http://bugs.gnu.org/17179 shred: shred one block even for empty files * src/shred.c (do_wipefd): Shred one block of empty regular files. This reverts an unintended part of the previous change. 2014-04-02 Paul Eggert shred: port to GNU/Linux behavior with tape drives See: http://bugs.gnu.org/17149 * src/shred.c [__linux__]: Include . (dorewind): New function, which works around the lseek problem with tape drives on GNU/Linux, the same way that dd does. (dopass): Use it. New arg ST, needed for dorewind. All uses changed. (do_wipefd): Don't rely on undefined behavior on integer overflow of file sizes. Use INT_ADD_OVERFLOW instead. head: port to Darwin and use simpler seeks This removes an unportable assumption that if lseek succeeds, the file is capable of seeking. See: http://bugs.gnu.org/17145 * src/head.c (elseek): New function, for consistency in reporting lseek failures. (elide_tail_bytes_file, elide_tail_lines_seekable) (elide_tail_lines_file, head_lines, head): Use it. (elide_tail_bytes_file, elide_tail_lines_file): New args CURRENT_POS and SIZE. All uses changed. Don't bother invoking lseek, since we know the file's pos and size now. (elide_tail_bytes_file): Change a local from uintmax_t to off_t, since it fits. (head): Use lseek only on regular files, since its behavior on unseekable devices is implementation-defined. * NEWS: Document this. 2014-03-31 Paul Eggert head: fix bug with head -c-N when stdin is not at start * src/head.c (elide_tail_bytes_file): Fix typo in lseek invocation. * tests/misc/head-c.sh: Add test for this bug. * NEWS: Document this. 2014-03-26 Pádraig Brady mv: fix issues with previous prompt adjustments * src/copy.c (overwrite_ok): Fix the gettext calls so that the second string is tagged for translation. Display the correct "replace ..." prompt when in move_mode. * tests/mv/i-3.sh: Display the output on failure to ease debugging. 2014-03-26 Bernhard Voelker ptx: avoid --format long option falling through into --help * src/ptx.c (main): Add a 'break' after the --format handling case. Otherwise it would fall through into the usage case. * tests/misc/ptx.pl: Add test cases for --format=tex and --format=roff. * NEWS (Bug fixes): Mention the fix. Bug introduced in 1999-04-04 commit, SH-UTILS-1_16f-269-gd815c15. Spotted by coverity (MISSING_BREAK). 2014-03-26 Bernhard Voelker basename: mention that the -s option implies -a in usage * src/basename.c (usage): Mention that -s implies -a. (main): Add "fall through" comment to case 's'. Spotted by coverity: MISSING_BREAK. 2014-03-24 Paul Eggert cp: simplify overwrite logic * src/copy.c (overwrite_ok): Rename from overwrite_prompt. Invoke yesno instead of having the caller do it; that's cleaner. Return bool, not void. All callers changed. cp: improve quality of overwrite prompt * src/copy.c (overwrite_prompt): New arg X. All callers changed. Use X to improve the quality of the prompt (Bug#17087). * tests/mv/i-2.sh, tests/mv/i-3.sh: Change test to match new prompt. 2014-03-24 Pádraig Brady split: avoid unnecessary input buffering Input buffering is best avoided because it introduces delayed processing of output for intermittent input, especially when the output size is less than that of the input buffer. This is significant when output is being further processed which could happen if split is writing to precreated fifos, or through --filter. If input is arriving quickly from a pipe then this will already be buffered before we read it, so fast arriving input shouldn't be a performance issue. * src/split.c (lines_split, lines_bytes_split, bytes_split, lines_chunk_split, bytes_chunk_extract): s/full_read/safe_read/. * THANKS.in: Mention the reporter. * NEWS: Mention the improvement. 2014-03-19 Pádraig Brady doc: clarify the operation of uniq -uD The description of -u was inaccurate when combined with -D: $ printf '%s\n' '1 a' '2 a' | uniq -uD -f1 1 a * doc/coreutils.texi (uniq invocation): Clarify that it's the last repeated line that is suppressed from the output. Fixes http://bugs.gnu.org/17022 2014-03-19 Edward Welbourne doc: avoid regex ambiguity in chmod mode description * man/chmod.x: s/[+-=]/[-+=]/ * doc/perm.texi (Symbolic Modes): Likewise. * THANKS.in: Remove patch owner. Fixes http://bugs.gnu.org/17041 2014-03-18 Pádraig Brady chroot: always diagnose failure to set supplemental groups * src/chroot.c (setgroups): Change this replacement to fail when called so that platforms like Interix without support for supplemental groups don't silently ignore a --groups option. 2014-03-18 Dylan Simon chmod: fix erroneous warnings with -R --changes For files with "special" bits set, we would stat the relative file name in the wrong directory, giving an erroneous ENOENT diagnostic. This issue was introduced with commit v5.92-653-gc1994c1 which changed fts to not change directory on traversal. * src/chmod.c (mode_changed): Use fts->fts_cwd_fd with fstatat rather than stat. All callers changed. * tests/chmod/c-option.sh: Add a test case. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/17035 2014-03-18 Bernhard Voelker maint: fix typo in comment in fadvise.c * gl/lib/fadvise.c: s/the the/the/, indroduced in commit v8.22-40-g4f21182. Promted by sc_prohibit_doubled_word. While at it, also s/be candidate/be a candidate/. 2014-03-18 Paul Eggert stat: port birthtime to Solaris 11 Problem reported by Rich Burridge. * src/stat.c [HAVE_GETATTRAT]: Include , . (print_statfs, print_stat, print_it): Pass fd, too, for the benefit of get_birthtime. All uses changed. (get_birthtime): New function, for porting to Solaris 11. (print_stat): Use it. * configure.ac (getattrat, LIB_NVPAIR): New checks. * src/local.mk (src_stat_LDADD): Add $(LIB_NVPAIR). 2014-03-13 Jim Meyering ln: with -sr, don't segfault for a TARGET of '' Prior to this change, "ln -sr '' F" would segfault, attempting to read path2[1] in relpath.c's path_common_prefix function. This problem arises whenever canonicalize_filename_mode returns NULL. * src/ln.c (convert_abs_rel): Call relpath only when both canonicalize_filename_mode calls return non-NULL. * tests/ln/relative.sh: Add a test to trigger this failure. * THANKS.in: List reporter's name/address. * NEWS (Bug fixes): Mention it. Reported by Erik Bernstein in 739752@bugs.debian.org. Fixes http://bugs.gnu.org/17010. 2014-03-13 Jim Meyering maint: avoid "attribute-const"-suggesting warning from gcc * gl/lib/fadvise.c: Use a pragma to turn off this warning option: -Wsuggest-attribute=const. Without this change, building with --enable-gcc-warnings would evoke this error: lib/fadvise.c:25:1: error: function might be candidate for\ attribute 'const' [-Werror=suggest-attribute=const] 2014-03-13 Jim Meyering scripts: autotools-install: update * scripts/autotools-install: Update version numbers of latest automake and gettext packages. 2014-03-13 Pádraig Brady chroot: improve --userspec and --groups look-up - Support arbitrary numbers in --groups, consistent with what is already done for --userspec - Avoid look-ups entirely for --groups items with a leading '+' - Support names that are actually numbers in --groups - Ignore an empty --groups="" option for consistency with --userspec - Look up both inside and outside the chroot with inside taking precedence. The look-up outside may load required libraries to complete the look-up inside the chroot. This can happen for example with a 32 bit chroot on a 64 bit system, where the 32 bit NSS plugins within the chroot fail to load. * src/chroot.c (parse_additional_groups): A new function refactored from set_addition_groups(), to just do the parsing. The actual setgroups() call is separated out for calling from the chroot later. (main): Call parse_user_spec() and parse_additional_groups() both outside and inside the chroot for the reasons outlined above. * tests/misc/chroot-credentials.sh: Ensure arbitrary numeric IDs can be specified without causing look-up errors. * NEWS: Mention the improvements. * THANKS.in: Add Norihiro Kamae who initially reported the issue with a proposed patch. Also thanks to Dmitry V. Levin for his diagnosis and sample patch. 2014-03-09 Aleksej Serdjukov doc: fix typo in texinfo docs * doc/coreutils.texi (Operating on characters): s/This/These/. Fixes http://bug.gnu.org/16973 2014-03-07 Pádraig Brady doc: fix readpath typo in texinfo contents * docs/coreutils.texi: s/readpath/realpath/ Fixes http://bugs.gnu.org/16964 2014-03-05 Pádraig Brady tests: avoid the :> construct which can hide errors On most shells `:>file || framework_failure_` will not evaluate the framework_failure_ even if there was an error writing the file. shells which do evaluate the failure are ksh 93u+ and bash 4.2, while shells wich don't include bash 4.3, solaris, freebsd, dash. Furthermore this construct is problematic on Solaris 10 sh, which will try to optimize away a `:' command in a loop after the first iteration, even if it is redirected. * tests/cp/link-deref.sh: Remove the leading colon on redirections. * tests/cp/reflink-perm.sh: Likewise. * tests/id/zero.sh: Likewise. * tests/install/install-C.sh: Likewise. * tests/misc/env.sh: Likewise. * tests/misc/md5sum-bsd.sh: Likewise. * tests/misc/runcon-no-reorder.sh: Likewise. * tests/mv/partition-perm.sh: Likewise. * tests/rm/r-root.sh: Likewise. * tests/split/l-chunk.sh: Likewise. * tests/split/line-bytes.sh: Likewise. * tests/tail-2/inotify-rotate.sh: Likewise. * tests/tail-2/retry.sh: Likewise. * tests/tail-2/symlink.sh: Likewise. * tests/tail-2/wait.sh: Likewise. * tests/touch/read-only.sh: Likewise. + cfg.mk (sc_prohibit_colon_redirection): A new syntax check to avoid further instances of this creeping in. 2014-03-05 Pádraig Brady tests: fix false failure in nohup.sh in non tty builds * tests/misc/nohup.sh: When running tests without a controlling tty, an exec failure is triggered in a subshell, which causes POSIX shells to immediately exit the subshell. This was brought to notice by the newly conforming bash 4.3. Fixes http:/bugs.gnu.org/16940 2014-03-03 Pádraig Brady doc: improve df --human and --si, help and man page * src/df.c (usage): Adjust the --human and --si descriptions to not depend on each other. Also include an example that is illustrative of the rounding, suffix, width, and localized fractions. * src/system.h (emit_size_note). Adjust so that it's obvious the description is pertaining to the input SIZE argument, and not to any sizes that might be output by df for example. Fixes http://bugs.gnu.org/16922 2014-02-27 Pádraig Brady date: fix crash or infinite loop when parsing a malformed TZ="" * NEWS: Mention the fix. * gnulib: Update to incorporate the fix. This is the only change in this gnulib update. * tests/misc/date.pl: Add a test for this case. Fixes http://bugs.gnu.org/16872 2014-02-27 Paul Eggert cp: copy files by inode only if that facility is available * src/copy.c (copy_dir): Use the new SAVEDIR_SORT_FASTREAD, not SAVEDIR_SORT_INODE. Problem reported by Bernhard Voelker in: http://lists.gnu.org/archive/html/coreutils/2014-02/msg00037.html build: update gnulib submodule to latest 2014-02-26 Paul Eggert cp: copy files by inode Problem reported by Bernhard Voelker in: http://lists.gnu.org/archive/html/coreutils/2014-02/msg00034.html * src/copy.c (copy_dir): Adjust to recent gnulib change. build: update gnulib submodule to latest 2014-02-26 Bernhard Voelker shuf: convert error diagnostic to lowercase * src/shuf.c (main): s/No/no/, introduced by commit v8.22-25-g9f60f37. * NEWS: Also adjust the NEWS for that recent commit to make it clear this was new bug rather than a regression. Prompted by the syntax-check rule sc_error_message_uppercase 2014-02-25 Bernhard Voelker doc: add dd example for failing disks * doc/coreutils.texi (dd invocation): Add an example for how to call dd to save data from a failing disk. Mention GNU 'ddrescue' as one of the more specialized tools in such a case. 2014-02-23 Paul Eggert shuf: with -r, don't dump core if the input is empty Problem reported by valiant xiao in . * NEWS: Document this. * src/shuf.c (main): With -r, report an error if the input is empty. * tests/misc/shuf.sh: Test for the bug. 2014-02-20 Paul Eggert doc: fix problems with @w and @kbd (Bug#16802) * doc/coreutils.texi: Prefer '@allowcodebreaks false' to '@w'. Also, don't use @kbd except for keyboard input. 2014-02-10 Pádraig Brady cp: with --link always use linkat() if available * src/copy.c (copy_reg): If linkat() is available it doesn't matter about the gnulib emulation provided, and thus the LINK_FOLLOWS_SYMLINKS should not have significance here. This was noticed on FreeBSD and the consequence is that cp --link will create hardlinks to symlinks there, rather than emulating with symlinks to symlinks. * tests/cp/link-deref.sh: Adjust the checks to cater for all cases where hardlinks to symlinks are supported. 2014-02-09 Pádraig Brady head,tail: consistently diagnose write errors If we can't output more data, we should immediately diagnose the issue and exit rather than consuming all of input (in some cases). * src/tail.c (xwrite_stdout): Also diagnose the case where only some data is written. Also clearerr() to avoid the redundant less specific error from atexit (close_stdout); * src/head.c (xwrite_stdout): Copy this new function from tail, and use it to write all output. * tests/misc/head-write-error.sh: A new test to ensure we exit immediately on write error. * tests/local.mk: Reference the new test. 2014-02-09 Алексей Шилин head: fix --lines=-0 outputting nothing if no newline at EOF * src/head.c (elide_tail_lines_pipe): Just output all input in this case to avoid the issue and also avoid redundant '\n' processing. (elide_tail_lines_seekable): Likewise. * tests/misc/head-elide-tail.pl: Add tests for no '\n' at EOF. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/16329 2014-02-09 John dircolors: add a new entry to colorize 'm4a' files * src/dircolors.hin: Add entry for mpeg4 audio files. Fixes http://bugs.gnu.org/16700 2014-02-09 Niels Möller od: add an --endian option to control byte swapping * src/od.c (main): Handle the new --endian option, taking "little" and "big" as parameters. (usage): Describe the new option. (PRINT_FIELDS): Adjust to swap bytes if required. * tests/misc/od-endian.sh: A new test to verify the byte swapping operations for hex (ints) and floats for all sizes between 1 and 16 inclusive. * test/local.mk: Reference the new test. * doc/coreutils.texi (od invocation): Describe the new option. * NEWS: Mention the new feature. 2014-01-28 Pádraig Brady stat,tail: improve support for HFS+ and HFSX * src/stat.c (human_fstype): Add new file system ID definitions. * NEWS: Mention the improvement. Fixes http://bugs.gnu.org/16336 2014-01-24 Assaf Gordon tests: skip chown/separator with conflicting group IDs * tests/chown/separator.sh: skip test if the user's group has multiple entries. Fixes http://bugs.gnu.org/16532 2014-01-21 Pádraig Brady build: suppress an erroneous warning --with-selinux=no * m4/jm-macros.m4: Don't check the SELinux cached variables --without-selinux. Reported-by: Bernhard Voelker 2014-01-17 Pádraig Brady build: remove no longer used spawn-pipe files from POTFILES.in This should have been part of the previous commit v8.22-14-gf18999b * po/POTFILES.in: Remove gnulib files no longer used. Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/8288209/ 2014-01-17 Pádraig Brady build: avoid unneeded spawn-pipe gnulib module * bootstrap.conf: Reference the used pipe-posix module rather than the uneeded and larger spawn-pipe module, which the pipe module is the deprecated equivalent of. 2014-01-17 Chengwei Yang doc: clarify SMACK related --help and documentation * src/ls.c: Remove all mention of SELinux since ls should treat all security context labels equally. * doc/coreutils.texi (ls invocation): Likewise. (id invocation): Clarify that -Z outputs the context inherited by the process, rather than one specific to a user. Note for SMACK this can be set instead by the SMACK64EXEC label, in the unusual case where this is set on the id executable. * src/id.c (usage): Likewise. * src/mkdir.c (usage): Clarify that -Z is specific to SELinux, while --context=CTX is also supported for SMACK. * src/mkfifo.c (usage): Likewise. * src/mknod.c (usage): Likewise. 2014-01-17 Pádraig Brady ln: fix replacing symbolic links whose targets can't exist * src/ln.c (errno_nonexisting): A new function to determine if the errno implies that a file doesn't or can't (currently) exist. (target_directory_operand): Use the new function to expand the set of errors we handle. * tests/ln/sf-1.sh: Add test cases for the newly handled errors. * THANKS.in: Mention the reporter. * NEWS: Mention the bug fix. 2014-01-14 Pádraig Brady tests: restrict a recent SELinux test to SELinux systems * tests/cp/no-ctx.sh: Since the test diagnoses whether the intercepted lgetfilecon() calls are actually called or not, restrict the test to systems where that occurs. The test cases are minimal on non SELinux systems and should be well covered by other tests. Reported-by: Bernhard Voelker 2014-01-13 Nicolas Iooss copy: fix a segfault in SELinux context copying code * src/selinux.c (restorecon_private): On ArchLinux the `fakeroot cp -a file1 file2` command segfaulted due to getfscreatecon() returning a NULL context. So map this to the sometimes ignored ENODATA error, rather than crashing. * tests/cp/no-ctx.sh: Add a new test case. * tests/local.mk: Reference the new test. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/16335 2014-01-13 Pádraig Brady maint: tests: refactor gcc commands for building shared lib * init.cfg (gcc_shared_): A new function refactored from tests. (require_gcc_shared_): Adjust to call gcc_shared_() to build the test library, and remove that library before the function returns. * tests/cp/nfs-removal-race.sh: Call the new gcc_shared_(). * tests/df/no-mtab-status.sh: Likewise. * tests/df/skip-duplicates.sh: Likewise. * tests/ls/getxattr-speedup.sh: Likewise. * tests/rm/r-root.sh: Likewise. copy: fix SELinux context preservation for existing directories * src/copy.c (copy_internal): Use the global process context to set the context of existing directories before they're populated. This is more consistent with the new directory case, and fixes a bug for existing directories where we erroneously set the context to the last copied descendent, rather than to that of the source directory itself. * tests/cp/cp-a-selinux.sh: Add a test for this case. * NEWS: Mention the fix. * THANKS.in: Add reporter Michal Trunecka. 2014-01-10 Bernhard Voelker tests: improve test for a working setfacl Prompted by a test framework failure of tests/mkdir/p-acl.sh on armv7l: The previous test for a working setfacl was not sufficient in some circumstances. * init.cfg (require_setfacl_): Call setfacl twice with conflictive ACL specs, and use ACL specs which can't be mapped into regular file permission bits. Document the reasons. 2014-01-08 Bernhard Voelker tests: avoid FP failure when cp fails for /proc/cpuinfo On emulated aarch64 systems like in the qemu-based OpenBuildService of openSUSE, cp fails to copy /proc/cpuinfo because the inode number changes between the initial stat() call and copying the file: $ cp /proc/cpuinfo exp cp: skipping file '/proc/cpuinfo', as it was \ replaced while being copied * tests/cp/proc-zero-len.sh: When cp fails, check on the above error message to skip the test. 2014-01-02 Bernhard Voelker maint: fix copyright holder and year in selinux sources * src/selinux.c: As the copyright is assigned to the FSF for all of coreutils, replace the copyright holder from "Red Hat, Inc." to "Free Software Foundation, Inc.". Prompted by a warning of 'make update-copyright'. Also update the copyright year. maint: update all copyright year number ranges Run "make update-copyright", but then also run this, perl -pi -e 's/2\d\d\d-//' tests/sample-test to make that one script use the single most recent year number. 2014-01-02 Bernhard Voelker maint: prevent update-copyright from updating files from gnulib The files 'bootstrap' and 'tests/init.sh' are taken from the gnulib submodule. Therefore, 'make update-copyright' there should already have updated the copyright year in these 2 files. * .x-update-copyright: Add entries for the above mentioned files. 2014-01-02 Bernhard Voelker build: update gnulib submodule, bootstrap and tests/init.sh to latest * gnulib: Update to latest. * bootstrap: Update from gnulib. * tests/init.sh: Likewise. 2013-12-13 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.22 * NEWS: Record release date. 2013-12-13 Pádraig Brady stat,tail: improve support for hostfs and smackfs hostfs is provided by the Linux UML subsystem. smackfs is provided by the Linux Smack security module. * src/stat.c (human_fstype): Add new file system ID definitions. * NEWS: Mention the improvement, and adjust for the fact that SNFS is a remote file system. 2013-12-12 Pádraig Brady tests: avoid unlikely deadlock in dd/no-allocate on some shells * test/dd/no-allocate.sh: Use 'wait' to ensure we don't have multiple writers to the fifo, which was seen to trigger a very hard to reproduce deadlock with make -j20 on solaris. Also avoid writing to the fifo with the shell; instead using dd. (check_dd_seek_alloc): A new function refactored from the various cases, which are now constructed from function parameters. tests: restrict cp --link inode comparisons to compatible systems * tests/cp/link-deref.sh: On systems were cp can't determine if gnulib linkat() emulation might create a symlink instead of a hardlink to a symlink, copy.c will create a symlink to the symlink so that it has more control over its metadata. Also even if the system supports this operation, the particular file system under test may not. So avoid the hardlinked symlink verification in these cases. This fixes a false failure on aix, solaris and freebsd. build: default to avoiding openssl system libraries * configure.ac: Don't change the gnulib default of 'no' for whether to link with openssl system libraries if available. Distributions can explicitly enable this as their policy allows. * NEWS: Adjust accordingly. 2013-12-12 Tiger Lee doc: fix typo in numfmt invocation * doc/coreutils.texi (numfmt invocation): s/nun/num/. Fixes http://bugs.gnu.org/16122 2013-12-11 Pádraig Brady tests: df/total-unprocessed: fix false failure with lofs * tests/df/total-unprocessed.sh: Skip the test when we can't determine the file system type as the exclusion filter is not applied in that case. "lofs" being ignored is effectively an unknown file system type. 2013-12-10 Pádraig Brady tests: avoid warnings due to new filefrag -v format * tests/cp/sparse-fiemap.sh: Newer versions of filefrag output more fields, in different orders, so handle that to avoid warnings. 2013-12-09 Pádraig Brady build: update gnulib to fix a tests build failure * gnulib: Update to incorporate a fix for building tests with -Werror=old-style-declaration tests: fix false failure due to xargs usage * tests/readlink/multi.sh: Ensure there is a trailing delimeter or xargs may (on AIX 7 at least) suppress the last argument. tests: fix false failure on systems without /dev/stdin * tests/rm/interactive-once.sh: Ensure the expected output matches with the output on systems without /dev/stdin (like AIX 7). Also change some fail=1 to a more appropriate framework_failure_. build: update gnulib and improve --with-openssl configure help * configure.ac: Use gl_SET_CRYPTO_CHECK_DEFAULT() to set the coreutils default for --with-openssl early, so that the help text can be provided in a standard and complete form. * gnulib: Update to incorporate a build fix on platforms with only some of md5 and sha* implemented by libcrypto. 2013-12-09 Pádraig Brady tests: fix false failure due to gdb inline function handling Inline functions are awkward to breakpoint as mentioned at: https://sourceware.org/bugzilla/show_bug.cgi?id=10738 The normal case here was for the breakpoint on the inline function to fail, resulting in a 10s delay before skipping the test. However on GCC 4.7.2 on ppc64 at least it was seen that the test failed erroneously due to the breakpoint being successfully set on an "out of line" function, but an inline function was actually being called. * tests/tail-2/inotify-race.sh: Switch to a line based breakpoint, rather than a symbol base one, which avoids issues with breakpoints on inline functions. Also skip_ on the initial breakpoint check in case the breakpoint is not traversed which would be the case on remote file systems for example. 2013-12-09 Pádraig Brady tail: use consistent diagnostics with and without inotify * src/tail.c: With inotify, when a file is initially absent, we fstat(-1) for that file spec, thus recording an errnum of EBADF, which caused the "has become accessible" diagnostic to be issued, when the file first appears. Instead we avoid the fstat(-1) and thus emit the more natural and consistent "has appeared" diagnostic. * tests/tail-2/retry.sh: Use the new diagnostic which also causes this test to pass on systems without inotify. 2013-12-09 Bernhard Voelker tests: avoid false df failure with nfs and lofs * tests/df/total-unprocessed.sh: -t nfs and --local are _not_ mutually exclusive on solaris, with lofs mounts. 2013-12-07 Pádraig Brady tests: avoid problematic $(< file) shell construct * tests/cp/link-deref.sh: This was seen to cause an issue with dash v0.5.6 at least, so change to a simpler expression. tests: fix false failure on platforms using newfstatat * tests/ls/stat-free-color.sh: Add newfstatat to the list of syscalls to trace. Also add all "stat" syscalls to the list of syscalls that we verify that strace supports. Also only create a single dangling symlink to check, since we already only check for a single "stat" call. Fixes http://bugs.gnu.org/16075 seen on AArch64 tests: avoid a false memory leak test failure * tests/misc/shuf-reservoir.sh: Restrict the valgrind "exit on leak" behavior to developer environments where specific "lint" code is enabled to avoid inconsequential leaks. 2013-12-06 Bernhard Voelker tests: fix shuf test verifying that -i and -e can not be combined Previously, the test triggered another error diagnostic: shuf: invalid input range ‘-e’ and therefore eclipsed the expected one: shuf: cannot combine -e and -i options While at it, reindent a line with more than 80 characters, present since the previous commit to silence sc_long_lines. * tests/misc/shuf.sh: Pass a valid range to the -i option. 2013-12-06 Paul Eggert shuf: --repeat, not --repetitions; default --head-count is infinity Original problem reported by Philipp Thomas in . * NEWS: shuf --repeat, not shuf --repetitions. * doc/coreutils.texi (shuf invocation): * src/shuf.c (usage, long_opts, main): * tests/misc/shuf.sh: Likewise. Also, the default head-count is infinity. 2013-12-05 Bernhard Voelker doc: clarify that df now generally processes special files correctly Since v8.21-172-g33660b4, df not only treats symbolic link arguments differently, as stated there, but now generally processes special file arguments in a non-canonicalized form correctly: $ cd /dev && df-old sdb Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1014572 48 1014524 1% /dev $ cd /dev && df-new sdb Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdb 10190136 6039532 3609932 63% /home Document df's new behavior. * doc/coreutils.texi (df invocation): In the paragraph describing df's behavior regarding special file arguments, relax the condition for such special files from "... is an absolute name of ..." to "... resolves to ...". * NEWS (Bug fixes): Mention the new behavior also here. 2013-12-04 Pádraig Brady tests: fix false failure with disabled SELinux support This could trigger on SELinux systems where we build --qithout-selinux or where the SELinux development libraries are not installed. * init.cfg (require_selinux_enforcing_): Call require_selinux_() to determine if the current build supports SELinux. This avoids a false failure in tests/mkdir/selinux.sh where only mkdir would determine that SELinux was disabled and thus ignore invalid contexts. (require_selinux_): Refactor a little to distinguish whether it's the build or the (file) system that doesn't support SELinux. 2013-12-04 Pádraig Brady selinux: fix --context=CTX for cp and diagnose defaultcon() errors * src/selinux.h (ignorable_ctx_err): A new function used to determine if a warning should be given after a call to defaultcon() or restorecon(). * src/cp.c (main): Fix the setfscreatecon() call to use the argument passed by the user. * src/mkdir.c (make_ancestor): Show all but "ignoreable" errors from defaultcon() and restorecon(). * tests/misc/selinux.sh: Add a test run as root in selinux enforcing mode, to ensure cp --context=invalid is honored and fails immediately. 2013-12-04 Pádraig Brady md5sum, sha*sum: use libcrypto where available libcrypto is generally available and has well optimized crypto hash routines particular to various systems. For example, testing sha1sum with openssl-1.0.0j on an i3-2310M, gives a performance boost of about 40%: $ time sha1sum.old --tag ~/test.iso SHA1 (/home/padraig/test.iso) = 3c27f7ed01965fd2b89e22128fd62dc51a3bef30 real 0m4.692s user 0m4.499s sys 0m0.162s $ time sha1sum.new --tag ~/test.iso SHA1 (/home/padraig/test.iso) = 3c27f7ed01965fd2b89e22128fd62dc51a3bef30 real 0m2.685s user 0m2.512s sys 0m0.170s * configure.ac: By default, enable use of libcrypto if available. * src/local.mk: Link with libcrypto. * NEWS: Mention the md5sum and sha*sum improvements. 2013-12-04 Pádraig Brady df: handle bind mounts when processing device nodes * src/df.c (get_disk): Use the same heuristic used in get_point() to select the shortest file system mount point, in an attempt to show the base mounted file system. * NEWS: Mention the bug fix. 2013-12-04 Pádraig Brady df: dereference symlinks to disk device nodes This is so the matching for the device is done on the canonical name of the disk node, rather than on the path of the symlink. In any case the user will generally want to use the symlink target. * src/df.c (get_disk): Canonicalize the passed file, before matching against the list of mounted file system devices. Note we pass the original symlink name to the "file" output field, as the symlink target is usually available through the "source" field. * tests/df/df-symlink.sh: Test the dereferencing operation. * tests/local.mk: Mention the new test. * NEWS: Mention the fix. Reported by Ondrej Oprala 2013-12-04 Pádraig Brady build: update gnulib submodule to latest Note tests/init.sh and bootstrap are still in sync with gnulib. * gnulib: Sync two configure check fixes. - Avoid generating core dumps from regex configure check - Fix compile error in getcwd configure check 2013-12-03 Pádraig Brady build: update gnulib submodule to latest Note tests/init.sh and bootstrap are still in sync with gnulib. * gnulib: Sync many fixes/changes, including the base64 encoding speedup mentioned in commit v8.21-145-g9120845 Also included is support for enabling use of libcrypto hash routines which are generally well optimized for particular systems. 2013-12-03 Bernhard Voelker maint: avoid '%s' quoting notation in diagnostic messages Add a new rule to ensure the use of quote() instead of '%s' or `%s' in format strings of diagnostics messages. * cfg.mk (sc_prohibit_quotes_notation): Add rule. * TODO: Remove the entry regarding the '%s' notation. * src/mkfifo.c (main): Remove the offending and in this case even duplicate quoting in the format string of the error diagnostic. * src/mknod.c (main): Likewise. * src/df.c (decode_output_arg): Change two invocations of error() according to the above new rule. * src/numfmt.c: Fix numerous wrong quote notations to fit the above new rule, mostly in internal debugging diagnostic messages. 2013-12-02 Pádraig Brady doc: remove obsolete info on terminal alignment bugs * doc/coreutils.texi (ls invocation): Remove the note about OSX terminals not aligning properly as this is no longer the case. Tested by: Philipp Thomas 2013-11-29 Pádraig Brady build: fix potential factor build failure on arm and powerpc * src/longlong.h: Sync with the latest longlong.h from libgmp to: - avoid arm asm when being compiled for the thumb instruction [sub]set - avoid old powerpc assembly that is incompatible with newer GCC - add arm64 optimized count_trailing_zeros() - add sparc64 optimized add_ssaaaa() and umul_ppmm() 2013-11-29 Bernhard Voelker tests: add a test for rm -r "/" * tests/rm/r-root.sh: Add a non-root test. * tests/local.mk (all_tests): Mention the test. 2013-11-29 Bernhard Voelker maint: remove various FIXME comments * src/extent-scan.c (extent_need_sync): Remove the FIXME comment about removing the work around. As discussed in [1], the needed FIEMAP fix never made it into the Linux kernel. * src/realpath.c (longopts): Remove the FIXME comment about deprecating the --strip option as it's a valid alias for --no-symlinks option. Also discussed in [1]. [1] http://lists.gnu.org/archive/html/coreutils/2013-11/msg00103.html 2013-11-29 Bernhard Voelker df: remove long-obsolescent --megabytes option This option has been undocumented for 12 years [1], and warned about for a year [2]. [1] commit FILEUTILS-4_1_4-23-gd177203 [2] commit v8.17-43-g453ce92 * src/df.c (MEGABYTES_OPTION): Remove. (long_options): Remove "megabytes" element. (main): In the option parsing loop, remove the MEGABYTES_OPTION case. * NEWS: Mention the change. 2013-11-27 Pádraig Brady build: fix selinux.c build failure on 32 bit * src/selinux.c: Don't include the system "fts.h" as that disallows _FILE_OFFSET_BITS=64 which gnulib auto enables to support large files on 32 bit systems. Instead include our "xfts.h" which includes the less limited gnulib replacement, and also a checked version of xfts_open(). (restorecon): Use the checked xfts_open() rather than the standard fts_open(). Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/6934169 2013-11-27 Pádraig Brady build: avoid more selinux build failures Handle both newer selinux libraries with mode_to_security_class(), and systems without selinux at all. We could easily adjust gnulib to provide the necessary stubs for use by this module, but it's more efficient to just stub out the module completely, when not using selinux. * src/selinux.h: Define stubs for the two module functions, when SELinux is not available. * src/selinux.c: Exclude all logic in preference for the stubs when selinux isn't used. Also when newer selinux libs are used, don't use our conflicting static version of mode_to_security_class(). m4/jm-macros.m4: Check for the system mode_to_security_class(). 2013-11-27 Pádraig Brady build: avoid build failure on non selinux systems * src/selinux.c: This module introduced in commit v8.21-159-gd8e27ab doesn't need to include . That header file isn't catered for by gnulib, but is not needed as we're not explicitly referencing any class IDs. Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/6920020 2013-11-27 Pádraig Brady selinux: adjust utils to run restorecon with -Z cp, mv, install, mkdir, mkfifo, mknod are adjusted so that: -Z no longer accepts an argument. -Z or --context without an argument do not warn without SELinux. --context with an argument will warn without SELinux. * src/local.mk: Reference the new selinux module where required. * src/system.h: Make the argument to --context optional. * src/mkdir.c: Likewise. Also handle the SMACK case for --context. Note we currently silently ignore -Z with SMACK. * src/mkfifo.c: Likewise. * src/mknod.c: Likewise. * src/install.c: Likewise. Note install(1) by default already set the context for target files to their system default, albeit with an older method. Use the -Z option to select between the old and new context restoration behavior, and document the differences and details for how context restoration is done in new and old methods, with a view disabling the old method entirely in future. * src/cp.c: Make the argument to --context optional. Note -Z implies --no-preserve=context. I.E. -Z overrides that aspect of -a no matter what order specified. (struct cp_options): Document the context handling options. (main): Check/adjust option combinations after all options are processed, to both simplify processing and to make handling independent of order of options on the command line. Also improve the diagnostics from a failed call to setfscreatecon(). (set_process_security_ctx): A new function, refactored to set the default context from the source file, or with the type adjusted as per the system default for the destination path. (set_file_security_ctx): A new function refactored to set the security context of an existing file, either based on the process context or the default system context for a path. (copy_internal): Use the refactored functions to simplify error handling and consistently fail or warn as needed. (copy_reg): Likewise. (copy_internal): With --preserve=context, also copy context from non regular files. Note for directories this may impact the copying of subsequent files to that directory? (copy_attr): If we're handling SELinux explicitly, then exclude to avoid the redudant copy with --preserve=context, and the problematic copy with -Z. Note SELinux attribute exclusion also now honors cp -a --no-preserve=context. Note there was a very small window over 10 years ago, where attr_copy_file was available, while attr_copy_check_permissions was not, so we don't bother adding an explicit m4 check for the latter function. * src/mv.c: Support --context, but don't allow specifying an argument. * src/chcon.c: Adjust a comment to be specific to SELinux. * src/runcon.c: Likewise. * src/copy.c: Honor the context settings to "restorecon" as appropriate. * src/copy.h: Add a new setting to select "restorecon" functionality. * tests/mkdir/selinux.sh: s/-Z/--context=/ * tests/cp/cp-a-selinux.sh: Augment this test with cases testing basic -Z functionality, and also test the various invalid option combinations and option precedence. * tests/mkdir/restorecon.sh: Add a new test for the more involved mkdir -Z handling, since the directory changing and non existent directories need to be specially handled. Also check the similar but simpler handling of -Z by mk{nod,fifo}. * tests/local.mk: Reference the new test. * doc/coreutils.texi (cp invocation): Update as per interface changes. (mv invocation): Likewise. (install invocation): Likewise. (mkfifo invocation): Likewise. (mknod invocation): Likewise. (mkdir invocation): Likewise. * NEWS: Mention the new feature and change in behavior. 2013-11-27 Daniel J Walsh selinux: a new module implementing "restorecon" functionality * src/selinux.c: A new module implementing "restorecon" functionality. There are two main functions to adjust the type of the referenced file system item. defaultcon() will setup the process context so that new items will have the required context without races. This is the preferred method. For existing files, the equivalent restorecon() is available which has two modes. With the "local" parameter set to false, restorecon() will adjust the type according to the system configuration for that file, and set to true will update the context as per the context for the current process (disregarding type). * src/selinux.h: Likewise. * po/POTFILES.in: Reference the new module. 2013-11-26 Pádraig Brady shred: provide --remove methods to avoid excessive syncing A sync operation is very often expensive. For illustration I timed the following python script which indicated that each ext4 dir sync was taking about 2ms and 12ms, on an SSD and traditional disk respectively. import os d=os.open(".", os.O_DIRECTORY|os.O_RDONLY) for i in range(1000): os.fdatasync(d) So syncing for each character for each file can result in significant delays. Often this overhead is redundant, as only the data is sensitive and not the file name. Even if the names are sensitive, your file system may employ synchronous metadata updates, which also makes explicit syncing redundant. * tests/misc/shred-remove.sh: Ensure all the new parameters actually unlink the file. * doc/coreutils.texi (shred invocation): Describe the new parameters to the --remove option. * src/shred.c (Usage): Likewise. (main): Parse the new options. (wipename): Inspect the new enum to see which of the now optional tasks to perform. * NEWS: Mention the new feature. * THANKS.in: Add reporter Joseph D. Wagner 2013-11-26 Pádraig Brady tail: improve inotify handling of symlinks Previous behavior failed to read contents of a (re)appearing file, when symlinked by tail's watched file. Also we now diagnose other edge cases when running in inotify mode, where an initially missing or regular file changes to a symlink. * src/tail.c (main): If any arg is a symlink, use polling mode. (recheck): Diagnose the edge case where a symlink appears during inotify processing. * tests/tail-2/symlink.sh: Test the fix. Mention the edge cases. * tests/local.mk: Reference the new test. * NEWS: Mention the fix. Reported by: Ondrej Oprala 2013-11-26 Pádraig Brady df: add --output=file to directly output specified arguments * src/df.c (usage): Document the new 'file' --output field. (get_dev): Add a new parameter to pass the specified argument from the command line through. Use '-' if a command line parameter is not being used. * doc/coreutils.texi (df invocation): Describe the new 'file' field. * tests/df/df-output.sh: Adjust all fields test, and add a specific test for --output=file. * NEWS: Mention the new feature. timeout: avoid unlikely issues with --kill-after * src/timeout.c (cleanup): When calling settimeout() from this signal handler, ensure we don't call out to error() or gettext(), which are not async-signal-safe. Also reset the errno which may be cleared by settimeout(). tests: ensure factor tests can be regenerated * tests/local.mk (factor-tests:) Add -f to the mv command that replaces any existing generated tests. This is required to avoid prompts when root initially generates the tests, and they subsequently need to be regenerated by a non root user. sort: avoid issues when issuing diagnostics from child processes * src/sort.c: (async_safe_die): A new limited version of error(), that outputs fixed strings and unconverted errnos to stderr. This is safe to call in the limited context of a signal handler, or in this particular case, between the fork() and exec() of a multithreaded process. (move_fd_or_die): Use the async_safe_die() rather than error(). (maybe_create_temp): Likewise. (open_temp): Likewise. Fixes http://bugs.gnu.org/15970 doc: clarify the operation of ls -k * src/ls.c (usage): Mention -k only changes the display for disk usage (directory total, and ls -s), and imply that it can be overridden (by --block-size, and -h). * doc/coreutils.texi (block size): Mention that ls -k handling is different to other utilities. Addresses http://bugs.gnu.org/14525 2013-11-22 John dircolors: add a new entry to colorize 'lz4' files * src/dircolors.hin: Add entry for the speed optimized 'lz4' compressor. 2013-11-22 Bernhard Voelker tests: enhance rm test regarding "." and ".." Recent commit 2da7009d changed the error diagnostic of rm(1) trying to remove "." or "..". Enhance the corresponding test. * tests/rm/r-4.sh: Ensure rm(1) outputs the expected error diagnostic. 2013-11-21 Bernhard Voelker maint: avoid perl warning in sc_check-AUTHORS With newer perl, "make syntax-check" issues many warnings like: -i used with no filenames on the command line, reading from STDIN. * cfg.mk (sc_check-AUTHORS): Remove the -i flag in the perl invocation as it is reading from a pipe. 2013-11-21 Bernhard Voelker doc: enhance diagnostic when rm skips "." or ".." arguments The error diagnostic "rm: cannot remove directory: '.'" does not give the user a hint for the reason. Issue a clearer error message. * src/remove.c (rm_fts): Enhance the error diagnostic in the above case to emphasize that skipping is done deliberately. In the corresponding comment, mention that POSIX mandates this behavior. Likewise in the subsequent comment for skipping "/". * doc/coreutils.texi (rm invocation): In the paragraph describing the above behavior, mention that POSIX mandates it. 2013-11-17 Pádraig Brady tests: fix false rm -I test failure when run as root Fix a recent regression introduced in commit v8.21-127-g5ee7d8f Also related to this is the recent query about root run `rm -I` ignoring the mode bits of a file: https://bugzilla.redhat.com/1013171 * tests/rm/interactive-once.sh: Avoid the messages and corresponding file presence checks with write protected files when running as root. 2013-11-17 Colin Leitner stty: add support for mark/space parity This adds support for using a constant or "stick" parity bit. * src/stty.c (usage): Mention the new flag. * tests/misc/stty.sh: Adjust for the new flag. * NEWS: Mention the improvement. * docs/coreutils.texi (stty invocation): Mention the new flag. 2013-11-11 Pádraig Brady base64: improve encoding I/O efficiency Since the I/O overhead is significant to the relatively simple processing done by this utility, use fputs() rather than fputc() to output '\n'. Time to process a 100MiB file was measured to decrease from 0.417s to 0.383s, i.e. an 8% improvement. Related to these changes, is a processing improvement in gnulib, which increases throughput by 60% when processing full buffers, which improves processing of a 100MiB file with standard wrapped output, down to 0.256s. http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=commit;h=43fd1e7b Also increase the encoding buffer size from 3 to 30KiB. This was seen to give a further 8% improvement, taking processing time down to 0.235s in the wrapped output case. The decoding size buffer is not adjusted, due to the noted caveat with --ignore-garbage. * src/base64.c (BLOCKSIZE): Split into ENC_ and DEC_ variants, with the former increased from 3KiB to 30KiB. (wrap_write): Use the simpler fputc() rather than fputs() to output the '\n' character. Also check against EOF rather than < 0 for errors. (do_encode): Likewise. * NEWS: Mention the large increase in performance, which with the I/O improvements in coreutils and the processing improvement in gnulib, amount to about a 60% throughput increase. 2013-11-08 Pádraig Brady shred: avoid data write pass with --size=0 * src/shred.c (dopass): Exit early to avoid redundant heap allocation, and more importantly avoiding a file sync when we're writting no data, as this can have side effects. Also with --verbose, this avoids printing of "pass status" which could be confusing as to whether data was actually written. * tests/misc/shred-passes.sh: Ensure the status for data passes are not written when not doing any data writes. shred: increase I/O block size for periodic pattern case * src/shred.c (dopass): In the periodic pattern case increase the I/O block size from 12KiB to 60KiB (also a multiple of 3 and 4096). * NEWS: Adjust accordingly. 2013-11-08 Pádraig Brady shred: fix direct I/O failures for last write to file Since direct I/O is now enabled with commit v8.21-139-gebaf961 we must handle the case where we write an odd size at the end of a file (with --exact), or we specify an odd --size that is larger than 64KiB, or in the very unlikely case of a device with an odd size. This issue was present since direct I/O support was first added in v5.3.0, but latent since v6.0. Theoretically this could have also been an issue after that on systems which didn't have alignment constraints, but did have size constraints for direct I/O. * src/shred.c (dopass): On the first pass for a file, always retry a write that fails with EINVAL, so we handle direct I/O failure at either the start or end of the file. Adjust the comment as the original case is out of date and implicitly handled by this more general fix. * tests/misc/shred-exact.sh: Add a test case. * NEWS: Add a "bug fix" entry for shred since there are two related issues now fixed. 2013-11-07 Gian Piero Carrubba Bernhard Voelker cp: fix --link regarding the dereferencing of symbolic links * src/copy.c (create_hard_link): Add a bool 'dereference' parameter, and pass AT_SYMLINK_FOLLOW as 'flags' to linkat() when dereference is true. (should_dereference): Add new 'bool' function to determine if a file should be dereferenced or not. (copy_internal): Use the above new should_dereference() and remember its return value in a new local bool 'dereference' variable. Use that in all three calls to create_hard_link(). * src/cp.c (main): after parsing the options, if x.dereference is still DEFEF_UNDEFINED and the x.recursive is true, then only set x.dereference to DEREF_NEVER iff --link was not specified. * doc/coreutils.texi (cp invocation): Mention that cp(1) does not follow symbolic links in the source when --link is specified. Likewise in the description of the -R option when used together with that option. * tests/cp/same-file.sh: Adapt the expected results for the -fl, the -bl and the -bfl tests. * tests/cp/link-deref.sh: Add a new test. * tests/local.mk (all_tests): Reference the above new test. * NEWS (Changes in behavior): Mention the change. This fixes http://bugs.gnu.org/15173 2013-11-06 Pádraig Brady shred: write larger chunks when possible * src/shred.c (dopass): When not needing to write periodic patterns, use a 64KiB block size to reduce the number of write system calls. 2013-11-05 Pádraig Brady shred: enable direct I/O when possible Commit v5.92-1057-g43d487b introduced a regression in coreutils 6.0 where it removed the page alignment of the buffer to write, thus disabling direct I/O. We want to use direct I/O when possible to avoid impacting the page cache at least, as we know we don't want to cache the data we're writing. * src/shred.c (dopass): Allocate the buffer on the heap, while using a more general calculation to allow to have the output size independent from the fillpattern() size constraint of a multiple of 3. Also we dispense with the union as it's no longer needed given we're aligning on a page boundary and thus don't need to explicitly handle uint32_t alignment. 2013-11-01 Pádraig Brady md5sum, sha*sum: improve help for --check related options * src/md5sum.c (usage): s/three/four/ in the message pertaining to the --check related options. Also clarify that --strict is just significant for the formatting of the checksum lines. Also since we're changing both strings, move the --strict description in with the description of the other options and order alphabetically. * THANKS.in: Added reporter: Daniel Mach maint: simplify printing of md5sum file names * src/md5sum.c (main): Add a comment as to why we continue to escape names that do not have '\n' but do have '\\' chars. (print_filename): Use the predetermined boolean to decide whether to escape or not, so that in the common case we can output the file name directly, rather than inspecting each char. * tests/misc/md5sum.pl: Add case to show '\\' chars cause escaping. * tests/misc/sha1sum.pl: Likewise. 2013-11-01 Bernhard Voelker build: fix dependencies of man/sha*sum.1 Although the above man pages depend on src/md5sum.c as a shared source, the build of the man pages directly requires their own executables to exist. * man/local.mk (man/sha1sum.1): Change the dependency from 'src/md5sum' to 'src/sha1sum'. (man/sha224sum.1): s/md5sum/sha224sum/ (man/sha256sum.1): s/md5sum/sha256sum/ (man/sha384sum.1): s/md5sum/sha384sum/ (man/sha512sum.1): s/md5sum/sha512sum/ Reported by Pádraig Brady in http://lists.gnu.org/archive/html/coreutils/2013-11/msg00006.html 2013-10-15 Mike Frysinger dircolors: add putty-256color terminal type * src/dircolors.hin: Add putty-256color Reported-by: Thomas D. , via http://bugs.gentoo.org/486786 Fixes http://bugs.gnu.org/15624 2013-10-15 Pádraig Brady maint: update the instructions for building prerequisites * README-prereq: Update as per the latest required versions in bootstrap.conf. Also add a missing cd command. Reported by Aaron Davies Fixes http://bugs.gnu.org/15612 2013-10-11 Ondřej Vašík cp: correct error message for invalid arguments of '--no-preserve' * src/cp.c (decode_preserve_arg): Correct error message for invalid arguments of '--no-preserve'. Reported by M.Vadkerti in http://bugzilla.redhat.com/1018206 Fixes http://bugs.gnu.org/15588 2013-10-10 Pádraig Brady doc: clarify the example for cp --preserve=links * doc/coreutils.texi (cp invocation): Give more detail about what's happening in the example, explicitly calling out the --no-dereference option required to make the -H and -L options significant. Also mention the option order significance of the -H and -L options. Fixes http://bugs.gnu.org/15579 2013-10-07 Pádraig Brady mktemp: fix incorrect exit status from previous commit * src/mktemp.c (main): Use an exit() strategy consistent with the previous clauses dealing with optional error messages to ensure we exit with the correct status in all cases. Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/6412979 2013-10-06 Pádraig Brady mktemp: with --quiet, only suppress I/O errors The reason for having a --quiet option is to suppress only some subset of possible errors. The most useful separation here is with usage/internal errors, and errors due to file creation etc. (i.e. I/O errors). * src/mktemp.c (main): Match the --help and info docs and only suppress the file/dir creation error messages. * tests/misc/mktemp.pl: Adjust accordingly. 2013-10-06 Pádraig Brady mktemp: synchronize the -p option with docs * src/mktemp.c (usage): Synchronize the -p option description with the logic and info docs. I.E. that -p is just an alias of --tmpdir. Also for consistency treat --tmpdir='' the same with or without -t. I.E. always ignore the --tmpdir option if the param is empty. Fixes http://bugs.gnu.org/15425 2013-10-06 Karl Berry doc: expand the texinfo examples for the paste command * doc/coreutils.texi (paste invocation): Move the synopsis to the top. Provide examples for the different type of operations possible. Add a specific common example to join consecutive lines with a space. 2013-09-26 Sergio Durigan Junior rm: with -I, prompt before deleting a write protected file This regression was introduced in commit v6.7-71-g0928c24 * src/rm.c (main): Make the -I option behave like --interactive=once. * tests/rm/interactive-once.sh: Add cases for single and multiple files. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/9308 2013-09-21 Pádraig Brady maint: move all id(1) tests to the same directory * tests/misc/id-context.sh -> tests/id/context.sh * tests/misc/id-setgid.sh -> tests/id/setgid.sh * tests/misc/id-uid.sh -> tests/id/uid.sh * tests/misc/id-zero.sh -> tests/id/zero.sh * tests/local.mk: Reference the renamed tests tests: parameterize the "root" username * tests/misc/chroot-credentials.sh: Don't assume uid 0 has the "root" name, nor any name for that matter. maint: remove unused parameters to parse_user_spec() * src/chown.c (main): Since "name" parameters to parse_user_spec() are now optional, just pass NULL for those unused parameters. * src/chroot.c (main): Likewise. id: support specifying the user by user ID * src/id.c (usage): Remove 'name' from the synopsis, implying that one can also specify by user ID. (main): Like chown(1), call parse_user_spec() to implement user name or ID lookup with appropriate precedence. * doc/coreutils.texi (id invocation): Mention that a user ID is supported and how '+' affects lookup order. * tests/misc/id-groups.sh: Remove test now subsumed into... * tests/misc/id-uid.sh: New test covering new interface. * tests/local.mk: Rename the test. * NEWS: Mention the new feature. Addresses http://bugs.gnu.org/15421 2013-09-21 Pádraig Brady build: update gnulib submodule to latest; also bootstrap to latest Notes tests/init.sh is still in sync with gnulib. * bootstrap: Update to latest. * gnulib: Sync many fixes/changes, including: a fix for http://bugs.gnu.org/15066 and preparation for http://bugs.gnu.org/15421 2013-09-21 Pádraig Brady tests: avoid a failure when there isn't a name for all user IDs * tests/misc/id-zero.sh: Don't check exit status when in -n mode. Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/6196762 2013-09-21 Bernhard Voelker id: add -z, --zero option * src/group-list.h (print_group_list): Add a parameter for the delimiter of type char. * src/group-list.c (print_group_list): Likewise, and use it instead of a white space character to delimit the group entries. * src/groups.c (main): Pass white space character to print_group_list(). * src/id.c (longopts): Add array element for the new long option. (usage): Document the new option. While at it, fix the alignment of the descriptions to match that of HELP_OPTION_DESCRIPTION. (main): Define the bool flag opt_zero indicating the use of the new option. In the getopt_long loop, handle it. Output an error diagnostic in the case the --zero option has been specified together with the default format. In the case of -gG, pass either a NUL or a white space character to print_group_list() - depending on the above new flag. Likewise change the printing of the final newline character: output a NUL instead if the --zero option has been specified. * doc/coreutils.texi (id invocation): Document the new option. While at it, move the @exitstatus macro down after the macro @primaryAndSupplementaryGroups in order to be consistent with other texinfo documents. (groups invocation): Move @exitstatus down after the macro @primaryAndSupplementaryGroups here, too. * tests/misc/id-zero.sh: Add new test exercising the new option. * tests/local.mk (all_tests): Reference it. * NEWS (New features): Mention the new option. Fixes http://bugs.gnu.org/9987 2013-09-11 Bernhard Voelker maint: avoid patching help2man Commit cde1ea0e separated the coreutils-specific patches from help2man. Most changes had been made to accommodate to the coreutils style guide, i.e., to avoid syntax-check failures like sc_long_lines. Yet 2 changes had to be put into the patch help2man.diff. But this added the dependency to patch(1) in distribution builds. Incidentally, the 2 remaining parts of the patch can easily be done outside of help2man. Therefore, this commit partly reverts the recent separation of help2man into 'help2man.in' and 'help2man.diff', and instead uses the original help2man script. * man/help2man.in: Rename to ... * man/help2man: ... this file. * man/help2man.diff: Remove. * man/local.mk (mandeps): Remove man/help2man. (man/help2man): Remove recipe. (.x.1): Add the --info-page option when calling help2man in order to change the name of the texinfo manual from the default, "info PRG", to "info coreutils 'PRG invocation'". Furthermore, use an sed pattern to remove the sentence starting with "For complete documentation". * .gitignore (/man/help2man): Remove entry. * .x-update-copyright: Replace the entries for the files 'man/help2man.diff' and 'man/help2man.in' by 'man/help2man'. * cfg.mk (sc_long_lines): Instead of 'man/help2man.in', exempt 'man/help2man' from this test. (sc_po_check): Likewise. (sc_space_tab): Instead of 'man/help2man.diff', exempt 'man/help2man' from this test. (sc_trailing_blank): Likewise. (sc_prohibit_tab_based_indentation): Instead of 'man/help2man.in' and 'man/help2man.diff', exempt 'man/help2man'. * man/dummy-man: Recognize the option --info-page=... as no-op. 2013-09-11 Bernhard Voelker maint: use help2man configured with --disable-nls Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/6038769 The previously committed 'help2man' requires a Perl module which does not seem to be installed everywhere - and which is not needed for our purposes: Can't locate Locale/gettext.pm in @INC BEGIN failed--compilation aborted at ./man/help2man line 28. This module was pulled in automatically by the default configure call. Use the NLS-disabled version instead. * man/help2man.in: Use help2man configured with the --disable-nls option to avoid the dependency to the above Perl module. * man/help2man.diff: Adapt the line numbers of the hunks in the coreutils-specific patch for help2man to apply without fuzz. Reported by Pádraig Brady. 2013-09-10 Pádraig Brady build: fix corrupted patch causing bootstrap failures The patch was corrupted in commit v8.21-50-g7b65f8e * gl/modules/tempname.diff: Fix the offsets so that the patch applies cleanly. Note that this was only apparent with patch < 2.6. With patch >= 2.6, patch will not indicate an error applying the second hunk of the patch and silently ignore it. I double checked that all patches now apply cleanly by adjusting gnulib-tool to run patch with --fuzz=0 which might be advisable going forward, even on a per project basis. The silent ignoring of hunks by newer patch(1) has been reported. Fixes http://bugs.gnu.org/15255 2013-09-09 Bernhard Voelker maint: update help2man to 1.43.3 Instead of diverging further from the upstream GNU help2man project (http://www.gnu.org/software/help2man/), hold a copy of the original script and keep track of our changes in a separate patch file. The man pages created with the new version show the following, non-invasive differences: a) command options in the EXAMPLE sections are no longer in bold format, b) file names are underlined now consistently. * man/help2man: Rename to ... * man/help2man.in: ... this file, and update content from the upstream GNU help2man project. * man/help2man.diff: Add patch file for help2man to remove the sentence "For complete documentation ..." (see commit 5d4f09d8), and to emit "info coreutils 'PROG invocation'" into the man pages (77abf69a). * man/local.mk (mandeps): Add help2man to the dependencies of the man pages. (man/help2man): Add rule to generate this script from the upstream help2man.in file and the help2man.diff patch. * .gitignore: Add man/help2man as it is no longer version controlled. * cfg.mk (sc_long_lines): Exempt help2man.in from this check. (sc_po_check): Likewise. (sc_space_tab): Likewise. (sc_trailing_blank): Exempt man/help2man.diff from this check. (sc_prohibit_tab_based_indentation): Instead of help2man, now exempt both help2man.in and help2man.diff from this test. * .x-update-copyright: Add new file and add the above new help2man files as well as the COPYING file. 2013-09-04 FUJIWARA Katsunori ls: fix possible incorrect exit status when recursing directories If there is an error reading a directory that was referenced through recursion, rather than directly on the command line, then exit with the "less serious" exit code, rather than the "serious" exit code reserved for command line arguments. This issue was introduced in commit v5.2.1-1908-gb58dea5 * src/ls.c (print_dir): Ensure that the command_line_arg param is false for directories being recursed into. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/15249 2013-09-01 Pádraig Brady maint: update out of date confusing comments * src/copy.c (copy_internal): Change mention of the removed --reply=no option, to the similar in this context --no-clobber. * src/sort.c: SI and IEC suffixes can now be mixed when --human-numeric. 2013-08-28 Pádraig Brady df: fix "blocks" translation in header * src/df.c (get_header): Get the translation of "blocks" here, rather than just marking the string for translation. Fixes http://bugs.gnu.org/15054 2013-08-10 Bernhard Voelker doc: clarify when seq ends * doc/coreutils.texi (seq invocation): Add a sentence clarifying that seq terminates when LAST becomes smaller than the current number plus INCREMENT. * src/seq.c (usage): Likewise. Fixes http://bugs.gnu.org/15068 2013-08-07 Bernhard Voelker doc: use more modern file system type examples in df description * doc/coreutils.texi (df invocation): In the example list of common file system types, exchange the entries which are not so common anymore (4.2, ufs, efs, hsfs, pcfs) by far more prominent ones (ext2, ext3, ext4, xfs, btrfs, iso9660, ntfs, fat). 2013-08-07 Filipus Klutiero doc: fix typo in description of df * doc/coreutils.texi (df invocation): s/pseude/pseudo/ * THANKS.in (Filipus Klutiero): Remove entry, now that it will be automatically included in the generated THANKS file. Fixes http://bugs.gnu.org/15041 2013-08-04 Benno Schulenberg doc: more semicolons instead of periods in option descriptions Also slightly rephrase some descriptions for extra clarity, and add more consistent indentation. * src/df.c (usage): Semicolon, no final period. * src/du.c (usage): Likewise, plus indentation and clarifying words. * src/ls.c (usage): Semicolon, rephrasings, added parentheses for clarity, indentation. * src/rm.c (usage): Semicolons. * src/tail.c (usage): Adjust -f description to prefer explanatory language instead of option syntax. 2013-08-03 Pádraig Brady doc: clarify that uniq -d outputs a single entry per group * src/uniq.c (usage): Clarify the -d option. Fixes http://bugs.gnu.org/14996 2013-08-01 Bernhard Voelker maint: update copyright year number ranges of numfmt sources Run "make update-copyright". * src/numfmt.c: Update copyright year number range. This file has obviously been added to coreutils after and without the annual update. * tests/misc/numfmt.pl: Likewise. 2013-08-01 Benno Schulenberg doc: use semicolon instead of period in option descriptions Also do not end option descriptions with a period, properly indent continuation lines, and make some tiny clarifications. * src/du.c (usage): Lowercase after semicolon. * src/ls.c (usage): Semicolons instead of periods, small rephrasing and two hyphens for clarity, proper indentation. * src/mktemp.c (usage): Semicolons and lowercase. * src/od.c (usage): Semicolons. * src/ptx.c (usage): Use the standard phrase, clarify default option. * src/setuidgid.c (usage): Properly indent continuation line. * src/split.c (usage): Semicolons, lowercase, no final period. * src/stat.c (usage): Semicolons, lowercase. * src/tail.c (usage): Proper indentation, one shorter rephrasing, semicolons, no final periods. * src/timeout.c (usage): Properly indent, semicolons, no final periods. Fixes http://bugs.gnu.org/14976 2013-08-01 Pádraig Brady doc: clarify that cp by default doesn't adjust existing file perms * doc/coreutils.texi (cp invocation): Mention explicitly what happens to permissions of existing files when -p is not specified. Fixes http://bugs.gnu.org/14972 2013-07-28 Bernhard Voelker tests: fix another test failure in test/du/inodes.sh Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/5584121 du(1) uses the first file object of the two test files linked to the same inode, 'd/f' and 'd/h', whatever the system returns first. Use 'd/f' in both the expected and the actual output. * test/du/inodes.sh: Change the expected output as described above when returning the --all directory entries (without -l). Also replace the name of the hardlink 'd/h' by 'd/f' in the actual output. 2013-07-28 Paul Eggert doc: fix typo * doc/coreutils.texi (du invocation): @itemx -> @item. 2013-07-28 Bernhard Voelker maint: cleanup compare calls in test/du/inodes.sh * test/du/inodes.sh: In the cases where compare() fails, that function would show the unified diff automatically. Therefore, remove the excess "cat out". In the cases where expecting an empty file, use compare() again rather than the simpler "test -s" because possible error reports will then include the file's content for the same reason as above. 2013-07-27 Pádraig Brady maint: avoid clang static analysis issues in csplit * src/csplit.c (find_lines): Assert that load_buffer() updates the global buffers, thus "b" will be non NULL, thus suppressing subsequent NULL pointer derefence warnings. (process_regexp): Avoid a redundant assignment of the "line" pointer. (process_line_count): Likewise. Also reduce the "line" pointer scope. 2013-07-27 Pádraig Brady tests: fix intermittent failure in test/du/inodes.sh Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/5582213 * test/du/inodes.sh: Due to undefined order in returned directory entries, the expected output might not match, so sort both expected and actual output when returning --all directory entries. Also use a simpler test for ensuring no errors are output. 2013-07-27 Bernhard Voelker du: add --inodes option This new option can be used to find directories with a huge amount of files. The GNU find utility has the printf format "%h" which prints the number of entries in a directory, but this is non-cumulative and doesn't handle hard links. * src/du.c (struct duinfo): Add new member for counting inodes. (duinfo_init): Initialize inodes member with Zero. (duinfo_set): Set inodes counter to 1. (duinfo_add): Sum up the 2 given inodes counters. (opt_inodes): Add new boolean flag to remember if the --inodes option has been specified. (INODES_OPTION): Add new enum value to be used ... (long_options): ... here. (usage): Add description of the new option. (print_size): Pass inodes counter or size to print_only_size, depending on the inodes mode. (process_file): Adapt threshold handling: with --inodes, print or elide the entries according to the struct member inodes. (main): Add a case for accepting the new INODES_OPTION. Print a warning diagnostic when --inodes is used together with the option --apparent-size or -b. Reset the output_block_size to 1 ... and thus ignoring the options -m and -k. * tests/du/inodes.sh: Add a new test. * tests/local.mk (all_tests): Mention it. * doc/coreutils.texi (du invocation): Document the new option. * NEWS: Mention the new option. 2013-07-25 Ken Booth mv: replace empty directories in cross file system move src/copy.c (copy_internal): Use rmdir() rather than unlink() when the source is a directory, so that empty directories are replaced in the destination as per POSIX. * tests/mv/part-rename.sh: Augment with various combinations. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/14763 2013-07-22 Bernhard Voelker maint: use new gnulib function to free mount entries in du * src/du.c (fill_mount_table): Use free_mount_entry() instead of freeing struct members manually. 2013-07-21 Bernhard Voelker maint: make some shell and perl scripts executable in 'tests/' Some newer test scripts - partially ones from me - are not executable. It does not seem to be a problem, but for consistency and to avoid future problems on unusual platforms or shells change the permissions by adding the executable bit. * cfg.mk (sc_tests_executable): Add new syntax-check rule to ensure that all test scripts are executable. * tests/df/df-output.sh: Change file mode from 644 to 755. * tests/du/threshold.sh: Likewise. * tests/factor/run.sh: Likewise. * tests/init.sh: Likewise. * tests/misc/csplit-suppress-matched.pl: Likewise. * tests/misc/numfmt.pl: Likewise. * tests/tail-2/retry.sh: Likewise. 2013-07-19 Rasmus Villemoes rm: output number of arguments at the interactive prompt Include the number of arguments which rm received in the "Remove all arguments?" prompt. This is useful in the, presumably, common case where the arguments were not provided by hand, but instead were the result of various shell expansions. A simple, if somewhat contrived, example (assuming rm is aliased to rm -I) could be: rm * .o where the prompt "Remove 120 arguments?" is more likely to make the user catch the problem. * src/rm.c (main): Include correctly pluralized n_files in the output message. Also remove the now redudant "all". * tests/rm/interactive-always.sh: Adjust to the new prompt. * tests/rm/interactive-once.sh: Likewise. 2013-07-19 Pádraig Brady dd: make status=none suppress all diagnostics * src/dd.c (STATUS_NONE): Simplify the enum so that it's more general than just suppressing transfer counts. Then test this in all locations where non fatal diagnostics are output. * tests/dd/misc.sh: Ensure the diagnostic about being unable to skip past the end of input is suppressed. * NEWS: Mention the change in behavior. Fixes http://bugs.gnu.org/14897 2013-07-11 Pádraig Brady head: avoid redundant allocations when reading empty files * src/head.c (elide_tail_lines_file): For seekable empty files, or seekable files where the current offset is after the end of the file, return immediately. Previously the short circuit code could not be reached due to logic error. Spotted by coverity. 2013-07-11 Pádraig Brady maint: avoid a valgrind memory leak warning from pinky Similarly to commit v8.21-84-g8d2da3f in src/uptime.c avoid a "definitely lost" error from valgrind. Note this only happens with pinky when compiled without optimization, in which case certain paths aren't eliminated casuing valgrind to trigger the message. Note also that coverity flags this "resource leak" too. * src/pinky.c (short_pinky): free utmp_buf for developer builds. 2013-07-11 Anton Ovchinnikov df: reduce memory usage when filtering mount entries Avoid Valgrind reports of "definitely lost" items and while at it, free all discarded mount entries to minimize the amount of memory used. * src/df.c (filter_mount_list): Use the newly exported free_mount_entry() from gnulib to free all mount entries as they're discarded. 2013-07-11 Pádraig Brady build: update gnulib submodule to latest 2013-07-10 Pádraig Brady build: fix a build warning on 32 bit systems in shuf.c Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/5508873 * src/shuf.c (write_random_numbers): Convert to an int type that matches the prinft format spec. 2013-07-10 Assaf Gordon shuf: add --repetition to support repetition in output main(): Process new option. Replace input_numbers_option_used() with a local variable. Re-organize argument processing. usage(): Describe the new option. (write_random_numbers): A new function to generate a permutation of the specified input range with repetition. (write_random_lines): Likewise for stdin and --echo. (write_permuted_numbers): New function refactored from write_permuted_output(). (write_permuted_lines): Likewise. * tests/misc/shuf.sh: Add tests for --repetitions option. * doc/coreutils.texi: Mention --repetitions, add examples. * TODO: Mention an optimization to avoid needing to read all of the input into memory with --repetitions. * NEWS: Mention new shuf option. tests: add more tests for shuf option combinations * test/misc/shuf.sh: Add tests for erroneous conditions like multiple '-o' and '--random-source'. 2013-07-10 Pádraig Brady maint: adjust indentation in df.c * src/df.c: Adjust indentation and spacing with has gotten quite out of line in a couple of places. 2013-07-08 Pádraig Brady df: fix mount list processing with unstatable mount dirs * src/df.c (filter_mount_list): Initialize devlist->dev_num correctly when unable to stat() a mount point. This will avoid possible invalid deduplication done on the list due to use of uninitialized memory. * tests/df/skip-duplicates.sh: Ensure this code path is exercised. Also refactor the test to be table driven. * NEWS: Mention the bug fix. 2013-07-07 Anton Ovchinnikov maint: avoid a valgrind memory leak warning from uptime * src/uptime.c (uptime): Free utmp_buf returned from read_utmp, to avoid a "definitely lost" warning from valgrind. 2013-07-06 Paul Eggert build: update gnulib submodule to latest 2013-07-04 Pádraig Brady tests: fix "shuf unreadable" test to work when run as root * tests/misc/shuf.sh: Restrict the test to the significant case where we can't in fact read the "unreadable" file. 2013-07-02 Pádraig Brady stat: ensure --context emits an "unrecognized option" diagnostic * src/stat.c (long_options): Remove the "context" option as it's no longer referenced since commit v8.5-65-g13f3237 2013-07-02 Bernhard Voelker mkdir,mkfifo,mknod: give scontext-related variable a better scope * src/mkdir.c (main): Move the variable ret and issuing the error message into the body of the scontext-related if-block. * src/mkfifo.c (main): Likewise. * src/mknod.c (main): Likewise. 2013-07-02 Pádraig Brady maint: refactor SMACK interface to a separate module Consolidate all smack routines and checks in a module. We replace and wrap the most commonly used smack routines, which allows removing ifdefs throughout the code. * gl/lib/smack.h: A new header containing the implementation of the wrapped and replacement routines. Note the is_smack_enabled() routine should be optimized out at compile time when compiled on a system without libsmack. * gl/modules/smack: Describe the new module and move the configure time code here from ... * m4/jm-macros.m4: ... here. * bootstrap.conf: Reference the new module. * src/id.c: Use the routines without ifdefs where possible. * src/ls.c: Likewise. * src/mkdir.c: Likewise. * src/mkfifo.c: Likewise. * src/mknod.c: Likewise. 2013-07-02 Pádraig Brady id: don't show SMACK errors unless -Z is specified * src/id.c (main): Be consistent with the SELinux case, and only show errors in getting the security context when -Z is specified. 2013-07-01 Jarkko Sakkinen mkdir,mkfifo,mknod: with -Z, create SMACK security context Enable creation of SMACK security context with -Z command-line switch if SMACK is enabled. * mkdir.c (main): Set process security context to given SMACK label. * mkfifo.c (main): Likewise. * mknod.c (main): Likewise. * src/local.mk: link mk{dir, fifo, nod} with libsmack. * NEWS: Mention the new feature. 2013-06-27 D. Hugh Redelmeier doc: fix details on the interaction of df with device nodes * man/df.x: Don't say that a dev node is always on the root file system. * doc/coreutils.texi (df invocation): Likewise. Also state that the device node to mounted file system interpretation is only done when passed absolute paths to device nodes. 2013-06-24 Jarkko Sakkinen ls: with -Z, show SMACK security context Enable showing of file SMACK security with '-Z' command-line switch if SMACK is enabled. Showing SMACK context of a file does not strictly require SMACK to be enabled but this required to make choice whether to show SELinux or SMACK security context. * src/ls.c (getfilecon_cache): Retrieve SMACK context if available. (gobble_file): Handle SMACK context similarly to SELinux context. * src/local.mk: Link lsl with libsmack. * NEWS: Mention the new feature. * .mailmap: Merge the Author's 2 email addresses. 2013-06-21 Pádraig Brady stdbuf: make it mandatory to specify a buffering option This is consistent with the documented interface and avoids any ambiguity in a user thinking that stdbuf without options might reset to a "standard" buffering setup. * src/stdbuf.c (set_libstdbuf_options): Indicate with the return value whether any env variables were actually set. (main): Fail unless some env variables were set. * tests/misc/stdbuf.sh: Ensure this constraint is enforced. * NEWS: Mention the small change in behavior. 2013-06-21 Pádraig Brady doc: clarify that truncate --size represents bytes by default * src/truncate.c (usage): Mention that --size is in bytes which is by far the most common usage. * doc/coreutils.texi (truncate invocation): Likewise. Also cross reference the --io-blocks option. Reported in http://bugs.gnu.org/14686 2013-06-20 Jarkko Sakkinen id: adjust/restrict smack support to newer versions of libsmack There was slight change to libsmack such that positive values are reserved for returning length of the label for smack_new_label_from_* functions. * m4/jm-macros.m4: Set HAVE_SMACK when both smack_new_label_from_self() and recently added smack_new_label_from_path() are present. The latter's presence indicates the newer API of the former. * src/id.c (main): Check that smack_new_label_from_self() < 0, and not just non-zero. 2013-06-18 Pádraig Brady tests: use appropriate precision when printing float limits * src/getlimits.c (print_float): Adjust to use the ftoastr module, which uses the appropriate precision so that no info is lost. * cfg.mk (sc_prohibit_continued_string_alpha_in_column_1): Exclude od.c fixes http://bugs.gnu.org/14650 2013-06-13 Pádraig Brady maint: update gitignore entries * .gitignore: Exclude the generated doc/manual directory, and also the src/fs-latest-magic.h file referenced in README-release. 2013-06-12 Pádraig Brady doc: clarify the description of du --separate-dirs * src/du.c (usage): Clarify that --separate-dirs doesn't exclude all directories. * doc/coreutils.texi (du invocation): Avoid implying that -S excludes the size of any non directory entries for a directory. Also don't mention st_size as it's dependent on --apparent-size. Reported by C de-Avillez in 2013-06-05 Eric Blake doc: mention prior stat change For a file of size 1234 bytes, commit ca9aa759 had the side effect of changing 'stat -c "%'s" file' from outputting "?s" to the nicer "1,234", depending on locale. This is worth mentioning in the NEWS. Resolves part of http://bugs.gnu.org/14556. * NEWS: Mention 8.7 improvement in stat. * cfg.mk (old_NEWS_hash): Adjust accordingly. 2013-06-05 Bernhard Voelker build: update gnulib submodule; also bootstrap to latest Notes tests/init.sh is still in sync with gnulib. * bootstrap: Update to latest. * gnulib: Update avoiding gnulib-test failures. 2013-06-03 Pádraig Brady tests: avoid a race in tail --retry testing Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/5221053 * tests/tail-2/retry.sh: Ensure the 'out' file is truncated, as it's used to arbitrate the run order of commands. Relying on the truncation in the background tail command is racy because the truncation can occur after the fork of the background shell and thus wait4lines would not wait for output to occur in 'out', which would mean that the 'missing' file could be populated by the time tail(1) gets to process it. 2013-06-02 Andreas Mohr doc: mention 'UTC' in date --utc help src/date.c (usage): Make -u apparent in searches for UTC. 2013-06-02 Jim Meyering od: -wN, N>64K, avoid misbehavior on systems with 32-bit size_t * src/od.c (PRINT_FIELDS): Declare "i" to be of type uintmax_t, so that the numerator in the expression for "next_pad" does not overflow. (print_named_ascii): Likewise. (print_ascii): Likewise. Bug introduced via commit v6.12-42-g20c0b87. * tests/misc/od.pl: Exercise each of the three affected code paths. * NEWS (Bug fixes): Mention it. Reported by Rich Burridge. 2013-06-01 Jim Meyering tests: head-c: avoid spurious failure with a 32-bit size_t * tests/misc/head-c.sh: Don't try to elide 1 exabytes, since on 32-bit systems, that number is not representable as a size_t. This command would fail on 32-bit systems, where SIZE_MAX < 1E: head --bytes=-E < /dev/null Instead of "E", use $SSIZE_MAX. For discussion, see http://bugs.gnu.org/13530 2013-05-27 Jim Meyering tests: avoid spurious failure when SIZE_MAX is 2^32-1 * tests/split/line-bytes.sh: Since we've limited virtual memory to 20MB, choose a smaller size, 1GiB (which is <= SIZE_MAX) rather than 1EiB, which is larger than SIZE_MAX on 32-bit systems. I confirmed that this test still fails when the split.c-modifying part of v8.21-58-gfec363c is backed out. 2013-05-26 Jim Meyering tests: fix an ls test not to fail when user or group name contains SP * tests/ls/block-size.sh (size_etc): The sed expression through which we filtered the output of "ls -l ..." assumed that the user and group name components of each line would not contain spaces. Avoid the problem by using -og instead of -l, thus not printing either of those fields. Adjust the sed expression accordingly. 2013-05-26 Stefano Lattarini tests: numfmt: use the printf program, not the shell builtin * tests/misc/numfmt.pl: Avoid a spurious failure when /bin/sh is dash (as can happen on Debian systems). 2013-05-24 Paul Eggert stat: tune for large-LDAP case * src/stat.c (print_stat): Omit unnecessary calls to setpwent, setgrent. Problem reported by Fridolín Pokorný in . 2013-05-23 Pádraig Brady split: with --line-bytes only allocate memory as needed * src/split.c (line_bytes_split): Rewrite to only buffer when necessary. I.E. only increase the buffer when we've already lines output in a split and we encounter a line larger than the input buffer size, in which case a hold buffer will be increased in increments of the input buffer size. (lines_rr): Use the more abstract xalloc_die() just like we did in line_bytes_split(), rather than explicitly printing the "memory exhausted" message and exiting. * tests/split/line-bytes.sh: Add a new test for this function which previously had no test coverage. * tests/local.mk: Reference the new test. * NEWS: Mention the improvement. Fixes http://bugs.gnu.org/13537 2013-05-18 Paul Eggert maint: port --enable-gcc-warnings to clang * configure.ac: If clang, add -Wno-format-extra-args and -Wno-tautological-constant-out-of-range-compare. * gl/lib/rand-isaac.c (ind): * gl/lib/randread.c (readisaac): * src/ls.c (dev_ino_push, dev_ino_pop): * src/sort.c (buffer_linelim): * src/system.h (is_nul): * src/tail.c (tail_forever_inotify): Rewrite to avoid casts that clang dislikes. It's good to avoid casts anyway. * src/expr.c (integer_overflow): Declare only if it exists. (die): Remove; unused. * src/ls.c (dev_ino_push): New function, replacing ... (DEV_INO_PUSH): ... this removed macro. All uses changed. (decode_switches): Rewrite "str"+i to &str[i]. build: update gnulib submodule to latest tests: don't assume expr was built with GMP * tests/misc/cut-huge-range.sh (subtract_one): New string. (CUT_MAX): Don't pass a too-large integer to 'expr'. 2013-05-12 Pádraig Brady doc: link to the list of rejected feature requests * README: Update the email address best suited to discussing feature requests, and also link to the list of previously discussed and rejected requests. 2013-05-11 Jim Meyering build: avoid new syntax-check failure * po/POTFILES.in: Reflect renaming. 2013-05-11 Paul Eggert maint: add FIXME comment mkdir: don't assume umask equals POSIX default ACL mask This fixes Bug#14371, reported by Killer Bassist. * NEWS: Document this. * src/mkdir.c (struct mkdir_options): Remove member ancestor_mode. New member umask_value. All uses changed. * src/mkdir.c (make_ancestor): Fix umask assumption. * src/mkdir.c, src/mkfifo.c, src/mknod.c (main): Leave umask alone. This requires invoking lchmod after creating the file, which introduces a race condition, but this can't be avoided on hosts with "POSIX" default ACLs, and there's no easy way with network file systems to tell what kind of host the directory is on. * tests/local.mk (all_tests): Add tests/mkdir/p-acl.sh. * tests/mkdir/p-acl.sh: New file. build: update gnulib submodule to latest 2013-05-08 Jarkko Sakkinen id: with -Z, show SMACK security context Adds an optional dependency on libsmack. * m4/jm-macros.m4: Look for the smack library/header. * src/id.c (main): Output the smack context if available. * src/local.mk: Link with libsmack if available. * NEWS: Mention the new feature. 2013-05-08 Cojocaru Alexandru cut: improve performance, especially with --output-delimiter Use a sentinel value that's checked implicitly, rather than a bit array, to determine if an item should be output. Benchmark results for this change are: $ yes abcdfeg | head -n1MB > big-file $ for c in orig sentinel; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 big-file > /dev/null done == orig == real 0m0.049s user 0m0.044s sys 0m0.005s == sentinel == real 0m0.035s user 0m0.032s sys 0m0.002s ## Again with --output-delimiter ## $ for c in orig sentinel; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 --output-delimiter=: big-file > /dev/null done == orig == real 0m0.106s user 0m0.103s sys 0m0.002s == sentinel == real 0m0.055s user 0m0.052s sys 0m0.003s eol_range_start: Removed. 'n-' is no longer treated specially, and instead SIZE_MAX is set for the 'hi' limit, and tested implicitly. complement_rp: Used to complement 'rp' when '--complement' is specified. ADD_RANGE_PAIR: Macro renamed to 'add_range_pair' function. * tests/misc/cut-huge-range.sh: Adjust to the SENTINEL value. Also remove the overlapping range test as this is no longer dependent on large ranges and also is already handled with the EOL-subsumed-3 test in cut.pl. 2013-05-07 Cojocaru Alexandru cut: fix handling of overlapping ranges This issue was introduced in commit v8.21-43-g3e466ad * src/cut.c (set_fields): Process all range pairs when merging. * tests/misc/cut-huge-range.sh: Add a test for this edge case. Also fix an issue where we could miss reported errors due to truncation of the 'err' file. 2013-04-30 Pádraig Brady doc: correct a URL to older textutils source * doc/coreutils.texi (Putting the tools together): Adjust the textutils-1.22 URL, and add a URL for newer sources. 2013-04-29 Pádraig Brady cut: reduce CPU usage for the the common case Ensure appropriate functions are inlined. This was seen to be required with gcc 4.6.0 with -O2 on x86_64 at least. It was reported that gcc 4.8.0 did inline these functions though. Also reinstate the bit vector for the common case, to further improve performance. Benchmark results for both aspects of this change are: $ yes abcdfeg | head -n1MB > big-file $ for c in orig inline inline-array; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 big-file > /dev/null done == orig == real 0m0.088s user 0m0.081s sys 0m0.007s == inline == real 0m0.070s user 0m0.060s sys 0m0.009s == inline-array == real 0m0.049s user 0m0.044s sys 0m0.005s * src/cut.c (set_fields): Set up the printable_field bit vector for performance, but only when it's appropriate. I.E. not when either --output-delimeter or huge ranges are specified. (next_item): Ensure it's inlined and avoid unnecessary processing. (print_kth): Ensure it's inlined and add a branch for the fast path. Related to http://bugs.gnu.org/13127 2013-04-29 Cojocaru Alexandru cut: reduce CPU overhead in determining item to output print_kth() is the central function of cut used to determine if an item is to be output or not, so simplify it by moving some logic outside. Benchmark results for this change are: $ yes abcdfeg | head -n1MB > big-file $ for c in orig split; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 big-file > /dev/null done == orig == real 0m0.111s user 0m0.108s sys 0m0.002s == split == real 0m0.088s user 0m0.081s sys 0m0.007s * src/cut.c (print_kth): Refactor a branch to outside the function. Related to http://bugs.gnu.org/13127 2013-04-29 Cojocaru Alexandru cut: make memory allocation independent of range width The current implementation of cut, uses a bit array, an array of `struct range_pair's, and (when --output-delimiter is specified) a hash_table. The new implementation will use only an array of `struct range_pair's. The old implementation is memory inefficient because: 1. When -b with a big num is specified, it allocates a lot of memory for `printable_field'. 2. When --output-delimiter is specified, it will allocate 31 buckets. Even if only a few ranges are specified. Note CPU overhead is increased to determine if an item is to be printed, as shown by: $ yes abcdfeg | head -n1MB > big-file $ for c in with-bitarray without-bitarray; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 big-file > /dev/null done == with-bitarray == real 0m0.084s user 0m0.078s sys 0m0.006s == without-bitarray == real 0m0.111s user 0m0.108s sys 0m0.002s Subsequent patches will reduce this overhead. * src/cut.c (set_fields): Set and initialize RP instead of printable_field. * src/cut.c (is_range_start_index): Use CURRENT_RP rather than a hash. * tests/misc/cut.pl: Check if `eol_range_start' is set correctly. * tests/misc/cut-huge-range.sh: Rename from cut-huge-to-eol-range.sh, and add a test to verify large amounts of mem aren't allocated. Fixes http://bugs.gnu.org/13127 2013-04-27 Pádraig Brady stat,tail: improve support for snfs The StorNext distributed file system was previously known as CVFS. * src/stat.c (human_fstype): Add new file system ID definition. * NEWS: Mention the improvement. Fixes http://bugs.gnu.org/14251 2013-04-25 Pádraig Brady tests: fix usage of require_ulimit_ * init.cfg (require_ulimit_v_): Renamed from require_ulimit_ as this only checks for ulimit -v support. Other uses of ulimit -t and ulimit -n in tests shouldn't cause false failures if not supported. * cfg.mk (sc_prohibit_test_ulimit_without_require_): A new syntax check to ensure that require_ulimit_v_() is used iff required. * tests/misc/head-c.sh: Add missing call to require_ulimit_v_. * tests/rm/many-dir-entries-vs-OOM.sh: Likewise. * tests/split/r-chunk.sh: Remove non mandatory require_ulimit_ call. * tests/misc/sort-merge-fdlimit.sh: Likewise. * tests/cp/link-heap.sh: Adjust to renamed require_ulimit_v_. * tests/dd/no-allocate.sh: Likewise. * tests/misc/csplit-heap.sh: Likewise. * tests/misc/cut-huge-to-eol-range.sh: Likewise. * tests/misc/printf-surprise.sh: Likewise. 2013-04-22 Jim Meyering scripts: tweak URLs in autotools-install * scripts/autotools-install (tarballs): Use http:// URLs rather than ftp:// ones. The former are more likely to work, these days. Update URLs to point to the latest versions. 2013-04-20 Bernhard Voelker tail: exit following by descriptor when no tailable file left As a side effect of the previous commit which fixes 'tail -f --retry' to wait for a file to appear, tail would not exit when the last file appears untailable and gives up on this file. This can happen, for example, when the argument file name appears as directory. Tail sets the 'ignore' flag of this file to true, but instead of exiting the program, tail would continue the loop. * src/tail.c (any_live_files): Change the function to return true if any of the files is still tailable or if tail should continue to try to check again. (tail_forever): Change the condition to break the loop in the "no files remaining" case, because now any_live_files() will care about it, as mentioned above. (parse_options): When --retry is used without any follow mode, then reset reopen_inaccessible_files to false. * tests/tail-2/retry.sh: Add test case. 2013-04-20 Bernhard Voelker tail: let -f --retry wait for inaccessible files The --retry option is indeed useful for both following modes by name and by file descriptor. The difference is that in the latter case, it is effective only during the initial open. As a regression of the implementation of the inotify support, tail -f --retry would immediately exit if the given file is inaccessible. * src/tail.c (usage): Change the description of the --retry option: remove the note that this option would mainly be useful when following by name. (main): Change diagnosing dubios uses of --retry option: when the --retry option is used without following, then issue a warning that this option is ignored; when it is used together with --follow=descriptor, then issue a warning that it is only effective for the initial open. Disable inotify also in the case when the initial open in tail_file() failed (which is the actual bug fix). * init.cfg (retry_delay_): Pass excess arguments to the test function. * tests/tail-2/retry.sh: Add new tests. * tests/local.mk (all_tests): Mention it. * doc/coreutils.texi (tail invocation): Enhance the documentation of the --retry option. Clarify the difference in tail's behavior regarding the --retry option when combined with the following modes name versus descriptor. * NEWS (Bug fixes): Mention the fix. Reported by Noel Morrison in: http://lists.gnu.org/archive/html/coreutils/2013-04/msg00003.html 2013-04-13 Pádraig Brady tests: avoid false failures with non default groups On OS X it was seen that the group ID used for new files, are set to a that of the directory rather than the current user. It's not currently understood when this happens, but it was confirmed that ACLs, extended attributes and setgid bits are _not_ involved. * init.cfg (skip_if_nondefault_group_): A new function to detect and avoid this situation. Document with links to the discussions for hopefully future clarification. * tests/install/install-C-root.sh: Use the new function. * tests/install/install-C-selinux.sh: Likewise. * tests/install/install-C.sh: Likewise. 2013-04-13 Pádraig Brady doc: mention caveats with using install --compare * doc/coreutils.texi (install invocation): Mention that install(1) may not correctly determine the default user or permissions for installed files, and so is best used with options specifying these attributes. head: with --bytes=-N only allocate memory as needed * src/head.c (elide_tail_bytes_pipe): Don't use calloc as that bypasses memory overcommit due to the zeroing requirement. Also realloc rather than malloc the pointer array to avoid dependence on overcommit entirely. * tests/misc/head-c.sh: Add a test case. Fixes http://bugs.gnu.org/13530 2013-04-13 Ondrej Oprala dd: avoid buffer allocations unless needed * src/dd.c: Add new static global variable ibuf. (alloc_ibuf, alloc_obuf): New functions factored from dd_copy(). (dd_copy): Call the new functions to allocate memory for ibuf and obuf when necessary. (skip): Likewise. * tests/dd/no-allocate.sh: New test. * tests/local.mk: Reference the test. 2013-04-10 Assaf Gordon csplit: add the --suppress-matched option With --suppress-matched, the lines that match the pattern will not be printed in the output files. I.E. the first line from the second and subsequent splits will be suppressed. * src/csplit.c: process_regexp(),process_line_count(): Don't output the matched lines. Since csplit includes "up to but not including" matched lines in each split, the first line (in the next group) is the matched line - so just skip it. main(): Handle new option. usage(): Mention new option. * doc/coreutils.texi (csplit invocation): Mention new option, examples. * tests/misc/csplit-suppress-matched.pl: New test script. * tests/local.mk: Reference the new test. * NEWS: Mention new feature. 2013-04-07 Enrico Scholz build: fix man page build failure with some permissions setups Use the more portable 'chmod a-w', instead of the 'chmod -w' form. The latter is not always supported. Also its operation is dependent on umask controlling the permissions bits for new files, which is not the case in the presence of POSIX default ACLs for e.g. In that case, chmod may print a warning like the following, and exit with failure status: chmod: man/hostid.1-t: new permissions are r--rw-r--, not r--r--r-- * man/local.mk: s/-w/a-w/ 2013-04-03 Pádraig Brady tests: avoid shared lib tests on unsupported platforms * init.cfg (require_gcc_shared_): A new function to check that we can build shared libraries in the particular manner we use in our tests. * tests/cp/nfs-removal-race.sh: Use require_gcc_shared_. Then fail rather than skip, if the actual shared lib build fails. * tests/df/no-mtab-status.sh: Likewise. * tests/df/skip-duplicates.sh: Likewise. * tests/ls/getxattr-speedup.sh: Likewise. Reported in http://bugs.gnu.org/14024 tail: exit without reading input if would never output * src/tail.c (main): If -n0 or -c0 were specified without -f, then no data would ever be output, so exit without reading input. * tests/tail-2/tail-n0f.sh: Augment the related test with this case. shuf: exit without reading if would never output * src/shuf.c (main): If -n0 specified then no data would ever be output, so exit without reading input. * tests/misc/shuf.sh: Augment the related test with this case. doc: add details on ln --relative symlink resolution * doc/coreutils.texi (ln invocation): Describe how symlinks are resolved with --relative, and give an example showing the greater control available through realpath(1). * tests/ln/relative.sh: Add a test to demonstrate full symlink resolution, in a case where it might not be wanted. 2013-04-03 Rémy Lefevre ln: --relative: fix updating of existing symlinks Don't dereference an existing symlink being replaced. I.E. generate the symlink relative to the symlink's containing dir, rather than to some arbitrary place it points to. * src/ln.c (convert_abs_rel): Don't consider the final component of the symlink name when canonicalizing, as we want to avoid dereferencing the final component. * tests/ln/relative.sh: Add a test case. * NEWS: Mention the fix. Resolves http://bugs.gnu.org/14116 2013-03-25 Assaf Gordon shuf: use reservoir-sampling for large or unknown sized inputs Reservoir sampling optimizes selecting K random lines from large or unknown-sized input: http://en.wikipedia.org/wiki/Reservoir_sampling Note this also avoids reading any input when -n0 is specified. * src/shuf.c (main): Use reservoir-sampling when the number of output lines is known, and the input size is large or unknown. (input_size): A new function to get the input size for regular files. (read_input_reservoir_sampling): New function to read lines from input, keeping only K lines in memory, replacing lines with decreasing prob. (write_permuted_output_reservoir): New function to output reservoir. * tests/misc/shuf-reservoir.sh: An expensive_ test using valgrind to exercise the reservoir-sampling code. * tests/local.mk: Reference new test. * NEWS: Mention the improvement. 2013-03-23 Pádraig Brady stat,tail: improve support for efivarfs, exofs, f2fs and ubifs * src/stat.c (human_fstype): Add new file system ID definitions. * NEWS: Mention the improvement. Fixes http://bugs.gnu.org/14020 maint: improve make src/fs-magic-compare * README-release: fix the `make` command, and mention how to get the latest results without requring running a system with the latest kernel. * src/local.mk (src/fs-latest-magic.h): A new target to document how/where to place the latest magic header. (src/fs-kernel-magic): Adjust to include separately downloaded header if available. (src/fs-magic): Undefine MANPAGER as it may impact the ability to pipe the output of man(1). (fs-magic-compare): Don't echo the commands run as they're distracting from the output which needs to be examined. build: fix issues when building with GMP * m4/gmp.m4 (cu_GMP): Add an extra check that gmp.h is available which is required on one Mac OS X 10.5.8 system at least, where the lib was available but the header wasn't. Also enable our GMP code on systems where GMP is not in a separate lib. 2013-03-22 Pádraig Brady doc: clarify the printable characters output by od * src/od.c (usage): Mention any printable character is output, Not just ASCII. * doc/coreutils.texi (od invocation): Further clarify that only single byte characters are output (due to the alignment requirement). Also mention the fact that 3 digit octal sequences are output for non printable characters without a corresponding C escape. Reported in http://bugs.gnu.org/13947 doc: clarify stat the meaning of --format="%t %T" * src/stat.c (usage): Mention that the values are only defined for character and block special files. * doc/coreutils.texi (stat invocation): Likewise. Also mention st_rdev. Reported in http://bugs.gnu.org/13927 2013-03-06 Pádraig Brady doc: mention `numfmt` as an alternative to `sort -h` * doc/coreutils.texi (sort invocation): Mention that numfmt can achieve the same results with a possibly more accurate sort. doc: remove a redundant numfmt heading from texinfo * doc/coreutils.texi (detailmenu): Remove the redundant numfmt heading. maint: ensure proper backslash quoting in texinfo macros * doc/coreutils.texi (ambiguousGroupNote): Ensure '\' is escaped appropriately within the macro. This was verified to generate a single '\' in both "info" and "pdf" outputs. 2013-03-04 Torbjörn Granlund build: fix factor build failure on aarch64 * src/longlong.h (__aarch64__): Make add_ssaaaa and sub_ddmmss work. * NEWS: Mention the build fix. Reported at https://bugzilla.redhat.com/917735 2013-02-28 Assaf Gordon tests: don't skip all uniq tests when locale is missing * tests/misc/uniq.pl: Previously, if LOCALE_FR was not defined, all tests would be skipped. Modified to skip only the relevant test. uniq: add the --group option * src/uniq.c (usage): Summarize the new option, and adjust the --all-repeated option to be more consistent. (check_file): Merge the --group functionality into the core loop for the default uniq operation since it's very similar and can output lines immediately upon reading. (main): Handle the new --group option and make it mutually exclusive with other selection options. * tests/misc/uniq.pl: Add tests. * NEWS: Mention the new feature. * doc/coreutils.texi (uniq invocation): Describe --group. 2013-02-28 Pádraig Brady doc: move some info from all --help messages, online * src/system.h (emit_ancillary_info): Link to the bug report email addresses and general help URLs online rather than specifying directly. This give us greater scope to present better info like describing the difference between bug-coreutils@gnu.org and coreutils@gnu.org etc. * tests/misc/help-version.sh: Remove the check for bug-coreutils@gnu.org * tests/local.mk: Remove the no longer needed PACKAGE_BUGREPORT. 2013-02-27 Assaf Gordon tests: add '--ignore-case' tests for uniq * tests/misc/uniq.pl: add tests for --ignore-case. 2013-02-27 Javier López dircolors: add a new entry to colorize 'arc' files * src/dircolors.hin: Add an entry for arc. Suggested in: https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1088131 2013-02-27 Assaf Gordon join: Add the -z, --zero-terminated option * NEWS: Mention join's new option: --zero-terminated (-z). * src/join.c: Add new option, --zero-terminated (-z), to make join use the NUL byte as separator/delimiter rather than newline. (get_line): Use readlinebuffer_delim in place of readlinebuffer. (main): Handle the new option. (usage): Describe new option the same way sort does. * doc/coreutils.texi (join invocation): Describe the new option. * tests/misc/join.pl: add tests for -z option. 2013-02-27 Ondřej Vašík dircolors: add new entries to colorized archive formats * src/dircolors.hin: Add .cab, .alz, .lzo, .lrz, .t7z, .tzo, .lha to colorized archives. Suggested by Ville Skyttä in https://bugzilla.redhat.com/868510 2013-02-22 Ondrej Oprala install: cleanup properly if the strip program failed for any reason * src/install.c (strip): Indicate failure with a return code instead of terminating the program. (install_file_in_file): Handle strip's return code and unlink the created file if necessary. * tests/install/strip-program.sh: Add a test to cover the changes. * NEWS (Bug fixes): Mention the fix. Reported by John Reiser in http://bugzilla.redhat.com/632444. 2013-02-19 Joachim Schmitz copy: ensure the correct root ID is checked on all platforms * src/copy.c (copy_reg): Check ROOT_UID rather than 0, which is significant on HP-NonStop. 2013-02-19 Pádraig Brady maint: cleanup up various uses of __attribute__ * src/cfg.mk (sc_prohibit-gl-attributes): Disallow the __attribute() form without trailing underscores as that is not elided where required. Also ensure we use gnulib macros rather than defining our own. * src/system.h: Remove gnulib provided macros. * src/chown-core.c: Likewise. * src/chroot.c: Likewise. * src/copy.c: Likewise. * src/csplit.c: Likewise. * src/dd.c: Likewise. * src/expr.c: Likewise. * src/extent-scan.c: Likewise. * src/factor.c: Likewise. * src/ls.c: Likewise. * src/od.c: Likewise. * src/paste.c: Likewise. * src/ptx.c: Likewise. * src/sort.c: Likewise. * src/stat.c: Likewise. * src/stty.c: Likewise. * src/system.h: Likewise. * src/tac.c: Likewise. * src/test.c: Likewise. * src/tsort.c: Likewise. 2013-02-19 Joachim Schmitz build: fix numfmt build error on compilers without __attribute * src/numfmt.c (): Use the more standard _GL_ATTRIBUTE_PURE which is elided where required. Reported in http://bugs.gnu.org/10305 2013-02-19 Pádraig Brady tests: avoid false failures on file systems with smaller NAME_MAX * tests/du/long-from-unreadable.sh: This test requires a NAME_MAX of at least 200, so skip the test otherwise. * tests/rm/deep-2.sh: Likewise. Reported by C de-Avillez with ecryptfs where NAME_MAX = 143. 2013-02-17 Stefano Lattarini maint: choose editor in the commit-msg git hook the same way git does Git honours the GIT_EDITOR environment variable, the "core.editor" Git configuration variable, and the EDITOR environment variable (in that order, and defaulting to "vi" if none of them is set) to decide which editor should be invoked for the user when he has to or want to edit his commit message. However, our commit-msg hook, when invoking an editor on behalf of the user to allow him to fix a non-policy-complaint commit message, only honoured the EDITOR environment variable. To avoid potential annoying inconsistencies, we should really use the same logic used by Git in the selection of the editor. Luckily, we don't have to duplicate this logic (that would be brittle in the long term), as we can rely on the "git var" command, designed exactly to be used in situations like this. * scripts/git-hooks ($editor): Adjust definition. 2013-02-17 Stefano Lattarini maint: don't reset PATH in the commit-msg git hook I have a custom 'editor' script in ~/bin, and a system-provided 'editor' program in /usr/bin (on Debian, this is a link set up the "debian alternatives" subsystem). My '$EDITOR' and '$GIT_EDITOR' variables are set simply to 'editor' (no absolute path), which I expect should point to my 'editor' script, since ~/bin precedes /usr/bin in my PATH definition. But the 'commit-msg' hook used in coreutils unconditionally resets its PATH to '/bin:/usr/bin', which causes it to call the "wrong" editor (the one in /usr/bin, not the one in ~/bin) when it makes me update a botched commit message. * scripts/git-hooks: Don't reset $ENV{PATH} to '/bin:/usr/bin', which was only done to avoid failure when enabling Perl's taint checking. 2013-02-14 Bernhard Voelker maint: prevent trailing period at first line of a commit message * scripts/git-hooks/commit-msg (bad_first_line): Return an error message if the first line of a commit message ends with a period. 2013-02-14 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.21 * NEWS: Record release date. tests: avoid non portable sed use of \t * tests/du/threshold.sh: use `cut` rather than sed to avoid using the non portable \t which fails on sed on openbsd 5 at least. Also remove a redundant call to `tr` and avoid explicit setting of LANG=C which is done globally. 2013-02-13 Pádraig Brady build: avoid link failure in devmsg() on older linkers On linkers that don't remove unused functions, there will be a reference to a missing dev_debug symbol in the devmsg() function. So for now ... * src/system.h: ... move devmsg() from here ... * src/numfmt.c: ... to here, and document future cleanup. * src/factor.c: Likewise. 2013-02-13 Assaf Gordon numfmt: fix strtol() return code handling src/numfmt.c (parse_format_string): On some systems, strtol() returns EINVAL if no conversion was performed. So only handle ERANGE here, and handle other format errors directly. 2013-02-11 Bernhard Voelker maint: avoid running check-root tests in gnulib * tests/local.mk (check-root): Restrict to SUBDIRS=. as traversing into gnulib-tests induces a false failure. 2013-02-11 Pádraig Brady doc: improve the numfmt man page format * src/numfmt.c (usage): Keep a single space between the "K = 1000", so that it's not displayed on a separate line. Also place ',' between each unit entry to improve readability. 2013-02-10 Benno Schulenberg doc: standardize helptext of numfmt and slice into single options * src/numfmt.c (usage): Correct synopsis and make command description clearer. Start option descriptions with lowercase letter; use semicolon instead of period where needed; indent continuation lines; gettextize single options for ease of translation and maintenance; sort options alphabetically. * doc/coreutils.texi (numfmt invocation): Sort numfmt options alphabetically. Enforce double-blank-after-period style. This addresses http://bugs.gnu.org/13681. Improved-by: Bernhard Voelker 2013-02-10 Pádraig Brady maint: consolidate developer debug messages Both factor and numfmt recently introduced debug messages for developers, enabled by --verbose and ---devdebug respectively. There were a few issues though: 1. They used different mechanisms to enable these messages. 2. factor used --verbose which might be needed for something else 3. They used different methods to output the messages, and numfmt used error() which added an unwanted newline 4. numfmt marked all these messages for translation and factor marked a couple. We really don't need these translated. So we fix the above issues here while renaming the enabling option for both commands to ---debug (still undocumented). * src/factor.c (verbose): Rename to dev_debug and change from int to bool as it's just a toggle flag. (long_options): Rename --verbose to ---debug. * src/system.h (devmsg): A new inline function to output a message if enabled by a global dev_debug variable in the compilation unit. * src/numfmt.c: Use devmsg() rather than error(). Also remove the translation tags from these messages. Also change debug flag to bool from int. * tests/misc/numfmt.pl: Adjust for the ---devdebug to ---debug change. * cfg.mk (sc_marked_devdiagnostics): Add a syntax check to ensure translations are not added to devmsg calls. Reported by Göran Uddeborg in http://bugs.gnu.org/13665 2013-02-09 Pádraig Brady tests: tail-2/inotify-rotate: fix a false failure on NFS * tests/tail-2/inotify-rotate.sh: Avoid a subshell with bash, which in turn causes the `kill` to be ineffective to the tail processes (as the SIGTERM is sent to the subshell which doesn't propagate the signal on to its children). On NFS the test cleanup will then fail as there will be .nfs files maintained in the directory for the files still opened by the tail processes. Reported by Bernhard Voelker. tests: skip numfmt grouping tests on some systems * tests/misc/numfmt.pl: When the system locale grouping doesn't match our expected format for grouping 1234 in the fr_FR locale, reset the locale to 'C' so as to skip all locale tests. 2013-02-07 Jim Meyering tests: avoid actual/expected mismatch due to changed diagnostic * tests/cp/fail-perm.sh: Adjust expected diagnostic to match just-changed cp diagnostic. * tests/ln/hard-to-sym.sh: Likewise. * .mailmap: Also map my new address. 2013-02-07 Pádraig Brady build: update gnulib submodule; also bootstrap to latest Notes tests/init.sh is still in sync with gnulib * bootstrap: update to latest * gnulib: update avoiding secure_getenv and subsequent patches as these are reported to fail on FreeBSD at least. 2013-02-06 Benno Schulenberg maint: improve error messages upon failed read, write, access, close Note we use "failed to {access,close}" for those single operations, and "error {read,writ}ing" for those partial operation failures. * src/copy.c: Improve error messages for failing read, write and close. * src/cp.c: Improve error messages for failing access. * src/dd.c: Improve error messages for failing read, write and open. * src/head.c: Improve error message for failing close. * src/install.c: Improve error messages for failing access. * src/ln.c: Likewise. * src/mv.c: Likewise. * src/touch.c: Improve error message for failing close. * src/truncate.c: Likewise. 2013-02-05 Assaf Gordon doc: fix a numfmt help section typo * src/numfmt.c (usage): Change erroneous "G" to "M". 2013-02-05 Ondřej Vašík stty: add support for DTR/DSR hardware control flow Originally requested in Red Hat bugzilla #445213. * src/stty.c (mode_info): Add support for DTR/DSR hardware flow control, if available. * doc/coreutils.texi: Document it. * tests/misc/stty.sh: Add it to the list of serial options to avoid. * NEWS: Mention the improvement. 2013-02-05 Pádraig Brady numfmt: correct a printf format Prompted by the continuous integration build failure at: http://hydra.nixos.org/build/4010493 * src/numfmt.c (parse_format_string): Correct both sign and size of a printf format, which only gives a warning on 32 bit builds. 2013-02-04 Assaf Gordon numfmt: a new command to format numbers * AUTHORS: Add my name. * NEWS: Mention the new program. * README: Reference the new program. * src/numfmt.c: New file. * src/.gitignore: Ignore the new binary. * build-aux/gen-lists-of-programs.sh: Update. * scripts/git-hooks/commit-msg: Allow numfmt: commit prefix. * po/POTFILES.in: Add new c file. * tests/misc/numfmt.pl: A new test file giving >93% coverage. * tests/local.mk: Reference the new test. * man/.gitignore: Ignore the new man page. * man/local.mk: Reference the new man page. * man/numfmt.x: A new template. * doc/coreutils.texi: Document the new command. 2013-02-04 Pádraig Brady cut: fix a segfault with disjoint open ended ranges Fixes the issue introduced in unreleased commit v8.20-60-gec48bea. * src/cut.c (set_fields): Don't access the bit array if we've an open ended range that's outside any finite range. * tests/misc/cut.pl: Add tests for this case. Reported by Marcel Böhme in http://bugs.gnu.org/13627 2013-02-01 Paul Eggert doc: say how to tac char-by-char This fixes Bug#12115, reported by Reuben Thomas. * doc/coreutils.texi (tac invocation): Document how to reverse a file character by character. Break out MS-DOS into a separate section, like 'cat' does. 2013-01-28 Bernhard Voelker df: do not treat rootfs specially Like any other pseudo file system, df should show rootfs only when the -a option is specified, i.e. specifying -trootfs alone is not sufficient. As the rootfs entry is now elided by the general deduplication in filter_mount_list (commit v8.20-103-gbb116d3), all other references to rootfs can be removed again. * src/df.c (show_rootfs): Remove global variable. (ROOTFS): Remove constant. (filter_mount_list): Remove case to handle rootfs specially. (main): In the case for handling the -t option, remove setting of the show_rootfs variable. * tests/df/skip-rootfs.sh: Adapt the test case "df -t rootfs": the rootfs file system must not be printed (because no -a). * doc/coreutils.texi (df invocation): Correct the documentation about eliding mount entries: it is not the first occurrence of the the device which wins, but now rather the entry with the shortest mount point name. Also adapt the description about eliding pseudo file system types like rootfs. * NEWS (Changes in behavior): Adapt entry. 2013-01-27 Ondrej Oprala Bernhard Voelker df: prefer fullpath entries when deduplicating * src/df.c (struct devlist): Add a new element for storing pointers to mount_entry structures. (devlist_head, dev_examined): Remove. (filter_mount_list): Add new function to filter out the rootfs entry (unless -trootfs is specified), and duplicities. The function favors entries with a '/' character in me_devname or those with the shortest me_mountdir string, if multiple entries fulfill the first condition. Use struct devlist to build up a list of entries already known, and finally rebuild the global mount_list. (get_all_entries): Call the above new function unless the -a option is specified. (get_dev): Remove the code for skipping rootfs and duplicities. * tests/df/skip-duplicates.sh: Add test cases. 2013-01-25 Stephan Krempel timeout: ensure a blocked SIGALRM from the parent is unblocked * src/timeout.c (unblock_signal): A new function to unblock a specified signal, or warn if not possible. (set_timeout): Ensure SIGALRM is unblocked before we setup the timer. * tests/misc/timeout-blocked.pl: A new test for the issue. * tests/local.mk: Reference the new test. * NEWS: Mention the fix. Fixes: http://bugs.gnu.org/13535 2013-01-25 Pádraig Brady seq: fix to always honor the step value * src/seq.c (main): With 3 positive integer args we were checking the end value was == "1", rather than the step value. * tests/misc/seq.pl: Add tests for this case. Reported by Marcel Böhme in http://bugs.gnu.org/13525 seq: fix misaligment with -w when no precision for start value * src/seq.c (get_default_format): Also account for the case where '.' is auto added to the start value, which is significant when the number sequence narrows. * tests/misc/seq.pl: Add two new tests for the failing cases. * NEWS: Mention the fix. Fixes http://bugs.gnu.org/13394 cut: fix -f to work with the -d$'\n' edge case * src/cut.c (cut_fields): Handle the edge case where '\n' is the delimiter, which could be used for example to suppress the last line if it doesn't contain a '\n'. * test/misc/cut.pl: Add tests for this edge case. 2013-01-25 Pádraig Brady cut: with -f, process each line independently Previously line N+1 was inspected before line N was fully output, which causes output ordering issues at the terminal or delays from intermittent sources like tail -f. * src/cut.c (cut_fields): Adjust so that we record the previous output character so we can use that info to determine wether to output a '\n' or not. * tests/misc/cut.pl: Add tests to ensure existing functionality isn't broken. * NEWS: Mention the fix. Fixes bug http://bugs.gnu.org/13498 2013-01-23 Paul Eggert build: update gnulib submodule to latest * bootstrap.conf (gnulib_modules): Add statat. The fstatat module was split in two, and we need both halves. 2013-01-23 Pádraig Brady doc: fix an example in the od man page * man/od.x: s/-w 16/-w16/ as -w takes an optional parameter and so the space is significant. 2013-01-22 Bernhard Voelker doc: fix order of du options in usage and texinfo manual * src/du.c (usage): Bring options into alphabetical order. * doc/coreutils.texi (du invocation): Likewise. Furthermore, use the @itemx macro for the long options --max-depth and --threshold instead of @item. 2013-01-22 Bernhard Voelker maint: define usage note about mandatory args centrally Each program with at least one long option which is marked as 'required_argument' and which has also a short option for that option, should print a note about mandatory arguments. Define that well-known note centrally and use it rather than literal printf/fputs, and add it where it was missing. * src/system.h (emit_mandatory_arg_note): Add new function. * src/cp.c (usage): Use it rather than literal printf/fputs. * src/csplit.c, src/cut.c, src/date.c, src/df.c, src/du.c: * src/expand.c, src/fmt.c, src/fold.c, src/head.c, src/install.c: * src/kill.c, src/ln.c, src/ls.c, src/mkdir.c, src/mkfifo.c: * src/mknod.c, src/mv.c, src/nl.c, src/od.c, src/paste.c: * src/pr.c, src/ptx.c, src/shred.c, src/shuf.c, src/sort.c: * src/split.c, src/stdbuf.c, src/tac.c, src/tail.c, src/timeout.c: * src/touch.c, src/truncate.c, src/unexpand.c, src/uniq.c: Likewise. * src/base64.c (usage): Add call of the above new function because at least one long option has a required argument. * src/basename.c, src/chcon.c, src/date.c, src/env.c: * src/nice.c, src/runcon.c, src/seq.c, src/stat.c, src/stty.c: Likewise. 2013-01-21 Jakob Truelsen Bernhard Voelker du: add -t SIZE, --threshold=SIZE option * src/du.c (opt_threshold): Add variable to hold the value of the --threshold option specified by the user. (long_options): Add a required_argument entry for the new --threshold option. (usage): Add --threshold option. (process_file): Elide printing the entry if its size does not meet the value specified by the --threshold option. (main): In the argument parsing loop, add a case for the new -t option. Convert the given argument by permitting the well-known suffixes for megabyte, gigabytes, etc. Handle the special case "-0": give an error as this value is not permitted. * doc/coreutils.texi (du invocation): Add documentation for the above new option. * tests/du/threshold.sh: Add new test to exercise the new option. * tests/local.mk (all_tests): Mention the above test. 2013-01-20 Bernhard Voelker tests: remove test case du/slink This test tried to ensure that not all symlinks (across all file system types) have Zero size and refers to a change in system.h from 2002-08-31 (commit SH-UTILS-2_0_15-55-g62808a7). The test used to do this by working on symlinks to long file names. This assumption is dependant on the underlying file system, and in some environments like XEN does not even work on file systems known to work otherwise. The test for dereferencing and no-dereferencing symlinks is already covered by other tests (du/deref.sh, du/deref-args.sh, and du/no-deref.sh). Therefore, remove it. * tests/du/slink.sh: Remove file. * tests/local.mk (all_tests): Remove the above test. Discussed in: http://lists.gnu.org/archive/html/coreutils/2013-01/msg00053.html 2013-01-20 Bernhard Voelker maint: fix alphabetical order in .gitignore Since commit v8.20-67-g0f525b6, .gitignore sometimes showed up as changed because the entries "*.gcda" and "*.gcno" had not been in alphabetical order. * .gitignore: Exchange the entries "*.gcda" and "*.gcno". 2013-01-12 Benno Schulenberg uptime: gettextize an overlooked string, and normalize another * src/uptime.c: Add calls to gettext() and select_plural(). 2013-01-10 Benno Schulenberg doc: make a --help text fragment identical to three others * src/ln.c (usage): Move a newline to the next text fragment, so the preceding fragment about backup methods becomes the same as the ones for cp, mv, and install. A bit easier for translators. 2013-01-06 Pádraig Brady factor: apply a more general fix to enable correct assembly In addition to the previous 64 bit guards we've placed in longlong.h there are additional _LP64 guards required for mips with -mcpu >= 3, to avoid a build failure (http://bugs.gnu.org/13353) and on sparc with -mcpu >= v9 in 32 bit mode where for example, `factor 2123123123123123123123` would go into an infinite loop. Since factor.c currently operates on uintmax_t, we restrict the use of the assembly in longlong.h to when 'long' has the same width, to provide a more general guard for this code. * src/factor.c: Restrict the use of longlong.h assembly code, to when the width of intmax_t == long. * src/longlong.h: Remove the previous _LP64 guards to avoid divergence from GMP's longlong.h * NEWS: Adjust the info on build and runtime fixes. 2013-01-06 Paul Eggert doc: sync parse-datetime from gnulib * doc/coreutils.texi (Top): Sync from gnulib parse-datetime.texi menu. 2013-01-06 Karl Berry doc: avoid @sc in texinfo; it is unnecessary * doc/coreutils.texi: avoid @sc and use explicit capitals. * doc/local.mk (sc-use-small-caps-NUL): Remove, as no longer applicable. 2013-01-04 Karl Berry doc: remove stale pr news in the manual * coreutils.texi (pr invocation): remove list of ancient news items; the main documentation already covers what is needed. 2012-12-31 Jim Meyering maint: update all copyright year number ranges Run "make update-copyright", but then also run this, perl -pi -e 's/2\d\d\d-//' tests/sample-test to make that one script use the single most recent year number. build: update gnulib submodule to latest maint: adjust NEWS entry wording * NEWS: Adjust wording in a few entries. 2012-12-27 Pádraig Brady doc: improve od --help and man page * src/od.c: Redorder the information output by --help to ease interpretation and so that appropriate sections are generated by help2man. * doc/coreutils.texi (od invocation): Fix an incorrect reference to @var{n}, which should be @var{bytes}. * man/od.x: Add an "Examples" section, and move the default od format to there, and add a commonly required format to generate hexdumps. Reported by Akim Demaille in http://bugs.gnu.org/13280. 2012-12-21 Mike Frysinger factor: disable x86_64 assembly code for x32 builds The current x86_64 asm code does not work for x32 (__ILP32__) ABIs, so disable it. Note simply deleting the q suffix is not enough. * src/longlong.h: Restrict x86_64 assembly to _LP64 targets, which is consistent with other checks in longlong.h and avoids this code on x32. * NEWS: Mention the build fix. 2012-12-21 Pádraig Brady tests: avoid false positive valgrind failures * init.cfg (require_valgrind_): Check the `true` program, which will check more valgrind failure cases as now detailed in the function comments. dd: fix a printf format mismatch in an error message * src/dd.c (dd_copy): To print an off_t portably we need to use PRIdMAX and cast to intmax_t, otherwise there could be a mismatch between say a 32 bit off_t and uintmax_t. This was flagged by -Wformat on a 64 bit host when compiling with CFLAGS=-m32. 2012-12-21 Pádraig Brady seq: fix newline output when -s specified This regression was introduced in commit v8.19-132-g3786fb6. * src/seq.c (seq_fast): Don't use puts() to output the first number, and instead insert it into the buffer as for other numbers. Also output the terminator unconditionally. * tests/misc/seq.pl: Add some basic tests for the -s option. * NEWS: Mention the fix. * THANKS.in: Reported by Philipp Gortan. 2012-12-20 Bernhard Voelker tests: add tests for basename's --zero option The -z option has been introduced in commit v8.15-60-ga3eb71a, i.e. in coreutils-8.16. Time to add some tests for it. * tests/misc/basename.pl: Add tests exercising the -z option. In the foreach loop to append a newline to the end of each expected 'OUT' string, skip the -z tests. 2012-12-20 Assaf Gordon doc: tweak 'lcov' in HACKING * HACKING: In the paragraph about creating coverage statistics, use the correct -b (--base-directory) parameter. 2012-12-19 Pádraig Brady maint: rewrap a long line noticed by make syntax-check * configure.ac: Wrap the recently introduced long line. doc: clarify when dd iflag=fullblock is useful * doc/coreutils.texi (dd invocation): Explain that iflag=fullblock ensures that count= will count blocks rather than reads, and reference that in both the count= and iflag=fullblock descriptions. Suggested by John Reiser. tests: avoid a race in timeout-group.sh * tests/misc/timeout-group.sh: The kernel might possibly delay signal propagation to timeout.cmd long enough, that it exits normally without running the signal handler (as sleep will be in the same process group and so get the signal too). So avoid this by explicitly checking that the signal handler is called, which should always happen under normal circumstances. Reported by Stefano Lattarini on linux-2.6.30-2-686 and bash-4.2.36. build: avoid --enable-gcc-warnings on GCC <= 4.5 * configure.ac: Only enable warnings automatically when on GCC >= 4.6 (and when building from a git checkout) as that was the first GCC version to support fine-grained control of warnings, allowing them to be adjusted around certain code sections. gnulib relies on this for certain warnings, so avoid auto enabling this option lest we trigger build failures on now over two year old compilers. Reported by Zartaj Majeed with GCC 4.5.3 on cygwin. 2012-12-19 Zartaj Majeed build: fix cygwin build issues * doc/local.mk (doc/constants.texi): Ensure the doc directory is present which is needed when doing a non source dir build, when the doc/constants.texi target is built before other doc targets. * src/local.mk: Add $(EXEEXT) to the make-prime-list calls. 2012-12-17 Pádraig Brady maint: consistently use @dots{} in coreutils.texi * doc/coreutils.texi: s/\.\.\./@dots{}/ except in code and examples. 2012-12-17 Pádraig Brady readlink: support multiple command line arguments This allows efficient processing of multiple files, while also increasing compatibility with BSD's readlink(1). We also add the -z, --zero option to delimit output items with the NUL character which disambiguates output in the presence of '\n' characters. * src/readlink.c (usage): Add the --zero description, and also adjust the description of --no-newline accordingly. (main): Handle the -z option and iterate over multiple arguments. Also as in commit v8.15-24-g9d46b25 we use fputs() and putchar() rather than printf() for performance reasons. * doc/coreutils.texi (readlink invocation): Document the new --zero option, adjust the --no-newline description, and tweak the general info to indicate multiple files are supported. * tests/readlink/multi.sh: A new test for the new functionality. * tests/local.mk: Reference the new test. * man/readlink.x: Adjust the summary and also reference realpath. * NEWS: Mention the improvement. * THANKS.in: Suggested by Aaron Davies. 2012-12-14 Assaf Gordon maint: update gitignore entries * .gitignore: ignore GCC coverage data files. doc: mention "git stash" in HACKING * HACKING: In the paragraph about switching branches, mention "git stash" as a way to continue while preserving uncommitted changes. 2012-12-14 Bernhard Voelker tail,stat: improve support for the ceph file system Teach tail -f that it must use polling on ceph file systems, and let stat -f --format=%T report the file system type name, "ceph". Website: http://ceph.com/ * src/stat.c (human_fstype): Add a case: ceph, 0x00C36400, remote. * NEWS (Improvements): Mention it. * THANKS.in: Update. Reported by Konrad Wróblewski in http://bugs.gnu.org/13172. 2012-12-14 Bernhard Voelker maint: fix typos found by misspellings * doc/coreutils.texi (df invocation): s/occurence/occurrence/. * tests/df/skip-rootfs.sh: s/supressed/suppressed/ 2012-12-09 Jim Meyering tests: add test case and note that last week's cut change is a bug fix * tests/misc/cut-huge-to-eol-range.sh: New test, showing that the change in v8.20-51-g7d03466 is a bug fix after all. * tests/local.mk (all_tests): Add it. * NEWS (Bug fixes): Mention it. 2012-12-08 Bernhard Voelker cp: fix --no-preserve=mode to not exit 1 cp --no-preserve=mode exited 1 unconditionally. Furthermore, the tests which would have detected this error - namely link-preserve.sh and reserve-mode.sh - failed to test cp's exit code. * src/copy.c (copy_reg): In the case x->explicit_no_preserve_mode, do only set return_val to false iff the previous set_acl () failed. * tests/cp/link-preserve.sh: Check cp's exit code. * tests/cp/link-symlink.sh: Likewise. * tests/cp/preserve-mode.sh: Likewise. * NEWS: Mention the fix. Bug introduced in commit v8.19-145-g24ebca6. Reported by Florian Pritz in http://bugs.gnu.org/13119. 2012-12-07 Ondrej Oprala Bernhard Voelker. df: do not print duplicate entries and rootfs by default * src/df.c (struct devlist): Add new struct for storing already- examined device numbers. (devlist_head): Add new store of the above type. (show_rootfs): Add new global boolean to not skip rootfs. (dev_examined): Add new function to check if the device has already been traversed. (get_dev): Filter out rootfs unless "-t rootfs" or the -a option is specified. Filter out duplicate entries by calling the above new dev_examined unless the -a option is specified. (main): Set the show_rootfs variable appropriately when the -t option is specified for rootfs. Free device list (guarded by IF_LINT). * tests/df/skip-duplicates.sh: Add test to exercise the skipping of duplicate entries. * tests/df/skip-rootfs.sh: Add test to exercise the skipping of the rootfs pseudo file system. * tests/local.mk: Add the above new tests. * NEWS (Changes in behavior): Mention the changes. * doc/coreutils.texi (df invocation): Document df's behavior about skipping rootfs and duplicate entries. 2012-12-06 Cojocaru Alexandru cut: avoid a redundant heap allocation * src/cut.c (set_fields): Don't allocate memory for `printable_field' if there are no finite ranges. This is achieved by keeping max_range_endpoint as 0 when there are no finite ranges. max_range_endpoint is then used throughout the code to guard against allocation of, and access to the bit array. The extra allocation was introduced via commit v8.10-3-g2e636af. 2012-12-06 Pádraig Brady maint: fix a referenced coreutils version in a test comment * tests/misc/cut.pl: This particular bug existed up to v8.10. 2012-12-06 Bernhard Voelker tests: fix regex to match "-" in ipcent field in df/total-verify.sh The regular expression failed to match for file systems that do not provide inode statistics, e.g. VFAT or CIFS (depending on the underlying peer file system). * tests/df/total-verify.sh: Fix the regular expression to match a dash in the ipcent field again. Reported by Assaf Gordon in http://bugs.gnu.org/13099. Bug introduced in commit v8.20-18-gdae8d22. 2012-12-06 Assaf Gordon maint: adjust HACKING instructions to run a single test * HACKING: Adjust as per the recent changes introduced when switching to non recursive make. 2012-12-06 Pádraig Brady tests: cut.pl: adjust for changed diagnostic * tests/misc/cut.pl: Since we now output the more complete error message irrespective of running in a multi-byte locale or not, adjust the test accordingly. 2012-12-06 Cojocaru Alexandru cut: improve error reporting * src/cut.c (main): Treat a NUL delimiter (-d '') consistently with non NUL delimiters, and disallow such a delimiter option, unless a field is also specified. (set_fields): Provide a more accurate error message when a given list is invalid. * tests/misc/cut.pl: Add a test case. 2012-12-06 Pádraig Brady maint: adjust instructions to run a single test * README: Adjust as per the recent changes introduced when switching to non recursive make. 2012-12-05 Bernhard Voelker maint: remove now auto-added entry from THANKS.in The syntax-check sc_THANKS_in_duplicates complained about that excess entry. * THANKS.in (Colin Watson): Remove entry, now that it will be automatically included in the generated THANKS file. 2012-12-04 Pádraig Brady maint: remove a redundant odd sized alloc from factor.c * src/factor.c (mp_factor_init): Init the pointers with NULL, which xrealloc (or realloc) handles fine. 2012-12-04 Colin Watson factor: fix infinite loop on 32 bit powerpc Both Debian and Ubuntu builds of coreutils 8.20 hang while running the test suite on powerpc, which is reproducible using 'factor 122'. This turns out to be somewhat related to http://bugs.gnu.org/12754, but not quite the same. uintmax_t is 64 bits, but the cntlzw instruction takes 32-bit operands, and the cntlzd option is only available on 64-bit hardware. * src/longlong.h: Add an _LP64 check around the PPC64 code, so that this falls back to the C implementations. * NEWS: Mention the fix. 2012-11-24 Jim Meyering cut: do not print extraneous delimiters in some unusual cases When printing output delimiters, and when a to-EOL range subsumes at least one other range, cut would mistakenly print delimiters for the subsumed range. This bug was probably introduced via commit v5.2.1-639-g847e066. * src/cut.c (set_fields): Ignore any range that is subsumed by a to-EOL range. Also, move two declarations down. * tests/misc/cut.pl: Add tests to exercise this. * NEWS (Bug fixes): Mention it. Reported by Marcel Böhme in http://bugs.gnu.org/12966 cut: treat -b2-,3- like -b2-, not like -b3- * src/cut.c (set_fields): When two right-open-ended ranges are specified, don't blindly let the latter one take precedence over the former. Instead, use the union of the ranges. * tests/misc/cut.pl: Add tests to exercise this. * NEWS (Bug fixes): Mention it. Reported by Marcel Böhme in http://bugs.gnu.org/12966 Thanks to Berhard Voelker for catching log and NEWS typos. 2012-11-24 Bernhard Voelker doc: cp: clarify behavior of the --preserve=xattr option * doc/coreutils.texi (cp invocation): Enhance documentation of the --preserve=xattr option regarding the preservation of ACLs, SELinux contexts and capabilities: the user may notice this only when not specifying --preserve=mode and --preserve=context, too, i.e., otherwise, these attributes are preserved anyway. 2012-11-24 Pádraig Brady doc: update info cross reference to bash * doc/coreutils.texi (tee invocation): Reference the bash manual rather than bashref, as the latter is more likely to work on newer systems. tests: use sub-second timeouts to speed up a timeout test * tests/misc/timeout.sh: Take advantage of recent support for sub-second timeouts to decrease runtime from about 6s to 2s. doc: clarify that renice is needed to adjust niceness * doc/coreutils.texi (nice invocation): Ensure there is no ambiguity in the summary in relation to nice being able to adjust the niceness of an existing process. Reference the renice command. * man/nice.x: Reference renice (1) tests: accept EEXIST from rm -d * tests/rm/d-2.sh: EEXIST is a valid error on some systems. Reported by Michael Felt on AIX 6.1 tests: don't rely on sed -i being available * tests/df/df-output.sh: sed --in-place is not generally available. Also add a couple of simplifications from Bernhard Voelker. Removing all spaces, rather than just leading spaces, suffices. Searching for ' --output' in unadjusted df --help, suffices. seq: ensure correct output width for scientific notation input * src/seq.c (scan_arg): Calculate the width more accurately for numbers specified using scientific notation. * tests/misc/seq.pl: Add tests for cases that were mishandled. * NEWS: Mention the fix. * THANKS.in: Reported by Marcel Böhme. Fixes http://bugs.gnu.org/12959 2012-11-22 Bernhard Voelker df: reorder default field list of --output option As the inodes information is usually not so much of interest, and some file systems including btrfs do not even provide it, reading of the full df --output is easier when the block statistic fields come just left of the last field, the mount point. * src/df.c (all_args_string): Move the inodes fields before the block fields. (usage): Likewise. * tests/df/df-output.sh: Likewise. * doc/coreutils.texi (df invocation): Likewise. Additionally, explicitly mention the default order of the --output option. 2012-11-20 Paul Eggert install: fix security race * NEWS: Document this. install: fix security race * src/copy.c (copy_internal): Use DST_MODE_BITS, not SRC_MODE. See Bernhard R. Link in and in . 2012-11-20 Bernhard Voelker nl: remove deprecated --page-increment option The above option has been deprecated since coreutils-7.5 by commit v7.4-129-g718b279. * src/nl.c (PAGE_INCREMENT_OPTION_DEPRECATED): Remove enum. (longopts): Remove "page-increment" entry. (main): Remove PAGE_INCREMENT_OPTION_DEPRECATED case. * NEWS (Changes in behavior): Mention the change. Reported by Marcel Böhme in . 2012-11-19 Bernhard Voelker doc: remove already implemented items from TODO * TODO (renice): The renice program is part of util-linux for years now. Remove entry. (dd): The option status=none has been implemented in commit v8.19-143-g7331ab5. Remove entry. 2012-11-18 Bernhard Voelker cut: do not accept the invalid range 0- The command "echo 12345 | cut -b 0-" prints an empty line while it should fail with "fields and positions are numbered from 1". * src/cut.c (set_fields): Add a diagnostic for the invalid open range which starts with Zero, i.e., the range 0-. * tests/misc/cut.pl: Add tests to ensure the range 0- fails for fields (-f) and for positions (-b, -c). * NEWS: Mention the fix. Reported by Marcel Böhme in . 2012-11-18 Daniel Schepler build: fix compilation failure on x32 * src/factor.c [HAVE_ATTRIBUTE_MODE]: Fix typo in #if test: s/HAVE_LONG_LONG/HAVE_LONG_LONG_INT/. Otherwise, factor.c would elicit assembler errors on x32: it was incorrectly defining DItype to long instead of long long. Patch and report in http://bugs.debian.org/693337; Mike Stone notified upstream. 2012-11-18 Jim Meyering maint: reenable accidentally-disabled cppi-check syntax-check rule * cfg.mk (exclude_file_name_regexp--sc_preprocessor_indentation): Fix typo that disabled the sc_preprocessor_indentation syntax-check rule by exempting all files. s/__ll/_ll/ The typo was introduced in commit v8.19-157-g759ebcb. maint: correct indentation of W_TYPE_SIZE-defining cpp directives * src/factor.c: Indent cpp directives to reflect their nesting. maint: avoid i686-specific syntax-check failure * cfg.mk (_gl_TS_unmarked_extern_vars): Define. 2012-11-17 Paul Eggert doc: explain why dd is called "dd" * doc/coreutils.texi (dd invocation): Mention JCL. 2012-11-16 Paul Eggert factor: tidy up primes.h again See Stefano Lattarini in . * src/local.mk (BUILT_SOURCES): Put $(top_srcdir)/src/primes.h here (MAINTAINERCLEANFILES): ... instead of here. 2012-11-13 Paul Eggert factor: cleanup for primes.h * .gitignore: Add src/primes.h back, since it's no longer in the repository. 2012-11-12 Jim Meyering maint: avoid unnecessary #include to fix syntax-check failure * src/factor.c: Remove unneeded #inclusion of"verify.h". It's already included via system.h. 2012-11-12 Paul Eggert factor: improve primes.h change This follows suggestions by Jim Meyering in . * src/make-prime-list.c (print_wide_uint): Change "nested" argument to "nesting", and use it to avoid outputting lines that are too long. * src/primes.h: Remove from git. This can be generated by a maintainer. It's nicer to do so on a host with at least 128-bit arithmetic. factor: maintainer builds primes.h, not builder With this change, the maintainer builds primes.h and it is part of the tarball. primes.h's contents are not architecture-specific. * .gitignore: Remove /src/primes.h. * src/factor.c: Include verify.h. (W): New constant. Verify that uintmax_t lacks holes and that W is no wider than the integers used to generate primes.h. * src/local.mk (EXTRA_DIST): Add src/primes.h. (BUILT_SOURCES, CLEANFILES): Remove src/primes.h. ($(top_srcdir)/src/primes.h): Rename from src/primes.h. Do not depend on src/make-prime-list. Instead, use sub-make to build, so that we build primes.h only if it does not exist. * src/make-prime-list.c: Include , for ULONG_MAX. (wide_uint): Define to uintmax_t or unsigned __int128 if not #defined. (struct prime, binvert, process_prime): Use it instead of uintmax_t. (print_wide_uint): New function. This generates the proper pinv value regardless of the width of uintmax_t on the target, so long as the width doesn't exceed that of the width of wide_uint on the maintainer host that generated src/primes.h. (output_primes): Use it. Output WIDE_UINT_BITS, too. Let the target compute its own lim, since its uintmax_t may be narrower than ours. (SZ): Remove. * src/primes.h: New file, generated with 128-bit integers and usable on any host where uintmax_t's width is no greater than 128 bits. factor: port to systems with rpl_malloc * src/make-prime-list.c (malloc): Undef. This fixes a problem on AIX 8.20 reported by Michael Felt in . 2012-11-09 Bernhard Voelker tests: fix factor's tests to use coreutil's own sha1sum The test used the shasum utility which seems to belong to the perl package. On SLES-10.4, perl doesn't include this yet: + seq 0 10000000 + factor + shasum -c --status exp ./tests/factor/t00.sh: line 30: shasum: command not found + Exit 1 It is better to use our own stuff anyway. * tests/factor/run.sh: s/shasum/sha1sum/. Additionally, add sha1sum to the print_ver_ call. 2012-11-09 Benno Schulenberg doc: chcon: add descriptions for three undocumented options These were missed in this related change v8.14-104-g44e20cd * src/chcon.c (usage): Mention the two --preserve-root options. * doc/coreutils.texi (chcon invocation): Plus the --dereferece option. 2012-11-09 Benno Schulenberg doc: with cp -n, option -f is not redundant but ignored * src/cp.c (usage): Replace "redundant" with "ignored". * doc/coreutils.texi (cp invocation): Likewise. 2012-11-09 Paul Eggert df: port the new df test to POSIX sed, larger file systems * tests/df/df-output.sh: For the test "df -B1K --output=size", do not assume that the file system size fits in 9 bytes; it might be larger than that, so omit leading space. Also, use portable 'sed' commands: POSIX says sed commands inside { } should all end in newline. 2012-11-08 Bernhard Voelker df: add a test for the --output option * tests/df/df-output.sh: Add a test case. * tests/local.mk (all_tests): Mention the test. * cfg.mk (sc_file_system): Exempt the test from this syntax-check. df: document the new --output option * src/df.c (usage): Add a short description of --output and its available field names for use in the optional argument. * doc/coreutils.texi (df invocation): Add the new option with more details and a few examples. * NEWS (New features): Mention the new option. (Changes in behvaior): Mention the new placeholder for fields in the "total" line. 2012-11-08 Bernhard Voelker df: add --output to select which fields to display This supports changing the order of the fields displayed, and also allows the simultaneous display of inode and block fields. src/df.c (get_dev): Factor out calling get_header to ... (main): ... here. Call print_table only if file_systems_processed. src/df.c (Displayable fields): Rename DEV_FIELD to SOURCE_FIELD. Rename TYPE_FIELD to FSTYPE_FIELD. Rename FREE_FIELD to AVAIL_FIELD. Rename MNT_FIELD to TARGET_FIELD. * src/df.c (display_field_t): Turn loose enum definition of the displayable fields into a typedef. Add the inode fields ITOTAL_FIELD, IUSED_FIELD, IAVAIL_FIELD, IPCENT_FIELD. (field_data_t): Define structure to hold the display field, the caption, the width and the alignment for each field of the above type. (field_data): Add array the values of field data for each display field. (headers, alignments, widths): Remove arrays. (columns): Add a pointer to the storage for the array of the actual output columns, i.e., fields. (ncolumns): Add counter for the current output columns. (alloc_table_row): Allocate the dynamic ncolumns value of strings. (print_table): Loop over ncolumns instead of constant NFIELDS. Rename loop variable 'field' to 'col' to avoid ambiguity with the 'field' element in the columns structure. Adjust the condition for printing the last column by comparing with the column number instead of the field name (TARGET_FIELD). Use the width and the alignment stored in the columns data. (alloc_field): Add new function to allocate a field in the columns array. (get_field_list): Add new function to fill the array of output columns for each mode. (get_header): Loop over ncolumns instead of constant NFIELDS. Rename the loop variable 'field' to 'col' to avoid ambiguity with the 'field' element in the columns structure. Remove the code for continuing the loop if the current column is the file system type and print_type is not active (which is now impossible). Store the cell in the columns store along with the new width. (get_dev): Loop over ncolumns instead of the constant NFIELDS. Rename the loop variable 'field' to 'col' to avoid ambiguity with the 'field' element in the columns structure; move the definition down to where it is used first. Add cases for the inode fields ITOTAL_FIELD, IUSED_FIELD, IAVAIL_FIELD and IPCENT_FIELD. Store the cell in the columns store along with the new width. (main): Use new get_field_list function to fill the list of output columns. * src/df.c (print_table): Instead of fputs()ing directly, apply ambsalign on the last field, too. Use the new MBA_NO_RIGHT_PAD flag for this. * src/df.c (TOTAL_OPTION): Add new enum value. (long_options): Use it for the "total" option instead of 'c'. (main): Likewise. * src/df.c (get_dev): Remove condition to copy the fstype into the FSTYPE_FIELD - based on whether print_type is non-Null. Since the introduction of get_field_list(), there are only fields added to the columns array which have to be added. * src/df.c (get_dev): Guard the summing up of the values for the grand total: only do it if we have to print the total and if the current invocation is not for processing it. * src/df.c (main): Pass a hyphen "-" for the mount point name to get_dev. (get_dev): As the mount_point is now always there, remove the condition and the else case for the TARGET_FIELD. Instead, simply copy the mount_point. All cells are now always present. Therefore, add an assertion statement if one was not. Furthermore, hide the problematic characters unconditionally. (print_table): Remove the skipping of empty cells. * tests/df/total-verify.sh: Accommodate to the new "-" in the target field of the summary line. * NEWS: Mention the change in behavior. * src/df.c (field_type_t): Add new typedef of 3 enums to distinguish between block, inode and other fields. (field_data_t): Add field_type member of the above new type. (field_data): Add default values for the above field_type, indicating whether a field contains block values, inode values or other, generic values. (field_values_t): Add this struct to store the field values, used by and factored out from get_dev to be able to define such a struct for both the inode and the block values. (get_field_values): Add this function to obtain the block values and the inode values from the file system usage, used by and factored out from get_dev. (add_to_grand_total): Add this function to sum the values of the current mount point up for the grand total, used by and factored out from get_dev. (get_dev): Move the definition of the variables fsu, buf, pct and cell down to where they are used first to give them a better scope. Factor out input_units, output_units, total, available, negate_available, available_to_root, used and negate_used into the above struct field_values_t. Factor out the mapping of the fsu values to the above variables into above function get_field_values. Factor out the summing up of the grand total values into the above function add_to_grand_total. Define block_values and inode_values of the new type and call the new get_field_values to fill them from the fsu values. Call the above function add_to_grand_total for summing up the values for the grand total. Inside the loop over all fields, define a variable 'v' to point to either the block_values or the inode_values, depending on the current field's field_type. Change the code in the cases TOTAL_FIELD/ITOTAL_FIELD, USED_FIELD/IUSED_FIELD, AVAIL_FIELD/ IAVAIL_FIELD and PCENT_FIELD/IPCENT_FIELD to use the field values where 'v' is pointing to, i.e., either the block_values or the inode_values. * src/df.c (main): Remove setting of grand_fsu.fsu_blocks in the inode_format case as this is no longer needed and would lead to wrong results once when mixed block/inode fields will be used. * src/df.c (main): Cleanup the code at the end regarding file_systems_processed to make the code clearer. * src/df.c (inode_format): Remove variable. (main): Remove initialization of the above variable. In getopts loop, directly set the header_mode to INODES_MODE instead of using the above variable. Afterwards, remove the mapping to INODES_MODE as it is already set. * src/df.c (posix_format): Move variable ... (main): ... to here. * src/df.c (print_table): Enhance the comment about 2-line format in cases where the SOURCE_FIELD exceeds 20 chars, as such behavior has been removed long ago by commit v8.10-40-g99679ff. * src/df.c (Display modes): Add OUTPUT_MODE, remove unused NMODES. (display_field_t): Remove unnecessary NFIELDS. (field_data_t): Add member 'arg' for the field name in the --output argument. Add member 'used' to remember if a field is already used in the columns array. (field_data): Add values for the above new members arg and used. (all_args_string): Add variable which represents the argument for the --output option which includes all fields. (OUTPUT_OPTION): Add enum to identify the long --output option. (long_options): Add optional-argument --output option. (alloc_field): Assert that the field is not already used. Mark the field as used. (decode_output_arg): Add function to parse the comma-separated field list passed to the --output option in order to add the appropriate fields to the columns array. (get_field_list): Add case for the new OUTPUT_MODE to add all available fields to columns in the case the --output option has been passed without any values. Use the comma-separated field list form to pass to decode_output_arg to keep the field header mapping for the OUTPUT_MODE only on one place. (main): Define format string msg_mut_excl to be used in the following checks whether the use of --output and the other option is mutually exclusive. In the getopt_long loop, add a check to the case for the -i option to issue an error message when it is used together with --output; Likewise for -T and -P. Add a new case for OUTPUT_OPTION, together with similar checks as above and eventually passing the optarg to decode_output_arg. After the getopt_long loop, consider the OUTPUT_MODE case in order not to run into -h or -P mode. * src/df.c (get_dev): Also xstrdup the dev_name, and free it afterwards to silence a valgrind warning about definitely lost memory. (main): Free the columns store to silence valgrind, guarded by the IF_LINT macro. * src/df.c (main): Pass "total" as the mount point to get_dev if the SOURCE_FIELD is not among the columns to output. * tests/df/df-output.sh: Change the test to ensure the content of the target field of the grand total line: if the source field is present, then the target should be "-", else the target field should be "target". * NEWS (Changes in behavior): Enhance the exiting NEWS entry. * doc/coreutils.texi (df invocation): Document the content of the source and target field in the grand total line. * src/df.c (main): Add another condition to the need_fs_type parameter of read_file_system_list whether the FSTYPE_FIELD is used or not. * src/df.c (get_header): Indicate the block size used, in the "size" header, when using --output without -h. * tests/df/df-output.sh: Adjust for, and add an extra test for, the new behavior. 2012-11-05 Bernhard Voelker maint: fix factor's test file name pattern in .gitignore * .gitignore: Adapt the entry for the files generated during the factor tests (see v8.20-9-g1cace4a). 2012-11-03 Ángel González timeout: add --preserve-status to always propagate the exit status It's useful for commands that support running for an indeterminite amount of time, to not return a specific timeout exit status (124), and instead let the command handle the timeout signal and return a status for the work done so far. * doc/coreutils.texi (timeout invocation): Describe the new option. * src/timeout.c (preserve_status): A new global boolean to enable the --preserve-status behavior. (usage): Describe the new option. (main): Don't return EXIT_TIMEOUT of preserve_status is set. * tests/misc/timeout.sh: Add a test for the new option. 2012-11-03 Pádraig Brady maint: fix the error message from sc_check-I18N-AUTHORS * cfk.mk (sc_check-I18N-AUTHORS): Fix the quoting so you don't get a 'command not found' error. build: avoid build failure on some HPPA systems * src/longlong.h: Restrict some HPPA assembly variants to PA RISC V2.0. Note we also avoid this assembly for ilp32 runtimes, since even though the assembly is accepted there, it's not safe as the context can get clobbered between the 'add' and 'add,dc'. This fixes a compile failure with newer HPPA systems with default GCC CPU options. Reported by John David Anglin 2012-11-03 Pádraig Brady build: don't assume uintmax_t is 64 bits This was not seen to be an issue in practise, but to make the code more robust, don't assume uintmax_t is 64 bits. * src/factor.c (W_TYPE_SIZE): Define based on integer limits. * src/make-prime-list.c (output_primes): Define format width based on integer limits. 2012-11-03 Pádraig Brady build: don't rely on support for '%j' printf format * src/factor.c (print_uintmaxes): Replace with PRIuMAX. * src/join.c (check_order): Likewise. * cfg.mk (sc_check-j-printf-format): Add a syntax-check rule to flag new cases of this. build: don't rely on support for '%*j' printf format * src/make-prime-list.c: Hardcode the uintmax_t width to 16 hex digits (64 bit), to avoid this compile issue on HPUS systems at least. 2012-11-01 Bernhard Voelker tests: fix syntax error in setuidgid_has_perm_ An invalid sed expression lead to the following error: + sed -n 'ls/.* //p' sed: -e expression #1, char 2: extra characters after command Use "one" instead of "el". Bug introduced in v8.19-111-g51a4b04. * init.cfg (setuidgid_has_perm_): s/ls/1s/. * NEWS (Build-related): Mention the fix. 2012-10-27 Bernhard Voelker tests: shorten factor's inordinately-long test file names Besides what the subject says, this commit moves the test data for the factor tests from tests/local.mk into the directory tests/factor/ where it belongs. * tests/local.mk (EXTRA_DIST): Add new tests/factor/create-test.sh. (p,q,t1,t2) Factor out the factor-related magic numbers. (factor_tests): Rename the test names to t{00..36}.sh. Factor out the triples of test data. ($(factor_tests)): Add dependency to new tests/factor/create-factor.sh. Call that script to generate the test scripts. * tests/factor/run.sh: Turn this script into a template, and therefore remove it's executable permission bit. Add template variables START, END and CKSUM, replacing the code to split the test data from the test script's file name. Use the new template variables in the call to seq and for creating the exp file. * tests/factor/create-test.sh: Add new script to create the test scripts from the template tests/factor/run.sh. Use test data and magic numbers factored out from the above files. Let the script also change the __TEMPLATE__ line in run.sh to make clear that the test scripts are generated. * cfg.mk (sc_tests_list_consistency): Exempt the new test. (exclude_file_name_regexp--sc_prohibit_test_backticks): Likewise. Improved-by: Stefano Lattarini Improved by: Jim Meyering 2012-10-26 Stefano Lattarini build: man page generation: degrade gracefully when perl is lacking Since commit v8.19-118-g57da212, our 'dist-hook' rule tweaked the distributed Makefile.in to make each man page 'man/foo.1' depend on the corresponding source 'src/foo.c' rather than the corresponding program 'src/foo'. That was done to accommodate systems without perl, which cannot run help2man to regenerate the manpage after its corresponding program has been built. This seems a right and proper graceful degradation, in that the man pages dependencies are still 100% correct in a git checkout, while being more lax but "more portable" in a distribution tarball. Alas, that is not the case in practice, as it turns out the tweaked Makefile makes the building of man pages unreliable and potentially incorrect! In fact, assume that instead of the correct a dependency: man/ls.1: src/ls we have the laxer one: man/ls.1: src/ls.c and think of what happens if a user modifies, say, 'src/ls.c', and then runs "make -j4" to rebuild everything. The make process will see that it has to rebuild the man page 'man/ls.1' (because its prerequisite 'src/ls.c' has changed), but won't see that it has to rebuild 'src/ls' *before* re-running 'help2man' to generate that man page; so, if 'man/ls.1' is rebuilt before 'src/ls' (which can happen with concurrent make), our user will get either a build error (if 'src/ls' did non exist) or, worse, a man page with an up-to-date timestamp but an out-of-date content. And what's even worse in all of this is that this problem will be present also for users who have perl installed: this is not a "graceful degradation" at all! In our situation, the best and simplest way to implement a graceful degradation it to keep the correct dependencies for man pages (that is, "man/ls.1: src/ls"), and if perl is not present, just generate dummy man pages reporting that built-time issue and redirecting the user back to either the info documentation or the '--help' output. As a consequence of this change, we also stop distributing man pages, since they will now be generated unconditionally. * Makefile.am (do-not-require-help2man): Remove. (dist-hook): Don't depend on it. * man/local.mk: Remove an obsolete comment. (EXTRA_DIST): Stop distributing generated man pages. ($(EXTRA_MANS)): This no longer needs to depend on $(all_programs). (MAINTAINERCLEANFILES): $(ALL_MANS) Do not list it here, and ... (CLEANFILES): ... list it here, instead. (.x.1): Instead of warning if perl is missing, but then trying to run 'help2man' unconditionally, simply run ... (run_help2man): ... the command referenced by this new variable, that expands to a proper invocation of 'help2man' if perl is present, and to an invocation of a shell script generating a dummy manpage if it is not. (EXTRA_DIST): Distribute that shell script. * man/dummy-man: New shell script. * NEWS (Build-related): Mention this. Fixes coreutils http://bugs.gnu.org/12715. 2012-10-25 Jim Meyering maint: tweak sc_check-I18N-AUTHORS * cfg.mk (sc_check-I18N-AUTHORS): Remove unnecessary subshell; add a self-explanatory diagnostic. maint: remove an obsolete test comment * tests/du/bind-mount-dir-cycle.sh: Remove obsolete comment spotted by Gilles Espinasse. 2012-10-25 Pádraig Brady pr: fix -n to pad consistently and not crash * src/pr.c: Replace the code to truncate the most significant digits of line numbers, with much simpler string manipulation that supports arbitrary widths. Before this, specifying a width >= 32 to -n would result in a divide by zero error. Also remove the inconsistent padding with zeros and spaces, which would result in zero padding for widths 12 and 15. * tests/pr/pr-tests.pl: Added a test to ensure no zero padding, and also a test for the divide by zero case. * NEWS: Mentioned the fix Reported by Ondrej Oprala build: ensure factor links the iconv library * src/local.mk (src_factor_LDADD): Append $(LIBICONV). * crg.mk (sc_check-I18N-AUTHORS): A new syntax check rule to ensure we add LIBICONV where appropriate. * THANKS.in: Add the reporter. Reported by Christian Jullien Syntax check suggested by Jim Meyering maint: refactor tests/misc/pr.pl into tests/pr/pr-tests.pl * tests/misc/pr.pl: Refactor this test into ... * tests/pr/pr-tests.pl: ... here. * tests/local.mk: Remove the reference to the removed test Improved by Jim Meyering 2012-10-24 Mike Frysinger build: do not redefine _FORTIFY_SOURCE When _FORTIFY_SOURCE is defined by the compiler or via flags, as it is on Gentoo, do not override it. Otherwise we get many redefinition warnings. * configure.ac (FORTIFY_SOURCE): Check if _FORTIFY_SOURCE is defined. This addresses http://bugs.gnu.org/12714 2012-10-23 Pádraig Brady maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 8.20 * NEWS: Record release date. See the source repo for older entries