Before this change, only plain directories, tar or zip archives were supported
as the source of a package for the GNU build system; anything else would cause
the unpack phase to fail. Origins relying on snippets would suffer from the
same problem.
This change adds the support to use files of the following extensions: .gz,
.Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unknown
extensions are treated as uncompressed files and supported as well.
* guix/packages.scm (patch-and-repack): Only add the compressor utility to the
PATH when the file is compressed. Bind more inputs in the mlet, and use them
for decompressing single files. Adjust the decompression and compression
routines.
[decompression-type]: Remove nested variable.
* guix/build/utils.scm (compressor, tarball?): New procedures. Move
%xz-parallel-args to the new 'compression helpers' section.
* tests/packages.scm: Add tests. Add missing copyright year for Jan.
* guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no
sub-directory was found.
(unpack): Support more file types, including uncompressed plain files.
Fixes <https://issues.guix.gnu.org/30116>.
Before this change, the presence of a NUL character on a line meant that
the (glibc) regexp engine used by Guile would either 1. stop scanning the
string or 2. crash with the error "string contains #\\nul character",
depending on the locale used.
This change works around this limitation by first replacing the NUL character
by an unused Unicode code point, doing the substitution, then reverting the
replacement.
* guix/build/utils.scm (unused-private-use-code-point)
(replace-char): New procedures.
(substitute): Make use of the above procedures to work around the NUL
character regexp engine limitation.
* tests/build-utils.scm: Add tests.
Co-authored-by: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Signed-off-by: Maxim Cournoyer <maxim.cournoyer@gmail.com>
* guix/scripts/hash.scm (guix-hash): When 'recursive? is true, use
'open-hash-port' instead of 'open-sha256-port'.
* tests/guix-hash.sh: Add test for 'guix hash -r -H sha512'.
Suggested by Philippe Swartvagher <philippe.swartvagher@inria.fr>.
* guix/transformations.scm (transform-package-patches): New procedure.
(%transformations): Add it as 'with-patch'.
(%transformation-options, show-transformation-options-help/detailed):
Add '--with-patch'.
* tests/transformations.scm ("options->transformation, with-patch"): New
test.
* doc/guix.texi (Package Transformation Options): Document it.
Fixes <https://bugs.gnu.org/45453>.
* guix/import/utils.scm (alist->package): Find plain license names in (guix
licenses) first, and only fall back to SPDX names on error.
* tests/import-utils.scm ("alist->package with SPDX license name 1/2",
"alist->package with SPDX license name 2/2"): New tests.
* gnu/bootloader/grub.scm (grub-configuration-file): New parameter
store-crypto-devices.
[crypto-devices]: New helper function.
[builder]: Use crypto-devices.
* gnu/machine/ssh.scm (roll-back-managed-host): Use
boot-parameters-store-crypto-devices to provide its contents to the
bootloader configuration generation process.
* gnu/tests/install.scm (%encrypted-root-not-boot-os,
%encrypted-root-not-boot-os): New os declaration.
(%encrypted-root-not-boot-installation-script): New script, whose contents
were initially taken from %encrypted-root-installation-script.
(%test-encrypted-root-not-boot-os): New test.
* gnu/system.scm (define-module): Export
operating-system-bootoader-crypto-devices and
boot-parameters-store-crypto-devices.
(<boot-parameters>): Add field store-crypto-devices.
(read-boot-parameters): Parse store-crypto-devices field.
[uuid-sexp->uuid]: New helper function extracted from
device-sexp->device.
(operating-system-bootloader-crypto-devices): New function.
(operating-system-bootcfg): Use
operating-system-bootloader-crypto-devices to provide its contents to
the bootloader configuration generation process.
(operating-system-boot-parameters): Add store-crypto-devices to the
generated boot-parameters.
(operating-system-boot-parameters-file): Likewise to the file with
the serialized structure.
* guix/scripts/system.scm (reinstall-bootloader): Use
boot-parameters-store-crypto-devices to provide its contents to the
bootloader configuration generation process.
* tests/boot-parameters.scm (%default-store-crypto-devices): New
variable.
(%grub-boot-parameters, test-read-boot-parameters): Use
%default-store-crypto-devices.
(tests store-crypto-devices): New tests.
Guile-JSON 4.3.2 would parse in spite of these typos, but 4.4.1 is
stricter.
* tests/swh.scm (%directory-entries): Add missing comma.
* tests/cve-sample.json: Likewise.
* tests/store.scm ("substitute")
("substitute + build-things with output path")
("substitute + build-things with specific output"): Call 'canonical-file?'.
* tests/substitute.scm ("substitute, authorized key"): Check the mtime
and permissions of "substitute-retrieved".
This way, the hash of the store item can be computed as it is restored,
thereby avoiding an additional file tree traversal ('hashPath' call)
later on in the daemon. Consequently, it should reduce latency between
subsequent substitute downloads.
This is a followup to 5ff521452b.
* guix/scripts/substitute.scm (narinfo-hash-algorithm+value): New
procedure.
(process-substitution): Wrap INPUT into a hash input port, 'hashed', and
read from it. Compare the actual and expected hashes, and print a
"hash-mismatch" status line when they differ. When they match, print
not just "success" but also the nar hash and size.
* nix/libstore/build.cc (class SubstitutionGoal)[expectedHashStr]:
Remove.
(SubstitutionGoal::finished): Tokenize 'status'. Parse it and handle
"success" and "hash-mismatch" accordingly. Call 'hashPath' only when
the returned hash is not SHA256.
(SubstitutionGoal::handleChildOutput): Remove 'expectedHashStr'
handling.
* tests/substitute.scm ("substitute, invalid hash"): Rename to...
("substitute, invalid narinfo hash"): ... this.
("substitute, invalid hash"): New test.
This avoids repeated (mkdir-p "/gnu/store/.links") calls when
deduplicating lots of files.
* guix/store/deduplication.scm (deduplicate): Remove initial call to
'mkdir-p'. Add ENOENT case in 'link' exception handler. Reindent.
* tests/store-deduplication.scm ("deduplicate, ENOSPC"): Check
for (<= links 4) to account for the initial 'link' call.
Until now deduplication was performed as an additional pass after
copying files, which involve re-traversing all the files that had just
been copied.
* guix/store/deduplication.scm (copy-file/deduplicate): New procedure.
* tests/store-deduplication.scm ("copy-file/deduplicate"): New test.
* guix/build/store-copy.scm (populate-store): Add #:deduplicate?
parameter and honor it.
* tests/gexp.scm ("gexp->derivation, store copy"): Pass #:deduplicate? #f
to 'populate-store'.
* gnu/build/image.scm (initialize-root-partition): Pass #:deduplicate?
to 'populate-store'. Pass #:deduplicate? #f to 'register-closure'.
* gnu/build/vm.scm (root-partition-initializer): Likewise.
* gnu/build/install.scm (populate-single-profile-directory): Pass
#:deduplicate? #f to 'populate-store'.
* gnu/build/linux-initrd.scm (build-initrd): Likewise.
* guix/scripts/pack.scm (self-contained-tarball)[import-module?]: New
procedure.
[build]: Pass it as an argument to 'source-module-closure'.
* guix/scripts/pack.scm (squashfs-image)[build]: Wrap in
'with-extensions'.
* gnu/system/linux-initrd.scm (expression->initrd)[import-module?]: New
procedure.
[builder]: Pass it to 'source-module-closure'.
* gnu/system/install.scm (cow-store-service-type)[import-module?]: New
procedure. Pass it to 'source-module-closure'.
Until now, 'populate-store' would reset permissions but not timestamps,
so callers would resort to going through an extra directory traversal to
reset timestamps.
* guix/build/store-copy.scm (reset-permissions): Remove.
(copy-recursively): New procedure.
(populate-store): Pass #:keep-permissions? to 'copy-recursively'.
Remove call to 'reset-permissions'.
* tests/gexp.scm ("gexp->derivation, store copy"): In BUILD-DRV, check
whether 'populate-store' canonicalizes permissions and timestamps.
* gnu/build/image.scm (initialize-root-partition): Pass #:reset-timestamps? #f
to 'register-closure'.
* gnu/build/vm.scm (root-partition-initializer): Likewise.
This avoids having to traverse and re-read the files that we have just
restored, thereby reducing I/O.
* guix/serialization.scm (dump-file): New procedure.
(restore-file): Add #:dump-file parameter and honor it.
* guix/store/deduplication.scm (tee, dump-file/deduplicate): New
procedures.
* guix/nar.scm (restore-one-item): Pass #:dump-file to 'restore-file'.
(finalize-store-file): Pass #:deduplicate? #f to 'register-items'.
* tests/nar.scm <top level>: Call 'setenv' to set "NIX_STORE".
* guix/serialization.scm (restore-file): Set the permissions and mtime
of FILE.
* guix/nar.scm (finalize-store-file): Pass #:reset-timestamps? #f to
'register-items'.
* tests/nar.scm (rm-rf): Add 'chmod' calls to ensure files are writable.
("write-file + restore-file with symlinks"): Ensure every file in OUTPUT
passes 'canonical-file?'.
* tests/guix-archive.sh: Run "chmod -R +w" before "rm -rf".
* guix/serialization.scm (fold-archive): Call PROC with a
'directory-complete tag when done with a directory.
(restore-file): Handle it.
* guix/scripts/archive.scm (list-contents): Likewise.
* guix/scripts/challenge.scm (archive-contents): Likewise.
* tests/nar.scm ("write-file-tree + fold-archive"): Adjust accordingly.
This is a followup to a8dccd4bdc, which
broke the test.
* guix/import/opam.scm (get-opam-repository): Prevent inlining.
* tests/opam.scm ("opam->guix-package"): Mock 'get-opam-repository'.
This avoids spawning one substitute process per substitution.
* nix/libstore/build.cc (class Worker)[substituter]: New field.
[outPipe, logPipe, pid]: Remove.
(class SubstitutionGoal)[expectedHashStr, status, substituter]: New fields.
(SubstitutionGoal::timedOut): Adjust to check 'substituter'.
(SubstitutionGoal::tryToRun): Remove references to 'outPipe' and
'logPipe'. Run "guix substitute --substitute" as an 'Agent'. Send the
request with 'writeLine'.
(SubstitutionGoal::finished): Likewise.
(SubstitutionGoal::handleChildOutput): Change to fill in
'expectedHashStr' and 'status'.
(SubstitutionGoal::handleEOF): Call 'wakeUp' unconditionally.
(SubstitutionGoal::~SubstitutionGoal): Adjust to check 'substituter'.
* guix/scripts/substitute.scm (process-substitution): Write "success\n"
to stdout upon success.
(%error-to-file-descriptor-4?): New variable.
(guix-substitute): Set 'current-error-port' to file descriptor 4
unless (%error-to-file-descriptor-4?) is false.
Remove "--substitute" arguments. Loop reading line from stdin.
* tests/substitute.scm <top level>: Call '%error-to-file-descriptor-4?'.
(request-substitution): New procedure.
("substitute, no signature")
("substitute, invalid hash")
("substitute, unauthorized key")
("substitute, authorized key")
("substitute, unauthorized narinfo comes first")
("substitute, unsigned narinfo comes first")
("substitute, first narinfo is unsigned and has wrong hash")
("substitute, first narinfo is unsigned and has wrong refs")
("substitute, two invalid narinfos")
("substitute, narinfo with several URLs"): Adjust to new "guix
substitute --substitute" calling convention.
* nix/libstore/local-store.hh (RunningSubstituter): Remove.
(LocalStore)[runningSubstituter]: Change to unique_ptr<Agent>.
[setSubstituterEnv, didSetSubstituterEnv]: Remove.
[getLineFromSubstituter, getIntLineFromSubstituter]: Take an 'Agent'.
* nix/libstore/local-store.cc (LocalStore::~LocalStore): Remove
reference to 'runningSubstituter'.
(LocalStore::setSubstituterEnv, LocalStore::startSubstituter): Remove.
(LocalStore::getLineFromSubstituter): Adjust to 'run' being an 'Agent'.
(LocalStore::querySubstitutablePaths): Spawn substituter agent if
needed. Adjust to 'Agent' interface.
(LocalStore::querySubstitutablePathInfos): Likewise.
* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Remove call to
'setSubstituterEnv' and add 'setenv' call for "_NIX_OPTIONS" instead.
(SubstitutionGoal::finished): Remove 'readLine' call for 'dummy'.
* guix/scripts/substitute.scm (%allow-unauthenticated-substitutes?):
Remove second argument to 'make-parameter'.
(process-query): Call 'warn-about-missing-authentication'
when (%allow-unauthenticated-substitutes?) is #t.
(guix-substitute): Wrap body in 'parameterize'. Set 'guix-warning-port'
too. No longer exit when 'substitute-urls' returns the empty list. No
longer print newline initially.
* tests/substitute.scm (test-quit): Parameterize 'current-error-port' to
account for the port changes in 'guix-substitute'.
In particular, "<origin> patches: same file name -> no warnings" would
have caught the issue which was fixed in commit
21887021b9.
* tests/lint.scm (patches: file names): Rename this test case...
("file patches: different file name -> warning"): ... to this.
("file patches: same file name -> no warnings")
("<origin> patches: different file name -> warning")
("<origin> patches: same file name -> no warnings"): New test cases.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes <https://bugs.gnu.org/23874>.
Reported by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>.
* guix/profiles.scm (manifest-transaction-effects): Delete duplicates in
install and remove. Let multiple upgrades and downgrades shadow previous
transactions of the same kind.
* tests/profiles.scm
("manifest-transaction-effects no double install or upgrades")
("manifest-transaction-effects no double downgrade")
("manifest-transaction-effects no double removal"): New tests.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
* guix/import/crate.scm: Add comment for the 'module-autoload!' calls.
* tests/crate.scm (have-guile-semver?): New variable.
("crate->guix-package", "cargo-recursive-import")
("cargo-recursive-import-hoors-existing-packages"): Skip when
HAVE-GUILE-SEMVER? is false.
If a package satisfying the dependency's semver requirement already exists,
use it. Prior to this change the highest version matching the semver
requirement was used (and imported in case it was not defined as package
already).
When resolving a dependency (now done in `sort-map-dependencies`), first
search for a package matching the semver requirement and only if this fails
reach out for a crate.
* guix/import/crate.scm (crate->guix-package)[find-package-version]: New
function. [dependency-name+version]: New function.
[sort-map-dependencies]: Use it instead of lambda function.
* tests/crate.scm (test-doctool-crate, test-doctool-dependencies): New
variables.
("self-test …", "cargo-recursive-import-hoors-existing-packages"): New
tests.
This complies to how versions are matched for caret requirements in crates:
An update is allowed if the new version number does not modify the left-most
non-zero digit in the major, minor, patch grouping.
* guix/import/crate.scm (version->semver-prefix): New function.
(make-crate-sexp)[format-inputs]: Use it.
(make-crate-sexp): Use it to pass shorter version to package->definition.
* guix/import/utils.scm (package->definition): Change optional parameter
APPEND-VERSION? into APPEND-VERSION?/STRING. If it is a string, append its
value to name.
* tests/crate.scm: Adjust tests accordingly.
This remove the patch version from generated package names. For example
'rust-my-crate-1.1.2' now becomes 'rust-my-crate-1.1'.
* guix/import/utils.scm (package->definition): Trim patch version from
generated package names.
* tests/crate.scm: (cargo>guix-package, cargo-recursive-import): Likewise.
The recursive crate importer will now include development dependencies only
for the top level package, but not for any of the recursively imported
packages. Also #:skip-build will be false for the top-most package.
* guix/import/crate.scm (make-crate-sexp): Add the key BUILD?.
(crate->guix-package): Add the key INCLUDE-DEV-DEPS?.
(crate-recursive-import): Likewise.
* guix/scripts/import/crate.scm (guix-import-crate): Likewise.
* tests/crate.scm (cargo-recursive-import): Likewise.
* guix/import/crate.scm: Add guile-semver as a soft dependency.
(make-crate-sexp): Don't allow other keys. Add '#:skip-build?' to build
system args. Pass a VERSION argument to 'cargo-inputs'.
(crate->guix-package): Use guile-semver to resolve the correct module
versions. Treat "build" dependencies as normal dependencies.
(crate-name->package-name): Reuse the procedure 'guix-name' instead of
duplicating its logic.
* guix/import/utils.scm (package-names->package-inputs): Implement
handling of (name version) pairs.
* guix/scripts/import/crate.scm (guix-import-crate): Use
crate-recursive-import instead of duplicate code.
* tests/crate.scm (recursive-import): Change test packages versions to be
distinguishable. Add version data to the test. Check created symbols, too.
Co-authored-by: Hartmut Goebel <h.goebel@crazy-compilers.com>
This adds a key VERSION to 'recursive-import' and moves the parameter REPO to
a key. This also changes all the places that rely on 'recursive-import'.
* guix/import/utils.scm (recursive-import): Add the VERSION key. Make REPO a
key.
(package->definition): Add optional 'append-version?'.
* guix/scripts/import/crate.scm (guix-import-crate): Add the VERSION key.
* guix/import/crate.scm (crate->guix-package): Add the VERSION key.
(crate-recursive-import): Pass VERSION to recursive-import, remove now
unnecessary code.
* guix/import/cran.scm (cran->guix-package, cran-recursive-import): Change the
REPO parameter to a key.
* guix/import/elpa.scm (elpa->guix-package, elpa-recursive-import): Likewise.
* guix/import/gem.scm (gem->guix-package, recursive-import): Likewise.
* guix/import/opam.scm (opam-recurive-import): Likewise.
* guix/import/pypi.scm (pypi-recursive-import): Likewise.
* guix/import/stackage.scm (stackage-recursive-import): Likewise.
* guix/scripts/import/cran.scm (guix-import-cran): Likewise.
* guix/scripts/import/elpa.scm (guix-import-elpa): Likewise.
* tests/elpa.scm (eval-test-with-elpa): Likewise.
* tests/import-utils.scm (recursive-import): Likewise.
Co-authored-by: Hartmut Goebel <h.goebel@crazy-compilers.com>
* guix/lint.scm (check-haskell-stackage): New procedure.
(%network-dependent-checkers): Add 'haskell-stackage' checker.
* guix/import/hackage.scm (%hackage-url): New variable.
(hackage-source-url, hackage-cabal-url): Use it in place of a
hard-coded string.
* guix/import/stackage.scm (%stackage-url): Make it a parameter.
(stackage-lts-info-fetch): Update accordingly.
* tests/lint.scm ("hackage-stackage"): New test.
Fixes <https://bugs.gnu.org/44626>.
Reported by Vagrant Cascadian <vagrant@debian.org>.
* tests/build-utils.scm ("wrap-script, simple case"): Pass
SCRIPT-CONTENTS to 'display' rather than 'format'.
* gnu/services/base.scm (file-system->shepherd-service-name)
[valid-characters, mount-point]: New variables.
Filter out invalid store file name characters from the mount point of
FILE-SYSTEM.
Reported by Ricardo Wurmus <rekado@elephly.net>.
* guix/scripts/publish.scm (compress-nar): Add 'chmod' call to ensure
PORT is #o644, in the uncompressed case.
(bake-narinfo+nar): Likewise for the narinfo file.
* tests/publish.scm ("with cache"): Check permissions on CACHED and NAR.
While importing those modules from the host system is valid, it is often
a mistake that introduces non-reproducibility. This patch prints a
warning when that happens.
* guix/gexp.scm (gexp-attribute): Add #:validate parameter and honor it.
(gexp-modules)[validate-modules]: New procedure.
Pass it to 'gexp-attribute'.
* tests/gexp.scm ("gexp-modules, warning"): New test.
* guix/gexp.scm (<gexp>)[location]: New field.
(gexp-location): New procedure.
(write-gexp): Print the location of GEXP.
(gexp->derivation): Adjust call to 'make-gexp'.
(gexp): Likewise.
Fixes <https://bugs.gnu.org/44442>.
Regression introduced in 2b2ab7796a.
* guix/scripts/publish.scm (render-nar/cached): Do not call
'bypass-cache?' when ITEM is the empty string.
* tests/publish.scm ("with cache, cache bypass, unmapped hash part"):
New test.
Fixes <http://issues.guix.gnu.org/44196>
* gnu/machine/ssh.scm (roll-back-managed-host): Use
boot-parameters-store-directory-prefix.
* gnu/system.scm (define-module): Export
boot-parameters-store-directory-prefix.
(<boot-parameters>)[store-directory-prefix]: New field. It is used to
generate the correct paths when /gnu/store is installed on a btrfs
subvolume whose name doesn't match the final runtime path, as the
bootloader doesn't have knowledge about the final mounting points.
[boot-parameters-store-directory-prefix]: New accessor.
(read-boot-parameters): Read directory-prefix from store field.
(operating-system-boot-parameters-file): Add directory-prefix to
store field.
* guix/scripts/system.scm (reinstall-bootloader): Use
boot-parameters-store-directory-prefix.
* test/boot-parameters.scm (%default-btrfs-subvolume,
%default-store-directory-prefix): New variables.
(%grub-boot-parameters): Use %default-store-directory-prefix.
(%default-operating-system): Use %default-btrfs-subvolume.
(test-boot-parameters): Add directory-prefix.
(test optional fields): Add test for directory-prefix.
(test os store-directory-prefix): New test.
Fixes <https://bugs.gnu.org/44261>.
Reported by Jan Nieuwenhuizen <janneke@gnu.org>.
* gnu/packages/aux-files/run-in-namespace.c (exec_in_user_namespace):
Add call to 'prctl'. Call 'mount' for NEW_ROOT and define 'is_tmpfs'.
When IS_TMPFS is true, call 'umount' and 'rmdir' after 'waitpid';
otherwise, call 'rm_rf' only when 'waitpid' returns -1 the second time.
(exec_with_loader): Call 'prctl'. Remove NEW_ROOT only when 'waitpid'
returns -1 the second time, otherwise leave it behind.
* tests/guix-pack-relocatable.sh (wait_for_file): New function.
Add test.
