There have been a few patches to the mailing list lately
not respecting this, and this linter detects 630 package
definitions that could be modified to support the --without-tests
package transformation.
* guix/lint.scm
(check-optional-tests): New linter.
(%local-checkers)[optional-tests]: Add it.
* tests/lint.scm
(package-with-phase-changes): New procedure.
("optional-tests: no check phase")
("optional-tests: check hase respects #:tests?")
("optional-tests: check phase ignores #:tests?")
("optional-tests: do not crash when #:phases is invalid")
("optional-tests: allow G-exps (no warning)")
("optional-tests: allow G-exps (warning)")
("optional-tests: complicated 'check' phase")
("optional-tests: 'check' phase is not first phase"): New tests.
Signed-off-by: Mathieu Othacehe <othacehe@gnu.org>
It will be used in the 'optional-tests' linter.
* guix/gexp.scm (gexp->approximate-sexp): New procedure.
* tests/gexp.scm
("no references", "unquoted gexp", "unquoted gexp (native)")
("spliced gexp", "unspliced gexp, approximated")
("unquoted gexp, approximated"): Test it.
* doc/gexp.scm ("G-Expressions"): Document it.
Signed-off-by: Mathieu Othacehe <othacehe@gnu.org>
* .dir-locals.el (scheme-mode)[gexp->derivation]: Define indentation rule.
* guix/scripts/pack.scm (debian-archive): New procedure.
(%formats): Register the new deb format.
(show-formats): Add it to the usage string.
* tests/pack.scm (%ar-bootstrap): New variable.
(deb archive with symlinks): New test.
* doc/guix.texi (Invoking guix pack): Document it.
* NEWS: Add news entry.
Tar translate duplicate files in the archive into hard links. These can cause
problems, as not every tool support them; for example dpkg doesn't.
* gnu/system/file-systems.scm (reduce-directories): New procedure.
(file-prefix?): Lift the restriction on file prefix. The procedure can be
useful for comparing relative file names. Adjust doc.
(file-name-depth): New procedure, extracted from ...
(btrfs-store-subvolume-file-name): ... here.
* guix/scripts/pack.scm (self-contained-tarball/builder): Use
reduce-directories.
* tests/file-systems.scm ("reduce-directories"): New test.
Sometimes two configurations might have the same types for their field values,
but the values might be serialized in two completely different
ways (e.g. because the two programs have different configuration languages).
An example of this would be the ‘serialize-boolean’ procedure in (gnu services
mail) and (gnu services getmail). They both serialize a boolean value, but
because the Dovecot’s configuration language has a different syntax to the
configuration language for Getmail, two different procedures have to be
defined.
One way to workaround this would be to specify custom serializers for many
fields in order to separate the serialization of the values that have the same
type but serialize in different ways. This could get very tedious, especially
if there are many configurations in the same module.
Another way would be to move one of the configurations to its own module, like
what was done with (gnu services getmail). However, this would mean that
there would be multiple modules containing configurations for related
programs, e.g. we have (gnu services mail) and (gnu services getmail), it
doesn’t make much sense to keep the Getmail configuration in its own module.
This patch will allow one to write something like this:
(define-configuration foo-configuration
(bar
(string "bob")
"Option bar.")
(prefix bar-))
and the value of the ‘bar’ field would be serialized using a procedure named
‘bar-serialize-string’ instead of just ‘serialize-string’.
* gnu/services/configuration.scm (define-maybe-helper): Accept ‘prefix’
argument for using serializer with custom prefix.
(define-maybe): Pattern match on ‘prefix’ literal.
(define-configuration-helper): Accept ‘prefix’ argument for using serializer
with custom prefix.
(define-configuration): Pattern match on ‘prefix’ literal.
* tests/services/configuration.scm ("serialize-configuration with prefix"):
New test.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes <https://bugs.gnu.org/49102>.
Reported by Mathieu Othacehe <othacehe@gnu.org>
and Tobias Geerinckx-Rice <me@tobias.gr>.
Fixes a regression introduced in
8cef92d063, whereby in case of file
collisions, the "wrong" one would take precedence.
* guix/build/profiles.scm (manifest-sexp->inputs+search-paths): Perform
a breadth-first traversal. Reverse INPUTS and SEARCH-PATHS in the base
case.
* tests/profiles.scm ("profile-derivation, ordering & collisions"):
New test.
This procedure lost its only user in commit
710854304b.
* guix/store.scm (references/substitutes): Remove.
* tests/store.scm ("references/substitutes missing reference info")
("references/substitutes with substitute info"): Remove.
* guix/lint.scm (check-tests-true): New linter.
(%local-checkers)[tests-true]: Add it.
* tests/lint.scm ("tests-true: #:tests? must not be set to #t")
("tests-true: absent #:tests? is acceptable")
("tests-true: #:tests? #f is acceptable")
("tests-true: #:tests? #t acceptable when compiling natively"): Test it.
Signed-off-by: Mathieu Othacehe <othacehe@gnu.org>
* guix/import/hackage.scm (string->license): Prefix the value of the license
field with ‘license:’.
* tests/hackage.scm (match-ghc-foo, match-ghc-foo-6, match-ghc-foo-revision):
Adjust accordingly.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
* guix/lint.scm (lookup-disarchive-spec): New procedure.
(check-archival): When 'lookup-content' returns #f, call
'lookup-disarchive-spec'. Call 'lookup-directory' on the result of
'lookup-directory'.
* guix/download.scm (%disarchive-mirrors): Make public.
* tests/lint.scm ("archival: missing content"): Set
'%disarchive-mirrors'.
("archival: content unavailable but disarchive available"): New test.
Before this change, using define-maybe along define-configuration with the
no-serialization syntactic keyword would result in the following warning:
warning: possibly unbound variable `VARIABLE-NAME'
This change introduces the define-maybe/no-serialization variant that does
away with defining a serialization helper procedure, which makes it possible
to avoid the above warning.
* gnu/services/configuration.scm (define-maybe/no-serialization): New syntax.
(define-maybe-helper): New procedure.
(define-maybe): Define syntax using the above procedure.
* tests/services/configuration.scm (tests): Fix module name.
(custom-number-serializer): Do not print to standard output.
(maybe-number?, serialize-maybe-number): New procedures defined via the
define-maybe macro.
(config-with-maybe-number): New configuration.
(serialize-number): New procedure.
("maybe value serialization"): New test.
(maybe-string?): New procedure defined via the define-maybe/no-serialization
macro.
(config-with-maybe-string/no-serialization): New configuration.
("maybe value without serialization no procedure bound"): New test.
I'm looking at this to help with adding support for looking up package
replacements to store in the Guix Data Service.
* guix/inferior.scm (inferior-package-replacement): New procedure.
* tests/inferior.scm ("inferior-package-replacement"): New test.
The duplicate test being removed has not much value given that
go-version->git-ref is a no-op for version strings not matching the
%go-pseudo-version-rx regexp; that case is already tested by the test above.
* tests/go.scm ("go-version omited 'v' character"): Remove test.
Reported-by: Tobias Geerinckx-Rice <me@tobias.gr>
This importer has suffered from bitrot and no longer works with current
Nix and Nixpkgs. See <https://bugs.gnu.org/32339> and
<https://bugs.gnu.org/36255>.
* guix/import/snix.scm, guix/scripts/import/nix.scm,
tests/snix.scm: Remove.
* Makefile.am (MODULES, SCM_TESTS): Remove them.
* guix/scripts/import.scm (importers): Remove "nix".
* build-aux/test-env.in: Remove NIXPKGS variable.
* configure.ac: Remove '--with-nixpkgs' option.
* doc/guix.texi (Invoking guix import): Remove bit about "guix import
nix".
* etc/completion/fish/guix.fish: Likewise.
As per section '16.4.2 Package Naming' in the manual, use hyphens
instead of underscores in package names.
* guix/lint.scm (check-name): Check whether the package name contains
underscores.
* tests/lint.scm ("name: use underscore in package name"): New test.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Partially fixes <https://bugs.gnu.org/33848>.
* guix/build/graft.scm (replace-store-references): Add support for
finding and rewriting UTF-16 and UTF-32 store references.
* tests/grafts.scm: Add tests.
This module allows for communicating with the IPFS
gateway over the HTTP interface. The commit has been
cherry-picked from <https://issues.guix.gnu.org/33899>.
The procedures for adding and restoring file trees have
been removed as according to a reply issue 33899, a different
format will be used. The procedure 'add-data' has been
exported as it will be used in the system test for IPFS.
* guix/ipfs.scm: New file.
* Makefile.am (MODULES): Add it.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
The ability to pin versions is handy when having to deal to packages that
bootstrap themselves through a chain of former versions. Not using pinned
versions in these case could introduce dependency cycles.
* guix/build-system/go.scm (guix)
(%go-version-rx): Rename to...
(%go-pseudo-version-rx): ... this. Simplify the regular expression, which in
turns makes it more robust.
* guix/build-system/go.scm (go-version->git-ref): Adjust following the above
rename.
(go-pseudo-version?): New predicate.
(go-module-latest-version): Rename to ...
(go-module-version-string): ... this. Rename goproxy-url argument to just
goproxy. Add a VERSION keyword argument, update docstring and adjust to have
it used.
(go-module-available-versions): New procedure.
(%go.mod-require-directive-rx): Document regexp.
(parse-go.mod): Harmonize the way dependencies are recorded to a list of lists
rather than a list of pairs, as done for other importers. Rewrite to directly pass
multiple values rather than a record object. Filter the replaced modules in a
functional style.
(go-module->guix-package): Add docstring.
[version, pin-versions?]: New arguments. Rename the GOPROXY-URL argument to
GOPROXY. Adjust to the new returned value of fetch-go.mod, which is a string.
Fail when the provided version doesn't exist. Return a list dependencies and
their versions when in pinned versions mode, else just the dependencies.
(go-module-recursive-import)[version, pin-versions?]: New arguments.
Honor the new arguments and guard against network errors.
* guix/scripts/import/go.scm (%default-options): Register a default value for
the goproxy argument.
(show-help): Document that a version can be specified. Remove the --version
argument and add a --pin-versions argument.
(%options)[version]: Remove option.
[pin-versions]: Add option.
(guix-import-go): Adjust so the version provided from the module name is
honored, along the new pin-versions? argument.
* tests/go.scm: Adjust and add new tests.
* guix/import/go.scm (%strict-tokenizer?): Set parameter to #t.
(go-path-escape): Redefine to prevent inlining.
(http-get*): Replace by ...
(http-fetch*): this ...
(json-fetch*): New procedure.
(go.pkg.dev-info): Use http-fetch*.
(go-package-licenses): Rewrite in terms of go.pkg.dev-info.
(go-package-description): Likewise.
(go-package-synopsis): Likewise.
(fetch-go.mod): Use the memoized http-fetch*.
(parse-go.mod): Adjust to receive content as a string.
(fetch-module-meta-data): Adjust to use http-fetch*.
(go-module->guix-package): Adjust to the modified fetch-go.mod return value.
[inputs]: Use propagated inputs, which is the most common situations for Go
libraries.
[description]: Beautify description.
[licenses]: Do no check for #f. The result of the license parsing is always a
list.
* tests/go.scm: Adjust following above changes.
This avoids the situation where error messages would unintentionally go
to stderr and be wrongfully interpreted as a reply by the daemon.
Fixes <https://bugs.gnu.org/46362>.
This is a followup to ee3226e9d5.
* guix/scripts/substitute.scm (display-narinfo-data): Add 'port'
parameter and honor it.
(process-query): Likewise.
(process-substitution): Likewise.
(%error-to-file-descriptor-4?, with-redirected-error-port): Remove.
(%reply-file-descriptor): New variable.
(guix-substitute): Remove use of 'with-redirected-error-port'. Define
'reply-port' and pass it to 'process-query' and 'process-substitution'.
* nix/libstore/build.cc (SubstitutionGoal::handleChildOutput): Swap
'builderOut' and 'fromAgent'.
* nix/libstore/local-store.cc (LocalStore::getLineFromSubstituter):
Likewise.
* tests/substitute.scm <top level>: Set '%reply-file-descriptor'
rather than '%error-to-file-descriptor-4?'.
Fixes <https://bugs.gnu.org/47239>.
Reported by Konrad Hinsen <konrad.hinsen@fastmail.net>.
* tests/publish.scm ("with cache"): Pass the result of 'stat:perms' to
'logand' to be umask-insensitive.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
This makes powerpc64le-linux a supported architecture for Guix, but not for
Guix System.
* Makefile.am (SUPPORTED_SYSTEMS): Add an entry for powerpc64le-linux.
* etc/guix-install.sh (chk_sys_arch): Same.
* guix/packages.scm (%supported-systems): Same.
* m4/guix.m4 (GUIX_ASSERT_SUPPORTED_SYSTEM): Same.
* tests/guix-build.sh (all_systems): Same.
Otherwise, the test crashes (not fails) when run in `guix environment --pure guix`.
Fixes <https://bugs.gnu.org/46445>.
* tests/store.scm (%shell): Fallback to "/bin/sh".
Previously, test cases could fail if some process was listening
at a hard-coded port. This patch eliminates most of these potential
failures, by automatically assigning an unbound port. This should
allow for building multiple guix trees in parallel outside a build
container, though this is currently untested.
The test "home-page: Connection refused" in tests/lint.scm still
hardcodes port 9999, however.
* guix/tests/http.scm
(http-server-can-listen?): remove now unused procedure.
(%http-server-port): default to port 0, meaning the OS
will automatically choose a port.
(open-http-server-socket): remove the false statement claiming
this procedure is exported and also return the allocated port
number.
(%local-url): raise an error if the port is obviously unbound.
(call-with-http-server): set %http-server-port to the allocated
port while the thunk is called.
* tests/derivations.scm: adjust test cases to use automatically
assign a port. As there is no risk of a port conflict now,
do not make any tests conditional upon 'http-server-can-listen?'
anymore.
* tests/elpa.scm: likewise.
* tests/lint.scm: likewise, and add a TODO comment about a port
that is still hard-coded.
* tests/texlive.scm: likewise.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Previously, transformations applied from a manifest (rather than via
"guix install") would be lost. This change fixes that and simplifies
things.
Reported by zimoun at
<https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00153.html>.
* guix/profiles.scm (default-properties): New procedure.
(package->manifest-entry): Use it for #:properties.
* guix/scripts/pack.scm (guix-pack)[with-transformations]: Remove.
Remove caller.
* guix/scripts/package.scm (transaction-upgrade-entry): Remove calls to
'manifest-entry-with-transformations'.
* tests/guix-package.sh: Add test.
* tests/transformations.scm ("options->transformation + package->manifest-entry"):
New test.
Fixes a regression introduced in
c6d6aee6659acb293eb33f498fdac3b47a19a48, where #:reference-graphs would
end up referring to native inputs.
This would notably break the compilation of systems using a childhurd,
because they would attempt to build the 'hurd' package natively.
* guix/gexp.scm (lower-reference-graphs)[tuple->gexp-input]: Honor TARGET.
* tests/gexp.scm ("gexp->derivation #:references-graphs cross-compilation"):
New test.
* guix/build/syscalls.scm (<mount>): New record type.
(option-string->mount-flags, mount-flags)
(octal-decode, mounts): New procedures.
(mount-points): Rewrite in terms of 'mount'.
* tests/syscalls.scm ("mounts"): New test.
This avoids double traversal of references and extra bookkeeping,
thereby further reducing memory allocations.
* guix/gexp.scm (lower-gexp): Include only one call to 'lower-inputs'.
(gexp-inputs): Remove #:native? parameter.
[set-gexp-input-native?]: New procedure.
[add-reference-inputs]: Use it.
(gexp-native-inputs): Remove.
* tests/gexp.scm (gexp-native-inputs): Remove.
(gexp-input->tuple): Include 'gexp-input-native?'.
("let-system")
("let-system, nested")
("ungexp + ungexp-native")
("ungexp + ungexp-native, nested")
("ungexp + ungexp-native, nested, special mixture")
("input list")
("input list + ungexp-native")
("input list splicing")
("input list splicing + ungexp-native-splicing")
("gexp list splicing + ungexp-splicing"): Adjust accordingly.
This slightly reduces memory allocation.
* guix/gexp.scm (lower-inputs): Expect a list of <gexp-input> rather
than a list of tuples.
(lower-reference-graphs)[tuple->gexp-input]: New procedure.
Use it.
(gexp-inputs): Return a list of <gexp-input> rather than a list of
tuples.
* tests/gexp.scm (gexp-input->tuple): New procedure.
("one input package")
("one input package, dotted list")
("one input origin")
("one local file")
("one local file, symlink")
("one plain file")
("two input packages, one derivation, one file")
("file-append")
("file-append, output")
("file-append, nested")
("let-system")
("let-system, nested")
("ungexp + ungexp-native")
("ungexp + ungexp-native, nested")
("ungexp + ungexp-native, nested, special mixture")
("input list")
("input list + ungexp-native")
("input list splicing")
("input list splicing + ungexp-native-splicing")
("gexp list splicing + ungexp-splicing"): Adjust accordingly.
Fixes <https://bugs.gnu.org/46390>.
Reported by zimoun <zimon.toutoune@gmail.com>.
This is similar to the fix in d10474c38d.
* guix/packages.scm (package-field-location): Handle FILE not in %LOAD-PATH.
* tests/guix-lint.sh: Add test.
Remove the 'vm-image' command that has been superseded by the 'image'
command.
* gnu/system/vm.scm (system-qemu-image): Remove it.
* guix/scripts/system.scm (system-derivation-for-action): Mark 'vm-image'
command as deprecated and use the image API to produce the VM image.
(perform-action, show-help): Adapt accordingly.
* tests/guix-system.sh: Ditto.
* doc/guix.texi (Invoking guix system,
Running Guix in a VM): Ditto.
* etc/completion/fish/guix.fish: Ditto.
* etc/completion/zsh/_guix: Ditto.
* guix/scripts/package.scm (export-manifest): New procedure.
(show-help, %options): Add '--export-manifest'.
(process-query): Honor it.
* guix/build/profiles.scm (build-profile): Mention it.
* tests/guix-package.sh: Test it.
* doc/guix.texi (Invoking guix package): Document it.
This test goes from 58s to 10s wall-clock time.
Reported by Maxim Cournoyer <maxim.cournoyer@gmail.com>.
* tests/packages.scm ("fold-available-packages with/without cache"):
Remove 'find-duplicates'. Add 'list->set*' and use it instead of
'find-duplicates', 'delete-duplicates', and 'lset='.
* guix/store/database.scm (assert-integer): New procedure.
(update-or-insert): Use it to validate NAR-SIZE and TIME.
* tests/store-database.scm ("sqlite-register with incorrect size"): New
test.
* guix/serialization.scm (read-string-pairs): New procedure.
* guix/store.scm (read-arg): Add support for 'string-pairs'.
(find-roots): New procedure.
* tests/store.scm ("add-indirect-root and find-roots"): New test.
The 'stack' variable could be #f when code is interpreted, which in
practice happens when running in "legacy" mode--i.e., when
'open-inferior' invokes "guile" instead of "guix repl".
* guix/repl.scm (send-repl-response)[handle-exception]: Check whether
STACK is true before passing it to 'stack->frames'.
* tests/inferior.scm ("&inferior-exception, legacy mode"): New test.
The path normalization of `--root` option of `guix environment` was
buggy as it appended full argument after normalized directory. This
patch fixes it.
* guix/scripts/environment.scm (register-gc-root): Fix gc-root path
normalization.
* tests/guix-environment.sh: Add test.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>
This separation between the code for dealing with narinfos from the code doing
that for a purpose should make things clearer, and better support components
other that the substitute script in using this code.
This is just moving the code around, no code should have been significantly
changed.
* guix/scripts/substitute.scm (<narinfo>): Move record type to (guix narinfo).
(fields->alist, narinfo-hash-algorithm+value, narinfo-hash->sha256,
narinfo-signature->canonical-sexp, narinfo-maker, read-narinfo,
narinfo-sha256, valid-narinfo?, write-narinfo, narinfo->string,
string->narinfo, equivalent-narinfo?, supported-compression?,
compresses-better?, narinfo-best-uri): Move procedures to (guix narinfo).
(%compression-methods): Move variable to (guix narinfo).
* guix/narinfo.scm: New file.
* Makefile.am (MODULES): Add it.
* po/guix/POTFILES.in: Add 'guix/narinfo.scm'.
* guix/scripts/publish.scm (compress-nar)[write-compressed-file]: New
procedure.
Use it for 'gzip' and 'lzip'. Add 'zstd.
(nar-response-port, string->compression-type): Add case for 'zstd'.
* tests/publish.scm (zstd-supported?): New procedure.
("/nar/zstd/*"): New test.
* doc/guix.texi (Invoking guix publish): Document zstd compression.
(Base Services): Add cross-reference to the above node.
* guix/utils.scm (lzip-port): Return a single value.
(zstd-port): New procedure.
(decompressed-port, compressed-output-port): Add 'zstd' case.
* tests/utils.scm (test-compression/decompression): Test 'zstd' when
the (zstd) module is available.
This procedure was unused except in one test.
* guix/utils.scm (compressed-port): Remove.
* tests/utils.scm (test-compression/decompression): Rewrite to use
'compressed-output-port' instead.
* guix/scripts/hash.scm (guix-hash): When 'recursive? is true, use
'open-hash-port' instead of 'open-sha256-port'.
* tests/guix-hash.sh: Add test for 'guix hash -r -H sha512'.
Suggested by Philippe Swartvagher <philippe.swartvagher@inria.fr>.
* guix/transformations.scm (transform-package-patches): New procedure.
(%transformations): Add it as 'with-patch'.
(%transformation-options, show-transformation-options-help/detailed):
Add '--with-patch'.
* tests/transformations.scm ("options->transformation, with-patch"): New
test.
* doc/guix.texi (Package Transformation Options): Document it.
Fixes <https://bugs.gnu.org/45453>.
* guix/import/utils.scm (alist->package): Find plain license names in (guix
licenses) first, and only fall back to SPDX names on error.
* tests/import-utils.scm ("alist->package with SPDX license name 1/2",
"alist->package with SPDX license name 2/2"): New tests.
* gnu/bootloader/grub.scm (grub-configuration-file): New parameter
store-crypto-devices.
[crypto-devices]: New helper function.
[builder]: Use crypto-devices.
* gnu/machine/ssh.scm (roll-back-managed-host): Use
boot-parameters-store-crypto-devices to provide its contents to the
bootloader configuration generation process.
* gnu/tests/install.scm (%encrypted-root-not-boot-os,
%encrypted-root-not-boot-os): New os declaration.
(%encrypted-root-not-boot-installation-script): New script, whose contents
were initially taken from %encrypted-root-installation-script.
(%test-encrypted-root-not-boot-os): New test.
* gnu/system.scm (define-module): Export
operating-system-bootoader-crypto-devices and
boot-parameters-store-crypto-devices.
(<boot-parameters>): Add field store-crypto-devices.
(read-boot-parameters): Parse store-crypto-devices field.
[uuid-sexp->uuid]: New helper function extracted from
device-sexp->device.
(operating-system-bootloader-crypto-devices): New function.
(operating-system-bootcfg): Use
operating-system-bootloader-crypto-devices to provide its contents to
the bootloader configuration generation process.
(operating-system-boot-parameters): Add store-crypto-devices to the
generated boot-parameters.
(operating-system-boot-parameters-file): Likewise to the file with
the serialized structure.
* guix/scripts/system.scm (reinstall-bootloader): Use
boot-parameters-store-crypto-devices to provide its contents to the
bootloader configuration generation process.
* tests/boot-parameters.scm (%default-store-crypto-devices): New
variable.
(%grub-boot-parameters, test-read-boot-parameters): Use
%default-store-crypto-devices.
(tests store-crypto-devices): New tests.
Guile-JSON 4.3.2 would parse in spite of these typos, but 4.4.1 is
stricter.
* tests/swh.scm (%directory-entries): Add missing comma.
* tests/cve-sample.json: Likewise.
* tests/store.scm ("substitute")
("substitute + build-things with output path")
("substitute + build-things with specific output"): Call 'canonical-file?'.
* tests/substitute.scm ("substitute, authorized key"): Check the mtime
and permissions of "substitute-retrieved".
This way, the hash of the store item can be computed as it is restored,
thereby avoiding an additional file tree traversal ('hashPath' call)
later on in the daemon. Consequently, it should reduce latency between
subsequent substitute downloads.
This is a followup to 5ff521452b.
* guix/scripts/substitute.scm (narinfo-hash-algorithm+value): New
procedure.
(process-substitution): Wrap INPUT into a hash input port, 'hashed', and
read from it. Compare the actual and expected hashes, and print a
"hash-mismatch" status line when they differ. When they match, print
not just "success" but also the nar hash and size.
* nix/libstore/build.cc (class SubstitutionGoal)[expectedHashStr]:
Remove.
(SubstitutionGoal::finished): Tokenize 'status'. Parse it and handle
"success" and "hash-mismatch" accordingly. Call 'hashPath' only when
the returned hash is not SHA256.
(SubstitutionGoal::handleChildOutput): Remove 'expectedHashStr'
handling.
* tests/substitute.scm ("substitute, invalid hash"): Rename to...
("substitute, invalid narinfo hash"): ... this.
("substitute, invalid hash"): New test.
This avoids repeated (mkdir-p "/gnu/store/.links") calls when
deduplicating lots of files.
* guix/store/deduplication.scm (deduplicate): Remove initial call to
'mkdir-p'. Add ENOENT case in 'link' exception handler. Reindent.
* tests/store-deduplication.scm ("deduplicate, ENOSPC"): Check
for (<= links 4) to account for the initial 'link' call.
Until now deduplication was performed as an additional pass after
copying files, which involve re-traversing all the files that had just
been copied.
* guix/store/deduplication.scm (copy-file/deduplicate): New procedure.
* tests/store-deduplication.scm ("copy-file/deduplicate"): New test.
* guix/build/store-copy.scm (populate-store): Add #:deduplicate?
parameter and honor it.
* tests/gexp.scm ("gexp->derivation, store copy"): Pass #:deduplicate? #f
to 'populate-store'.
* gnu/build/image.scm (initialize-root-partition): Pass #:deduplicate?
to 'populate-store'. Pass #:deduplicate? #f to 'register-closure'.
* gnu/build/vm.scm (root-partition-initializer): Likewise.
* gnu/build/install.scm (populate-single-profile-directory): Pass
#:deduplicate? #f to 'populate-store'.
* gnu/build/linux-initrd.scm (build-initrd): Likewise.
* guix/scripts/pack.scm (self-contained-tarball)[import-module?]: New
procedure.
[build]: Pass it as an argument to 'source-module-closure'.
* guix/scripts/pack.scm (squashfs-image)[build]: Wrap in
'with-extensions'.
* gnu/system/linux-initrd.scm (expression->initrd)[import-module?]: New
procedure.
[builder]: Pass it to 'source-module-closure'.
* gnu/system/install.scm (cow-store-service-type)[import-module?]: New
procedure. Pass it to 'source-module-closure'.
Until now, 'populate-store' would reset permissions but not timestamps,
so callers would resort to going through an extra directory traversal to
reset timestamps.
* guix/build/store-copy.scm (reset-permissions): Remove.
(copy-recursively): New procedure.
(populate-store): Pass #:keep-permissions? to 'copy-recursively'.
Remove call to 'reset-permissions'.
* tests/gexp.scm ("gexp->derivation, store copy"): In BUILD-DRV, check
whether 'populate-store' canonicalizes permissions and timestamps.
* gnu/build/image.scm (initialize-root-partition): Pass #:reset-timestamps? #f
to 'register-closure'.
* gnu/build/vm.scm (root-partition-initializer): Likewise.
This avoids having to traverse and re-read the files that we have just
restored, thereby reducing I/O.
* guix/serialization.scm (dump-file): New procedure.
(restore-file): Add #:dump-file parameter and honor it.
* guix/store/deduplication.scm (tee, dump-file/deduplicate): New
procedures.
* guix/nar.scm (restore-one-item): Pass #:dump-file to 'restore-file'.
(finalize-store-file): Pass #:deduplicate? #f to 'register-items'.
* tests/nar.scm <top level>: Call 'setenv' to set "NIX_STORE".
* guix/serialization.scm (restore-file): Set the permissions and mtime
of FILE.
* guix/nar.scm (finalize-store-file): Pass #:reset-timestamps? #f to
'register-items'.
* tests/nar.scm (rm-rf): Add 'chmod' calls to ensure files are writable.
("write-file + restore-file with symlinks"): Ensure every file in OUTPUT
passes 'canonical-file?'.
* tests/guix-archive.sh: Run "chmod -R +w" before "rm -rf".
* guix/serialization.scm (fold-archive): Call PROC with a
'directory-complete tag when done with a directory.
(restore-file): Handle it.
* guix/scripts/archive.scm (list-contents): Likewise.
* guix/scripts/challenge.scm (archive-contents): Likewise.
* tests/nar.scm ("write-file-tree + fold-archive"): Adjust accordingly.
This is a followup to a8dccd4bdc, which
broke the test.
* guix/import/opam.scm (get-opam-repository): Prevent inlining.
* tests/opam.scm ("opam->guix-package"): Mock 'get-opam-repository'.
This avoids spawning one substitute process per substitution.
* nix/libstore/build.cc (class Worker)[substituter]: New field.
[outPipe, logPipe, pid]: Remove.
(class SubstitutionGoal)[expectedHashStr, status, substituter]: New fields.
(SubstitutionGoal::timedOut): Adjust to check 'substituter'.
(SubstitutionGoal::tryToRun): Remove references to 'outPipe' and
'logPipe'. Run "guix substitute --substitute" as an 'Agent'. Send the
request with 'writeLine'.
(SubstitutionGoal::finished): Likewise.
(SubstitutionGoal::handleChildOutput): Change to fill in
'expectedHashStr' and 'status'.
(SubstitutionGoal::handleEOF): Call 'wakeUp' unconditionally.
(SubstitutionGoal::~SubstitutionGoal): Adjust to check 'substituter'.
* guix/scripts/substitute.scm (process-substitution): Write "success\n"
to stdout upon success.
(%error-to-file-descriptor-4?): New variable.
(guix-substitute): Set 'current-error-port' to file descriptor 4
unless (%error-to-file-descriptor-4?) is false.
Remove "--substitute" arguments. Loop reading line from stdin.
* tests/substitute.scm <top level>: Call '%error-to-file-descriptor-4?'.
(request-substitution): New procedure.
("substitute, no signature")
("substitute, invalid hash")
("substitute, unauthorized key")
("substitute, authorized key")
("substitute, unauthorized narinfo comes first")
("substitute, unsigned narinfo comes first")
("substitute, first narinfo is unsigned and has wrong hash")
("substitute, first narinfo is unsigned and has wrong refs")
("substitute, two invalid narinfos")
("substitute, narinfo with several URLs"): Adjust to new "guix
substitute --substitute" calling convention.
* nix/libstore/local-store.hh (RunningSubstituter): Remove.
(LocalStore)[runningSubstituter]: Change to unique_ptr<Agent>.
[setSubstituterEnv, didSetSubstituterEnv]: Remove.
[getLineFromSubstituter, getIntLineFromSubstituter]: Take an 'Agent'.
* nix/libstore/local-store.cc (LocalStore::~LocalStore): Remove
reference to 'runningSubstituter'.
(LocalStore::setSubstituterEnv, LocalStore::startSubstituter): Remove.
(LocalStore::getLineFromSubstituter): Adjust to 'run' being an 'Agent'.
(LocalStore::querySubstitutablePaths): Spawn substituter agent if
needed. Adjust to 'Agent' interface.
(LocalStore::querySubstitutablePathInfos): Likewise.
* nix/libstore/build.cc (SubstitutionGoal::tryToRun): Remove call to
'setSubstituterEnv' and add 'setenv' call for "_NIX_OPTIONS" instead.
(SubstitutionGoal::finished): Remove 'readLine' call for 'dummy'.
* guix/scripts/substitute.scm (%allow-unauthenticated-substitutes?):
Remove second argument to 'make-parameter'.
(process-query): Call 'warn-about-missing-authentication'
when (%allow-unauthenticated-substitutes?) is #t.
(guix-substitute): Wrap body in 'parameterize'. Set 'guix-warning-port'
too. No longer exit when 'substitute-urls' returns the empty list. No
longer print newline initially.
* tests/substitute.scm (test-quit): Parameterize 'current-error-port' to
account for the port changes in 'guix-substitute'.
In particular, "<origin> patches: same file name -> no warnings" would
have caught the issue which was fixed in commit
21887021b9.
* tests/lint.scm (patches: file names): Rename this test case...
("file patches: different file name -> warning"): ... to this.
("file patches: same file name -> no warnings")
("<origin> patches: different file name -> warning")
("<origin> patches: same file name -> no warnings"): New test cases.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Fixes <https://bugs.gnu.org/23874>.
Reported by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>.
* guix/profiles.scm (manifest-transaction-effects): Delete duplicates in
install and remove. Let multiple upgrades and downgrades shadow previous
transactions of the same kind.
* tests/profiles.scm
("manifest-transaction-effects no double install or upgrades")
("manifest-transaction-effects no double downgrade")
("manifest-transaction-effects no double removal"): New tests.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
