offload: Modify the build-machine record to accept multiple systems.

* guix/scripts/offload.scm (<build-machine>)[systems]: New field.
[system]: Accessor changed to %build-machine-system.  Default to #f.
* guix/scripts/offload.scm (build-machine-system): Wrap %build-machine-system
with a deprecation warning.
(build-machine-systems): Access the new systems field or fallback to use
build-machine-system, for backward compatibility.
(machine-matches?): Adjust.
* tests/offload.scm: Add tests...
* Makefile.am (SCM_TESTS): ...and register them.
* doc/guix.texi (Daemon Offload Setup): Update doc.
This commit is contained in:
Maxim Cournoyer 2020-08-24 16:26:14 -04:00
parent a5ccf1b522
commit 4b5a6fbc9b
No known key found for this signature in database
GPG key ID: 1260E46482E63562
4 changed files with 111 additions and 20 deletions

View file

@ -433,6 +433,7 @@ SCM_TESTS = \
tests/monads.scm \ tests/monads.scm \
tests/nar.scm \ tests/nar.scm \
tests/networking.scm \ tests/networking.scm \
tests/offload.scm \
tests/opam.scm \ tests/opam.scm \
tests/openpgp.scm \ tests/openpgp.scm \
tests/packages.scm \ tests/packages.scm \

View file

@ -1043,29 +1043,31 @@ When desired, the build daemon can @dfn{offload} derivation builds to
other machines running Guix, using the @code{offload} @dfn{build other machines running Guix, using the @code{offload} @dfn{build
hook}@footnote{This feature is available only when hook}@footnote{This feature is available only when
@uref{https://github.com/artyom-poptsov/guile-ssh, Guile-SSH} is @uref{https://github.com/artyom-poptsov/guile-ssh, Guile-SSH} is
present.}. When that present.}. When that feature is enabled, a list of user-specified build
feature is enabled, a list of user-specified build machines is read from machines is read from @file{/etc/guix/machines.scm}; every time a build
@file{/etc/guix/machines.scm}; every time a build is requested, for is requested, for instance via @code{guix build}, the daemon attempts to
instance via @code{guix build}, the daemon attempts to offload it to one offload it to one of the machines that satisfy the constraints of the
of the machines that satisfy the constraints of the derivation, in derivation, in particular its system types---e.g., @code{x86_64-linux}.
particular its system type---e.g., @file{x86_64-linux}. Missing A single machine can have multiple system types, either because its
prerequisites for the build are copied over SSH to the target machine, architecture natively supports it, via emulation (@pxref{Transparent
which then proceeds with the build; upon success the output(s) of the Emulation with QEMU}), or both. Missing prerequisites for the build are
build are copied back to the initial machine. copied over SSH to the target machine, which then proceeds with the
build; upon success the output(s) of the build are copied back to the
initial machine.
The @file{/etc/guix/machines.scm} file typically looks like this: The @file{/etc/guix/machines.scm} file typically looks like this:
@lisp @lisp
(list (build-machine (list (build-machine
(name "eightysix.example.org") (name "eightysix.example.org")
(system "x86_64-linux") (systems (list "x86_64-linux" "i686-linux"))
(host-key "ssh-ed25519 AAAAC3Nza@dots{}") (host-key "ssh-ed25519 AAAAC3Nza@dots{}")
(user "bob") (user "bob")
(speed 2.)) ;incredibly fast! (speed 2.)) ;incredibly fast!
(build-machine (build-machine
(name "armeight.example.org") (name "armeight.example.org")
(system "aarch64-linux") (systems (list "aarch64-linux"))
(host-key "ssh-rsa AAAAB3Nza@dots{}") (host-key "ssh-rsa AAAAB3Nza@dots{}")
(user "alice") (user "alice")
(private-key (private-key
@ -1075,8 +1077,8 @@ The @file{/etc/guix/machines.scm} file typically looks like this:
@noindent @noindent
In the example above we specify a list of two build machines, one for In the example above we specify a list of two build machines, one for
the @code{x86_64} architecture and one for the @code{aarch64} the @code{x86_64} and @code{i686} architectures and one for the
architecture. @code{aarch64} architecture.
In fact, this file is---not surprisingly!---a Scheme file that is In fact, this file is---not surprisingly!---a Scheme file that is
evaluated when the @code{offload} hook is started. Its return value evaluated when the @code{offload} hook is started. Its return value
@ -1096,8 +1098,9 @@ builds. The important fields are:
@item name @item name
The host name of the remote machine. The host name of the remote machine.
@item system @item systems
The system type of the remote machine---e.g., @code{"x86_64-linux"}. The system types the remote machine supports---e.g., @code{(list
"x86_64-linux" "i686-linux")}.
@item user @item user
The user account to use when connecting to the remote machine over SSH. The user account to use when connecting to the remote machine over SSH.
@ -25025,7 +25028,7 @@ Maximum number of backup files to keep.
Defaults to @samp{3} Defaults to @samp{3}
@end deftypevr @end deftypevr
@node Transparent Emulation with QEMU
@subsubheading Transparent Emulation with QEMU @subsubheading Transparent Emulation with QEMU
@cindex emulation @cindex emulation

View file

@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU ;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net> ;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;; ;;;
;;; This file is part of GNU Guix. ;;; This file is part of GNU Guix.
;;; ;;;
@ -66,14 +67,16 @@ (define-module (guix scripts offload)
;;; ;;;
;;; Code: ;;; Code:
(define-record-type* <build-machine> (define-record-type* <build-machine>
build-machine make-build-machine build-machine make-build-machine
build-machine? build-machine?
(name build-machine-name) ; string (name build-machine-name) ; string
(port build-machine-port ; number (port build-machine-port ; number
(default 22)) (default 22))
(system build-machine-system) ; string (systems %build-machine-systems ; list of strings
(default #f)) ; drop default after system is removed
(system %build-machine-system ; deprecated
(default #f))
(user build-machine-user) ; string (user build-machine-user) ; string
(private-key build-machine-private-key ; file name (private-key build-machine-private-key ; file name
(default (user-openssh-private-key))) (default (user-openssh-private-key)))
@ -91,6 +94,19 @@ (define-record-type* <build-machine>
(features build-machine-features ; list of strings (features build-machine-features ; list of strings
(default '()))) (default '())))
;;; Deprecated.
(define (build-machine-system machine)
(warning (G_ "The 'system' field is deprecated, \
please use 'systems' instead.~%"))
(%build-machine-system machine))
;;; TODO: Remove after the deprecated 'system' field is removed.
(define (build-machine-systems machine)
(or (%build-machine-systems machine)
(list (build-machine-system machine))
(leave (G_ "The build-machine object lacks a value for its 'systems'
field."))))
(define-record-type* <build-requirements> (define-record-type* <build-requirements>
build-requirements make-build-requirements build-requirements make-build-requirements
build-requirements? build-requirements?
@ -359,8 +375,8 @@ (define store
(define (machine-matches? machine requirements) (define (machine-matches? machine requirements)
"Return #t if MACHINE matches REQUIREMENTS." "Return #t if MACHINE matches REQUIREMENTS."
(and (string=? (build-requirements-system requirements) (and (member (build-requirements-system requirements)
(build-machine-system machine)) (build-machine-systems machine))
(lset<= string=? (lset<= string=?
(build-requirements-features requirements) (build-requirements-features requirements)
(build-machine-features machine)))) (build-machine-features machine))))

71
tests/offload.scm Normal file
View file

@ -0,0 +1,71 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
(define-module (tests offload)
#:use-module (guix scripts offload)
#:use-module (srfi srfi-64))
(test-begin "offload")
(define-syntax-rule (expose-internal-definitions s1 s2 ...)
(begin
(define s1 (@@ (guix scripts offload) s1))
(define s2 (@@ (guix scripts offload) s2)) ...))
(expose-internal-definitions machine-matches?
build-requirements-system
build-requirements-features
build-machine-system
build-machine-systems
%build-machine-system
%build-machine-systems
build-machine-features)
(define (deprecated-build-machine system)
(build-machine
(name "m1")
(user "dummy")
(host-key "some-key")
(system system)))
(define (new-build-machine systems)
(build-machine
(name "m1")
(user "dummy")
(host-key "some-key")
(systems systems)))
;;; Test that deprecated build-machine definitions still work.
(test-assert (machine-matches? (deprecated-build-machine "i686-linux")
(build-requirements
(system "i686-linux"))))
(test-assert (machine-matches? (new-build-machine '("i686-linux"))
(build-requirements
(system "i686-linux"))))
;;; A build machine can act as more than one system type, thanks to QEMU
;;; emulation.
(test-assert (machine-matches? (new-build-machine '("armhf-linux"
"aarch64-linux"
"i686-linux"
"x86_64-linux"))
(build-requirements
(system "armhf-linux"))))