ryan77627/guix
1
0
Fork 0
mirror of https://git.in.rschanz.org/ryan77627/guix.git synced 2024-12-24 21:38:07 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

shell: Cache profiles even when using package specs.

Browse source 
This enables profile caching not just when '-m' or '-f' is used, but
also when package specs are passed on the command line, as in:

  guix shell -D guix git

It also changes profile cache keys to include the system type, which was
previously ignored.

* guix/scripts/shell.scm (options-with-caching)[single-file-for-caching]:
Remove.
Call 'profile-cached-gc-root' instead; adjust to accept two values.
(profile-cache-primary-key): New procedure.
(profile-cache-key): Remove.
(profile-file-cache-key, profile-spec-cache-key): New procedures.
(profile-cached-gc-root): Rewrite to include functionality formally in
'single-file-for-caching', but extend to handle package specs.
* gnu/packages.scm (cache-is-authoritative?): Export.
* guix/transformations.scm (transformation-option-key?): New procedure.
* doc/guix.texi (Invoking guix shell): Move '--rebuild-cache'
documentation to the bottom, just above '--root'.  Explain caching and
how these two options relate to that.
This commit is contained in:
Ludovic Courtès 2022-01-05 19:29:50 +01:00
parent 6128c27478
commit 0552dcb294
No known key found for this signature in database
GPG key ID: 090B11993D9AEBB5
4 changed files with 142 additions and 76 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

45
doc/guix.texi
View file

@ -5806,17 +5806,6 @@ This is similar to the same-named option in @command{guix package}
(@pxref{profile-manifest, @option{--manifest}}) and uses the same (@pxref{profile-manifest, @option{--manifest}}) and uses the same
manifest files. manifest files.
@item --rebuild-cache
When using @option{--manifest}, @option{--file}, or when invoked without
arguments, @command{guix shell} caches the environment so that
subsequent uses are instantaneous. The cache is invalidated anytime the
file is modified.
The @option{--rebuild-cache} forces the cached environment to be
refreshed even if the file has not changed. This is useful if the
@command{guix.scm} or @command{manifest.scm} has external dependencies,
or if its behavior depends, say, on environment variables.
@item --pure @item --pure
Unset existing environment variables when building the new environment, except Unset existing environment variables when building the new environment, except
those specified with @option{--preserve} (see below). This has the effect of those specified with @option{--preserve} (see below). This has the effect of
@ -5932,6 +5921,21 @@ directory:
guix shell --container --expose=$HOME=/exchange guile -- guile guix shell --container --expose=$HOME=/exchange guile -- guile
@end example @end example
@item --rebuild-cache
@cindex caching, of profiles
@cindex caching, in @command{guix shell}
In most cases, @command{guix shell} caches the environment so that
subsequent uses are instantaneous. Least-recently used cache entries
are periodically removed. The cache is also invalidated, when using
@option{--file} or @option{--manifest}, anytime the corresponding file
is modified.
The @option{--rebuild-cache} forces the cached environment to be
refreshed. This is useful when using @option{--file} or
@option{--manifest} and the @command{guix.scm} or @command{manifest.scm}
file has external dependencies, or if its behavior depends, say, on
environment variables.
@item --root=@var{file} @item --root=@var{file}
@itemx -r @var{file} @itemx -r @var{file}
@cindex persistent environment @cindex persistent environment
@ -5942,11 +5946,20 @@ register it as a garbage collector root.
This is useful if you want to protect your environment from garbage This is useful if you want to protect your environment from garbage
collection, to make it ``persistent''. collection, to make it ``persistent''.
When this option is omitted, the environment is protected from garbage When this option is omitted, @command{guix shell} caches profiles so
collection only for the duration of the @command{guix shell} that subsequent uses of the same environment are instantaneous---this is
session. This means that next time you recreate the same environment, comparable to using @option{--root} except that @command{guix shell}
you could have to rebuild or re-download packages. @xref{Invoking guix takes care of periodically removing the least-recently used garbage
gc}, for more on GC roots. collector roots.
In some cases, @command{guix shell} does not cache profiles---e.g., if
transformation options such as @option{--with-latest} are used. In
those cases, the environment is protected from garbage collection only
for the duration of the @command{guix shell} session. This means that
next time you recreate the same environment, you could have to rebuild
or re-download packages.
@xref{Invoking guix gc}, for more on GC roots.
@end table @end table
@command{guix shell} also supports all of the common build options that @command{guix shell} also supports all of the common build options that

3
gnu/packages.scm
View file

@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU ;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2012-2020, 2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2013 Mark H Weaver <mhw@netris.org> ;;; Copyright © 2013 Mark H Weaver <mhw@netris.org>
;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org> ;;; Copyright © 2014 Eric Bavier <bavier@member.fsf.org>
;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com> ;;; Copyright © 2016, 2017 Alex Kost <alezost@gmail.com>
@ -51,6 +51,7 @@ (define-module (gnu packages)
%auxiliary-files-path %auxiliary-files-path
%package-module-path %package-module-path
%default-package-module-path %default-package-module-path
cache-is-authoritative?
fold-packages fold-packages
fold-available-packages fold-available-packages

161
guix/scripts/shell.scm
View file

@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU ;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2021-2022 Ludovic Courtès <ludo@gnu.org>
;;; ;;;
;;; This file is part of GNU Guix. ;;; This file is part of GNU Guix.
;;; ;;;
@ -21,7 +21,8 @@ (define-module (guix scripts shell)
#:use-module ((guix diagnostics) #:select (location)) #:use-module ((guix diagnostics) #:select (location))
#:use-module (guix scripts environment) #:use-module (guix scripts environment)
#:autoload (guix scripts build) (show-build-options-help) #:autoload (guix scripts build) (show-build-options-help)
#:autoload (guix transformations) (show-transformation-options-help) #:autoload (guix transformations) (transformation-option-key?
show-transformation-options-help)
#:use-module (guix scripts) #:use-module (guix scripts)
#:use-module (guix packages) #:use-module (guix packages)
#:use-module (guix profiles) #:use-module (guix profiles)
@ -40,6 +41,7 @@ (define-module (guix scripts shell)
#:use-module ((guix build utils) #:select (mkdir-p)) #:use-module ((guix build utils) #:select (mkdir-p))
#:use-module (guix cache) #:use-module (guix cache)
#:use-module ((ice-9 ftw) #:select (scandir)) #:use-module ((ice-9 ftw) #:select (scandir))
#:autoload (gnu packages) (cache-is-authoritative?)
#:export (guix-shell)) #:export (guix-shell))
(define (show-help) (define (show-help)
@ -201,51 +203,35 @@ (define (authorized-shell-directory? directory)
(const #f))) (const #f)))
(define (options-with-caching opts) (define (options-with-caching opts)
"If OPTS contains exactly one 'load' or one 'manifest' key, automatically "If OPTS contains only options that allow us to compute a cache key,
add a 'profile' key (when a profile for that file is already in cache) or a automatically add a 'profile' key (when a profile for that file is already in
'gc-root' key (to add the profile to cache)." cache) or a 'gc-root' key (to add the profile to cache)."
(define (single-file-for-caching opts) ;; Attempt to compute a file name for use as the cached profile GC root.
(let loop ((opts opts) (let* ((root timestamp (profile-cached-gc-root opts))
(file #f)) (stat (and root (false-if-exception (lstat root)))))
(match opts (if (and (not (assoc-ref opts 'rebuild-cache?))
(() file) stat
((('package . _) . _) #f) (<= timestamp (stat:mtime stat)))
((('load . ('package candidate)) . rest) (let ((now (current-time)))
(and (not file) (loop rest candidate))) ;; Update the atime on ROOT to reflect usage.
((('manifest . candidate) . rest) (utime root
(and (not file) (loop rest candidate))) now (stat:mtime stat) 0 (stat:mtimensec stat)
((('expression . _) . _) #f) AT_SYMLINK_NOFOLLOW)
((_ . rest) (loop rest file))))) (alist-cons 'profile root
(remove (match-lambda
;; Check whether there's a single 'load' or 'manifest' option. When that is (('load . _) #t)
;; the case, arrange to automatically cache the resulting profile. (('manifest . _) #t)
(match (single-file-for-caching opts) (('package . _) #t)
(#f opts) (('ad-hoc-package . _) #t)
(file (_ #f))
(let* ((root (profile-cached-gc-root file)) opts))) ;load right away
(stat (and root (false-if-exception (lstat root))))) (if (and root (not (assq-ref opts 'gc-root)))
(if (and (not (assoc-ref opts 'rebuild-cache?)) (begin
stat (if stat
(<= (stat:mtime ((@ (guile) stat) file)) (delete-file root)
(stat:mtime stat))) (mkdir-p (dirname root)))
(let ((now (current-time))) (alist-cons 'gc-root root opts))
;; Update the atime on ROOT to reflect usage. opts))))
(utime root
now (stat:mtime stat) 0 (stat:mtimensec stat)
AT_SYMLINK_NOFOLLOW)
(alist-cons 'profile root
(remove (match-lambda
(('load . _) #t)
(('manifest . _) #t)
(_ #f))
opts))) ;load right away
(if (and root (not (assq-ref opts 'gc-root)))
(begin
(if stat
(delete-file root)
(mkdir-p (dirname root)))
(alist-cons 'gc-root root opts))
opts))))))
(define (auto-detect-manifest opts) (define (auto-detect-manifest opts)
"If OPTS do not specify packages or a manifest, load a \"guix.scm\" or "If OPTS do not specify packages or a manifest, load a \"guix.scm\" or
@ -308,28 +294,87 @@ (define %profile-cache-directory
(make-parameter (string-append (cache-directory #:ensure? #f) (make-parameter (string-append (cache-directory #:ensure? #f)
"/profiles"))) "/profiles")))
(define (profile-cache-key file) (define (profile-cache-primary-key)
"Return the \"primary key\" used when computing keys for the profile cache.
Return #f if no such key can be obtained and caching cannot be
performed--e.g., because the package cache is not authoritative."
(and (cache-is-authoritative?)
(match (current-channels)
(()
#f)
(((= channel-commit commits) ...)
(string-join commits)))))
(define (profile-file-cache-key file system)
"Return the cache key for the profile corresponding to FILE, a 'guix.scm' or "Return the cache key for the profile corresponding to FILE, a 'guix.scm' or
'manifest.scm' file, or #f if we lack channel information." 'manifest.scm' file, or #f if we lack channel information."
(match (current-channels) (match (profile-cache-primary-key)
(() #f) (#f #f)
(((= channel-commit commits) ...) (primary-key
(let ((stat (stat file))) (let ((stat (stat file)))
(bytevector->base32-string (bytevector->base32-string
;; Since FILE is not canonicalized, only include the device/inode ;; Since FILE is not canonicalized, only include the device/inode
;; numbers. XXX: In some rare cases involving Btrfs and NFS, this can ;; numbers. XXX: In some rare cases involving Btrfs and NFS, this can
;; be insufficient: <https://lwn.net/Articles/866582/>. ;; be insufficient: <https://lwn.net/Articles/866582/>.
(sha256 (string->utf8 (sha256 (string->utf8
(string-append (string-join commits) ":" (string-append primary-key ":" system ":"
(number->string (stat:dev stat)) ":" (number->string (stat:dev stat)) ":"
(number->string (stat:ino stat)))))))))) (number->string (stat:ino stat))))))))))
(define (profile-cached-gc-root file) (define (profile-spec-cache-key specs system)
"Return the cached GC root for FILE, a 'guix.scm' or 'manifest.scm' file, or "Return the cache key corresponding to SPECS built for SYSTEM, where SPECS
#f if we lack information to cache it." is a list of package specs. Return #f if caching is not possible."
(match (profile-cache-key file) (match (profile-cache-primary-key)
(#f #f) (#f #f)
(key (string-append (%profile-cache-directory) "/" key)))) (primary-key
(bytevector->base32-string
(sha256 (string->utf8
(string-append primary-key ":" system ":"
(object->string specs))))))))
(define (profile-cached-gc-root opts)
"Return two values: the file name of a GC root for use as a profile cache
for the options in OPTS, and a timestamp which, if greater than the GC root's
mtime, indicates that the GC root is stale. If OPTS do not permit caching,
return #f and #f."
(define (key->file key)
(string-append (%profile-cache-directory) "/" key))
(let loop ((opts opts)
(system (%current-system))
(file #f)
(specs '()))
(match opts
(()
(if file
(values (and=> (profile-file-cache-key file system) key->file)
(stat:mtime (stat file)))
(values (and=> (profile-spec-cache-key specs system) key->file)
0)))
(((and spec ('package . _)) . rest)
(if (not file)
(loop rest system file (cons spec specs))
(values #f #f)))
((('load . ('package candidate)) . rest)
(if (and (not file) (null? specs))
(loop rest system candidate specs)
(values #f #f)))
((('manifest . candidate) . rest)
(if (and (not file) (null? specs))
(loop rest system candidate specs)
(values #f #f)))
((('expression . _) . _)
;; Arbitrary expressions might be non-deterministic or otherwise depend
;; on external state so do not cache when they're used.
(values #f #f))
((((? transformation-option-key?) . _) . _)
;; Transformation options are potentially "non-deterministic", or at
;; least depending on external state (with-source, with-commit, etc.),
;; so do not cache anything when they're used.
(values #f #f))
((('system . system) . rest)
(loop rest system file specs))
((_ . rest) (loop rest system file specs)))))
;;; ;;;

9
guix/transformations.scm
View file

@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU ;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2016-2022 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2021 Marius Bakke <marius@gnu.org> ;;; Copyright © 2021 Marius Bakke <marius@gnu.org>
;;; ;;;
;;; This file is part of GNU Guix. ;;; This file is part of GNU Guix.
@ -56,6 +56,7 @@ (define-module (guix transformations)
tuned-package tuned-package
show-transformation-options-help show-transformation-options-help
transformation-option-key?
%transformation-options)) %transformation-options))
;;; Commentary: ;;; Commentary:
@ -796,6 +797,12 @@ (define (transformation-procedure key)
(and (eq? k key) proc))) (and (eq? k key) proc)))
%transformations)) %transformations))
(define (transformation-option-key? key)
"Return true if KEY is an option key (as returned while parsing options with
%TRANSFORMATION-OPTIONS) corresponding to a package transformation option.
For example, (transformation-option-key? 'with-input) => #t."
(->bool (transformation-procedure key)))
;;; ;;;
;;; Command-line handling. ;;; Command-line handling.