Commit Graph

46 Commits

Author SHA1 Message Date
Ludovic Courtès 81c3dc3224
maint: Switch to Guile-JSON 3.x.
Guile-JSON 3.x is incompatible with Guile-JSON 1.x, which we relied on
until now: it maps JSON dictionaries to alists (instead of hash tables),
and JSON arrays to vectors (instead of lists).  This commit is about
adjusting all the existing code to this new mapping.

* m4/guix.m4 (GUIX_CHECK_GUILE_JSON): New macro.
* configure.ac: Use it.
* doc/guix.texi (Requirements): Mention the Guile-JSON version.
* guix/git-download.scm (git-fetch)[guile-json]: Use GUILE-JSON-3.
* guix/import/cpan.scm (string->license): Expect vectors instead of
lists.
(module->dist-name): Use 'json-fetch' instead of 'json-fetch-alist'.
(cpan-fetch): Likewise.
* guix/import/crate.scm (crate-fetch): Likewise, and call 'vector->list'
for DEPS.
* guix/import/gem.scm (rubygems-fetch): Likewise.
* guix/import/json.scm (json-fetch-alist): Remove.
* guix/import/pypi.scm (pypi-fetch): Use 'json-fetch' instead of
'json-fetch-alist'.
(latest-source-release, latest-wheel-release): Call 'vector->list' on
RELEASES.
* guix/import/stackage.scm (stackage-lts-info-fetch): Use 'json-fetch'
instead of 'json-fetch-alist'.
(lts-package-version): Use 'vector->list'.
* guix/import/utils.scm (hash-table->alist): Remove.
(alist->package): Pass 'vector->list' on the inputs fields, and default
to the empty vector.
* guix/scripts/import/json.scm (guix-import-json): Remove call to
'hash-table->alist'.
* guix/swh.scm (define-json-reader): Expect pair? or null? instead of
hash-table?.
[extract-field]: Use 'assoc-ref' instead of 'hash-ref'.
(json->branches): Use 'map' instead of 'hash-map->list'.
(json->checksums): Likewise.
(json->directory-entries, origin-visits): Call 'vector->list' on the
result of 'json->scm'.
* tests/import-utils.scm ("alist->package with dependencies"): New test.
* gnu/installer.scm (build-compiled-file)[builder]: Use GUILE-JSON-3.
* gnu/installer.scm (installer-program)[installer-builder]: Likewise.
* gnu/installer/locale.scm (iso639->iso639-languages): Use 'assoc-ref'
instead of 'hash-ref', and pass vectors through 'vector->list'.
(iso3166->iso3166-territories): Likewise.
* gnu/system/vm.scm (system-docker-image)[build]: Use GUILE-JSON-3.
* guix/docker.scm (manifest, config): Adjust for Guile-JSON 3.
* guix/scripts/pack.scm (docker-image)[build]: Use GUILE-JSON-3.
* guix/import/github.scm (fetch-releases-or-tags): Update docstring.
(latest-released-version): Use 'assoc-ref' instead of 'hash-ref'.  Pass
the result of 'fetch-releases-or-tags' to 'vector->list'.
* guix/import/launchpad.scm (latest-released-version): Likewise.
2019-07-25 00:16:41 +02:00
Maxim Cournoyer 4b60ab8c00
import: pypi: Preserve package name case when forming pypi-uri.
Fixes <https://bugs.gnu.org/33046>.

* guix/build-system/python.scm (pypi-uri): Update the host URI to
"files.pythonhosted.org".
* guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when
the source URL calls for it.
2019-07-02 10:08:00 +09:00
Maxim Cournoyer d514276b93
import: pypi: Include optional test inputs as native-inputs.
* guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use it.
(test-section?): New predicate.
(parse-requires.txt): Collect the optional test inputs, and return them as the
second element of the returned list.
(parse-wheel-metadata): Likewise.
(guess-requirements): Adapt.
(make-pypi-sexp): Likewise, and include the test inputs requirements as native
inputs in the returned package expression.

* tests/pypi.scm (test-requires.txt): Include a test section in the
test-requires.txt data.
(test-requires.txt-beaker): New variable.
("parse-requires.txt"): Adapt.
("parse-requires.txt - Beaker"): New test.
("parse-wheel-metadata, with extras"): Adapt.
("parse-wheel-metadata, with extras - Jedi"): Adapt.
("pypi->guix-package, no wheel"): Re-indent, and add the expected
native-inputs.
("pypi->guix-package, wheels"): Likewise.
2019-07-02 10:08:00 +09:00
Maxim Cournoyer f801c6215d
import: pypi: Completely mute the output of the "unzip" command.
* guix/import/pypi.scm (guess-requirements): Completely mute the output of the
"unzip" command.
2019-07-02 10:08:00 +09:00
Maxim Cournoyer 73e8373060
import: pypi: Fix typo in docstring.
* guix/import/pypi.scm (guess-requirements): Fix typo.
2019-07-02 10:08:00 +09:00
Maxim Cournoyer f0190a5dcd
import: pypi: Parse wheel METADATA instead of metadata.json.
With newer Wheel releases, there is no more metadata.json file; the METADATA
file should be used instead (see: https://github.com/pypa/wheel/issues/195).

This change updates our PyPI importer so that it uses the latter.

* guix/import/pypi.scm (define-module): Remove unnecessary modules and export
the PARSE-WHEEL-METADATA procedure.
(parse-wheel-metadata): Add procedure.
(guess-requirements): Use it.
* tests/pypi.scm (test-metadata): Test it.
2019-07-02 10:08:00 +09:00
Maxim Cournoyer c799ad7276
import: pypi: Support more types of archives.
This change enables the PyPI importer to look for requirements in a source
archive of a different type than "tar.gz" or "tar.bz2".  Also, scan the source
archive to find a requires.txt file.

* guix/import/pypi.scm: (guess-requirements)[tarball-directory]: Remove procedure.
[guess-requirements-from-source]: Use COMRESSED-FILE? to determine if an
archive type is supported, and some file extension logic that chooses either
"tar" or "unzip" as the extractor.  Search for the requires.txt file in the
archive instead of using a static, expected location.
(guess-requirements): Rename the TARBALL argument to ARCHIVE, to denote the
archive format is no longer bound specifically to the Tar format.
(compute-inputs): Likewise.
* tests/pypi.scm ("pypi->guix-package, no wheel"): Mock the requires.txt at a
non-standard location.
("pypi->guix-package, no usable requirement file."): New test.
2019-07-02 10:07:59 +09:00
Maxim Cournoyer cc9a77cd39
import: pypi: Deduplicate requirements.
* guix/import/pypi.scm (parse-requires.txt): Remove potential duplicates.
2019-07-02 10:07:59 +09:00
Maxim Cournoyer 803fb336d6
import: pypi: Improve parsing of requirement specifications.
The previous solution was fragile and could leave unwanted characters in a
requirement name, such as '[' or ']'.

Partially fixes <https://bugs.gnu.org/33047>.

* guix/import/pypi.scm (use-modules): Export SPECIFICATION->REQUIREMENT-NAME
(%requirement-name-regexp): New variable.
(clean-requirement): Rename to...
(specification->requirement-name): this, which now uses
%requirement-name-regexp to select the requirement name from the requirement
specification.
(parse-requires.txt): Adapt.
2019-07-02 10:07:59 +09:00
Maxim Cournoyer c4797121be
import: pypi: Do not parse optional requirements from source.
* guix/import/pypi.scm: Export PARSE-REQUIRES.TXT.
(clean-requirement): Move procedure to the top level.
(guess-requirements): Move the READ-REQUIREMENTS procedure to the top level,
and rename it to PARSE-REQUIRES.TXT.  Move the CLEAN-REQUIREMENT procedure to
the top level.  Move the COMMENT? functions inside the PARSE-REQUIRES.TXT
procedure.
(parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to prevent
parsing optional requirements.

* tests/pypi.scm (test-requires-with-sections): New variable.
("parse-requires.txt, with sections"): New test.
2019-07-02 10:07:59 +09:00
Maxim Cournoyer 01589acc5e
import: pypi: Do not consider requirements.txt files.
PyPI packages are mandated to have a setup.py file, which contains a listing
of the required dependencies.  The setuptools/distutils machinery embed
metadata in the archives they produce, which contains this information. There
is no need nor gain to collect the requirements from a "requirements.txt"
file, as it is not the true record of dependencies for PyPI packages and may
contain extraneous requirements or not exist at all.

* guix/import/pypi.scm (guess-requirements): Update comment.
[guess-requirements-from-source]: Do not attempt to parse the file
requirements.txt.  Streamline logic.
* tests/pypi.scm (test-requires.txt): Rename from test-requirements, to hint
at the file being tested.
("pypi->guix-package"): Adapt so that the fake package contains a requires.txt
file rather than a requirements.txt file.
("pypi->guix-package, wheels"): Likewise.
2019-07-02 10:07:59 +09:00
Leo Famulari 7277d06d8b
gnu: Use pypi.org.
<pypi.io> redirects to <pypi.org>.

* guix/build-system/python.scm (pypi-uri): Replace pypi.io with pypi.org.
* guix/import/pypi.scm (pypi-url?): Likewise.
* tests/pypi.scm: Likewise.
2018-10-18 23:46:33 -04:00
Ricardo Wurmus ac906cb7bb
import: pypi: Support recursive importing.
* guix/import/pypi.scm (guess-requirements): Use upstream names.
(compute-inputs): Return the upstream dependency names as an additional value.
(make-pypi-sexp): Likewise.
(pypi->guix-package): Memoize it.
(pypi-recursive-import): New procedure.
* guix/scripts/import/pypi.scm (show-help, %options): Accept "recursive"
option.
(guix-import-pypi): Use pypi-recursive-import.
* doc/guix.texi (Invoking guix import): Document it.
2018-08-30 15:22:22 +02:00
Ricardo Wurmus e37f889404
import: pypi: Also guess dependencies from *.egg-info/requires.txt.
* guix/import/pypi.scm (guess-requirements): Extract "requires.txt" from the
egg-info directory in addition to "requirements.txt"; strip off version
constraints; use call-with-temporary-directory.
2018-08-30 15:22:14 +02:00
Marius Bakke 8440db459a
import: PyPI: Update redirected URL.
* guix/import/pypi.scm (guix-package->pypi-name, pypi->guix-package): Update
docstrings.
(pypi-package?): Test for pypi.org, too.
(pypi-fetch): s/pypi.python.org/pypi.org/
* tests/pypi.scm ("guix-package->pypi-name, new URL style",
"pypi->guix-package", "pypi->guix-package, wheels"): Likewise.
2018-07-24 17:42:04 +02:00
Jelle Licht 3edf0d53a4
import: json: Consolidate duplicate json-fetch functionality.
* guix/import/json.scm (json-fetch): Return a list or hash table.
  (json-fetch-alist): New procedure.
* guix/import/github.scm (json-fetch*): Remove.
  (latest-released-version): Use json-fetch.
* guix/import/cpan.scm (module->dist-name): Use json-fetch-alist.
  (cpan-fetch): Likewise.
* guix/import/crate.scm (crate-fetch): Likewise.
* guix/import/gem.scm (rubygems-fetch): Likewise.
* guix/import/pypi.scm (pypi-fetch): Likewise.
* guix/import/stackage.scm (stackage-lts-info-fetch): Likewise.
2018-06-10 22:15:12 +02:00
Ludovic Courtès b719ddbbbb
import: pypi: Remove unneeded import.
* guix/import/pypi.scm: Remove unneeded import.
2017-10-10 23:12:03 +02:00
Danny Milosavljevic d1e7ca2df8
import: pypi: Always use pypi.io URL with downcased package name.
* guix/import/pypi.scm (make-pypi-sexp): Always use pypi.io URL with
downcased package name.
2017-06-15 09:49:13 +02:00
Mathieu Othacehe 22fc581db3
import: pypi: Robustify latest-release.
* guix/import/pypi.scm (latest-release): Check if pypi-fetch has
  failed. If so return #f, else construct the <upstream-source>.
2017-05-13 12:40:20 +02:00
Ludovic Courtès 69daee23af
ui: Rename '_' to 'G_'.
This avoids collisions with '_' when the latter is used as a 'match'
pattern for instance.  See
<https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00464.html>.

* guix/ui.scm: Rename '_' to 'G_'.
* po/guix/Makevars (XGETTEXT_OPTIONS): Adjust accordingly.
* build-aux/compile-all.scm (warnings): Remove 'format'.
* gnu/packages.scm,
gnu/services.scm,
gnu/services/shepherd.scm,
gnu/system.scm,
gnu/system/shadow.scm,
guix/gnupg.scm,
guix/http-client.scm,
guix/import/cpan.scm,
guix/import/elpa.scm,
guix/import/pypi.scm,
guix/nar.scm,
guix/scripts.scm,
guix/scripts/archive.scm,
guix/scripts/authenticate.scm,
guix/scripts/build.scm,
guix/scripts/challenge.scm,
guix/scripts/container.scm,
guix/scripts/container/exec.scm,
guix/scripts/copy.scm,
guix/scripts/download.scm,
guix/scripts/edit.scm,
guix/scripts/environment.scm,
guix/scripts/gc.scm,
guix/scripts/graph.scm,
guix/scripts/hash.scm,
guix/scripts/import.scm,
guix/scripts/import/cpan.scm,
guix/scripts/import/cran.scm,
guix/scripts/import/crate.scm,
guix/scripts/import/elpa.scm,
guix/scripts/import/gem.scm,
guix/scripts/import/gnu.scm,
guix/scripts/import/hackage.scm,
guix/scripts/import/nix.scm,
guix/scripts/import/pypi.scm,
guix/scripts/import/stackage.scm,
guix/scripts/lint.scm,
guix/scripts/offload.scm,
guix/scripts/pack.scm,
guix/scripts/package.scm,
guix/scripts/perform-download.scm,
guix/scripts/publish.scm,
guix/scripts/pull.scm,
guix/scripts/refresh.scm,
guix/scripts/size.scm,
guix/scripts/substitute.scm,
guix/scripts/system.scm,
guix/ssh.scm,
guix/upstream.scm: Use 'G_' instead of '_'.  Most of this change was
obtained by running: "sed -i -e's/(_ "/(G_ "/g' `find -name \*.scm`".
2017-05-03 16:16:17 +02:00
Ludovic Courtès 4eaac4b722
import: pypi: Correctly handle multiple-URL origins.
Fixes <https://bugs.gnu.org/25635>.
Reported by Sergei Trofimovich <slyfox@inbox.ru>.

* guix/import/pypi.scm (guix-package->pypi-name)[url->pypi-name]: New
procedure.
Rewrite body to match lists in addition to strings.
* tests/pypi.scm ("guix-package->pypi-name, several URLs"): New test.
2017-02-08 16:17:05 +01:00
Carlo Zancanaro 2f977d92d3
import: pypi: Don't add setuptools to propagated-inputs.
* guix/import/pypi.scm (compute-inputs): Don't add setuptools to the imported
  package's propagated-inputs.
2017-01-26 13:44:31 +01:00
Marius Bakke 4cb7786eff
import: pypi: Match new 'pypi-uri' domain in updater.
* guix/import/pypi.scm (pypi-package?): Match pypi.io domain.
2016-12-20 12:57:26 +01:00
Eric Bavier 63773200d7
import: json: Silence json-fetch output.
* guix/import/json.scm (json-fetch): Use http-fetch instead of url-fetch
to avoid writing to stdout and a temporary file for each invocation.
* guix/import/gem.scm (rubygems-fetch): Do not redirect json-fetch
output to /dev/null.
* guix/import/pypi.scm (pypi-fetch): Likewise.
2016-12-12 22:22:02 -06:00
Hartmut Goebel b5c347ad3d
import: pypi: All inputs are propagated-inputs by default.
* guix/import/pypi.scm (maybe-inputs): Return inputs as
  "propagated-inputs".
2016-10-23 19:23:14 +02:00
David Craven 263ac57fc2
import: Move string->license to importers.
* guix/import/gem.scm (string->license): Move from (guix import utils).
* guix/import/pypi.scm (string->license): Move from (guix import utils).
2016-09-28 12:47:29 +02:00
Ludovic Courtès 8173ceee1f
import: pypi: Correctly handle new-style URLs.
Fixes <http://bugs.gnu.org/23997>.

* guix/import/pypi.scm (guix-package->pypi-name): Rewrite using
'basename' and 'hyphen-package-name->name+version'.
* tests/pypi.scm ("guix-package->pypi-name, old URL style")
("guix-package->pypi-name, new URL style"): New tests.
2016-07-26 17:49:34 +02:00
Cyril Roelandt aebd383d04 import: pypi: do not fail when 'run_requires' is missing from the metadata.
* guix/import/pypi.scm (read-wheel-metadata): do not crash when 'run_requires'
is missing from the metadata.
2016-06-25 15:33:06 +02:00
Cyril Roelandt 266785d21e import: pypi: read requirements from wheels.
* doc/guix.tex (Invoking guix import): Mention that the pypi importer
works better with "unzip".
* guix/import/pypi.scm (latest-wheel-release,
wheel-url->extracted-directory): New procedures.
* tests/pypi.scm (("pypi->guix-package, wheels"): New test.
2016-06-14 22:03:22 +02:00
Ludovic Courtès 7d27a0259b upstream: Pass a package object to updaters.
* guix/upstream.scm (package-update-path): Pass PACKAGE to
'latest-release'.
* guix/gnu-maintenance.scm (latest-release*)
(latest-gnome-release, latest-xorg-release): Adjust accordingly.
* guix/import/cran.scm (latest-cran-release):
(latest-bioconductor-release): Likewise.
* guix/import/elpa.scm (latest-release): Likewise.
* guix/import/gem.scm (latest-release): Likewise.
* guix/import/github.scm (latest-release): Likewise.
* guix/import/hackage.scm (latest-release): Likewise.
* guix/import/pypi.scm (latest-release): Likewise.
2016-04-15 00:32:18 +02:00
Ludovic Courtès 522773b700 import: pypi: Emit 'pypi-uri' only when it yields the right URL.
Fixes <http://bugs.gnu.org/23062>.
Reported by Danny Milosavljevic <dannym@scratchpost.org>.

* guix/import/pypi.scm (make-pypi-sexp): Check whether 'pypi-uri'
returns SOURCE-URL and fall back to the full URL otherwise.
* tests/pypi.scm ("pypi->guix-package"): Adjust expected URI
accordingly.

Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>
2016-03-19 23:54:19 +01:00
Ludovic Courtès 85dce718cb import: pypi: Updater silently ignores packages without source.
Reported by Andreas Enge <andreas@enge.fr>
at <https://lists.gnu.org/archive/html/guix-devel/2015-11/msg00764.html>.

* guix/import/pypi.scm (&missing-source-error): New error type.
(latest-source-release): Raise it instead of using 'error'.
(pypi->guix-package): Guard against it and use 'leave' to report the
error.
(latest-release): Guard against it and return #f when caught.
2015-12-01 23:18:48 +01:00
Ludovic Courtès 32728adb95 import: pypi: Make downloads silent.
* guix/import/pypi.scm (pypi-fetch): Wrap body in
  'call-with-output-file' and 'with-error-to-port'.
2015-11-04 00:12:19 +01:00
Ludovic Courtès d1cb7e9562 import: pypi: Add missing copyright line.
* guix/import/pypi.scm: Add missing copyright line for bab020d.
2015-11-04 00:12:19 +01:00
Cyril Roelandt bab020d7ca import: pypi: add updater
* guix/import/pypi.scm (guix-package->pypi-name,
  latest-release): New procedures.
  (%pypi-updater): New variable.
* guix/scripts/refresh.scm (%updaters): Add %PYPI-UPDATER.
* doc/guix.texi (Invoking guix refresh): Mention PyPI
2015-11-03 23:41:25 +01:00
Cyril Roelandt d8bdd38201 import: pypi: Use "pypi-uri" instead of building the URL manually.
* guix/import/pypi.scm (make-pypi-sexp): Use "pypi-uri".
* tests/pypi.scm: Update the tests accordingly.
2015-11-03 21:38:13 +01:00
Ludovic Courtès e3302fec18 import: pypi: Remove unused procedure.
* guix/import/pypi.scm (join): Remove.
2015-10-05 23:10:14 +02:00
David Thompson 140b304815 import: pypi: Move generally useful procedures to utils module.
* guix/import/pypi.scm (make-pypi-sexp): Factorize license to symbol
  conversion code.
  (string->license, snake-case, guix-hash-url): Move from here...
* guix/import/utils.scm: ... to here.
  (license->symbol): New procedure.
2015-08-27 18:02:35 -04:00
Ludovic Courtès dee2b3bdb9 import: pypi: Improve warning message.
* guix/import/pypi.scm (guess-requirements): Add missing newline and quotes in
  warning.
2015-07-21 21:14:33 +02:00
Cyril Roelandt ff986890ec import: pypi: Detect inputs.
* guix/import/pypi.scm (python->package-name, maybe-inputs, compute-inputs,
  guess-requirements): New procedures.
* guix/import/pypi.scm (guix-hash-url): Now takes a filename instead of an
  URL as input.
* guix/import/pypi.scm (make-pypi-sexp): Now tries to generate the inputs
  automagically.
* tests/pypi.scm: Update the test.
2015-06-24 22:47:59 +02:00
Eric Bavier 1ff2619bc1 import: Factorize utility functions.
* guix/import/pypi.scm (hash-table->alist, flatten, assoc-ref*,
  url-fetch, json-fetch): Pull procedures from here into...
* guix/import/utils.scm: Here and...
* guix/import/json.scm: Here.  New file.
* Makefile.am (MODULE)[HAVE_GUILE_JSON]: Add it.
* guix/import/gnu.scm (file-sha256): Move from here to...
* guix/hash.scm: Here.
* tests/pypi.scm (pypi->guix-package): Update mock module reference.
2015-01-09 10:33:24 -06:00
Ludovic Courtès 467a3c93db import: pypi: Gracefully handle non-existent packages.
Fixes <http://bugs.gnu.org/18831>.
Reported by Ian Denhardt <ian@zenhack.net>.

* guix/import/pypi.scm (url-fetch, json-fetch, pypi-fetch): Augment
  docstring to mention #f on failure.
  (pypi->guix-package): Likewise, and actually return #f on failure.
* guix/scripts/import/pypi.scm (guix-import-pypi): Call 'leave' when
  'pypi->guix-package' returns #f.
2014-11-05 17:56:39 +01:00
Cyril Roelandt 68b18aae6a guix import pypi: Fix a typo in a docstring.
* guix/import/pypi.scm (snake-case): Fix documentation.
2014-10-22 02:47:29 +02:00
Cyril Roelandt a5ffa812b8 guix import pypi: do not add "python-" to a package name if it's already there.
* guix/import/pypi.scm (make-pypi-sexp): test whether the package name starts
  with "python-" before modifying it.
2014-10-22 02:47:29 +02:00
Cyril Roelandt cf690129b4 guix import pypi: add the Apache License 2.0
* guix/import/pypi.scm (string->license): add ASL2.0.
2014-10-22 02:47:29 +02:00
David Thompson 1b3e968512 import: Add PyPI importer.
* guix/snix.scm: Delete.
* guix/import/snix.scm: New file.
* guix/import/pypi.scm: New file.
* guix/import/utils.scm: New file.
* guix/scripts/import/nix.scm: New file.
* guix/scripts/import/pypi.scm: New file.
* tests/pypi.scm: New file.
* tests/snix.scm: Import (guix import snix) module.
* guix/scripts/import.scm (%default-options, %options): Delete.
  (%standard-import-options, importers): New variables.
  (show-help): List importers.
  (guix-import): Factor out Nix-specific logic.  Delegate to correct importer
  based upon first argument.
* configure.ac (HAVE_GUILE_JSON): New conditional.
* Makefile.am (MODULES): Add new files and remove 'guix/snix.scm'.
  (SCM_TESTS): Add 'tests/pypi.scm' if guile-json is installed.
2014-09-29 19:30:28 -04:00