Remote package management

[uael]

A package manager or package management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer's operating system in a consistent manner.

A package manager deals with packages, distributions of software and data in archive files. Packages contain metadata, such as the software's name, description of its purpose, version number, vendor, checksum, and a list of dependencies necessary for the software to run properly. Upon installation, metadata is stored in a local package database. Package managers typically maintain a database of software dependencies and version information to prevent software mismatches and missing prerequisites. They work closely with software repositories, binary repository managers, and app stores.

Source: package manager

Remote package management roadmap:

• detect tools and environment and wrap them
  • git
  • curl
  • wget
  • unzip/tar/7z/gzip
  • ping (test speed)
• Extractor wrapper
  • tar
  • unzip
  • gzip
  • 7z
• Cache management
• semver
  • parse
  • satisfies
  • select
• Multi language
  • asm/c/c++
  • objc/c++
  • swift
  • rust
  • dlang
• Multi platform
  • linux
  • macOS
  • windows
    • git
    • downloader (http[s], ftp ..)
    • extractor (zip, tar.gz, 7z .. )
• Command line tools
  • require task
    • install
      • parse require info
      • parse and load packages from repositories
      • download package
        • optimization: select the fastest url if exists multiple urls
      • extract package
      • build package
        • xmake.lua
          • linux/macos
          • windows
        • makefile
          • linux/macos
          • windows
        • configure/make
          • linux/macos
        • cmakelists.txt
          • linux/macos
          • windows (in progress ..)
        • *.vcproj
          • windows
      • install package
      • install basic dependent tools (git, curl, unzip ...)
    • search
    • clear
    • remove
    • list
  • repo task
    • add repository
    • remove repository
    • list all repositories
    • clear all repositories and cache
    • global or local repositories
• xmake lua api
  • add_requires: add package require info
  • add_repositories: add repository url
  • add_packages: add package dependencies to target
  • package: package definition
    • set_urls
    • add_versions
    • set_homepage
    • set_description
    • add_deps
    • [on|before|after]_load: custom load script
    • [on|before|after]_install: custom install script
    • [on|before|after]_test: custom test script
    • add_patches
    • add_links
    • add_defines
    • add_includedirs

[uael]

References :

• yarn
• composer
• cocoapods
• ...

There is many other but they are good one

[waruqi]

Great! I have finished the xmake repo plugin task to manage repositories.

for example:

$ xmake repo --add origin https://github.com/tboox/xmake-repo.git
$ xmake repo --add other-repo https://github.com/xxxx/other-repo.git

Or use global add_repositories api in xmake.lua:

add_repositories("origin https://github.com/tboox/xmake-repo.git")
add_repositories("other-repo https://github.com/other/other-repo.git")

And the xmake-repo is the default repository, we need not add it manually.

[waruqi]

And we can add requires and add package dependencies (developing ..)

Add requires:

add_requires("tboox.tbox >=1.5.1", "zlib >=1.2.11")
add_requires("zlib master")
add_requires("xmake-repo@tboox.tbox >=1.5.1")
add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")

Add package dependencies:

target("test")
    add_packages("tboox.tbox", "zlib")

If you run xmake require or xmake require --install, xmake will install and update all required packages to local project repository directory.

[uael]

It's all great 👍
Actually this repo implementation is really nice and consistent. Even if I'm afraid about the future size of the xmake-repo.
A remaining task will be to automate packaging and publication on user repos.
I'm looking for semantic versioner, this is the yarn/npm one, and lua implementations.

[uael]

How did you plan to handle that case?

add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")

Excuse me, I'm wondering what's the advantage of xmake repo that git submodule doesn't have.

[waruqi]

@uael I just place xmake.lua and other some text files (.e.g *.patch) in xmake-repo and I did not intend to place the binary file in this repository.
And you can create yourself private repository to place some packages and binary files for each archs and platforms.So it will not be too large.

• add_requires("https://github.com/tboox/tbox.git@tboox.tbox >=1.5.1")

add_requires can process it directly If this project has complete tags like 1.2.1 or v1.x.x and it will define a package named tboox.tbox. We can use add_packages("tboox.tbox") to add package denpence to target.

If this git project has not tags, we can set a branch name, for example: add_requires("https://github.com/tboox/tbox.git@tboox.tbox master")

Or you can directly define package("tboox.tbox") to describe custem rules in xmake.lua if cannot get git tags, for example:

package("tboox.tbox")
      add_versions("1.5.1", "1.6.0", "1.6.1") -- we can add a version list if no git tags
      set_url("https://github.com/tboox/tbox/archive/v$(v1).$(v2).$(v3).tar.gz")
      on_build(function ()
           -- if need
      end)
      on_package(function ()
           -- if need
      end)

add_requires("tboox.tbox >=1.5.1")

We also use package("xxx") to describe every package in xmake-repo or other private repositories.

[waruqi]

@titansnow git submodule is great, but xmake repo and xmake require will provide more convenient cross-platform package management.

Some advantages:

• Do not need user to manage each third-party package
• Better version control
• Support .tar.gz or .zip package if the dependent package does not have a git repo, but git submodule need it.
• Automated package compilation and linking to the given target.
• Support multiple platforms, architectures and languanges.

In most cases, you only need to add two lines in xmake.lua to link dependent package automatically which exists in xmake-repo.

add_requires("pcre2 >=1.2.0", "zlib >= 1.2.11")
target("test")
    add_packages("pcre2", "zlib")

Then run:

$ xmake 

And it will show tips when the version of package dependences have been changed in xmake.lua

This project exists outdated package dependences, update them now? y (y / n)
y
installing and updating outdated package dependences ... (call `xmake require` automatically)
building project .. (`xmake build`)

[waruqi]

@uael semver.lua is great 👍. I will consider using it.

Sounds like fun. Like it:heart:. May it be hard work?

[uael]

@titansnow actually the following isn't possible with git submodule:
foo depends on bar;
foo depends on baz;
bar depends on baz;
then with package management the foo vendor dir look like:

vendor
|__ bar
   |__ ..
   :
|__ baz
   |__ ..
   :

[uael]

In the case of provided binary files i just dunno how to handle it properly, wondering what's your feeling about it