Всеволод Стахов: Новое в pkg и перспективы развития...
Post on 19-May-2015
2.719 Views
Preview:
DESCRIPTION
TRANSCRIPT
FreeBSD package management system
Vsevolod Stakhovvsevolod@FreeBSD.org
ruBSD conference December 14, 2013
Ports and packages
Ports is the comprehensive system of source packages.
I Mature
I Clear and well defined
I Simple (sometimes not)
I Configurable
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 2 of 26
Ports before pkg
Make�le
Dist�les
Patches
Ports
Installed
Files
Packages
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 3 of 26
Disadvantages of the old architecture
I Make cannot handle complex packages relationships
I Complicated upgrade procedure (hard to keep up-to-date)
I Hard to migrate between releases
I Long build time
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 4 of 26
Planned ports and pkg interaction
Build STAGE Packages Pkg Instal
Build Depends
Run Depends
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 5 of 26
Ports and packages
I Ports are used to build packages
I Dependencies are resolved by pkg, not make
I Stable branch of ports has an appropriate stable branch ofpackages
I Encourage users to install software from binary packages
I But do not prevent them from building custom packages fromthe ports
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 6 of 26
Repositories creation
Repo
Checksum
Checksum
Package
Checksum
Package
MirrorsBuildStable
portsChecksum
Signature
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 7 of 26
Pkg architecture
Mirrors Jobsfetch
Solver
Checksum
+ Manifest
Files...
Package Install
FS
PkgDB
User
Request
Remote
Repos
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 8 of 26
The current problems with pkg
I Legacy ports support (with no staging, for example)
I Plain dependencies style
I Naive solver
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 9 of 26
The problems of the solver in pkg
I Absence of conflicts resolving/handling
I No alternatives support
I Can perform merely a single task: install, upgrade or remove,so install task cannot remove packages for example
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 10 of 26
Existing systems
There are many examples of solvers used in different packagemanagement systems, for example:
I Zypper/SUSE - uses libsolv as the base
I Yum/RedHat - migrating to libsolv
I Apt/Debian - uses internal solver
I Pacman/Archlinux - uses naive internal solver
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 11 of 26
External solvers
To interact with an external solver we have chosen CUDF formatused in the Mancoosi research project http://mancoosi.org:
package: devel/libblah
version: 1
depends: x11/libfoo
package: security/blah
version: 2
depends: devel/libblah
conflicts: security/blah-devel
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 12 of 26
We need an internal solver!
Alternatives:
I Write own logic of dependencies and conflicts resolution?
I Use some existing solution?
I Use some known algorithm?
Use SAT solver for packages management
SAT expression︷ ︸︸ ︷(x1‖¬x2‖x3)︸ ︷︷ ︸
Clause
&(x3‖¬x1)&(x2)
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
We need an internal solver!
Alternatives:
I Write own logic of dependencies and conflicts resolution?
I Use some existing solution?
I Use some known algorithm?
Use SAT solver for packages management
SAT expression︷ ︸︸ ︷(x1‖¬x2‖x3)︸ ︷︷ ︸
Clause
&(x3‖¬x1)&(x2)
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
We need an internal solver!
Alternatives:
I Write own logic of dependencies and conflicts resolution?
I Use some existing solution?
I Use some known algorithm?
Use SAT solver for packages management
SAT expression︷ ︸︸ ︷(x1‖¬x2‖x3)︸ ︷︷ ︸
Clause
&(x3‖¬x1)&(x2)
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
We need an internal solver!
Alternatives:
I Write own logic of dependencies and conflicts resolution?
I Use some existing solution?
I Use some known algorithm?
Use SAT solver for packages management
SAT expression︷ ︸︸ ︷(x1‖¬x2‖x3)︸ ︷︷ ︸
Clause
&(x3‖¬x1)&(x2)
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
Making a SAT problem
I Assign a variable to each package: package A → a1, packageB → b1
I Interpret a request as a set of unary clauses:I Install/Upgrade package A → (a1)I Delete package B → (¬b1)
I Convert dependencies and conflicts to disjuncted clauses
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 14 of 26
Converting dependencies and conflicts
I If package A depends on package B (versions B1 and B2),then we can either have package A not installed or any of Binstalled:
(¬A‖B1‖B2)
I If we have a conflict between versions of B (B1, B2 and B3)then we ensure that merely one version is installed:
(¬B1‖¬B2)&(¬B1‖¬B3)&(¬B2‖¬B3)︸ ︷︷ ︸Conflicts chain
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 15 of 26
Converting dependencies and conflicts
I If package A depends on package B (versions B1 and B2),then we can either have package A not installed or any of Binstalled:
(¬A‖B1‖B2)
I If we have a conflict between versions of B (B1, B2 and B3)then we ensure that merely one version is installed:
(¬B1‖¬B2)&(¬B1‖¬B3)&(¬B2‖¬B3)︸ ︷︷ ︸Conflicts chain
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 15 of 26
The solving of SAT problem
Some rules to follow to speed up SAT problem solving.
I Trivial propagation - solve unary clauses
I Unit propagation - solve clauses with only a single unsolvedvariable
I Conflicts learning - if we assign some free variable and detecta conflict during unit propagation, we can fallback and learnthat this variable must be negated
I Package specific assumptions.
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 16 of 26
SAT problem propagation
I Trivial propagation - direct install or delete rules
(¬A‖B)& (A)︸︷︷︸true
& (¬C )︸ ︷︷ ︸false
&(¬A‖¬D)
I Unit propagation - simple depends and conflicts
Dependency
(¬A‖B)︸ ︷︷ ︸B→true
&
true︷︸︸︷(A) &
false︷ ︸︸ ︷(¬C ) &
Conflict
(¬A‖¬D)︸ ︷︷ ︸D→false
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 17 of 26
SAT problem propagation
I Trivial propagation - direct install or delete rules
(¬A‖B)& (A)︸︷︷︸true
& (¬C )︸ ︷︷ ︸false
&(¬A‖¬D)
I Unit propagation - simple depends and conflicts
Dependency
(¬A‖B)︸ ︷︷ ︸B→true
&
true︷︸︸︷(A) &
false︷ ︸︸ ︷(¬C ) &
Conflict
(¬A‖¬D)︸ ︷︷ ︸D→false
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 17 of 26
Conflicts driven learning
To handle alternatives it is required to test all variables unassigned:
1. full depth-first enumeration of possible values
2. fallback if a conflict found
3. remember which assignment caused conflict
4. make negative assignment for the learned variable and go tothe first step
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 18 of 26
Package specific assumptions
Pure SAT solvers cannot deal with package management as theydo not consider several packages peculiarities:
I try to keep installed packages (if no direct conflicts)
I do not install packages if they are not needed
I prefer high priority packages and repositories over low priorityones
These options also improve SAT performance providing a goodinitial assignment.
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 19 of 26
Packages universeWe convert all packages involved to a packages universe of thefollowing structure:
Name
Version
Con�icts Chain
Depend
Con�ict
N1 N2 N3 Nn
V1 V1 V1 V1
V2 V2 V2
V3
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 20 of 26
Package management task
I A request is splitted to install/upgrade and delete requestswhich could be passed simultaneously to the solver
I A conflicts between packages are detected with a repositorycreation
I All depends, reverse and conflicts of the requested packagesare analyzed and the package universe is created
I Each package is defined by its name and the digest ofsignificant fields (version, options and so on)
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 21 of 26
Solvers and Pkg
I Pkg may pass the formed universe to an external CUDFsolver:
I convert versionsI format requestI parse output
I Alternatively the internal SAT solver may be used:I convert the universe to SAT problemI formulate requestI ???I PROFIT
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 22 of 26
Perspectives
I Using pkg solver for ports management
I Better support of multiple repositories
I Test different solvers algorithms using CUDF
I New dependencies and conflicts format
I Provides and alternatives
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 23 of 26
New dependencies format
libblah >= 1.0 + option1,+option2‖libfoo! = 1.1
I Can depend on normal packages and virtual packages(provides)
I Easy to define the concrete dependency versions
I Alternative dependencies
P1Depends
P2 Vx>
Vy Vz !
P3
Or
Con ict
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 24 of 26
Alternatives
I Used to organize packages with the same functionality (e.g.web-browser)
I May be used to implement virtual dependencies(provides/requires)
Alternatives
Text EditorDepends
Package
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 25 of 26
Thank you for your attention!Questions?
vsevolod@FreeBSD.org
http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 26 of 26
top related