Всеволод Стахов: Новое в pkg и перспективы развития...

FreeBSD package management system

Vsevolod [email protected]

ruBSD conference December 14, 2013

[email protected]

Ports and packages

Ports is the comprehensive system of source packages.

I Mature

I Clear and well defined

I Simple (sometimes not)

I Configurable

http://highsecure.ru/pkg Vsevolod Stakhov [email protected] 2 of 26

http://highsecure.ru/pkg

[email protected]

Ports before pkg

Make�le

Dist�les

Patches

Ports

Installed

Files

Packages



[email protected]

Disadvantages of the old architecture

I Make cannot handle complex packages relationships

I Complicated upgrade procedure (hard to keep up-to-date)

I Hard to migrate between releases

I Long build time



[email protected]

Planned ports and pkg interaction

Build STAGE Packages Pkg Instal

Build Depends

Run Depends



[email protected]

Ports and packages

I Ports are used to build packages

I Dependencies are resolved by pkg, not make

I Stable branch of ports has an appropriate stable branch ofpackages

I Encourage users to install software from binary packages

I But do not prevent them from building custom packages fromthe ports



[email protected]

Repositories creation

Repo

Checksum

Checksum

Package

Checksum

Package

MirrorsBuildStable

portsChecksum

Signature



[email protected]

Pkg architecture

Mirrors Jobsfetch

Solver

Checksum

+ Manifest

Files...

Package Install

FS

PkgDB

User

Request

Remote

Repos



[email protected]

The current problems with pkg

I Legacy ports support (with no staging, for example)

I Plain dependencies style

I Naive solver



[email protected]

The problems of the solver in pkg

I Absence of conflicts resolving/handling

I No alternatives support

I Can perform merely a single task: install, upgrade or remove,so install task cannot remove packages for example



[email protected]

Existing systems

There are many examples of solvers used in different packagemanagement systems, for example:

I Zypper/SUSE - uses libsolv as the base

I Yum/RedHat - migrating to libsolv

I Apt/Debian - uses internal solver

I Pacman/Archlinux - uses naive internal solver



[email protected]

External solvers

To interact with an external solver we have chosen CUDF formatused in the Mancoosi research project http://mancoosi.org:

package: devel/libblah

version: 1

depends: x11/libfoo

package: security/blah

version: 2

depends: devel/libblah

conflicts: security/blah-devel


http://mancoosi.org


[email protected]

We need an internal solver!

Alternatives:

I Write own logic of dependencies and conflicts resolution?

I Use some existing solution?

I Use some known algorithm?

Use SAT solver for packages management

SAT expression︷︸︸︷(x1‖¬x2‖x3)︸︷︷︸

Clause

&(x3‖¬x1)&(x2)



[email protected]

Making a SAT problem

I Assign a variable to each package: package A → a1, packageB → b1

I Interpret a request as a set of unary clauses:I Install/Upgrade package A → (a1)I Delete package B → (¬b1)

I Convert dependencies and conflicts to disjuncted clauses



[email protected]

Converting dependencies and conflicts

I If package A depends on package B (versions B1 and B2),then we can either have package A not installed or any of Binstalled:

(¬A‖B1‖B2)

I If we have a conflict between versions of B (B1, B2 and B3)then we ensure that merely one version is installed:

(¬B1‖¬B2)&(¬B1‖¬B3)&(¬B2‖¬B3)︸︷︷︸Conflicts chain



[email protected]

The solving of SAT problem

Some rules to follow to speed up SAT problem solving.

I Trivial propagation - solve unary clauses

I Unit propagation - solve clauses with only a single unsolvedvariable

I Conflicts learning - if we assign some free variable and detecta conflict during unit propagation, we can fallback and learnthat this variable must be negated

I Package specific assumptions.



[email protected]

SAT problem propagation

I Trivial propagation - direct install or delete rules

(¬A‖B)& (A)︸︷︷︸true

& (¬C )︸︷︷︸false

&(¬A‖¬D)

I Unit propagation - simple depends and conflicts

Dependency

(¬A‖B)︸︷︷︸B→true

&

true︷︸︸︷(A) &

false︷︸︸︷(¬C ) &

Conflict

(¬A‖¬D)︸︷︷︸D→false



[email protected]

Conflicts driven learning

To handle alternatives it is required to test all variables unassigned:

1. full depth-first enumeration of possible values

2. fallback if a conflict found

3. remember which assignment caused conflict

4. make negative assignment for the learned variable and go tothe first step



[email protected]

Package specific assumptions

Pure SAT solvers cannot deal with package management as theydo not consider several packages peculiarities:

I try to keep installed packages (if no direct conflicts)

I do not install packages if they are not needed

I prefer high priority packages and repositories over low priorityones

These options also improve SAT performance providing a goodinitial assignment.



[email protected]

Packages universeWe convert all packages involved to a packages universe of thefollowing structure:

Name

Version

Con�icts Chain

Depend

Con�ict

N1 N2 N3 Nn

V1 V1 V1 V1

V2 V2 V2

V3



[email protected]

Package management task

I A request is splitted to install/upgrade and delete requestswhich could be passed simultaneously to the solver

I A conflicts between packages are detected with a repositorycreation

I All depends, reverse and conflicts of the requested packagesare analyzed and the package universe is created

I Each package is defined by its name and the digest ofsignificant fields (version, options and so on)



[email protected]

Solvers and Pkg

I Pkg may pass the formed universe to an external CUDFsolver:

I convert versionsI format requestI parse output

I Alternatively the internal SAT solver may be used:I convert the universe to SAT problemI formulate requestI ???I PROFIT



[email protected]

Perspectives

I Using pkg solver for ports management

I Better support of multiple repositories

I Test different solvers algorithms using CUDF

I New dependencies and conflicts format

I Provides and alternatives



[email protected]

New dependencies format

libblah >= 1.0 + option1,+option2‖libfoo! = 1.1

I Can depend on normal packages and virtual packages(provides)

I Easy to define the concrete dependency versions

I Alternative dependencies

P1Depends

P2 Vx>

Vy Vz !

P3

Or

Con ict



[email protected]

Alternatives

I Used to organize packages with the same functionality (e.g.web-browser)

I May be used to implement virtual dependencies(provides/requires)

Alternatives

Text EditorDepends

Package



[email protected]

Thank you for your attention!Questions?

[email protected]


[email protected]


[email protected]

Всеволод Стахов: Новое в pkg и перспективы развития...

Technology