Repository priorities for the real world user
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Berry, Jimmy
Use additional repositories with confidence The topic of additional repositories comes up on a regular basis. The official position is to submit everything to Factory to avoid the issue, but for a large number of reasons this cannot always be the case. As such users living in the real world have to navigate the unsupported landscape. By far the best approach is to use repository priorities available through libzypp, but unfortunately this is not well known. This quick talk will cover the basic usage of priorities, strategies for real usage, and examples of how effective the workflow can be. Additionally some pitfalls will also be covered.
 All right, I guess go ahead and get started. So the talk's obviously repository priorities for the real world user. So I guess my motivation for making this talk is seeing people in IRC and some of the other support channels have issues that I believe could be resolved by using priorities on repositories. So first let's go over the goals we're hoping to accomplish with this talk. So I want to cover why you might actually add extra repositories beyond the default ones. Some of the pitfalls you can run into when using different types of repositories without priorities and how you can avoid those issues by using priorities. So first of all, let's take a look at why you might add repositories. So the first one, which I think is probably the most common one, is you want packages compiled with different or additional features, something like Pacman or something like that. Another one being you want packages that are not available already in one of the main product repositories. So you might add a develop project or someone's own project. Otherwise, you might add four pre-release packages either again in develop project or somewhere else. Or you may be maintaining packages and you want to easily install them on your machine after you build them on OBS to try them out. So there's some reasons why you might add repositories. So let's take a look at a fresh install and we'll go through the first reason and kind of cover what I imagine a lot of people already do. So let's go ahead and add Pacman repository on default install. So hopefully you're familiar with adding a repository with Zipper. And then another next step being to dump from that package or the repository, which would switch all your packages to Pacman. And then an example here of trying to install Blender, which I believe shouldn't be in the default install. So then the question arises, where does Blender come from? Because it's in both Pacman and it's in the main product repository. So does it come from Tumbley or Pacman? And the answer is not necessarily obvious. So basically here's an example where we have, these are real versions I pulled from those two repositories. You can see a big long version string. So in this case, it would come from Pacman because the build number is larger. So basically it's always looking for overall larger number. So let's say that we had a new release of Blender that hits Tumbleweed. It hasn't yet propagated to Pacman yet. So if you were to be installing and you happen to hit this scenario, you would have the 3.0 version in Tumbleweed and not in Pacman. So in this case, it would come from Tumbleweed. So there's no clear rhyme or reason. So it's kind of, do you feel lucky? Where is it going to come from? You can obviously check this yourself manually, but generally you usually add repositories with a reason in mind. So let's talk about when we're dumping with allow vendor change, which is when a lot of this comes from. So this used to be the default behavior that would come out of the box, but it would mean that if you added something like Pacman, depending on when you decided to update your machine and the state of Pacman and the product repository, the packages might flip flop because of those build division numbers or the versions of the packages themselves. And that can cause a lot of problems. You could actually have weird splits of packages from both repositories. Going forward, that's been disabled. So the default now is to not allow vendor change by default, which prevents the flip flop, but you don't necessarily resolve the question of where does the package come from if I don't manually pick it. This is obviously further aggravated if you add more things besides just something like Pacman if you add develop projects or your own repositories. A lot of times, those repositories will have different versions of the same packages, which means that package will be available in multiple sources and you'll have to make a decision as to where it comes from. So the question is, can we communicate why are we adding a repository to zipper in such a way that it would make the right choices to where to take the package from? And the answer is yes. You can use repository priorities. So priorities allow us to specify an order of precedence for the repositories, which means which one to basically use over the other ones when they both have the same package. So by doing this, we eliminate the flip flop because we'll basically always use whichever repository has the higher precedence. Which also gives us consistent package sourcing because if we know that a package is in multiple repositories, it always comes from whichever one we have the higher priority. So again, it doesn't switch and it picks from the same place. It's also nicely documents on your own machine, kind of what your reasoning for is for adding repositories because a lot of times they can be there for a long time and you may forget. So helps with that. And as we'll cover more, it nicely allows kind of a hands-free switching to the product repository when packages become stable. So I'll go through that example some more. So how to use zipper repository priorities. You can either set them when you add a repository, the example there, or you can modify them after the fact. So in case you're not familiar, the default priority is 99. Lower is more important. Higher is less important. So basically one would always win over a 99. So it's an ascending order of importance. So here's an example of what your default setup might look like on Tumblyd where you basically have the three main repositories all enabled with default priority of 99. So now let's go ahead and we'll revisit our Pac-Man scenario where we add Pac-Man. But this time we'll add it with a priority of 90. So that being more important than the default packages, which is typically why you would add Pac-Man since you want it to replace packages with those compiled with additional features. So now we don't actually have to specify to dup from Pac-Man. We can just say dup in general because it will go ahead and pick any packages that come from both from Pac-Man because we told it it's a higher priority. So here again is the example of installing a package that wouldn't exist in the first place, so Blender. So then we have the question again of where does it come from? But this time it's easy to answer. It comes from Pac-Man because we added that with a higher priority. So again we'll take a look at the example where I gave earlier in Flux where there was an update in Tumblyd that hasn't yet propagated to Pac-Man. So in this case the version number is clearly larger for Tumblyd than it is in Pac-Man. But again it will come from Pac-Man regardless, which is actually what we want. So there's no flip-flop and clearly go through it. So next I'll cover a case study that I had personally which was using KDE Connect. So it was not yet in the official repositories when I started using it. So I actually added, I had already the KDE Extra repository and I added the KDE Extra unstable. So you'll see I added them with priorities of 105 and 106, so those being less important than the default priorities, which means I basically only want to use them for packages that aren't in the main product repositories. So I would go ahead and install KDE Connect in that case. And then initially it came from the unstable repository because the KDE maintainers only had it there since it had no official releases. Some months later it had a pre-release candidate, so the package was moved into the KDE Extra repository and nicely automatically for me when I went to dump my machine there was a change in repositories to that one. I didn't have to do anything, but I basically got the more stable one. Basically you might want to continue using the nightly builds, but normally that's not what I want. And then a few months later had official stable releases, so when I dumped then it automatically switched to the standard repository. So to review, basically we can automatically switch to using the main product repos without having to do anything. So again, this took like five months or something to occur. Nothing I had to manage. I just noticed it when I went to dump, which was kind of nice. Everything's dealt with for you. It also means that we can add lots of extra repositories. So in fact you could just about add all the repositories on OBS with lower priorities and in theory they wouldn't completely hose your machine because you wouldn't use anything from them if it didn't come, if it was already in the main product repos. So that's kind of nice. So let's bring it all together. How do we actually make use of all this? So the key here is that we want to always dump with a lot of interchange and we want to have priorities set on all our repos. So if you don't have priorities set then obviously dumping with a lot of interchange can have undesirable effects. So these are the ways that you can dump with that either whenever you run the command or you can change the config. And this basically gives us everything that we want. So we automatically utilize new packages when they're added to Pac-Man. So if there's some package like Blender that comes out with new feature and it can only be compiled in Pac-Man then we automatically switch to that. And when they're dropped everything babes basically the way you want and extra repos, all the ones with lower priority are basically only utilized when you have to. So let's review the reasons why we might add repositories again here and basically assign them priorities. These are priorities I use. The key here is more or less important than the defaults. I generally leave space. So I basically increment by five. That way if I have interesting cases that need to fit in the middle I can do that. So for example, compiled with additional or different features that would be Pac-Man. So I would add it with priority of 90. So it would be more important. Our repository that contains extra packages 105 so that was like the KDX extra ones. And then again, pre-release or things you want to test yourself. They're higher priority than the defaults but I don't generally want to override Pac-Man in this case but you might. So for example, this is what my desktop looked like before this presentation. Obviously I have a lot of other repositories but basically just there. So if you run list repositories with the P option you get them sorted by priority which is nice. That goes back to that documenting why you might have added these. So for example, the network and open age repositories I added because I wanted a package out of those that wasn't available in the main product which is why they are lower priority. So again, it's very easy to tell when you look at that list. Of course the standard caveats apply. If you're installing things from the extra additional repos they may or may not work. Obviously things in developed projects are in flux so packages may be broken and zipper priorities doesn't protect you from things that are broken so I always look at the dump output and make sure it's the same proposal. And that's the basic idea. So from personal use before the changes to allow vendor change where all that was changed in Tumblr this resolved issues with Pac-Man and other things like that. So I used to have issues where you would have to resolve package problems when you have lots of repositories like that and I haven't had issues since doing this several years. So any questions or comments? All right. Then I guess that's it. So thank you. Thanks a bunch.
 All right, I guess go ahead and get started.  So the talk's obviously repository priorities for the real world user.  So I guess my motivation for making this talk is seeing people in IRC and some of the other  support channels have issues that I believe could be resolved by using priorities on repositories.  So first let's go over the goals we're hoping to accomplish with this talk.  So I want to cover why you might actually add extra repositories beyond the default ones.  Some of the pitfalls you can run into when using different types of repositories without  priorities and how you can avoid those issues by using priorities.  So first of all, let's take a look at why you might add repositories.  So the first one, which I think is probably the most common one, is you want packages  compiled with different or additional features, something like Pacman or something like that.  Another one being you want packages that are not available already in one of the main product  repositories.  So you might add a develop project or someone's own project.  Otherwise, you might add four pre-release packages either again in develop project or  somewhere else.  Or you may be maintaining packages and you want to easily install them on your machine  after you build them on OBS to try them out.  So there's some reasons why you might add repositories.  So let's take a look at a fresh install and we'll go through the first reason and kind  of cover what I imagine a lot of people already do.  So let's go ahead and add Pacman repository on default install.  So hopefully you're familiar with adding a repository with Zipper.  And then another next step being to dump from that package or the repository, which would  switch all your packages to Pacman.  And then an example here of trying to install Blender, which I believe shouldn't be in the  default install.  So then the question arises, where does Blender come from?  Because it's in both Pacman and it's in the main product repository.  So does it come from Tumbley or Pacman?  And the answer is not necessarily obvious.  So basically here's an example where we have, these are real versions I pulled from those  two repositories.  You can see a big long version string.  So in this case, it would come from Pacman because the build number is larger.  So basically it's always looking for overall larger number.  So let's say that we had a new release of Blender that hits Tumbleweed.  It hasn't yet propagated to Pacman yet.  So if you were to be installing and you happen to hit this scenario, you would have the 3.0  version in Tumbleweed and not in Pacman.  So in this case, it would come from Tumbleweed.  So there's no clear rhyme or reason.  So it's kind of, do you feel lucky?  Where is it going to come from?  You can obviously check this yourself manually, but generally you usually add repositories  with a reason in mind.  So let's talk about when we're dumping with allow vendor change, which is when a lot of  this comes from.  So this used to be the default behavior that would come out of the box, but it would mean  that if you added something like Pacman, depending on when you decided to update your machine  and the state of Pacman and the product repository, the packages might flip flop because of those  build division numbers or the versions of the packages themselves.  And that can cause a lot of problems.  You could actually have weird splits of packages from both repositories.  Going forward, that's been disabled.  So the default now is to not allow vendor change by default, which prevents the flip  flop, but you don't necessarily resolve the question of where does the package come from  if I don't manually pick it.  This is obviously further aggravated if you add more things besides just something like  Pacman if you add develop projects or your own repositories.  A lot of times, those repositories will have different versions of the same packages, which  means that package will be available in multiple sources and you'll have to make a decision  as to where it comes from.  So the question is, can we communicate why are we adding a repository to zipper in such  a way that it would make the right choices to where to take the package from?  And the answer is yes.  You can use repository priorities.  So priorities allow us to specify an order of precedence for the repositories, which  means which one to basically use over the other ones when they both have the same package.  So by doing this, we eliminate the flip flop because we'll basically always use whichever  repository has the higher precedence.  Which also gives us consistent package sourcing because if we know that a package is in multiple  repositories, it always comes from whichever one we have the higher priority.  So again, it doesn't switch and it picks from the same place.  It's also nicely documents on your own machine, kind of what your reasoning for is for adding  repositories because a lot of times they can be there for a long time and you may forget.  So helps with that.  And as we'll cover more, it nicely allows kind of a hands-free switching to the product  repository when packages become stable.  So I'll go through that example some more.  So how to use zipper repository priorities.  You can either set them when you add a repository, the example there, or you can modify them  after the fact.  So in case you're not familiar, the default priority is 99.  Lower is more important.  Higher is less important.  So basically one would always win over a 99.  So it's an ascending order of importance.  So here's an example of what your default setup might look like on Tumblyd where you  basically have the three main repositories all enabled with default priority of 99.  So now let's go ahead and we'll revisit our Pac-Man scenario where we add Pac-Man.  But this time we'll add it with a priority of 90.  So that being more important than the default packages, which is typically why you would  add Pac-Man since you want it to replace packages with those compiled with additional features.  So now we don't actually have to specify to dup from Pac-Man.  We can just say dup in general because it will go ahead and pick any packages that come  from both from Pac-Man because we told it it's a higher priority.  So here again is the example of installing a package that wouldn't exist in the first  place, so Blender.  So then we have the question again of where does it come from?  But this time it's easy to answer.  It comes from Pac-Man because we added that with a higher priority.  So again we'll take a look at the example where I gave earlier in Flux where there was  an update in Tumblyd that hasn't yet propagated to Pac-Man.  So in this case the version number is clearly larger for Tumblyd than it is in Pac-Man.  But again it will come from Pac-Man regardless, which is actually what we want.  So there's no flip-flop and clearly go through it.  So next I'll cover a case study that I had personally which was using KDE Connect.  So it was not yet in the official repositories when I started using it.  So I actually added, I had already the KDE Extra repository and I added the KDE Extra  unstable.  So you'll see I added them with priorities of 105 and 106, so those being less important  than the default priorities, which means I basically only want to use them for packages  that aren't in the main product repositories.  So I would go ahead and install KDE Connect in that case.  And then initially it came from the unstable repository because the KDE maintainers only  had it there since it had no official releases.  Some months later it had a pre-release candidate, so the package was moved into the KDE Extra  repository and nicely automatically for me when I went to dump my machine there was a  change in repositories to that one.  I didn't have to do anything, but I basically got the more stable one.  Basically you might want to continue using the nightly builds, but normally that's not  what I want.  And then a few months later had official stable releases, so when I dumped then it automatically  switched to the standard repository.  So to review, basically we can automatically switch to using the main product repos without  having to do anything.  So again, this took like five months or something to occur.  Nothing I had to manage.  I just noticed it when I went to dump, which was kind of nice.  Everything's dealt with for you.  It also means that we can add lots of extra repositories.  So in fact you could just about add all the repositories on OBS with lower priorities and  in theory they wouldn't completely hose your machine because you wouldn't use anything  from them if it didn't come, if it was already in the main product repos.  So that's kind of nice.  So let's bring it all together.  How do we actually make use of all this?  So the key here is that we want to always dump with a lot of interchange and we want  to have priorities set on all our repos.  So if you don't have priorities set then obviously dumping with a lot of interchange can have  undesirable effects.  So these are the ways that you can dump with that either whenever you run the command or  you can change the config.  And this basically gives us everything that we want.  So we automatically utilize new packages when they're added to Pac-Man.  So if there's some package like Blender that comes out with new feature and it can only  be compiled in Pac-Man then we automatically switch to that.  And when they're dropped everything babes basically the way you want and extra repos,  all the ones with lower priority are basically only utilized when you have to.  So let's review the reasons why we might add repositories again here and basically assign  them priorities.  These are priorities I use.  The key here is more or less important than the defaults.  I generally leave space.  So I basically increment by five.  That way if I have interesting cases that need to fit in the middle I can do that.  So for example, compiled with additional or different features that would be Pac-Man.  So I would add it with priority of 90.  So it would be more important.  Our repository that contains extra packages 105 so that was like the KDX extra ones.  And then again, pre-release or things you want to test yourself.  They're higher priority than the defaults but I don't generally want to override Pac-Man  in this case but you might.  So for example, this is what my desktop looked like before this presentation.  Obviously I have a lot of other repositories but basically just there.  So if you run list repositories with the P option you get them sorted by priority which  is nice.  That goes back to that documenting why you might have added these.  So for example, the network and open age repositories I added because I wanted a package  out of those that wasn't available in the main product which is why they are lower priority.  So again, it's very easy to tell when you look at that list.  Of course the standard caveats apply.  If you're installing things from the extra additional repos they may or may not work.  Obviously things in developed projects are in flux so packages may be broken and zipper  priorities doesn't protect you from things that are broken so I always look at the dump  output and make sure it's the same proposal.  And that's the basic idea.  So from personal use before the changes to allow vendor change where all that was changed  in Tumblr this resolved issues with Pac-Man and other things like that.  So I used to have issues where you would have to resolve package problems when you have  lots of repositories like that and I haven't had issues since doing this several years.  So any questions or comments?  All right.  Then I guess that's it.  So thank you.  Thanks a bunch.
openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_0_131.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_132_275.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_276_335.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_336_419.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_420_491.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_492_599.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_600_695.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_696_3491.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_3492_3695.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_3696_9623.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_9624_9767.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_9768_11891.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_11892_16571.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_16572_16679.png openSUSE_Conference_2018_-_Repository_priorities_f_x8kEaJU6hlw_16680_16750.png