Distributed build systems Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 00:00UTC (8:00pm US/Eastern) 2019 Community Moderator Election Results Why I closed the “Why is Kali so hard” questionOCR on Linux systemslibcanberra is failing to buildLooking for tools to build minimal linux desktopYum adds the dir 'Package' to baseurlHow to install libesedb? Build errorHow to use Vagrant provisioning to install a package without download?Get dependencies to build package from sourceTemporarily cache and write-buffer a directory (to speed up a build process on an NFS share)Why has there been so little innovation with autoconf configure scripts in the Unix and Linux ecosystem?Why do intelligent build systems not exist?
Why are vacuum tubes still used in amateur radios?
How to dry out epoxy resin faster than usual?
How often does castling occur in grandmaster games?
Is this another way of expressing the side limit?
"Lost his faith in humanity in the trenches of Verdun" — last line of an SF story
Why limits give us the exact value of the slope of the tangent line?
What is the meaning of 'breadth' in breadth first search?
Project Euler #1 in C++
How to compare two different files line by line in unix?
If Windows 7 doesn't support WSL, then what does Linux subsystem option mean?
How do I find out the mythology and history of my Fortress?
Selecting user stories during sprint planning
Can anything be seen from the center of the Boötes void? How dark would it be?
Sum letters are not two different
What order were files/directories outputted in dir?
Why does the remaining Rebel fleet at the end of Rogue One seem dramatically larger than the one in A New Hope?
Using audio cues to encourage good posture
What is the appropriate index architecture when forced to implement IsDeleted (soft deletes)?
Do wooden building fires get hotter than 600°C?
What initially awakened the Balrog?
Find 108 by using 3,4,6
Why weren't discrete x86 CPUs ever used in game hardware?
Should I use a zero-interest credit card for a large one-time purchase?
Importance of からだ in this sentence
Distributed build systems
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 00:00UTC (8:00pm US/Eastern)
2019 Community Moderator Election Results
Why I closed the “Why is Kali so hard” questionOCR on Linux systemslibcanberra is failing to buildLooking for tools to build minimal linux desktopYum adds the dir 'Package' to baseurlHow to install libesedb? Build errorHow to use Vagrant provisioning to install a package without download?Get dependencies to build package from sourceTemporarily cache and write-buffer a directory (to speed up a build process on an NFS share)Why has there been so little innovation with autoconf configure scripts in the Unix and Linux ecosystem?Why do intelligent build systems not exist?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
Which tools for distributed builds are currently in use in popular Linux distros, like Ubuntu, Debian, RedHat, etc.?
For example, we have a huge amount of packages with dependencies and try to compile all of them for the new release of the distro. It takes a lot of time to build them one by one. I have heard about distcc, but it shares one project compilation between the servers for distributed parallel building.
Is there any tool which analyzes dependencies between the packages (creates a graph, for example) and creates a schedule for building with package-level parallelism? For example, we have packages p1, p2, p3, p4, and two servers. One server builds p1, another p2 at the same time if they are not dependent on each other.
I need examples of the projects which are being used in production.
linux compiling distributions opensource-projects
edited Apr 15 at 3:42
Rui F Ribeiro
42.2k1484142
asked Apr 14 at 23:06
heylin_nheylin_n
132
New contributor
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
Which tools for distributed builds are currently in use in popular Linux distros, like Ubuntu, Debian, RedHat, etc.?
For example, we have a huge amount of packages with dependencies and try to compile all of them for the new release of the distro. It takes a lot of time to build them one by one. I have heard about distcc, but it shares one project compilation between the servers for distributed parallel building.
Is there any tool which analyzes dependencies between the packages (creates a graph, for example) and creates a schedule for building with package-level parallelism? For example, we have packages p1, p2, p3, p4, and two servers. One server builds p1, another p2 at the same time if they are not dependent on each other.
I need examples of the projects which are being used in production.
linux compiling distributions opensource-projects
edited Apr 15 at 3:42
Rui F Ribeiro
42.2k1484142
asked Apr 14 at 23:06
heylin_nheylin_n
132
New contributor
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1
Most modern distros deliver pre-compiled packages. Unless I mis-understand your question, it sounds like you are compiling most of the software for your systems locally. Is this correct? If not, what distro(s) are you using?
– 0xSheepdog
Apr 14 at 23:24
1
I build packages for custom Debian-like distro. Sometimes I need to rebuild the whole set of packages from sources. The result is a set of .deb packages with binaries inside. But it is a long-running process, so I am interested in ways of solving this problem.
– heylin_n
Apr 15 at 0:44
Sounds awesome. And challenging. Thanks for the info, I see Stephen has already answered.
– 0xSheepdog
Apr 15 at 16:00
add a comment |
Which tools for distributed builds are currently in use in popular Linux distros, like Ubuntu, Debian, RedHat, etc.?
For example, we have a huge amount of packages with dependencies and try to compile all of them for the new release of the distro. It takes a lot of time to build them one by one. I have heard about distcc, but it shares one project compilation between the servers for distributed parallel building.
Is there any tool which analyzes dependencies between the packages (creates a graph, for example) and creates a schedule for building with package-level parallelism? For example, we have packages p1, p2, p3, p4, and two servers. One server builds p1, another p2 at the same time if they are not dependent on each other.
I need examples of the projects which are being used in production.
linux compiling distributions opensource-projects
edited Apr 15 at 3:42
Rui F Ribeiro
42.2k1484142
asked Apr 14 at 23:06
heylin_nheylin_n
132
New contributor
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Which tools for distributed builds are currently in use in popular Linux distros, like Ubuntu, Debian, RedHat, etc.?
For example, we have a huge amount of packages with dependencies and try to compile all of them for the new release of the distro. It takes a lot of time to build them one by one. I have heard about distcc, but it shares one project compilation between the servers for distributed parallel building.
Is there any tool which analyzes dependencies between the packages (creates a graph, for example) and creates a schedule for building with package-level parallelism? For example, we have packages p1, p2, p3, p4, and two servers. One server builds p1, another p2 at the same time if they are not dependent on each other.
I need examples of the projects which are being used in production.
linux compiling distributions opensource-projects
linux compiling distributions opensource-projects
edited Apr 15 at 3:42
Rui F Ribeiro
42.2k1484142
asked Apr 14 at 23:06
heylin_nheylin_n
132
New contributor
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Apr 15 at 3:42
Rui F Ribeiro
42.2k1484142
asked Apr 14 at 23:06
heylin_nheylin_n
132
New contributor
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Apr 15 at 3:42
Rui F Ribeiro
42.2k1484142
edited Apr 15 at 3:42
Rui F Ribeiro
42.2k1484142
edited Apr 15 at 3:42
Rui F Ribeiro
42.2k1484142
42.2k1484142
asked Apr 14 at 23:06
heylin_nheylin_n
132
New contributor
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Apr 14 at 23:06
heylin_nheylin_n
132
asked Apr 14 at 23:06
heylin_nheylin_n
132
132
New contributor
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
heylin_n is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1
Most modern distros deliver pre-compiled packages. Unless I mis-understand your question, it sounds like you are compiling most of the software for your systems locally. Is this correct? If not, what distro(s) are you using?
– 0xSheepdog
Apr 14 at 23:24
1
I build packages for custom Debian-like distro. Sometimes I need to rebuild the whole set of packages from sources. The result is a set of .deb packages with binaries inside. But it is a long-running process, so I am interested in ways of solving this problem.
– heylin_n
Apr 15 at 0:44
Sounds awesome. And challenging. Thanks for the info, I see Stephen has already answered.
– 0xSheepdog
Apr 15 at 16:00
add a comment |
1
Most modern distros deliver pre-compiled packages. Unless I mis-understand your question, it sounds like you are compiling most of the software for your systems locally. Is this correct? If not, what distro(s) are you using?
– 0xSheepdog
Apr 14 at 23:24
1
I build packages for custom Debian-like distro. Sometimes I need to rebuild the whole set of packages from sources. The result is a set of .deb packages with binaries inside. But it is a long-running process, so I am interested in ways of solving this problem.
– heylin_n
Apr 15 at 0:44
Sounds awesome. And challenging. Thanks for the info, I see Stephen has already answered.
– 0xSheepdog
Apr 15 at 16:00
1
1
Most modern distros deliver pre-compiled packages. Unless I mis-understand your question, it sounds like you are compiling most of the software for your systems locally. Is this correct? If not, what distro(s) are you using?
– 0xSheepdog
Apr 14 at 23:24
Most modern distros deliver pre-compiled packages. Unless I mis-understand your question, it sounds like you are compiling most of the software for your systems locally. Is this correct? If not, what distro(s) are you using?
– 0xSheepdog
Apr 14 at 23:24
1
1
I build packages for custom Debian-like distro. Sometimes I need to rebuild the whole set of packages from sources. The result is a set of .deb packages with binaries inside. But it is a long-running process, so I am interested in ways of solving this problem.
– heylin_n
Apr 15 at 0:44
I build packages for custom Debian-like distro. Sometimes I need to rebuild the whole set of packages from sources. The result is a set of .deb packages with binaries inside. But it is a long-running process, so I am interested in ways of solving this problem.
– heylin_n
Apr 15 at 0:44
Sounds awesome. And challenging. Thanks for the info, I see Stephen has already answered.
– 0xSheepdog
Apr 15 at 16:00
Sounds awesome. And challenging. Thanks for the info, I see Stephen has already answered.
– 0xSheepdog
Apr 15 at 16:00
add a comment |
1 Answer
1
active
oldest
votes
Each distribution tends to have its own set of tools for this, there’s not much shared software here.
Debian uses wanna-build, buildd and sbuild, which you’ll all find documented on the Debian site (follow the links too). wanna-build maintains the build queue, buildd picks a package to build, and sbuild builds it. wanna-build tracks packages with missing dependencies using the “dep-wait” state; packages can enter that state directly (if wanna-build itself can identify that dependencies are missing) or after a build fails because of missing dependencies. There’s a tutorial available if you want to set up a local build infrastructure.
Fedora uses Koji, which is extensively documented. It also involves a number of different components, including koji-hub, the centralised database front-end, and kojid , which drives the builds. I’m not as familiar with Koji though so I don’t know how it all integrates to handle build states.
Other distributions have other build systems, such as Launchpad. They all handle the same concerns you have: a centralised view of all the packages, multiple build systems, and a centralised repository which receives the build results.
answered Apr 15 at 9:11
Stephen KittStephen Kitt
182k25419497
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
heylin_n is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f512460%2fdistributed-build-systems%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Each distribution tends to have its own set of tools for this, there’s not much shared software here.
Debian uses wanna-build, buildd and sbuild, which you’ll all find documented on the Debian site (follow the links too). wanna-build maintains the build queue, buildd picks a package to build, and sbuild builds it. wanna-build tracks packages with missing dependencies using the “dep-wait” state; packages can enter that state directly (if wanna-build itself can identify that dependencies are missing) or after a build fails because of missing dependencies. There’s a tutorial available if you want to set up a local build infrastructure.
Fedora uses Koji, which is extensively documented. It also involves a number of different components, including koji-hub, the centralised database front-end, and kojid , which drives the builds. I’m not as familiar with Koji though so I don’t know how it all integrates to handle build states.
Other distributions have other build systems, such as Launchpad. They all handle the same concerns you have: a centralised view of all the packages, multiple build systems, and a centralised repository which receives the build results.
answered Apr 15 at 9:11
Stephen KittStephen Kitt
182k25419497
add a comment |
Each distribution tends to have its own set of tools for this, there’s not much shared software here.
Debian uses wanna-build, buildd and sbuild, which you’ll all find documented on the Debian site (follow the links too). wanna-build maintains the build queue, buildd picks a package to build, and sbuild builds it. wanna-build tracks packages with missing dependencies using the “dep-wait” state; packages can enter that state directly (if wanna-build itself can identify that dependencies are missing) or after a build fails because of missing dependencies. There’s a tutorial available if you want to set up a local build infrastructure.
Fedora uses Koji, which is extensively documented. It also involves a number of different components, including koji-hub, the centralised database front-end, and kojid , which drives the builds. I’m not as familiar with Koji though so I don’t know how it all integrates to handle build states.
Other distributions have other build systems, such as Launchpad. They all handle the same concerns you have: a centralised view of all the packages, multiple build systems, and a centralised repository which receives the build results.
answered Apr 15 at 9:11
Stephen KittStephen Kitt
182k25419497
add a comment |
Each distribution tends to have its own set of tools for this, there’s not much shared software here.
Debian uses wanna-build, buildd and sbuild, which you’ll all find documented on the Debian site (follow the links too). wanna-build maintains the build queue, buildd picks a package to build, and sbuild builds it. wanna-build tracks packages with missing dependencies using the “dep-wait” state; packages can enter that state directly (if wanna-build itself can identify that dependencies are missing) or after a build fails because of missing dependencies. There’s a tutorial available if you want to set up a local build infrastructure.
Fedora uses Koji, which is extensively documented. It also involves a number of different components, including koji-hub, the centralised database front-end, and kojid , which drives the builds. I’m not as familiar with Koji though so I don’t know how it all integrates to handle build states.
Other distributions have other build systems, such as Launchpad. They all handle the same concerns you have: a centralised view of all the packages, multiple build systems, and a centralised repository which receives the build results.
answered Apr 15 at 9:11
Stephen KittStephen Kitt
182k25419497
Each distribution tends to have its own set of tools for this, there’s not much shared software here.
Debian uses wanna-build, buildd and sbuild, which you’ll all find documented on the Debian site (follow the links too). wanna-build maintains the build queue, buildd picks a package to build, and sbuild builds it. wanna-build tracks packages with missing dependencies using the “dep-wait” state; packages can enter that state directly (if wanna-build itself can identify that dependencies are missing) or after a build fails because of missing dependencies. There’s a tutorial available if you want to set up a local build infrastructure.
Fedora uses Koji, which is extensively documented. It also involves a number of different components, including koji-hub, the centralised database front-end, and kojid , which drives the builds. I’m not as familiar with Koji though so I don’t know how it all integrates to handle build states.
Other distributions have other build systems, such as Launchpad. They all handle the same concerns you have: a centralised view of all the packages, multiple build systems, and a centralised repository which receives the build results.
answered Apr 15 at 9:11
Stephen KittStephen Kitt
182k25419497
answered Apr 15 at 9:11
Stephen KittStephen Kitt
182k25419497
answered Apr 15 at 9:11
Stephen KittStephen Kitt
182k25419497
answered Apr 15 at 9:11
Stephen KittStephen Kitt
182k25419497
182k25419497
add a comment |
add a comment |
heylin_n is a new contributor. Be nice, and check out our Code of Conduct.
heylin_n is a new contributor. Be nice, and check out our Code of Conduct.
heylin_n is a new contributor. Be nice, and check out our Code of Conduct.
heylin_n is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f512460%2fdistributed-build-systems%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Most modern distros deliver pre-compiled packages. Unless I mis-understand your question, it sounds like you are compiling most of the software for your systems locally. Is this correct? If not, what distro(s) are you using?
– 0xSheepdog
Apr 14 at 23:24
1
I build packages for custom Debian-like distro. Sometimes I need to rebuild the whole set of packages from sources. The result is a set of .deb packages with binaries inside. But it is a long-running process, so I am interested in ways of solving this problem.
– heylin_n
Apr 15 at 0:44
Sounds awesome. And challenging. Thanks for the info, I see Stephen has already answered.
– 0xSheepdog
Apr 15 at 16:00