nodejs packaging in Fedora

[Stuart D Gathman]

I've started working on packaging scuttlebot for Fedora. I see that we now have a Fedora package for every nodejs module. This makes it easy to map directories in node_modules to package names - however, it means submitting hundreds of packages to ever get scuttlebot submitted.

I'm wondering if there is a better way. A node module typically corresponds to a .o file in a C library (with exceptions like libsodium). It is like having a separate package for every function in glibc. Suppose we did this:

1) a nodejs-stdlib that includes all the common modules (a list to be argued over at length :-) ). There is no penalty other than a small amount of disk space for unused modules - just like with a C library.

2) other multi-module systems are combined - usually including all modules with the same first word. For instance, all the pull stream modules begin with 'pull': pull-abortable, pull-box-stream, pull-cat, pull-cont, etc. This would become nodejs-pull, and include all the pull modules.

When the package name matches the first word of the module name, then determining the package is still easy. When that is not the case, as with the proposed nodejs-stdlib, then dnf can still search for npm(...)
_______________________________________________
nodejs mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to

[Troy Dawson]

Post by Stuart D Gathman
I've started working on packaging scuttlebot for Fedora. I see that we now have a Fedora package for every nodejs module. This makes it easy to map directories in node_modules to package names - however, it means submitting hundreds of packages to ever get scuttlebot submitted.
1) a nodejs-stdlib that includes all the common modules (a list to be argued over at length :-) ). There is no penalty other than a small amount of disk space for unused modules - just like with a C library.
2) other multi-module systems are combined - usually including all modules with the same first word. For instance, all the pull stream modules begin with 'pull': pull-abortable, pull-box-stream, pull-cat, pull-cont, etc. This would become nodejs-pull, and include all the pull modules.
When the package name matches the first word of the module name, then determining the package is still easy. When that is not the case, as with the proposed nodejs-stdlib, then dnf can still search for npm(...)

Although this looks tempting, it's only looking at half the problem.
Versions
Nodejs modules get updated all the time, at different rates for
different packages.
If you had just one package for many modules, it would be getting
updated at an alarming rate.

Troy
_______________________________________________
nodejs mailing list -- ***@lists.fedoraproject.org
To unsubscrib

[Stephen Gallagher]

Post by Stuart D Gathman

Post by Stuart D Gathman
I've started working on packaging scuttlebot for Fedora. I see that we

now have a Fedora package for every nodejs module. This makes it easy to
map directories in node_modules to package names - however, it means
submitting hundreds of packages to ever get scuttlebot submitted.

Post by Stuart D Gathman
I'm wondering if there is a better way. A node module typically

corresponds to a .o file in a C library (with exceptions like libsodium).
It is like having a separate package for every function in glibc. Suppose

Post by Stuart D Gathman
1) a nodejs-stdlib that includes all the common modules (a list to be

argued over at length :-) ). There is no penalty other than a small amount
of disk space for unused modules - just like with a C library.

Post by Stuart D Gathman
2) other multi-module systems are combined - usually including all

modules with the same first word. For instance, all the pull stream
modules begin with 'pull': pull-abortable, pull-box-stream, pull-cat,
pull-cont, etc. This would become nodejs-pull, and include all the pull
modules.

Post by Stuart D Gathman
When the package name matches the first word of the module name, then

determining the package is still easy. When that is not the case, as with
the proposed nodejs-stdlib, then dnf can still search for npm(...)
Although this looks tempting, it's only looking at half the problem.
Versions
Nodejs modules get updated all the time, at different rates for
different packages.
If you had just one package for many modules, it would be getting
updated at an alarming rate.

I think the better approach is carefully-controlled bundling in Fedora. As
of a couple years ago, it is now permissible to bundle software together in
Fedora if it meets certain conditions:
1) If the dependency is already packaged in Fedora and this software is
compatible with that version, then this software must link against the
unbundled version.
2) If the dependency is not yet packaged in Fedora but is likely to be
useful to large amounts of Fedora software, it is strongly encouraged that
it be packaged separately.
3) Other dependencies MAY be carried internally by the package that needs
them, but that package MUST include `Provides: bundled(npm(modulename)) =
version`

Rule 3) is so that if there is a security vulnerability in npm(modulename),
we can find any and all software that is required to be updated.

I haven't had any time to work on it, but I'd very much like to develop an
automatic RPM dependency generator that will recurse down the node_modules
directories, read their package.json files and automatically create those
Provides. If anyone else wants to take a crack at doing that, it would be
an immense help.

[Stuart Gathman]

Post by Troy Dawson
Although this looks tempting, it's only looking at half the problem.
Versions
Nodejs modules get updated all the time, at different rates for
different packages.
If you had just one package for many modules, it would be getting
updated at an alarming rate.

Point taken.

Post by Troy Dawson
I think the better approach is carefully-controlled bundling in
Fedora. As of a couple years ago, it is now permissible to bundle
1) If the dependency is already packaged in Fedora and this software
is compatible with that version, then this software must link against
the unbundled version.
2) If the dependency is not yet packaged in Fedora but is likely to be
useful to large amounts of Fedora software, it is strongly encouraged
that it be packaged separately.
3) Other dependencies MAY be carried internally by the package that
bundled(npm(modulename)) = version`
Rule 3) is so that if there is a security vulnerability in
npm(modulename), we can find any and all software that is required to
be updated.
I haven't had any time to work on it, but I'd very much like to
develop an automatic RPM dependency generator that will recurse down
the node_modules directories, read their package.json files and
automatically create those Provides. If anyone else wants to take a
crack at doing that, it would be an immense help.

That sounds workable. Thanks! There are 502 npm dependencies of
scuttlebot, 196 of which are already in Fedora. I shudder to think of
the Requires: list and the Provides: list!!

[Stephen Gallagher]

Post by Stephen Gallagher
I haven't had any time to work on it, but I'd very much like to develop an
automatic RPM dependency generator that will recurse down the node_modules
directories, read their package.json files and automatically create those
Provides. If anyone else wants to take a crack at doing that, it would be
an immense help.

A big thank-you needs to go to Tomas Tomacek for
https://pagure.io/nodejs-packaging/pull-request/2 which is now in effect on
Fedora 28 (Rawhide) builds.

So now any Node.js package that is built in Fedora will automatically have
`Provides: bundled(nodejs-foo)` added automatically.

[Zuzana Svetlikova]

I was wondering, could we use bundling and clean up the mess that is
current nodejs stack in Fedora?

To be more precise, there are currently over 2000 packages and over 500
nodejs-* is available bugs,
that nobody has time to fix and/or feels like fixing, because updating one
module might/will break several others.

For the sake of maintainability, I'd like to leave as little packages as
possible.

Post by Stephen Gallagher

Post by Stephen Gallagher
I haven't had any time to work on it, but I'd very much like to develop
an automatic RPM dependency generator that will recurse down the
node_modules directories, read their package.json files and automatically
create those Provides. If anyone else wants to take a crack at doing that,
it would be an immense help.

A big thank-you needs to go to Tomas Tomacek for
https://pagure.io/nodejs-packaging/pull-request/2 which is now in effect
on Fedora 28 (Rawhide) builds.
So now any Node.js package that is built in Fedora will automatically have
`Provides: bundled(nodejs-foo)` added automatically.
_______________________________________________

[Stephen Gallagher]

That sounds like a Change Proposal for F28.

So your idea would be to mass-retire all but the most common or
security-sensitive NPMs from Fedora and rely on bundling in most cases?

That will also mean coming up with a strategy for removing those unbundled
packages on updates. Thoughts on that?

Post by Zuzana Svetlikova
I was wondering, could we use bundling and clean up the mess that is
current nodejs stack in Fedora?
To be more precise, there are currently over 2000 packages and over 500
nodejs-* is available bugs,
that nobody has time to fix and/or feels like fixing, because updating one
module might/will break several others.
For the sake of maintainability, I'd like to leave as little packages as
possible.

Post by Stephen Gallagher

Post by Stephen Gallagher
I haven't had any time to work on it, but I'd very much like to develop
an automatic RPM dependency generator that will recurse down the
node_modules directories, read their package.json files and automatically
create those Provides. If anyone else wants to take a crack at doing that,
it would be an immense help.

A big thank-you needs to go to Tomas Tomacek for
https://pagure.io/nodejs-packaging/pull-request/2 which is now in effect
on Fedora 28 (Rawhide) builds.
So now any Node.js package that is built in Fedora will automatically
have `Provides: bundled(nodejs-foo)` added automatically.
_______________________________________________
_______________________________________________

[Tom Hughes]

Binary compiled ones probably also need to be kept.

Presumably this also means identifying "top level" packages that are an
end goal in themselves and need to be kept.

Tom

Post by Stephen Gallagher
That sounds like a Change Proposal for F28.
So your idea would be to mass-retire all but the most common or
security-sensitive NPMs from Fedora and rely on bundling in most cases?
That will also mean coming up with a strategy for removing those
unbundled packages on updates. Thoughts on that?
I was wondering, could we use bundling and clean up the mess that is
current nodejs stack in Fedora?
To be more precise, there are currently over 2000 packages and over
500 nodejs-* is available bugs,
that nobody has time to fix and/or feels like fixing, because
updating one module might/will break several others.
For the sake of maintainability, I'd like to leave as little
packages as possible.
On Thu, Sep 14, 2017 at 6:45 PM, Stephen Gallagher
On Wed, Aug 9, 2017 at 8:06 AM Stephen Gallagher
I haven't had any time to work on it, but I'd very much like
to develop an automatic RPM dependency generator that will
recurse down the node_modules directories, read their
package.json files and automatically create those Provides.
If anyone else wants to take a crack at doing that, it would
be an immense help.
A big thank-you needs to go to Tomas Tomacek for
https://pagure.io/nodejs-packaging/pull-request/2 which is now
in effect on Fedora 28 (Rawhide) builds.
So now any Node.js package that is built in Fedora will
automatically have `Provides: bundled(nodejs-foo)` added
automatically.
_______________________________________________
To unsubscribe send an email to
_______________________________________________
_______________________________________________

--
Tom Hughes (***@compton.nu)
http://compton.nu/
_______________________________________________
nodejs mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to nodejs-lea