The problems outlined in the announcement are largely related to deployment and scaling the npm client in enterprise settings. Yarn seems to have re-solved some of the problems that the npm client has already addressed. Switch npm-shrinkwrap for a lockfile for example. It seems a shame that there was no aspiration to contribute to the open source npm client but multiple clients are no bad thing.
It is indisputable that npm has been a runaway success. The platform has seen explosive growth, has scaled and has become a daily tool for software developers.
npm users downloaded 1.4 billion packages in the last 7 days, 5.2 billion in the last 28 days, and 48 billion in the last 365 days. pic.twitter.com/pXaPTB6s1p— Laurie Voss (@seldo) September 30, 2016
150 new authors are now publishing packages every day.
This is a graph of how many people publish a package to npm for the very first time, per day – 150-200 new authors per day! pic.twitter.com/MEBY1qNFTJ— Laurie Voss (@seldo) October 5, 2016
The explosive growth is not surprising given that small modules are favoured over large monolithic code bases but the growth is indisputable and impressive. The npm service has matured to become stable and able to scale.
The Node.js ecosystem has promoted writing tiny modules. Along with the upside of tiny reusable modules comes massive dependency trees. Have a look at the yarn dependency graph as an example.
I have seen the large dependency trees cause build servers to lock up or fail which is frustrating when you are trying to ship code. Furthermore package versioning can lead to brittle builds if you do not specify exact versions or use npm shrinkwrap.
With so many tiny userland modules there are issues of modules authors not following semantic versioning or just removing packages. One such example is left-pad a 52 line module for left padding a string. This module was removed from the registry by the author breaking deployment pipelines and a number of major projects that depended on it. The package was later restored and the fallout documented by npm. This incident led to many running their own npm cache to mitigate against deployments being disrupted.
The history of npm is one moving from a hobby project and one of the original Node.js package managers. Remember kiwi anyone? npm eventually became the default choice and coupled with Isaac Schlueter taking over as the benevolent dictator of the Node.js project it became official.
The npm project then took $2.6 Million in seed funding and became a for-profit organisation whilst maintaining an open source approach. This has been universally good for the Node.js ecosystem but has resulted in one for-profit organisation owning the hosting of the registry. The npm team have done a fantastic job in scaling the registry and maintaining an excellent service but I can’t help feeling that the release of yarn is an expression that this model is less than perfect.
Far better for me would be to have a decentralised registry that allows replication and mirroring around the world similar to the way major Linux distributions run their registries. This reduces reliance one on single registry provider and reduces issues of scale. Packages are verified by checksums and signing so files can be hosted and downloaded from anywhere. Far from ruining npm’s business model I feel there would still be opportunities to offer private registries and generate revenue. Perhaps we are too far down the road now though.
Whilst npm does perform sha1sum checking packages are unsigned. Often authors will declare dependencies with a greater than syntax. Using something like
~1.2.3 will install anything greater than 1.2.3 and less than 1.3.0. As such it would be quite feasible for an npm author account to be compromised and for a new package version to be released that is less than 1.3.0. This version would proliferate around project installs, build servers and production sites as
npm install is run.
rimfafall was released to highlight this issue but the conclusion was ‘You are responsible for what you require’. With dependencies trees reaching to tens of thousands of modules this simply isn’t feasible.
Whilst signing packages has been discussed it has not progressed so package installers are still vulnerable to an npm account being compromised.
A while back I was asked to perform an audit on a relatively small project for an large enterprise who were interested in understanding the Open Source licences used in a project. There are packages available to output this information. Let us suppose that an organisation requires that no GPL licenses are used in a project. Given the epic dependency trees of projects it is very hard to know this and unless you lock versions of packages you install this can change as you run
Currently yarn is just a better npm client. The registry it uses is just a proxy to the main npm registry but the fact it does not point to the npm registry directly suggests that there may be some aspiration to include reworking the registry in the project.
Have an update or suggestion for this article? You can edit it here and send me a pull request.
Using HashiCorp Vault with LDAP
How to use HashiCorp Vault to setup an LDAP backed secret store with read-only access for users in groups and read-write access for specific users
Linux and Unix xargs command tutorial with examples
Tutorial on using xargs, a UNIX and Linux command for building and executing command lines from standard input. Examples of cutting by character, byte position, cutting based on delimiter and how to modify the output delimiter.
Copy a file in Go
How to copy a file in Go. The ioutil package does not offer a shorthand way of copying a file. Instead the os package should be used.