Comparing npm audit with Snyk

At NearForm, we specialize in building practical software solutions for our clients and part of designing and building a modern solution is making it secure.

In today’s world where almost everything is connected and operated by computers, adding security on-top, as an afterthought no longer works. As software architects and engineers, we also focus on securing our software development process every step of the way. When adding layers of security as far “left” as possible the cost of vulnerabilities is dramatically reduced. NearForm has a unique connection to the JavaScript Community, being a significant Open Source code contributor, financial sponsor and advocate of the full stack JavaScript application stack.

One of the important aspects that affect the security of modern solutions built on almost any platform (including Node.js and/or React.JS), also affects pretty much every other platform (including C#, Java, Python, Ruby, PHP and others). Developers no longer write software from scratch. While 20 years ago a typical program would primarily use libraries provided by the development tools vendors, modern solutions are usually formed using dozens, or more typically, hundreds of open source packages. This powerful practice enables developers to construct complex solutions a lot quicker than ever before. Yet, whether the development team is using Node.js with npm package manager, or Ruby with its Gems, the end result is conceptually the same. The final product is a bundle containing the code developed in-house, as well as code with potential security and stability issues or even malicious code specifically developed to create attack points within IT solutions.

Given this reality, there is ongoing work to address the security issues of this new development paradigm. We will start by examining software package scanners - tools that go inside modular solutions, helping to identify and fix the problems.

We will compare the security scanner provided by npm;  npm audit  and  Snyk , a more established player in the security arena.

We want our security scanner to report, and if possible, automatically fix any discovered vulnerabilities. In order to compare npm audit and Snyk, let's start by looking into the terminology both products employ.

By default, Snyk does not test ‘development dependencies’ unless you ask it to, with the –dev flag in the CLI or by enabling from the Web UI. These are the dependencies that will not be part of your application in production, only needed for things like building and packaging your application. The npm audit tool, on the other hand, checks for development dependencies by default, so if you want to only see your production vulnerabilities, you would need to disable them with “–only=prod”.

Another subtlety in language is the terms ‘vulnerabilities’ and ‘vulnerable paths’. In npm, every high-level dependency has its own dependency tree, thus if you have two top-level dependencies A and B, that both rely on dependency C, you will have two copies of dependency C (each copy potentially with a different library version). Now consider dependency C having a vulnerability. In this case Snyk will say you have “1 vulnerability” and “2 vulnerable paths” since this vulnerability can be exploited once through package A and once through B. npm on the other hand just says you have 2 vulnerabilities. For example, in the ‘goof’ application the ‘qs’ vulnerability appears twice: once through body-parser, and once through the use of express.

Snyk reports there are 36 issues and 82 vulnerable paths for this project, overall.

Npm reports that there are 53 vulnerabilities (but that number needs to be compared to the ‘82’ from the Snyk results above).

You can add a little grepping magic to the output of npm to find the number of unique vulnerabilities:

The npm package manager hosts one of the largest software packages collections in the world. It has been around since 2009. Recently npm added a special tool for auditing package security and stability - npm-audit. The audit tool assesses package dependencies for vulnerabilities helping to protect package users by finding and fixing known problems in dependencies that could cause data loss, service outages, unauthorized access to sensitive information, or other issues. The audit tool could be used to fix the problems when invoked as:

The report is based purely on the Node security advisories. The npm audit will analyze your entire dependency tree, but it will not do code analysis or what is under the hood in the node_modules directory. The main idea behind npm audit as of now is to make developers more aware of the potential problems with dependencies, and use that information to embed package security into the development process from the moment the package is being installed. Being a part of the npm package manager, npm-audit is free, and available with the recent npm builds. The audit module is working by scanning package.json and package-lock.json files to create the full list of dependencies, then comparing the packages from the dependencies-tree to list of known vulnerabilities. If an advisory is found affecting the package used by the project, an alert with detailed information is shown. The contents of locally installed packages are not actually examined.

The npm-audit component is a new addition to the NPM package manager. It follows a fresh acquisition of Node Security Platform - the entity providing the Node.JS security advisories. These steps show that security is becoming one of the focus areas for NPM.

In this post, we will also take a look at snyk.io. Snyk is a dependency analysis platform for multiple development stacks covering JavaScript, Java, .Net, Ruby, Python, PHP, Golang and Scala. Snyk maintains a comprehensive, Open Source vulnerability database which includes vulnerabilities found by Snyk’s own dedicated research team (see the recent widespread critical “Zip Slip” disclosure, impacting hundreds of projects like Apache Hadoop, Apache Hive and Pivotal Spring), vulnerabilities surfaced from community sources (like Github and bulletin boards) as well as vulnerabilities tracked from public data sources (such as NVD). Their database is hand-curated with rich content, code snippets and useful links to get more information.

Update: Snyk has just released container image scanning to its cli and web application. In this post, we are only looking at the application code scanning since npm does not support container image scanning. We will follow up this post when we have thoroughly tested the Snyk container scanning and compared it to other options.

A nifty feature of Snyk is their ‘precision patches’ which allows fixing of vulnerabilities which can’t be upgraded out of. One of the main advantages of Snyk, is their tight integration with modern git-based source control (like github, gitlab and bitbucket). With these integrations Snyk can seamlessly scan every commit and pull requests, efficiently flagging pull requests that introduce new vulnerabilities. This is part of the “shift left” movement of security, to give developers security feedback earlier in the lifecycle, increasing developer engagement in security. A “Fix Pull Request” can be initiated to automate fixes via upgrades and precision patches. Snyk is free for open source projects and has a free plan with limited usage for private code. Paid plans include unlimited scanning, as well as enterprise needed features such as SSO, reporting, user management, support for on-prem source code (like GitHub Enterprise), JIRA integration, Docker image scanning, etc. On top of the rich source control integration and CLI, Snyk provides APIs and access control mechanisms, which could be used to implement custom automation and integration scenarios.

Another important enterprise-grade feature of Snyk is the ability to audit software licenses. For example, if developers are releasing proprietary software without publishing sources, they cannot use GPL-compatible licenses. Snyk automates license compliance based on customer specified rules, and can produce an open source license usage report to satisfy your legal team’s compliance needs.

Snyk sees package vulnerability detection as a continuous process. Whenever a new issue is discovered, or remediation for a known issue becomes available, Snyk will notify the affected customers via email or slack or via a pull request sent directly to the affected repo. The latter has the added value, that if a fix is available for the vulnerability it will be included in the pull request itself!

With support for multiple development stacks, enterprise features, free licenses as well as paid professional support, Snyk seems like a great tool for software developers working on many types of projects.

One of the features present in Snyk, as well as several other commercial products, is the software-license auditing. This is an important feature, presently missing from npm audit. Some software licenses are incompatible. Without the license auditing functionality, DevSecOps teams will have to take manual steps in understanding the legal implications of having to accept each included agreement. Snyk audits all relevant licenses then summarizes the information in an easy-to-understand, user-friendly report.

The npm audit is a great free tool for package vulnerability awareness. It comes integrated with the main Node.JS package manager. Still, Snyk offers substantially more functionality with a more friendly user interface and its liberal licensing, it deserves consideration for DevSecOps arsenal, by enterprise IT teams that require a more complete security solution.