Release the Kraken: How PayPal is being revolutionised by Node.js and Lean-UX

At the beginning of September 2013, we had the pleasure of hearing Bill Scott from PayPal talk about the revolution that is happening there through the use of Node.js and Lean-UX practices.

Bill arrived at PayPal in 2011 to a risk-averse, “not invented here” culture where long shelf life was the norm.

Things were so slow, in fact, that in 2011 even a simple content copy change could take as much as 6 weeks to change on the PayPal site.

Due to the slow release cycle times, the door had been left open for new entrants like Stripe and Square to enter the payments market.

One of the root problems was that all the technology in PayPal was tangled up and not suited for rapid experimentation and build/measure/learn.

The existing stack was Java through and through. You didn’t write HTML, CSS or JavaScript – you wrote CSS in Java, you wrote HTML in Java and you wrote JavaScript in Java.

Later the stack was changed to be more reasonable (JSP for templating). But even in that state the stack was not conducive to prototyping, and most of the UI bits lived on the server.

Under Bill Scott, some sharp folks joined from Netflix and Yahoo, and in early 2012 they began to flesh out a UI layer that could support rapid experimentation.

In April 2012, David Marcus became president of PayPal and Bill was given a small team and a project to re-invent checkout.

David is a startup person and gave them 6 weeks to launch a new checkout system (bear in mind that checkout is a system that generates $3.5 Billion revenue for PayPal).

It wasn’t possible to get Node.js production ready within 6 weeks as a large number of PayPal’s subsystems needed to be integrated into the Node.js system so Node.js was initially used as a rapid prototyping framework.

David had a number of features in mind for the initial system, and he drew some mockups.

With Node.js, Bootstrap and other open source technology, it was possible to get the flow working in 3 days. This LeanUX project was called Hermes: God of Mobile (mobility), God of Agility (lean) and God of Commerce (checkout).

The team adopted a whiteboard-to-code build-measure-learn prototyping cycle where the concept becomes a living spec, or as Bill dubbed it, ‘a vicious cycle of goodness was embarked upon’.

Every week, the team went from code to usability test and feedback, where they built, got feedback and learning and then implemented what they learned.

Bill’s team did this for 8 weeks, while the Java application team had started to spin up the application controller layer and were able to get it up and running simultaneously.

In the meantime, most of the initial experience was already written in Node.js.

The strategy was to try to use the legacy systems already in existence in PayPal – but to do this from Node.js.

These are the steps involved in implementing a game-changing program inside PayPal:

Express, Connect, Require.js, bring in JavaScript templating and other open source UI Goodness.

Used Bootstrap to enable developers to create a newly branded frontend in a few hours, enabling sketch to code.

JavaScript Templating can be run in the client browser or server on the production stack.

It was possible to take the Dust Templates and use them on top of the old Java-Stack (this meant that quick results were possible from the frontend work that was being done).

This allowed the team to drag and drop the UI bits from the prototyping stack (Node) to the production stack (Java), with Rhino (JavaScript for Java VM) enabling stack parity between the two. Project Delorean (named after the car in Back to the Future):

As PayPal still has a number of experiences running on top of an XML/C++ stack in production, they were able to insert V8 into the C++ stack.

Just like with Rhino, this allowed them to create Dust templates and run these JavaScript templates on the legacy C++ stack.

JavaScript templating gave a clean way to cover across 3 stacks at once – Node.js prototype, Java Legacy and C++ Legacy – and provided a graceful way to nibble away at the old stacks, producing results all the way along.

Building totally on open source stack enabled new hires to start and grasp what was going on within a few hours. Contrast this with the custom Java Spring framework where it took a 20-day training course just to become proficient in that environment.

A key goal was hiding all the complexity of PayPal from developers and allowing them to just get stuff done without over-complicating things, plus creating an environment where it’s possible to embrace learning and experimentation.

This makes it easy to experiment with minimal investment. In other words, make it cheap and frictionless to experiment.

In order to do this, it was necessary to enable all of the standard PayPal services – without looking like PayPal. In other words, do it in a friendly NPM way, and simplify creating an app in a few minutes with all of the PayPal services.

Bringing Node.js into PayPal removed the barriers between the layers in the software organization, Node.js and JavaScript all the way up from just above the services layer to the frontend layer, meaning you need fewer engineers.

In one project there were initially 25-30 Java engineers required to build a simple controller layer.

Later this was simplified to a smaller team. But in parallel, the Node.js team did it with just 1-2 node engineers!

PayPal got it down to the level where developers could build an initial deployable app in a few minutes, while integrated with all the PayPal services.

Bill’s team (along with other key partners in PayPal) made the environment developer-friendly internally to the point where adoption teams that had been in PayPal in the past to help with the adoption of Java Stack were no longer necessary.

The Node.js stack was user-friendly to the point where they had removed all the bespoke complexity and scariness out of their toolset.

Once Node.js made it into PayPal, they had to almost have an anti-adoption team to slow things down.

To get people up and running, there are JavaScript classes, a nice set of GitHub internal docs and Yeoman scripts to produce flavours of apps.

A simple app that connects to authentication, account services, experimentation, logging and other services can be finished in 10-15 minutes.

Once engineers have gone through this experience, they become believers and in turn evangelists for Node.

One stack to rule them; all the new applications in PayPal are now being built using Node.js. This makes the prototyping and production stacks one and the same.

A core express module (Kraken), i18n: managing content at scale, security, app lifecycle middleware, and bootstrap capability.

Headcount:  Node.js. Typically they are seeing anywhere from ⅓ to 1/10 the number of developers needed on the new stack vs the old stack. Performance:  In one set of Load & Performance tests done on the same app built on both the Java stack vs the Node stack showed a 10x throughput in scale. All in on Node.js:  all the new applications in PayPal are now being built using Node.js Lines of code:  In general they have seen code size shrink by a factor of 3-5 by moving from the Java stack to the Node stack.

Even the number of files shrunk by a similar factor.

For more information on the modules and frameworks PayPal is moving to open source check out the kraken homepage .

The framework that started it all. This is the glue that holds everything together.

Kraken-js is built on top of the tried and true Express web app framework, and further streamlines it.

It gives you a clean web app structure that will induce developers to write neat, well-organized code.

When used in combination with the other modules, it makes it easy to set up robust applications.

To make things even easier for adopters, we are also providing a Yeoman generator for Kraken apps. By simply typing `yo kraken` you will have a ready-to-run basic application.

Out-of-the-box application security.

Lusca is middleware that can be deployed over Express, and configured to plug common attack vectors.

When used, it will:

A DustJS template rendering plugin for Express.

It is based on LinkedIn’s fork of DustJS, and includes DustJS-helpers by default.

An internationalization/localization support module.

It allows for loading content bundles based on a context (eg: a user connecting from the US would receive the en_US content bundle, while a Spanish user would receive the es_ES one ).

The loaded bundles can then be used to decorate DustJS templates.

One of the main challenges faced by the enterprise is the need to deploy private modules in an efficient manner.

Existing solutions often entail replicating the public npm registry in its entirety, in order to deploy a few packages privately.

Kappa is a smart npm proxy that allows public and private npm packages to be fetched from the right place. This means that companies can have a private npm server that only holds their modules.

When a project requires either private or public modules, Kappa will transparently deliver them. In addition, it will soon support licensing checks, blacklisting, statistics, and more!

Sometimes JSON just isn’t enough for configuration needs. Occasionally it would be nice to use arbitrary types as values, but JSON is necessarily a subset of all available JS types. shortstop enables the use of protocols and handlers to enable identification and special handling of json values.

A string tokenizer, configured to recognize a particular pattern: {@[A-Za-z._] [attrs...]/}.

A content bundle transcoder. It will convert to and from different formats, including .properties, .json, .4cb, etc.

Route configuration middleware for expressjs. This module changes where/how express loads its routing, making for more organized applications.

A collection of Development-time tools for Kraken applications.

Need help developing Node.js applications for your business? NearForm are experts in Node.js development . Contact us today to find out more!

Join the Digital Advantage Insider for more great content and exclusive offers. Antonella Lombardi