joepie91's Ramblings

CloudFlare, We Have A Problem

Thu, 14 Jul 2016 00:00:00 +0200

For the past few years, CloudFlare has been steadily gaining popularity - being used by a staggering amount of websites, big and small. One of their frequently repeated claims to fame is that they "make web properties faster and safer".

I disagree.

In reality, CloudFlare has been structurally making the web less secure during these years. And they are incredibly good at selling that as a feature.

The Solution To No Problems

Back in 2011, when I ran AnonNews.org, I had to cope with frequent DDoS attacks - not all that surprising, given that it was a very popular news site and community for Anonymous, which was seeing the peak of its media coverage at the time. In 2011, however, it was pretty much impossible to get working DDoS mitigation for less than $100 a month, and that was simply not a budget I had to spend on it.

I eventually ran across CloudFlare, and - despite it not advertising DDoS mitigation anywhere at the time - I realized that with it being essentially a reverse proxy on beefy infrastructure, it would make for a useful pincushion against most DDoS attacks. And it did - it got in the way of many attacks, saved me some traffic as a bonus, and was overall a good solution to the problem at the time, even if it wasn't "real" DDoS mitigation.

Fast-forward to today, in 2016. It's not so clear anymore whether CloudFlare really solves any problems. Single-homed bandwidth can be gotten for $0.35/TB, DDoS mitigation services are plentiful and sometimes even provided by default, and the web is generally Fast Enough. Of course this doesn't stop CloudFlare from marketing to AWS customers - who are still grossly overpaying for bandwidth - or simply to those who are not aware of the changes in the hosting landscape.

Essentially, there's not really a reason to use CloudFlare anymore, and the majority of sites won't see any real benefit from it at all. I'll go into the alternatives further down the article, but I want to address some of the problems that CloudFlare introduces first.

Encryption? What's That?

A big issue with CloudFlare today is their "Universal SSL" feature. Hailed as a way to "support SSL connections to every CloudFlare customer", it actually does the very thing that SSL/TLS is meant to prevent. One instance of this occurred today, when visitors of The Pirate Bay suddenly started seeing network blocking messages from Airtel. This wasn't a compromise of CloudFlare; rather, the connection between CloudFlare and the real TPB servers wasn't encrypted, and so CloudFlare's network provider could intercept and mess with the traffic.

This is a simplified schematic illustrating what happened:

The reason this could happen is CloudFlare's "Flexible SSL" option, which is one of the modes in which "Universal SSL" can operate. In this mode, the TLS connection only runs from the end user to CloudFlare, and is plaintext beyond that point.

See the problem?

To the end user, it will look like they are using TLS, and their connection is secure right up until the site they are trying to reach - after all, that's how TLS is meant to work, right? In reality, anybody could have still messed with the traffic between CloudFlare and the "origin server". The user is presented with a false sense of security, suggesting security that simply isn't there. This breaks the TLS model, and is extremely dangerous; users will behave more carelessly because they believe they are being protected, resulting in a greater compromise.

The issue is made even worse by CloudFlare claiming that "there are no security flaws on its side". Apparently, they don't consider "complete compromise of the SSL/TLS trust model" a security flaw. Right.

But let's pretend that CloudFlare realizes that Flexible SSL was a mistake, and removes the option. They'd then require TLS between CloudFlare servers and the origin server as well. While this solves the specific problem of other ISPs meddling with the connection, it leaves a bigger problem unsolved: the fact that CloudFlare itself acts as an MITM (man-in-the-middle). By the very definition of how their system works, they must decrypt and then re-encrypt all traffic, meaning they will always be able to see all the traffic on your site, no matter what you do.

This may not sound that bad - after all, they're just a service provider, right? - but let's put this in context for a moment. Currently, CloudFlare essentially controls 11% of the 10k biggest websites, over 8% of the 100k biggest websites (source), and almost 5% of sites on the entire web (source). According to their own numbers from 2012(!), they had more traffic than several of the most popular sites and services on earth combined, and almost half the traffic of Facebook. It has only grown since. And unlike every other backbone provider and mitigation provider, they can read your traffic in plaintext, TLS or not.

Could you claim with a straight face that all this intercepted data isn't used by intelligence agencies, whether with CloudFlare's cooperation or not? It would be the perfect intelligence source, and the only way to have a guarantee that target sites will never start encrypting the data - after all, that's what they're expecting the service to do for them!

And what if somebody wanted to serve malware? What better place to do that than injected directly into potentially billions of sites, without any cross-domain restrictions whatsoever?

And all of this is completely inevitable, because CloudFlare's very business model is based on the ability to intercept and read HTTP traffic. They can't offer any of their services without it.

Which brings us to...

Packets, Please!

CloudFlare is frequently hailed as a "free DDoS mitigation provider", ever since they started marketing themselves as such. The reality is very different; you won't get any actual DDoS mitigation, even if you pay $200/month for their Business plan.

Traditional DDoS mitigation services work by analyzing the packets coming in, spotting unusual patterns, and (temporarily) blocking the origin of that traffic. They never need to know what the traffic contains, they only need to care about the patterns in which it is received. This means that you can tunnel TLS-encrypted traffic through a DDoS mitigation service just fine, without the mitigation service ever seeing the plaintext traffic... and you're still protected.

In contrast, CloudFlare is just a reverse proxy with a very fast connection. Layer 3/4 attacks (those aimed at the underlying network infrastructure, rather than the application or protocol itself) will only ever reach up to the point where it's handled by a server rather than just passed through, and in a "reverse proxy"-type setup, that server is CloudFlare. They're not actually mitigating anything, it just so happens that they are the other side of the connection and thus "take the hit"!

This is also why CloudFlare only supports HTTP(S), and not other protocols - they never actually pass through any traffic. Their servers will make a request to your site on the behalf of a visitor, and forward the response, after potentially modifying it first. They would have to write a custom reverse proxy for each protocol to support.

At this point, you might be wondering "well okay, I get that, but why should I care as long as it protects my site?", and the answer to that would be: because it doesn't. You can't protect the rest of your infrastructure (mailservers, chat servers, gameservers, and so on), and even for your web-based services, CloudFlare will kick you off the Free and Pro plans if you get attacked too much and they can figure out that you are the target.

In other words: unless you pay them $200/month, they won't provide any protection that you wouldn't already have anyway. And if you do pay them $200/month, you'll get half-functional protection for a single protocol on a single domain, with all your users being completely exposed to CloudFlare and whatever other organizations might obtain access to their traffic or servers. As you'll see below, this is a pretty shitty deal, and there are far better options today.

Oh, and about that "I'm Under Attack" mode that you get on the Free plan as well? Yeah, well, it doesn't work. But don't take my word for it - here's proof. That code will solve the 'challenge' that it presents to your browser, in a matter of milliseconds. Any attacker can trivially do this. And the challenge can't be made more difficult, because it would make it prohibitively expensive for mobile and embedded devices to use anything hosted at CloudFlare.

But while it doesn't stop attackers, it does stop legitimate users. Which brings us to...

You Shall Not Pass

See, the "I'm Under Attack" system imposes some problems. By its very definition, it requires that you have JavaScript enabled to be able to view a site - note that I'm not talking about the CAPTCHA page here, but about the "Checking your browser..." page.

It's quite frequently claimed that "oh well, everybody has JS anyway", but this is simply not true. Eevee has written an excellent article about this problem, including many examples where this assumption doesn't hold true.

But I want to address an issue that I've had specifically with CloudFlare's "I'm Under Attack" mode. I'm involved in ArchiveTeam, essentially a loose collective of archivists that try to preserve culture and knowledge on an ever-rotting web - the rotting usually being a result of service providers throwing away user data on 2 weeks notice because it's no longer profitable to them, not really caring about the consequences for the users.

One of the services that ArchiveTeam operates is ArchiveBot, essentially an IRC bot that archives whatever is thrown at it, and adds it to the Internet Archive. You can kind of think of it as an on-demand, public service Wayback machine. To be able to do this, it needs to access websites - not as a browser, but just as a plain HTTP client - and spider their content. Somewhat predictably, ArchiveBot has very limited support for JavaScript.

Indeed it is essentially impossible to archive something that's in "I'm Under Attack" mode, despite that usually being the exact moment where archival is necessary!

I've been told that ArchiveBot can be added to the internal whitelist that CloudFlare has, but this completely misses the point. Why do I or anybody else need to talk to a centralized gatekeeper to be able to access content on the web, especially if there might be any number of such gatekeepers? This kind of approach defeats the very point of the web and how it was designed!

And for a volunteer-run organization like ArchiveTeam, it's far more tricky to implement support for these "challenge schemes" than it is for a botnet operator, who stands to profit from it. That problem only becomes worse as more services start implementing these kind of schemes, and often it takes a while for people to notice that their requests are being blocked - sometimes losing important information in the process.

Some might argue that these kind of archival bots are precisely what CloudFlare is meant to protect against, but that's not really true. If that were the case, why would there be an offer to add ArchiveBot to the whitelist to begin with? Why would the Wayback Machine be on that very same whitelist?

Speaking of which, perhaps you're using CloudFlare because of their blocking of spambots. Apart from the fact that blacklists for this are freely available and don't require sending your traffic through a centralized middleman, it's also a completely misguided approach. It's based entirely on the premise of "malicious IPs", but there is no such thing.

IP addresses change hands frequently, can be shared by tens of thousands of people, and can be reassigned to a different household 10 minutes later. In reality, there are only malicious clients and malicious users, and trying to identify them by IP will lead to a lot of false positives, and not just on Tor.

The effective way to deal with malicious clients and users isn't to block "known-bad IPs" - because again, those do not exist, and there's no correlation to clients or users. It's to detect patterns of abusive behaviour, and to encourage the behaviour that you desire. Blocking IPs is akin to banning trucks from the freeway - sure, you've reduced the amount of truck-on-car collisions to zero, but was the loss of commercial transport really worth it?

As somebody who has run various high-risk services over the years, attracting a lot of targeted abuse, I can confidently say that IP blocking is never necessary and rarely effective.

But The Speed! The Speed!

CloudFlare's original mass-market selling point, performance. Route your traffic through us, and everything will be magically faster! Well, as it turns out, that's not quite true. Where to start...

In most of the Western world, connectivity is pretty good. You can go from most places in the US to Europe and back - across the ocean! - in about 140 milliseconds. A commonly used metric in the web development industry is that your page and all your assets should be loaded in under 300 milliseconds.

Assuming you're declaring all the assets on your page directly, that would make it two roundtrips totalling about 280 milliseconds, since the assets can be retrieved in parallel. Even if you have to cross the Atlantic, you're still going to clock in under the guideline, without any CDN or geolocation whatsoever.

But some people still want to squeeze out more performance - for example, they might have assets referenced a few levels deep, or they consider every millisecond critical because they are in e-commerce. Whether that's a valid concern is something I'll leave in the middle, but let's assume for now that it is. Even in this case, the problem is still static assets - CloudFlare can't cache the actual pageloads locally, because they are dynamic and different for everybody.

So why not just use a CDN? Using a CDN means you can still optimize your asset loading, but you don't have to forward all your pageloads through CloudFlare. Static assets are much less sensitive, from a privacy perspective.

But perhaps you're also targeting users in regions with historically poor connectivity, such as large parts of Asia. Well, turns out that it doesn't really work there either - CloudFlare customers routinely report performance problems in these regions that are worse than they were before they switched to CloudFlare.

This is not really surprising, given the mess of peering agreements in Asia; using CloudFlare just means you're adding an additional hop to go through, which increases the risk of ending up on a strange and slow route.

And this is the problem with CloudFlare in general - you can't usually make things faster by routing connections through somewhere, because you're adding an extra location for the traffic to travel to, before reaching the origin server. There are some cases where these kind of techniques can make a real difference, but they are so rare that it's unreasonable to build a business model on it. Yet, that's precisely what CloudFlare has done.

But perhaps you're thinking of the extra features that they offer like bundling assets, minification, cache headers, different loading orders, and so on. However, all of these are things that you can do on your own infrastructure, and without compromising the privacy of your users. Sending your traffic through a third party is completely unnecessary for that.

To top it off, for most cases none of this matters anyway - more and more organizations are (unnecessarily) turning their sites into Single Page Applications, and don't realize that this adds entire seconds of rendering time on top of your milliseconds worth of asset retrieval. Why bother with those 50 milliseconds difference, especially at such a cost?

What Of My Web?

Unfortunately, all of these issues together mean that CloudFlare is essentially breaking the open web. Extreme centralization, breaking the trust model of SSL/TLS, a misguided IP blocking strategy, requiring specific technologies like JavaScript to be able to access sites, and so on. None of this benefits anybody but CloudFlare and its partners.

For you as a CloudFlare customer, the problem is bigger - by routing your traffic through CloudFlare, you are essentially exposing every single one of your users completely, to CloudFlare and more than likely to intelligence agencies as well. Browsing behaviour, credit card details, passwords, private conversations, everything. Even if you just have a small static site like a blog, your users can be tracked as having visited it, without either you or your users having any knowledge of it whatsoever.

We shouldn't want this, especially if it's completely unnecessary.

And What Of My Provider?

One argument I saw in response to this, was that you are trusting your hosting provider anyway, and so adding CloudFlare doesn't really do any harm. This is really not true, though - not only does CloudFlare run at a far larger scale than any hosting provider, they are also in a much better position to maliciously intercept traffic (or be forced to do so). It's considerably easier to do dragnet interception on a reverse proxy, than it is to compromise every single server in a datacenter. And either way, you now have two providers you need to trust, rather than one.

Another argument is that CloudFlare just does the same thing that load balancers have been doing for over a decade, and that this isn't really anything new. But while the functionality is the same, the context is not - traditional load balancers run on the same network as the servers they are balancing between, and so the risk of interception is almost non-existent. While there are some newer providers that offer similar services to CloudFlare - and I consider them bad on exactly the same grounds - they run on a much smaller scale, and have much less impact.

Whence Shall I Source My Solutions?

Of course, it'd be a bit strange for me to claim that CloudFlare has outlived its usefulness, and then not provide any alternatives. Thankfully I've had this discussion before on Hacker News, so I have some concrete alternatives handy.

I'll reiterate them here for your convenience:

DDoS mitigation

Use a real (network-level) mitigation provider.

Some providers include mitigation for free with your hosting service (OVH, Online.net, ServerCrate, ...). Others charge a small fee, typically between $1 and $5 (RamNode, BuyVM, SecureDragon, ...).

There are also dedicated mitigation providers for more demanding usecases (Akamai, Level3, Voxility, CNServers, Sharktech, ...) and some providers that resell and/or combine these services (eg. X4B.net).

If you have your own physical infrastructure, you can also pick a mitigation appliance provider. There are quite a few.

Easy and free SSL/TLS

Let's Encrypt offers free browser-recognized SSL/TLS certificates. If you don't want the hassle of setting it up, Caddy is a web server that will automatically set up SSL/TLS using Let's Encrypt certificates out of the box, no configuration required.

Web Application Firewall

Run one on your own backend server(s) and/or loadbalancer(s). There's no benefit to doing this remotely, really. Even something relatively simple like ModSecurity will cover a wide array of problems.

Better performance

Do server-side optimizations of your code. Don't build an SPA unless you need it - it will significantly slow things down for your users.

Use a (real) CDN for your static assets, not a proxy like CloudFlare. If you want to optimize your dynamic pageloads as well, look into Anycast hosting (BuyVM, BHost, and so on.)

Saving bandwidth

Don't bother, beyond the usual performance improvements described above. Use a provider that doesn't gouge you over it - a typical cost for both VPSes and dedicated servers is $1 to $5 per TB per month, with no additional fee for the connection.

Free DNS

Many hosting providers offer this for free with your plan. If you'd rather not put all your eggs into one basket, Hurricane Electric offers free dual-stack Anycast DNS.

Other things

If there's anything you're using CloudFlare (or similar services) for that isn't listed here, then please do let me know, and I'll do my best to find you a less harmful alternative. My contact details are at the bottom of this article.

You can also leave comments in the thread on Hacker News.

Stop using JWT for sessions, part 2: Why your solution doesn't work

Sun, 19 Jun 2016 00:00:00 +0200

Almost a week ago I published an article explaining why you shouldn't use JSON Web Tokens as a session mechanism.

Unfortunately, it seems I've found the upper limit on article length before people stop reading - many of the commenters on Reddit and Hacker News kept suggesting the same "solutions" over and over again, completely ignoring that they were already addressed and found impractical in the article itself.

So, this time, I'm going to illustrate it with a slightly sarcastic flowchart.

Footnote: microservice architectures

Another argument that came up a lot, was that using JWT for sessions is still fine in a microservice architecture. This one is also wrong, but is a bit too complex to fit into a flowchart.

In a microservice architecture where the client talks directly to the services, you will have roughly two types of services:

Stateful services: Something that has a concept of a session or persistence, like a chat service.
Stateless services: Something that does not have a concept of a session, but rather performs individual self-contained tasks, like a video transcoding service.

You don't need to use JWT token as a session in either case. For a stateless service, there's no session at all, so you simply have the application server hand out short-lived, single-use tokens for each individual authorized operation.

For a stateful service, you hand out a new short-lived, single-use token for each service - which is then exchanged on the service itself, for a session on that specific service. You never use the token itself as the session.

In a microservice architecture where the client only talks to the application server, none of this as relevant, as there's no concept of a "session" between services - it's all individual, self-contained actions from the same origin(s). It's probably fine to use JWT tokens there, even if they're not optimal for this kind of case - you're just not using them as sessions.

Stop using JWT for sessions

Mon, 13 Jun 2016 00:00:00 +0200

Update - June 19, 2016: A lot of people have been suggesting the same "solutions" to the problems below, but none of them are practical. I've published a new post with a slightly sarcastic flowchart - please have a look at it before suggesting a solution.

Unfortunately, lately I've seen more and more people recommending to use JWT (JSON Web Tokens) for managing user sessions in their web applications. This is a terrible, terrible idea, and in this post, I'll explain why.

Just to prevent any confusion, I'll define a few terms first:

Stateless JWT: A JWT token that contains the session data, encoded directly into the token.
Stateful JWT: A JWT token that contains just a reference or ID for the session. The session data is stored server-side.
Session token/cookie: A standard (optionally signed) session ID, like web frameworks have been using for a long time. The session data is stored server-side.

To be clear: This article does not argue that you should never use JWT - just that it isn't suitable as a session mechanism, and that it is dangerous to use it like that. Valid usecases do exist for them, in other areas. At the end of this article, I'll briefly go into those other usecases.

A note upfront

A lot of people mistakenly try to compare "cookies vs. JWT". This comparison makes no sense at all, and it's comparing apples to oranges - cookies are a storage mechanism, whereas JWT tokens are cryptographically signed tokens.

They aren't opposites - rather, they can be used either together or independently. The correct comparisons are "sessions vs. JWT" and "cookies vs. Local Storage".

In this particular article, I will be comparing sessions to JWT tokens, and occasionally go into "cookies vs. Local Storage" as well where it makes sense to do so.

Claimed advantages of JWT

When people recommend JWT, they usually claim one or more of the following benefits:

Easier to (horizontally) scale
Easier to use
More flexible
More secure
Built-in expiration functionality
No need to ask users for 'cookie consent'
Prevents CSRF
Works better on mobile
Works for users that block cookies

I'll address each of these claims - and why they are wrong or misleading - individually. Some of the explanations below may be a little vague; that's primarily because the claims themselves are vague. I'll happily update it to address more specific claims; you can find my contact details at the bottom of this article.

Easier to (horizontally) scale

This is the only claim in the list that is technically somewhat true, but only if you are using stateless JWT tokens. The reality, however, is that almost nobody actually needs this kind of scalability - there are many easier ways to scale up, and unless you are operating at the size of Reddit, you will not need 'stateless sessions'.

Some examples of scaling stateful sessions:

Once you run multiple backend processes on a server: A Redis daemon (on that server) for session storage.
Once you run on multiple servers: A dedicated server running Redis just for session storage.
Once you run on multiple servers, in multiple clusters: Sticky sessions.

These are all scenarios that are well-supported by existing software. Your application is very unlikely to ever go beyond the second step.

Perhaps you're thinking that you should "future-proof" your application, in case you do ever scale up beyond that. In practice, however, it's fairly trivial to replace the session mechanism at a later point, with the only cost being logging out every user once, when you make the transition. It's just not worth it to implement JWT upfront, especially considering the downsides that I'll get to later.

Easier to use

They really aren't. You will have to deal with session management yourself, on both the client and the server side, whereas standard session cookies just work, out of the box. JWT isn't easier in any way.

More flexible

I have yet to see somebody actually explain how JWT is more flexible. Almost every major session implementation lets you store arbitrary data for the session anyway, and this is no different from how JWT works. As far as I can tell, this is just used as a buzzword. If you disagree, feel free to contact me with examples.

More secure

A lot of people think that JWT tokens are "more secure" because they use cryptography. While signed cookies are more secure than unsigned cookies, this is in no way unique to JWT, and good session implementations use signed cookies as well.

"It uses cryptography" doesn't magically make something more secure either; it must serve a specific purpose, and be an effective solution for that specific purpose. Incorrectly used cryptography can, in fact, make something less secure.

Another explanation of the "more secure" argument that I hear a lot, is that "they are not sent as a cookie". This makes absolutely no sense - a cookie is just a HTTP header, and there's nothing insecure about using cookies. In fact, cookies are especially well-protected against eg. malicious client-side code, something I'll get into later.

If you are concerned about somebody intercepting your session cookie, you should just be using TLS instead - any kind of session implementation will be interceptable if you don't use TLS, including JWT.

Built-in expiration functionality

This is nonsense, and not a useful feature. Expiration can be implemented server-side just as well, and many implementations do. Server-side expiration is preferable, in fact - it allows your application to clean up session data that it doesn't need anymore, something you can't do if you use stateful JWT tokens and rely on their expiration mechanism.

No need to ask users for 'cookie consent'

Completely wrong. There's no such thing as a "cookie law" - the various laws concerning cookies actually cover any kind of persistent identifier that isn't strictly necessary for the functioning of the service. Any session mechanism you can think of will be covered by this.

In a nutshell:

If you are using a session or token for functional purposes (eg. keeping a user logged in), then you don't need to ask for user consent, regardless of how you store that session.
If you are using a session or token for other purposes (eg. analytics or tracking), then you do need to ask for user consent, regardless of how you store that session.

Prevents CSRF

It doesn't, really. There are roughly two ways to store a JWT:

In a cookie: Now you are still vulnerable to CSRF attacks, and still need protection against it.
Elsewhere, eg. Local Storage: Now you are not vulnerable to CSRF attacks, but your application or site now requires JavaScript to work, and you've just made yourself vulnerable to an entirely different, potentially worse class of vulnerabilities. More about this below.

The only correct CSRF mitigation is a CSRF token. The session mechanism is not relevant here.

Works better on mobile

Nonsense. Every mobile browser still in use supports cookies, and thus sessions. The same goes for every major mobile development framework, and any serious HTTP library. This is just not a problem at all.

Works for users that block cookies

Unlikely. Users don't just block cookies, they typically block all means of persistence. That includes Local Storage, and any other storage mechanism that would allow you to persist a session (with or without using JWT). Whether you use JWT simply doesn't matter here, it's an entirely separate problem - and trying to get authentication to work without cookies is a bit of a lost cause.

On top of that, users that block all cookies typically understand that this will break authentication functionality for them, and individually unblock cookies for sites where they care about this. It's simply not a problem that you, as a web developer, need to be solving; a much better solution is to explain to your users why your site requires cookies to work.

The drawbacks

Now that I've covered all the common claims and why they're wrong, you might think "oh, that's not a big deal, it still doesn't matter that I use JWT even if it doesn't help me", and you'd be wrong. There are quite a few downsides to using JWT as a session mechanism, several of them being serious security issues.

They take up more space

JWT tokens are not exactly small. Especially when using stateless JWT tokens, where all the data is encoded directly into the token, you will quickly exceed the size limit of a cookie or URL. You might decide to store them in Local Storage instead - however...

They are less secure

When storing your JWT in a cookie, it's no different from any other session identifier. But when you're storing your JWT elsewhere, you are now vulnerable to a new class of attacks, described in this article (specifically, the "Storing sessions" section):

We pick up where we left off: back at local storage, an awesome HTML5 addition that adds a key/value store to browsers and cookies. So should we store JWTs in local storage? It might make sense given the size that these tokens can reach. Cookies typically top out somewhere around 4k of storage. For a large-sized token, a cookie might be out of the question and local storage would be the obvious solution. However, local storage doesn’t provide any of the same security mechanisms that cookies do.

Local storage, unlike cookies, doesn’t send the contents of your data store with every single request. The only way to retrieve data out of local storage is by using JavaScript, which means any attacker supplied JavaScript that passes the Content Security Policy can access and exfiltrate it. Not only that, but JavaScript also doesn’t care or track whether or not the data is sent over HTTPS. As far as JavaScript is concerned, it’s just data and the browser will operate on it like it would any other data.

After all the trouble those engineers went through to make sure nobody is going to make off with our cookie jar, here we are trying to ignore all the fancy tricks they’ve given us. That seems a little backwards to me.

Simply put, using cookies is not optional, regardless of whether you use JWT or not.

You cannot invalidate individual JWT tokens

And there are more security problems. Unlike sessions - which can be invalidated by the server whenever it feels like it - individual stateless JWT tokens cannot be invalidated. By design, they will be valid until they expire, no matter what happens. This means that you cannot, for example, invalidate the session of an attacker after detecting a compromise. You also cannot invalidate old sessions when a user changes their password.

You are essentially powerless, and cannot 'kill' a session without building complex (and stateful!) infrastructure to explicitly detect and reject them, defeating the entire point of using stateless JWT tokens to begin with.

Data goes stale

Somewhat related to this issue, and yet another potential security issue. Like in a cache, the data in a stateless token will eventually 'go stale', and no longer reflect the latest version of the data in your database.

This can mean that a token contains some outdated information like an old website URL that somebody changed in their profile - but more seriously, it can also mean somebody has a token with a role of admin, even though you've just revoked their admin role. Because you can't invalidate tokens either, there's no way for you to remove their administrator access, short of shutting down the entire system.

Implementations are less battle-tested or non-existent

You might think that all these issues are just with stateless JWT tokens, and you'd be mostly right. However, using a stateful token is basically equivalent to a regular session cookie... but without the battle-tested implementations.

Existing session implementations (eg. express-session for Express) have been running in production for many, many years, and their security has been improved a lot because of that. You don't get those benefits when using JWT tokens as makeshift session cookies - you will either have to roll your own implementation (and most likely introduce vulnerabilities in the process), or use a third-party implementation that hasn't seen much real-world use.

Conclusion

Stateless JWT tokens cannot be invalidated or updated, and will introduce either size issues or security issues depending on where you store them. Stateful JWT tokens are functionally the same as session cookies, but without the battle-tested and well-reviewed implementations or client support.

Unless you work on a Reddit-scale application, there's no reason to be using JWT tokens as a session mechanism. Just use sessions.

So... what is JWT good for, then?

At the start of this article, I said that there are good usecases for JWT, but that they're just not suitable as a session mechanism. This still holds true; the usecases where JWT is particularly effective are typically usecases where they are used as a single-use authorization token.

From the JSON Web Token specification:

JSON Web Token (JWT) is a compact, URL-safe means of representing claims to be transferred between two parties. [...] enabling the claims to be digitally signed or integrity protected with a Message Authentication Code (MAC) and/or encrypted.

In this context, "claim" can be something like a 'command', a one-time authorization, or basically any other scenario that you can word as:

Hello Server B, Server A told me that I could <claim goes here>, and here's the (cryptographic) proof.

For example, you might run a file-hosting service where the user has to authenticate to download their files, but the files themselves are served by a separate, stateless "download server". In this case, you might want to have your application server (Server A) issue single-use "download tokens", that the client can then use to download the file from a download server (Server B).

When using JWT in this manner, there are a few specific properties:

The tokens are short-lived. They only need to be valid for a few minutes, to allow a client to initiate the download.
The token is only expected to be used once. The application server would issue a new token for every download, so any one token is just used to request a file once, and then thrown away. There's no persistent state, at all.
The application server still uses sessions. It's just the download server that uses tokens to authorize individual downloads, because it doesn't need persistent state.

As you can see here, it's completely reasonable to combine sessions and JWT tokens - they each have their own purpose, and sometimes you need both. Just don't use JWT for persistent, long-lived data.

What is Promise.try, and why does it matter?

Wed, 11 May 2016 00:00:00 +0200

A topic that frequently confuses users in the #Node.js channel, is the Promise.try method provided by Bluebird. People often struggle to understand what it is, or why they should use it - and this isn't helped by the fact that almost all guides to Promises fail to demonstrate its use.

In this brief article, I hope to provide a better explanation of what Promise.try is, and why you should always use it, without exceptions. I'll be assuming that you already have some knowledge of Promises, and specifically about the role of .then.

Even if you are using a different Promises implementation (such as ES6 Promises), this article will still be useful to you - at the end of this article, I will explain how to achieve the same functionality without Bluebird.

So, what is it?

Simply put, Promise.try is like .then, without requiring a previous Promise. Now that's still a little vague, so let's start out with an example.

This is what some typical code using Promises might look like:

function getUsername(userId) {
    return database.users.get({id: userId})
        .then(function(user) {
            return user.name;
        });
}

So far, so good. We'll assume that database.users.get will return a Promise of some sort, and that that Promise will eventually resolve to an object with a name property.

Now, this is the same code, but using Promise.try:

var Promise = require("bluebird");

function getUsername(userId) {
    return Promise.try(function() {
        return database.users.get({id: userID});
    }).then(function(user) {
        return user.name;
    });
}

As you can see, we've added Promise.try at the very start of the chain. Rather than chaining directly off database.users.get, we chain off Promise.try and simply return the result of database.users.get, like we would normally do from a .then.

So... what is all this good for?

The above may look like needless extra code. But in practice, there are several advantages:

Better error handling; synchronous errors are propagated as rejections, no matter where in the process they occur.
Better interoperability; no matter what Promises implementation the third-party method uses, you will always be working with your preferred implementation.
Easier to 'scan'; all code is horizontally at the same indentation level, so it's easier to see what's going on at a glance.

Below, I'll address each of these points individually.

1. Better error handling

One of the touted benefits of Promises is that we can treat synchronous and asynchronous errors in the same way - synchronous errors are simply caught, and 'repropagated' as a rejected Promise. But is this really the case? Let's have a look at a slightly modified version of our first example:

function getUsername(userId) {
    return database.users.get({id: userId})
        .then(function(user) {
            return uesr.name;
        });
}

In this modified version, we've mistyped user.name, so that it now says uesr.name. This would normally fail, as uesr is undefined, and so can't have any properties. Indeed, this will be caught within the .then and turn into a rejected Promise, as we expect.

But what if database.users.get throws synchronously? What if there's a typo or other error in the third-party database code, which we didn't write? The error-catching feature of Promises works by virtue of all your synchronous code being in a .then, so that it can wrap it in a giant try/catch block.

But... our database.users.get isn't in a .then block! The Promises implementation thus doesn't have access to that chunk of code, and can't wrap it. Our synchronous error will remain a synchronous error, and now we're back where we started - having to deal with two kinds of errors, synchronous and asynchronous.

Now, let's look back at our example using Promise.try again:

var Promise = require("bluebird");

function getUsername(userId) {
    return Promise.try(function() {
        return database.users.get({id: userID});
    }).then(function(user) {
        return user.name;
    });
}

I said before that Promise.try is like a .then, but without requiring a previous Promise. That holds true here, as well - it will catch the synchronous error in database.users.get, just like a .then would have!

By using Promise.try, we've streamlined our error handling to cover all of the synchronous errors, not just those after the first asynchronous operation (as that is where our first real .then callback would otherwise be).

Interjection: Promises/A+

Before we move on to the next point, let's cover what Promises/A+ is, and what role it plays in the ecosystem. The Promises/A+ site summarizes it as follows:

An open standard for sound, interoperable JavaScript promises—by implementers, for implementers.

Put differently, it's a way to ensure that different Promises implementations (Bluebird, ES6, Q, RSVP, ...) work together flawlessly, out of the box. This Promises/A+ specification is why you can use pretty much any Promises implementation you prefer, and why you don't have to care about which implementation a third-party library (like, for example, Knex) uses.

To illustrate what Promises/A+ does for you as a user:

All the functions highlighted in red return Bluebird Promises, the one in blue returns an ES6 Promise, and the one in green returns a Q Promise.

Note how even though we are returning an ES6 Promise from the first callback, the .then that wraps it will still return a Bluebird Promise. Similarly, even though the second callback returns a Q Promise, our doStuff method will still return a Bluebird Promise.

This happens because, aside from catching synchronous errors, .then will also wrap the return values and ensure that they end up being a Promise from the same implementation that the .then itself comes from. In practice, this means that the first Promise in the chain determines what implementation you will be working with.

This is a practical way to ensure that you are always working with a predictable API. The only Promises implementation you have to care about, is whatever the first function in the chain uses - no matter what comes afterwards.

2. Better interoperability

The above isn't always desirable, however - for example, look at this example:

var Promise = require("bluebird");

function getAllUsernames() {
    //     ⬐ This will return an ES6 Promise.
    return database.users.getAll()
        .map(function(user) {
            return user.name;
        });
}

If you are not familiar with map and you'd like to know more, you can read this article.

The .map feature we are using in this particular example is a Bluebird feature, and is not available on ES6 Promises. We can't use it here, even if we are using Bluebird in our project otherwise - simply because the first function in the chain (database.users.getAll) returned an ES6 Promise rather than a Bluebird Promise.

Now, let's look at the same example again, but this time using Promise.try:

var Promise = require("bluebird");

function getAllUsernames() {
    //     ⬐ This will return a Bluebird Promise.
    return Promise.try(function() {
        //     ⬐ This will return an ES6 Promise.
        return database.users.getAll();
    }).map(function(user) {
        return user.name;
    });
}

Now we can use .map! Because we are starting with Promise.try, which originates from Bluebird, all our subsequent Promises will also be Bluebird Promises, no matter what happens in the .then callbacks.

By using Promise.try like this, you can decide for yourself what implementation you want to be working with, by ensuring that the first Promise in the chain comes from your preferred implementation. You can't do this with .then, because that'd require you to chain off some other Promise - which would defeat the point.

3. Easier to scan

The final benefit is one of readability. By starting every chain with Promise.try, all the actual 'business logic' exists within a callback at the same (horizontal) level of indentation. While this might seem like a minor benefit, it actually makes a fairly significant difference, due to how humans scan large amounts of text.

To illustrate the difference, this is how you would visually scan the code without Promise.try, using a common indentation style:

... and the same code, but using Promise.try:

Even though the code looks slightly 'noisier', it's still easier to get a quick impression of the code, simply because your eyes have to 'seek' less.

What if I'm not using Bluebird?

To my knowledge, Bluebird is currently the only Promises implementation that ships with Promise.try out of the box. However, it's relatively easy to replicate the functionality in other implementations, as long as their version of new Promise catches synchronous errors.

For example, es6-promise-try is an implementation that I've written for ES6 Promises, and it works in the browser too. I have not gotten around to writing its documentation yet, but it works in essentially the same way as Promise.try - however, instead you call promiseTry like below:

var promiseTry = require("es6-promise-try");

function getUsername(userId) {
    return promiseTry(function() {
        return database.users.get({id: userID});
    }).then(function(user) {
        return user.name;
    });
}

Keep in mind that es6-promise-try will assume that your environment supports ES6 Promises - if that is not the case, you'll want to use something like es6-promise as a polyfill.

I'd seriously recommend looking at Bluebird instead, though, as it has much more robust error handling and debugging features. Usually, ES6 Promises are only the ideal option in constrained environments like old versions of Internet Explorer, or when trying to the reduce bundle size of frontend code.

If you know of Promise.try implementations for other Promises implementation, then please let me know, and I'll add them here! My contact details are at the bottom of this page.

I am currently offering Node.js code review and tutoring services, at affordable rates. More details can be found here.

Reflections on NPM-gate, one day later

Wed, 23 Mar 2016 00:00:00 +0100

Okay, so let's get these things out of the way first:

I am not a lawyer. However, I have to deal with intellectual property on a regular basis, because of my open-source work. I'm also very strongly opposed to copyright and patents, but have a more nuanced stance on trademarks. If any of this makes you uneasy, you can stop reading here.
This issue was about trademarks. Trademarks are designed to, in summary, "prevent consumers from getting confused or misled as to the origin or endorsement of a thing". This means that trademarks are not a form of copyright, they don't mean that you "have copyright on a name', and most importantly, it doesn't mean that you "own" the name. I will get back to this later.
This article is based on public sources, and it's possible that some things are inaccurate, if (for example) any of the parties withheld communications or details.

So, for the past day, I've been following the fallout around and reasons for Azer's decision to remove all of his modules from NPM. I've noticed that many people discussing the matter either don't understand the issue, don't understand trademarks, or underestimate the complexity of solving this problem.

Therefore, I've decided to write this post - both as a summary of the events, and as a correction of some particularly common misconceptions.

What happened?

Kik (the company) contacted Azer, asking 'politely' whether they could have the kik package name on NPM. Azer responded:

Sorry, I’m building an open source project with that name.

After that, the following message was sent by Kik (the company):

We don’t mean to be a dick about it, but it’s a registered Trademark in most countries around the world and if you actually release an open source project called kik, our trademark lawyers are going to be banging on your door and taking down your accounts and stuff like that — and we’d have no choice but to do all that because you have to enforce trademarks or you lose them.

Can we not come to some sort of a compromise to get you to change the name without involving lawyers? Is there something we could do for you in compensation to get you to change the name?

At this point, Azer responded with:

hahah, you’re actually being a dick. so, fuck you. don’t e-mail me back.

Kik (the company) contacted NPM Inc., asking them to help out, again alluding to trademarks and lawyers. At this point, NPM Inc. offered to take away the kik package name from Azer and give it to Kik (the company).

Azer felt that NPM no longer represented his interests as a user (and open-source developer), decided he did not want to be a part of NPM anymore, and 'unpublished' all of his modules from NPM.

Consequently, builds all around the world broke. One of the removed modules was the left-pad module, which was used in many, many popular projects, either directly or indirectly.

Azer's response

It's easy to blame Azer for 'being a dick' here. However, Azer initially responded politely, and only became rude after Kik threatened with lawyers, despite having been told no.

If your immediate response to a "no" is to threaten with lawyers, then that was your plan all along, and you were simply asking 'politely' first for appearances, and to save on legal costs. There was clearly no intention to let Azer make a decision in the first place.

Kik's claim

"But," you might say, "they hold the trademark - Azer didn't have a choice anyway, and had to comply". This argument in particular is one I've seen a lot, but it's most likely false.

Holding a trademark doesn't mean you own the name, nor does it work like copyright. It simply means that if somebody could be confused or misled about the origins of something, you can exercise your trademark to put it to a stop. It's a way to protect a brand name, and prevent scams.

That wasn't the case here at all, however - as far as I can tell, Azer's package was completely unrelated to Kik (the service), and simply happened to have the same three-letter name. There would be no room for confusion here at all, and that would make this a case of trademark trolling; where Kik (the company) just figured that trademark threats would be the easiest way to get hold of a nice and short package name.

It's very, very unlikely that this would have been a legitimate trademark infringement claim. Indeed, no lawyers were ever involved, and it was never more than a threat.

I also don't believe Kik's claim that it was "nothing more than a polite request" - if that were the case, they wouldn't have threatened with lawyers three(!) times.

And finally, I'd like to mention Startup Timelines, who removed Kik from their site for this very reason.

NPM Inc.'s response

NPM Inc. removed Azer's package without any real infringement claim ever having been sent, purely on the basis of the threat of getting lawyers involved.

By doing this, NPM Inc. has painted a giant target on their back towards trademark trolls, saying "here's a cheap target" - it is now known that they will not make an effort to represent the interests of their users, and will fold even without a legal notice being sent.

This is disastrous for the ecosystem, as every developer is now a potential target, and their package might be pulled out from under their feet as well, for an entirely frivolous claim.

Reclaimed packages

Then the next disaster struck, once people realized that not only could Kik (the company) push whatever code they wanted as a patch version to existing users of the kik library... but anybody could register any of the other now-removed NPM packages, and do the same thing.

This is a security issue so significant, that I can't believe it even happened. Had a malware author scooped up left-pad, for example, they could have infected potentially thousands to millions of users with a single publish. In fact, that still might happen - because who is nj48 anyway?

This really cannot ever, ever, ever be allowed. Global namespace or not, once an identifier has been used and removed, it should not ever be possible to reassign it to anything else.

But Azer could have published malware as well.

Yes, he could have. But he was the only person who could modify his packages (aside from NPM Inc. itself), and people decided to trust him as a legitimate developer. That trust breaks down when you quietly 'reallocate' a package to a different, entirely unrelated person.

Signed packages would be even better, of course - but even without that, preventing reallocation of package names is the most basic protection you should really have.

NPM Inc.'s history

Unfortunately, this wasn't the first incident with NPM Inc. - in the past, NPM employees have advocated for blocking IPs from the registry, when the users they belong to 'misbehave' (in their view) outside of the NPM registry. It's completely unacceptable to have a small group of people decide who can use critical infrastructure and who can't, based on personal dislike of off-site behaviour.

Returning to security. While I won't go into this too much, I've had poor experiences with NPM staff's handling of security in the past, as well - specifically, with Isaac, who apparently did not understand the severity of the Buffer security issue in Node, yet felt that he could proclaim it on Twitter as "no big deal". I was blocked after showing him a proof-of-concept of the vulnerability. I have heard similar stories from others.

Overall, I feel that both Kik and NPM have handled this issue very poorly, and that there is a history of doing so for NPM. Which brings us to the next point...

Fixing NPM

There have been many ideas, from many people, in many places, on how to 'fix' NPM. From replacing the client, to replacing the company, to replacing the registry with a decentralized equivalent.

Unfortunately, not all of these ideas are thought through or argued well. I'll address some of the common issues below.

Namespacing

A common complaint is that there's a single global namespace in NPM - that is, a single collection of packages, with a first-come-first-serve policy. While a valid concern in and of itself, many people seem to think that this is the reason of package takeovers that occurred today. It's not.

The problem with the package takeovers is one of mutability and, more specifically, who has permission to do what. Right now, removed packages can apparently be freely re-registered, but this should not be the case, for the reasons I've described before. Imagine for a moment that we had a namespaced NPM, where each package was prefixed with the username of the publisher. Imagine that once somebody removed their account, their username could be re-registered.

What would happen? Exactly the same thing.

Namespaces do not solve this, and if you can prevent reuse of usernames, then you can also do so for package names. While namespaces are worth discussing, it's completely unrelated to the issues that occurred today.

Decentralizing NPM

There have been various suggested solutions for decentralizing NPM. This is a very good idea in principle, as it would remove single points of failure, like what we're seeing now with NPM Inc. Unfortunately, most of these proposals won't actually work, because the author underestimated the complexities involved.

For example, you might argue for content-addressable packages on IPFS. But how do you handle semantic versioning? You definitely should support it like NPM does, as it's essential to receiving (security) patches. But purely content-addressable packages rule out this possibility by definition, as they require immutability. And there are many subtle issues like this.

There has been a discussion ongoing about this for a few months now, and it can be found here. If you wish to contribute to building a decentralized package manager, then please read that thread and those linked from it, because they list many of the issues that a naive decentralized implementation would run into. You can't replace NPM with a package manager that is missing features.

In conclusion...

I'm surprised you've made it this far through the article. Either way, the conclusion is quite simple:

We need to replace the central NPM registry, but we need to do so carefully. Also, Kik needs to stop pretending that it was a polite request.

To end on a more amusing note, from the Node.js IRC channel:

[20:34] <TheEmpath> Hey guys, I've created a package that converts characters into yellow/gold N'ko, a modern unifier of the Manding languages spoken by mansas throughout medieval North Africa. You can try it out with npm install goldmansachs

And yep, it actually exists.

Why you should never, ever, ever use MongoDB

Sun, 19 Jul 2015 00:00:00 +0200

MongoDB is evil. It...

... loses data (sources: 1, 2)
... in fact, for a long time, ignored errors by default and assumed every single write succeeded no matter what (which on 32-bits systems led to losing all data silently after some 3GB, due to MongoDB limitations)
... is slow, even at its advertised usecases, and claims to the contrary are completely lacking evidence (sources: 3, 4)
... forces the poor habit of implicit schemas in nearly all usecases (sources: 4)
... has locking issues (sources: 4)
... has an atrociously poor response time to security issues - it took them two years to patch an insecure default configuration that would expose all of your data to anybody who asked, without authentication (sources: 5)
... is not ACID-compliant (sources: 6)
... is a nightmare to scale and maintain
... isn't even exclusive in its offering of JSON-based storage; PostgreSQL does it too, and other (better) document stores like CouchDB have been around for a long time (sources: 7, 8)

... so realistically, there's nothing it's good at, and a bunch of stuff it's outright bad at. And these bulletpoints are facts, not 'just your opinion'. You can go out and verify them yourself.

For most cases, what you want is actually a relational database. PostgreSQL is a good option for these cases, and you can use a query builder or ORM to make working with it easier. In Node.js, some options are Knex (as a query builder), Bookshelf, Sequelize, or Waterline (as ORMs).

If your project involves user accounts, or any kind of relationship between two records, then you should use a relational database, and not a document store - after all, your data is relational.

If you find yourself using Mongoose, you should also be using a relational database. Libraries like Mongoose just try to (poorly) emulate schemaful relational databases using a document store, so you might as well just use a relational database directly!

Even if what you need is a document store (and again, in most cases it isn't), there are many better options available than MongoDB. Here is a list of document store databases. Note that that list is ranked by popularity, and says nothing about the quality of a database.

Never use a database because 'everybody else does', do your own research as to the advantages and drawbacks of a particular database. Popularity is largely subject to hype, and with the right marketing team, it's not hard to popularize an inferior solution.

And no, it isn't "good for prototyping" either - you just end up locking yourself into a database that can never reasonably make it into production. If your prototype is considered viable, then you're still going to have to rewrite everything using a different database.

Nearly every development ecosystem has migrations with a rollback mechanism by now, which provide a comparable ease of use for prototyping, and without the drawback of having to rewrite your code for production.

There are no valid usecases for MongoDB. It is technically inferior to other options, does not offer exclusive features that actually work, and the developers are failing to ensure data integrity and security - arguably two of the most important aspects of a database.

Using Promises (eg. bluebird) with Express

Thu, 14 May 2015 00:00:00 +0200

When using a promises library like bluebird in Express, you might have found it somewhat awkward to use them correctly - it's easy to forget to tack on a .catch statement at the end of a chain of promises.

Fortunately, there's a solution for that, and it's called express-promise-router!

Using `express-promise-router`

express-promise-router is a slightly modified version of the regular router that ships with Express, that expects a route to return a promise object. It then automatically attaches an error handler to the promise, that will forward any errors to your regular error handling middleware, so that you can handle them as if you called next(err).

With the regular Express router, your code might look something like this:

var express = require("express");
var router = express.Router();

router.get("/", function(req, res, next){
    Promise.try(function(){
        return doSomeAsyncThing();
    }).then(function(value){
        return doAnotherAsyncThing(value);
    }).then(function(newValue){
        res.send("Done!");
    }).catch(function(err){
        next(err);
    });
});

module.exports = router;

And this is what it looks like with express-promise-router:

var expressPromiseRouter = require("express-promise-router");
var router = expressPromiseRouter();

router.get("/", function(req, res){
    return Promise.try(function(){
        return doSomeAsyncThing();
    }).then(function(value){
        return doAnotherAsyncThing(value);
    }).then(function(newValue){
        res.send("Done!");
    });
});

module.exports = router;

Note the following:

We no longer need to define a .catch handler.
We now return the chain of promises from the route.

That's it! All errors in your chain of promises now end up in your regular error handling middleware, where you can log them and send an error page.

Does this support the same features as a regular Express router?

Yep. The express-promise-router module really just wraps the native router, and adds error handling to the route methods. That's all it does.

Can I also return promises from `.param()` calls?

Yep, since 0.0.7. They're handled in the same way as the regular route methods. If an error occurs in the .param() call, the request will be aborted, and the route after it won't be run - in other words, like it should be :)

And how about `.use()` and `.all()`?

Yes, same as for .param().

What if I don't return a chain of promises from my route? Will it break?

Nope. The express-promise-router module only implements special behaviour when a chain of promises is returned. Otherwise, it has the default behaviour of the Express router.

Can I still use `next()` manually?

Yes.

My rejections are ending up on my console anyway. What gives?

Make sure you're actually returning the chain of promises from your route or middleware. It can't handle promises that you don't give to it!

How do I send different kinds of errors? I want to send different HTTP status codes.

Ideally, you'd use a module like errors for this. You can either use the standard HTTP error methods that it provides, or even create your own errors - and then handle all of them in your error handling middleware, like normal.

You can just do something along these lines:

app.use(function(err, req, res, next)
    if(err.status != null) {
        var statusCode = err.status;
    } else {
        var statusCode = 500;
    }

    res.status(statusCode).send(err.message);
}

My next Node.js article will be an in-depth guide to promises, and how they can help you escape from callback hell. I know I promised that last time, but this time it's really going to be the next post :)

I am currently offering Node.js code review and tutoring services, at affordable rates. More details can be found here.

Functional programming in Javascript: map, filter and reduce

Mon, 04 May 2015 00:00:00 +0200

This article is meant to be an introduction to functional programming in Javascript - specifically, it will explain the map, filter and reduce methods.

While these are natively available in any recent browser and in Node.js, most articles on them are far too technical to understand, while the concept of these functions is actually really simple, and will benefit any developer - even those working on simple scripts.

Note that I'll be pretty verbose in this article, to make the concept understandable to developers of any skill level. If you're already familiar with some of the concepts described, you can safely skip over those bits. Similarly, if you're a more advanced developer, and examples alone are enough for you, just skip over the text entirely.

I won't address other aspects of functional programming here - these three functions fit well into most workflows on their own. I might explain other concepts in a later post.

So let's start with map, arguably the most common one in a typical project.

So, what is this `map` business, then?

Think of map as a "for each" loop, that is specifically for transforming values - one input value corresponds to one 'transformed' output value.

You may have found yourself writing code that looks something like this:

var numbers = [1, 2, 3, 4];
var newNumbers = [];

for(var i = 0; i < numbers.length; i++) {
    newNumbers[i] = numbers[i] * 2;
}

console.log("The doubled numbers are", newNumbers); // [2, 4, 6, 8]

While this code works, it's not very nice. It takes a lot of work to do a pretty simple task - double all the numbers in an array.

What if we could simply write our intention of doubling every number in an array?

var numbers = [1, 2, 3, 4];

var newNumbers = numbers.map(function(number){
    return number * 2;
});

console.log("The doubled numbers are", newNumbers); // [2, 4, 6, 8]

Suddenly, we don't need a loop anymore, and we don't have to manually add numbers to an array either! We can simply define our intention, and let map do the work. This can also be chained easily:

var numbers = [1, 2, 3, 4];

var newNumbers = numbers.map(function(number){
    return number * 2;
}).map(function(number){
    return number + 1;
});

console.log("The doubled and incremented numbers are", newNumbers); // [3, 5, 7, 9]

Every number is now doubled, and has 1 added to it. We don't have to manually manage any arrays - we simply specify functions that do some kind of transformation on a single value.

There are a few 'rules' for map functions, though. While these are not always strictly enforced, you should keep to them regardless - they make it a lot easier to understand what your code is doing. If you think these rules will get in your way, just keep reading - your questions will be answered.

The amount of input elements is equal to the amount of output elements

You can't give map 4 values, and only receive 3 back. If the 'source array' has an X amount of elements, then the resulting aray will also have X elements. Every output element corresponds to the input element in the same position - they're never shuffled around.

Your callbacks shouldn't 'mutate' values

What this means, is really just that you shouldn't modify objects or arrays directly from within your callbacks - if the input value is an object or an array, clone it instead, and modify the copy.

This way, there's a guarantee that your callback doesn't cause 'side effects' - that is, no matter what happens in your callback, it will only affect the specific value you're working with. This makes it much easier to write reliable code.

You can clone an array in Javascript by doing array.slice(0). Note that this is a 'shallow clone' - if the values in the array are themselves arrays or objects, they will still be the same values in both arrays.

Shallow-cloning an object is a little more complex. If you're using a CommonJS environment (Node.js, Webpack, Browserify, ...), you can simply use the xtend module. Otherwise, using a function like this should suffice:

function cloneObject(obj) {
    var newObj = {};

    for (var key in obj) {
        newObj[key] = obj[key];
    }

    return newObj;
}

var clonedObject = cloneObject(originalObject);

There is an exception to this rule, but we'll get to that later - just assume that this rule always applies, unless told otherwise.

Don't cause side-effects!

You should never do anything in a map call that modifies 'state' elsewhere. For example, while making a HTTP GET request is fine (although not really possible with plain Array.map), changing another array outside of the callback is not. Your callback can only modify the new value you're returning from that callback.

But... isn't this slow? All those callbacks and the cloning...

Don't worry about it. Such operations are actually rather fast in Javascript, especially in V8-based engines like Node.js, and it's extremely unlikely that it'll ever cause performance issues for you.

Always write code for readability first, and for performance second - it's much easier to optimize readable code, than it is to make optimized code readable.

But what if I only want to transform some of the values?

Perhaps your source array has some values that you want to transform, and some values that you just want to throw away entirely. That's not possible with map alone, as the number of input values and the number of output values for a map call is always equal.

Say you want to double the odd numbers, but throw away the even numbers. Your code may look something like this:

var numbers = [1, 2, 3, 4];
var newNumbers = [];

for(var i = 0; i < numbers.length; i++) {
    if(numbers[i] % 2 !== 0) {
        newNumbers[i] = numbers[i] * 2;
    }
}

console.log("The doubled numbers are", newNumbers); // [2, 6]

This is what it'd look like using map and filter:

var numbers = [1, 2, 3, 4];

var newNumbers = numbers.filter(function(number){
    return (number % 2 !== 0);
}).map(function(number){
    return number * 2;
});

console.log("The doubled numbers are", newNumbers); // [2, 6]

The filter callback is subject to the same "don't mutate" and "don't cause side-effects" rules as map, but the mechanics are different.

The return value from the filter callback should be a boolean, indicating whether to include the original value in the result (true) or whether to leave it out (false). You should not return the value itself, just a boolean - you can't modify the value from within the callback. The return value just decides whether the value will be included in the result, nothing else.

Chaining

As you might have noticed, you can chain map and filter calls indefinitely. Each of them just returns an array, and every array has these methods by default, so you can build a jQuery-like chain. There is no limit to how many of these calls you can chain, and it's usually pretty simple to build complex array transformations from just these functions alone.

An array goes in, an array comes out, and the callback operates on each value individually.

So... what if I just want to combine the values in an array?

'Combining' the values can mean a lot of different things. For example, you might want to sum all the numbers, doubled:

var numbers = [1, 2, 3, 4];
var totalNumber = 0;

for(var i = 0; i < numbers.length; i++) {
    totalNumber += numbers[i] * 2;
}

console.log("The total number is", totalNumber); // 20

But what if we want to do that in a map/filter chain? Easy enough, with reduce:

var numbers = [1, 2, 3, 4];

var totalNumber = numbers.map(function(number){
    return number * 2;
}).reduce(function(total, number){
    return total + number;
}, 0);

console.log("The total number is", totalNumber); // 20

It works like this:

The second argument to the function call is considered to be the 'starting value' for the total - this is what you start out with.
For each item in the array, it calls the callback, with the total value up to that point, and the item itself. For the first item, the 'total value' is the starting value.
You return a new 'total value'. In this case, it's the sum of all previous numbers plus the current number. This return value is used as the 'total value' for the next item.
After running out of items, the 'cumulative' total value is returned.

Again, the "don't mutate" and "don't cause side-effects" rules apply. Otherwise, it doesn't matter what kind of value you're working with - you could be summing a number, concatenating a string, putting together an object, and so on.

Just keep in mind that reduce always returns just the 'total value' - unless you've specifically made it an array, it won't be an array like for map and filter. That means that unless you're explicitly returning an array from it, you can't chain off it any further. But then again, it wouldn't really make sense to run map on a number!

But what if I want to add items? Like `filter`, but the opposite.

If you use Node.js, you may be familiar with transform streams - these are kind of like a map callback, but besides letting you 'push' (return) 0 or 1 values, they also let you push multiple values. This can be useful if you are, for example, 'flattening' an array of arrays down to a single array with all the values.

While this can't be done with map - after all, one input element means one output element - it can be done with reduce, despite what the name implies. It's a slightly unusual trick, but you'll occasionally find yourself needing it.

Let's say you want to add the even numbers to the resulting array twice, but the odd numbers only once. This is what it could look like:

var numbers = [1, 2, 3, 4];
var newNumbers = [];

for(var i = 0; i < numbers.length; i++) {
    var number = numbers[i];
    newNumbers.push(number);

    if(number % 2 == 0) {
        /* Add it a second time. */
        newNumbers.push(number);
    }
}

console.log("The final numbers are", newNumbers); // [1, 2, 2, 3, 4, 4]

This is how you'd do it with reduce:

var numbers = [1, 2, 3, 4];

var newNumbers = numbers.reduce(function(newArray, number){
    newArray.push(number);

    if(number % 2 == 0) {
        /* Add it a second time. */
        newArray.push(number);
    }

    return newArray; /* This is important! */
}, []);

console.log("The final numbers are", newNumbers); // [1, 2, 2, 3, 4, 4]

Note how we start out with [] as initial value, and just add the value to it one or more times. While this does violate the "no mutation" rule, it's okay in this particular situation - the array object is created explicitly for this reduce call, so it can't cause side-effects anywhere else in the code.

When you use this trick, don't forget to return the array! It's easy to forget this, and you'll end up with an unexpected output. The reduce callback always expects the new total value to be returned, even if that value is an array, and even if it's the same one as before!

The same trick also works with objects - however, in that case you'd specify {} as starting value, and you'd use property assignment rather than .push.

Bonus: forEach

Perhaps you really do want a 'for each' loop - you're not transforming and returning any values, you just want to cause some kind of side-effect (eg. creating a DOM element). It would be nice if you could chain it at the end of a map/reduce/filter 'pipeline', and indeed you can!

This is what your for loop might normally look like:

var numbers = [1, 2, 3, 4];

for(var i = 0; i < numbers.length; i++) {
    doSomethingWith(numbers[i]);
}

And this is what it would look like using forEach:

var numbers = [1, 2, 3, 4];

numbers.forEach(function(number){
    doSomethingWith(number);
});

You can do the same for an object, using Object.keys:

var numbers = {one: 1, two: 2, three: 3, four: 4};

Object.keys(numbers).forEach(function(key){
    var value = numbers[key];
    doSomethingWith(value);
    /* For example, key == "one" and value == 1 */
});

You should really always consider using forEach instead of a for loop . Because forEach uses callbacks, it prevents scope problems with asynchronous operations in a loop - that is, you don't have to worry that the value of the key variable changes inbetween the start and the completion of your operations.

Indexes

At some point, you may want to get the index of the current item in the source array - however, I've only demonstrated the use of the value so far.

Getting the index of the current element is trivial: it's the second argument for map, filter and forEach, and the third argument for reduce. In other words, it comes after the arguments I've shown in the examples so far.

If you need a reference to the original array for some reason, that is the argument after that - however, needing this usually means that you're doing something wrong. You should never directly modify the source array from any of these functions, for example.

Further reference

MDN has further documentation on map, filter, reduce, and forEach.

Other libraries

Some utility libraries like lodash and underscore offer similar methods, but you almost certainly don't need them. All modern Javascript runtimes and browsers have these functions available natively on array objects.

jQuery offers similar methods, but keep in mind that the jQuery equivalents switch around the value and the index - that is, the order of arguments for the callback is (index, value). A more reliable way is to use this within your callback where possible - it is always set to the value in jQuery.

My next Javascript and Node.js article will be about how Promises can help you escape callback hell. I will be discussing the bluebird library specifically, including its equivalents of map, filter, reduce and forEach (named each in bluebird).

I am currently offering Node.js code review and tutoring services, at affordable rates. More details can be found here.

On Mozilla's forced SSL

Fri, 01 May 2015 00:00:00 +0200

Yesterday, Mozilla made a blogpost, stating that they will be "deprecating non-secure HTTP" - in other words, forcing HTTPS. I believe that this decision is harmful to the open web, and this is why.

First of all, for those who are not familiar with me - I actively encourage people to use SSL/TLS wherever possible. I do not believe that there is data that is "not important enough to encrypt".

I do however believe that there are fundamental problems with the way TLS is currently deployed in practice, problems that absolutely need solving before a forced global deployment of TLS can happen.

TLS is broken

Yes, it's broken. Any certificate authority in the world can sign a certificate for any hostname in the world, regardless of whether that's the actual CA being used by the real owner of that hostname. This means that anybody with access to the signing infrastructure of any certificate authority anywhere in the world, can sign fake certificates for everybody else.

While TLS can still provide data confidentiality against passive attackers, and this problem can be circumvented in custom applications and API libraries by hardcoding a certificate or CA into the client, it is very much broken in the common "TLS on the web" implementation.

Certificate pinning is suggested as a 'solution', but still requires to trust the certificate on the first connection, and it doesn't provide an easy migration path for service operators who want to switch their certificates to a different certificate authority. It's also still not actually implemented by major browsers.

SSL certificates are a racket

It's really that simple. You are essentially forced to pay for a certificate if you want to use TLS. If you do not, and try to use a self-signed certificate, a browser will show a scary warning page to your users. You can't even embed (iframe) pages using self-signed certificates at all. The browser will simply refuse to load the page, not even showing a warning.

Domain-validated certificates have a cost of practically zero to generate. It's all automated. Higher-validation certificates have a non-zero, but still trivial cost to generate - the cost is purely in the validation work, which is generally a repetitive and largely automatable job.

Higher-validation certificates generally do come with insurance, but there is no way to 'opt out' of this if you don't need it - as is the case for the majority of non-corporate uses of an SSL certificate.

In short; the true cost of an SSL certificate is generally several magnitudes lower than what you're being charged for it. And everyone charges like that. It's a racket.

Startcom is not actually free

"But Startcom gives away free certificates!", you say. Well, no, not really.

The domain-validated certificates that Startcom provides are indeed free to generate. Revocation, however, even in the situation of a global security issue that is completely out of your control, will cost you money.

Revocation is a vital part of a secure TLS cryptosystem. If you don't pay Startcom, you are unable to revoke, and this breaks the security model of TLS. Your security is broken.

Startcom is not really free.

Let's Encrypt is not a complete solution either

"But," you say, "Let's Encrypt is just around the corner!"

It is. It will also - at least, initially - only offer basic domain-validated, multi-host certificates. It is apparently undecided as of yet whether wildcard domains will be supported. Other scenarios may also not be supported.

"But if you need a wildcard certificate, surely you have the money to pay for it?" Well, no, not necessarily. Right now, I'm building a free CDN for non-commercial projects. This would ideally let me provide a separate subdomain for every user - something that isn't feasible without a wildcard certificate, if TLS were to be enforced. I don't have the money for such a certificate - keep in mind that this is all non-commercial - and it would be ludicrous to expect otherwise.

"But you can just generate a new certificate for every subdomain!" Yes, you could. In complex setups, however, that quickly becomes a bookkeeping nightmare. You now have to add infrastructure for the specific purpose of propagating TLS certificates across multiple frontend servers. Not to mention that if Let's Encrypt goes down at any point, your entire registration system is down.

And on top of that, Let's Encrypt is only one certificate authority. If they are ever removed from browser trust stores - don't forget that it is still very experimental - you have exactly zero alternatives, other than to suddenly pay for some other CA.

"But you can still use HTTP!"

Yes, but you will not be able to use many new features. A quote from the original article:

For the first of these steps, the community will need to agree on a date, and a definition for what features are considered “new”. For example, one definition of “new” could be “features that cannot be polyfilled”. That would allow things like CSS and other rendering features to still be used by insecure websites, since the page can draw effects on its own (e.g., using <canvas>). But it would still restrict qualitatively new features, such as access to new hardware capabilities.

Those features are not disabled for 'security reasons', as some people in the comments on the Mozilla post have suggested. They are disabled to pressure you into using TLS.

"But you won't need those features!" Maybe you do, maybe you don't. Isn't the point of an 'open web' that the same features are available to everybody, regardless of financial means or other qualifications? Hell, I personally use many newer features to build non-commercial projects.

Conclusion

In conclusion; no, TLS certificates are not really free. Introducing forced TLS would create an imbalance between those who have the money and means to purchase a certificate (or potentially many certificates), and those who don't - all the while promoting a cryptosystem as being 'secure' when there are known problems with it. This is directly counter to an open web.

There are plenty of problems with TLS that need to be fixed before pressuring people to use it. Let's start with that first.

The EU may have just banned the anonymous sale of online goods and services

Tue, 13 Jan 2015 00:00:00 +0100

Well, not exactly a cheerful topic for my first post of 2015, but it has to be said.

The EU has introduced new VAT (Value Added Tax) legislation that is supposedly meant to prevent companies like Amazon from 'reducing their tax burden'.

Roughly summarized, the change of legislation means that VAT is now calculated based on the country of residence of the buyer, rather than that of the seller. Which sounds like a great idea, until you look at the way it's implemented.

I've just finished reading the official proposal for these changes, and as far as I can tell - the proposal does not actually define 'electronic service' anywhere like it claimed it would - these changes basically mean that it is no longer legally possible to, as a company, sell (online) goods or services to an anonymous customer.

Again, due to the lack of definition for 'electronic service' it is unclear whether this also entails physical goods that are bought over the internet.

The proposal lists nine options for establishing the country of residence, of which a minimum of two are required. I have quoted the list below, highlighting the options that are relevant to online sales.

(a) customer details such as the billing address of the customer;

(b) the Internet Protocol (IP) address of the device used by the customer or any method of geolocation;

(c) bank details such as the place where the bank account used for payment is and the billing address of the customer held by that bank;

(d) the Mobile Country Code (MCC) of the International Mobile Subscriber Identity (IMSI) stored on the Subscriber Identity Module (SIM) card used by the customer;

(e) the location of the residential fixed land line through which the service is supplied to the customer;

(f) in relation to a customer who is selling goods via the Internet or similar electronic network, the place where the transport or dispatch of the goods sold by that customer initially begins;

(g) in relation to a customer who is buying goods via the Internet or similar electronic network, the place where the transport or dispatch of the goods bought by that customer finally ends;

(h) registration details of the means of transport hired by the customer, if registration of that means of transport is required at the place where it is used, and other similar information;

(i) other commercially relevant information obtained by the supplier.

As you can see, practically every relevant item in the list would expose the identity of the customer, and businesses are required to collect this information "for VAT purposes".

The only thing that could potentially be spoofed is the IP address, and even there it is not unreasonable to assume that the tax office will complain about business who do not block known proxies or Tor relays.

While other forms of proof may also be admissible, it is very likely that they will be held to a similar standard of identity verification.

According to a well-known Dutch (IT) lawyer, Arnoud Engelfriet, this information is supposed to be kept for 10 years and accessible to any foreign tax office. While the proposal does list individual privacy as a consideration, this is not expanded upon in the rest of the proposal.

The consequences will be especially dire for businesses (and their customers) that use micropayments or cryptocurrencies like Bitcoin. Effectively, the only way to still serve your anonymous customers, is through tax evasion.

While I'm not a lawyer, I have read through the proposal a few times now, and I really see no other interpretation of it than the above. Whether this side-effect of banning anonymous sales was an oversight or intentional... I'll leave that as an exercise to the reader.

joepie91's Ramblings

CloudFlare, We Have A Problem

The Solution To No Problems

Encryption? What's That?

Packets, Please!

You Shall Not Pass

But The Speed! The Speed!

What Of My Web?

And What Of My Provider?

Whence Shall I Source My Solutions?

DDoS mitigation

Easy and free SSL/TLS

Web Application Firewall

Better performance

Saving bandwidth

Free DNS

Other things

Stop using JWT for sessions, part 2: Why your solution doesn't work

Footnote: microservice architectures

Stop using JWT for sessions

A note upfront

Claimed advantages of JWT

Easier to (horizontally) scale

Easier to use

More flexible

More secure

Built-in expiration functionality

No need to ask users for 'cookie consent'

Prevents CSRF

Works better on mobile

Works for users that block cookies

The drawbacks

They take up more space

They are less secure

You cannot invalidate individual JWT tokens

Data goes stale

Implementations are less battle-tested or non-existent

Conclusion

So... what is JWT good for, then?

What is Promise.try, and why does it matter?

So, what is it?

So... what is all this good for?

1. Better error handling

Interjection: Promises/A+

2. Better interoperability

3. Easier to scan

What if I'm not using Bluebird?

Reflections on NPM-gate, one day later

What happened?

Azer's response

Kik's claim

NPM Inc.'s response

Reclaimed packages

But Azer could have published malware as well.

NPM Inc.'s history

Fixing NPM

Namespacing

Decentralizing NPM

In conclusion...

Why you should never, ever, ever use MongoDB

Using Promises (eg. bluebird) with Express

Using express-promise-router

Does this support the same features as a regular Express router?

Can I also return promises from .param() calls?

And how about .use() and .all()?

What if I don't return a chain of promises from my route? Will it break?

Can I still use next() manually?

My rejections are ending up on my console anyway. What gives?

How do I send different kinds of errors? I want to send different HTTP status codes.

Functional programming in Javascript: map, filter and reduce

So, what is this map business, then?

The amount of input elements is equal to the amount of output elements

Your callbacks shouldn't 'mutate' values

Don't cause side-effects!

But... isn't this slow? All those callbacks and the cloning...

But what if I only want to transform some of the values?

Chaining

So... what if I just want to combine the values in an array?

But what if I want to add items? Like filter, but the opposite.

Bonus: forEach

Using `express-promise-router`

Can I also return promises from `.param()` calls?

And how about `.use()` and `.all()`?

Can I still use `next()` manually?

So, what is this `map` business, then?

But what if I want to add items? Like `filter`, but the opposite.