Skip to Content

Benchmark: Apollo Federation Gateway v1 vs v2 vs WunderGraph vs mercurius-js

Published: November 23, 2021
Jens Neuse

Jens Neuse

Apollo just released their newest version of the Apollo Federation Gateway (v2), so I was curious how it performs against v1 and the WunderGraph implementation.

Both Apollo Gateway v1 and v2 are implemented using NodeJS, the WunderGraph Gateway is written in Go.

So far, WunderGraph is the only Federation implementation aside from Apollo. It was brought to my attention that there's also mercurius-js that implements the Federation specification. It's built on top of Fastify (NodeJS). Thanks to @matteocollina for pointing out! If you know any other implementation, please let me know!

TLDR#

WunderGraph achieves up to 271x (132x) more requests per second vs. Apollo Gateway v1 (v2), 99th percentile latency is 292x (54x) lower. Apollo Gateway v2 achieves 2x more rps than v1, 99th percentile latency is 5.6x slower than v1. While Apollo Gateway v1 had problems with timeout errors, v2 solved this problem.

Apollo Federation with Subscriptions#

In contrast to Apollo Gateway, WunderGraph supports Subscriptions. This is possible because Go has green threads (goroutines) which enable services to scale easily across all cores of a computer. Each Subscription can run in its own goroutine, which only takes up a few kilobytes of memory in stack size, so scaling this solution is quite efficient.

That said, getting the architecture right for federated subscriptions is a complex problem. Most if not all GraphQL server implementations "interpret" GraphQL Operations at runtime. This means, they parse the Operation into an AST at runtime and work through this AST to resolve the Operations.

WunderGraph takes a different approach. We've built a Query Compiler that divides resolving a GraphQL Operation into multiple stages. On a high level, we differentiate between the planning, and the execution phase. During planning, we evaluate the AST and build an optimized execution Plan, hence the "Query Compiler". This execution Plan can be cached, which makes this approach very efficient. But efficinecy is not everything. More importantly, this approach allows us to solve complex problems like resolving federated GraphQL Operations with a multi step compiler, combined with an optimized execution engine.

Btw. this Query Compiler and Execution engine is open source under a MIT license. It's used by more and more companies in production. We're very proud that the developers of Khan Academy joined the ranks of maintainers recently.

One last word on open source, graphql-go-tools, the library we're building WunderGraph upon, has some amazing contributors. Amongst them is Vasyl Domanchuk from the Ukraine, he contributed the DataLoader implementation that plays an important role in making the engine so fast. This implementation solves the N+1 problem when resolving nested federated GraphQL Operations.

Thank you Vasyl, your work is highly appreciated!

Benchmarking methodology#

I've setup the basic Federation demo, more info at the end of the post. For benchmarking, I've used the cli "hey" with a concurrency of 50 over 10 seconds.

Results - Apollo Federation Gateway vs WunderGraph#

Requests per second (small Query)#

Gatewayrps
Apollo Gateway v1195
Apollo Gateway v2400
WunderGraph53000
mercurius-js1883

Requests per second (large Query)#

Gatewayrps
Apollo Gateway v1186
Apollo Gateway v2293
WunderGraph43834
mercurius-js208

Latency (small Query)#

Gatewayavg99th percentile95th percentile
Apollo Gateway v18110897
Apollo Gateway v297.1194127
WunderGraph0.94.41.9
mercurius-js264034

Latency (large Query)#

Gatewayavg99th percentile95th percentile
Apollo Gateway v1115287114
Apollo Gateway v21701609614
WunderGraph0.94.22.1
mercurius-js238344269

Observations#

Apollo Gateway v1 always has timeout errors under high load. The newer version (v2) fixed this problem. However, v2 seems to be not yet mature as requests per second ranged from 10 to 400 in some test runs.

I've also found that Apollo now configures their gateway to use Apollo Studio by default. As an alternative, they provide you a code snipped to use curl. Additionally, there's a link to the docs to enable the Playground again, running on your local machine:

import { ApolloServer } from "apollo-server";
import {
ApolloServerPluginLandingPageGraphQLPlayground
} from "apollo-server-core";
const server = new ApolloServer({
typeDefs,
resolvers,
plugins: [
ApolloServerPluginLandingPageGraphQLPlayground(),
],
});

mercurius-js is written in NodeJS, similarly to Apollos gateway. For the server, it's using the Fastify framework, which is visible from the results. On small payloads, it comes out on top of Apollo by almost 5x in terms of rps. It only seems that it struggles with the large Query. This could be either processing more data in general or due to the higher amount of network requests, the gateway has to make. Something must be going that here, which makes mercurius fall behind Apollo on the large Query.

Conclusion#

NodeJS is still not comparable in terms of performance vs. Golang. While the new version of the Apollo Gateway doesn't throw timeout errors anymore, it's visible that it doesn't scale well when GraphQL Operations become deeply nested.

Comparing the latencies of Apollo v2 for the small and large payload, it's observable that the numbers skyrocket when Operations become more nested.

WunderGraph on the other hand is not yet saturated with the workload. we could probably increase the nesting further until it has to give up.

If you want a fast Federation compatible Gateway solution, WunderGraph can save you a lot of money for hosting while increasing the security of your API.

What makes the difference?#

It's mainly two things. For one, WunderGraph is written in Go, a language that is much more capable when it comes to concurrent workloads like implementing an HTTP server. The second aspect is the architecture of WunderGraph. Instead of "interpreting" Operations, WunderGraph works with a Query Compiler that prepares the execution of an Operation at deployment time, removing all the complexity of working with the GraphQL AST at runtime.

If you want to learn more on this topic, have a look at the overview on the Query Compiler.

Demo Setup#

WunderGraph: https://github.com/wundergraph/wundergraph-demo

Apollo: https://github.com/StevenACoffman/federation-demo

In both cases, I was using the upstreams implemented using gqlgen to eliminate perfomance issues on the upstreams.

If you want to reproduce the results, just clone the repos and use hey or similar tools to benchmark yourself.

Test Query Small#

query {
topProducts {
upc
name
price
reviews {
id
body
author {
id
name
username
}
}
}
}

Test Query Large#

query {
topProducts {
upc
name
price
reviews {
id
body
author {
id
name
username
reviews {
id
body
author {
id
name
username
reviews {
id
body
product {
inStock
name
price
shippingEstimate
upc
weight
reviews {
body
id
author {
id
name
reviews {
body
author {
id
name
username
}
}
}
}
}
}
}
}
}
}
}
}
About the Author
Jens Neuse

Jens Neuse

Jens has experience in building native apps for iOS and Android, built hybrid apps with Xamarin, React Native and Flutter, worked on backends using PHP, Java and Go. He's been in roles ranging from development to architecture and led smaller and larger engineering teams.

Throughout his whole career he realized that working with APIs is way too complicated, repetitive and needs a lot more standardization and automation. That's why he started WunderGraph, to make usage of APIs and collaboration through APIs easier.

He believes that businesses of the future will be built on top of collaborative systems that are connected through APIs. Making usage, exploration, sharing and collaboration with and through APIs easier is key to achieve this goal.

Follow and connect with Jens to exchange ideas or simply participate in his feed of thoughts.

Comments

Product

Comparisons

Subscribe to our newsletter!

Stay informed when great things happen! Get the latest news about APIs, GraphQL and more straight into your mailbox.

© 2021 WunderGraph