<article><p>When I joined WunderGraph in March, my first full feature was to improve access control in Cosmo, WunderGraph’s platform for managing federated GraphQL APIs. I didn’t expect the scope to cover so much: frontend, backend, CLI. But the range of changes made it a good opportunity to improve how access control works across the platform. It's one thing to build something, but another to help define how teams manage who can do what.</p><p>Groups introduce a new system for managing <a href="https://cosmo-docs.wundergraph.com/studio/rbac">role-based access control (RBAC)</a> in Cosmo. Built to give you more control, especially when your organization gets bigger.</p><h2><strong>Where the Idea Came From</strong></h2><p>This began because customers requested more control over their members. Some needed to restrict publishing access to specific environments, like limiting production deployments to only lead developers. Others wanted to prevent users from even seeing certain namespaces or subgraphs unless they were explicitly granted access to them.</p><p>At the same time, there was inconsistency between user roles and API key permissions. API keys were managed in a separate interface and always had admin access by default. That made it harder to limit what automation or external tools could do. It also meant there was no way to apply the same policies to both users and keys.</p><p>Fine-grained GraphQL access control was technically possible before, but it required workarounds and manual setups that didn't scale well. We needed a centralized and flexible model that could define access rules once and apply them consistently across all systems.</p><p>That’s what led us to Groups.</p><h2><strong>What Groups Let You Do</strong></h2><p>Groups make it possible to assign roles and resources in one place. Both members and API keys can be added to a group. And then, the group defines what they can access.</p><p>You can:</p><ul><li>Assign access to specific namespaces, federated graphs, or subgraphs</li><li>Use different roles for different resources</li><li>Apply the same permissions to both users and API keys</li></ul><p>If someone does not have access to a resource, they won’t even see it in the UI. Namespaces, graphs, and subgraphs that aren’t assigned to them won’t show up in the sidebar or any of their views. In the CLI, restricted actions, such as publishing or introspecting, will return errors if the group doesn’t allow them.</p><h2><strong>How Cosmo Groups Work</strong></h2><p>You create a Group. Then you add rules to it. Each rule links a role to one or more resources.</p><p>Important behaviors:</p><ul><li>If a group has no rules, its members and keys have no access.</li><li>If a rule does not specify any resources, it gives access to all.</li><li>If a resource is deleted, the rule falls back to all resources.</li><li>Groups are synchronized through SSO (via OIDC mappers).</li></ul><p>Cosmo currently provides the following roles, grouped by category:</p><h3><strong>Organization:</strong></h3><ul><li><strong>Admin</strong>: Provides read and write access to all the organization resources, billing, and settings.</li><li><strong>Developer</strong>: Provides read and write access to all organizational resources and read-only access to organizational settings.</li><li><strong>API Key Manager</strong>: Provides access to managing the organization's API keys.</li><li><strong>Viewer</strong>: Provides read-only access to all the organization resources and settings.</li></ul><hr><h3><strong>Namespace:</strong></h3><ul><li><strong>Admin</strong>: Provides write access to the namespace configuration.<ul><li>When resources are not assigned to this role, members and API keys will be able to create new namespaces</li></ul></li><li><strong>Viewer</strong>: Provides read-only access to the namespace configuration.<ul><li>When resources are not assigned to this role, members and API keys will be able to create new namespaces.</li></ul></li></ul><hr><h3><strong>Graph:</strong></h3><ul><li><strong>Admin</strong>: Provides write access to the graph, like configuring check overrides, creating cache warmer operations and so on.<ul><li>When resources are not assigned to this role, the members and API key will be able to create new graphs.</li><li>If the role is assigned to a namespace instead of a graph, the members and API keys will be able to create new graphs under that namespace.</li></ul></li><li><strong>Viewer</strong>: Provides read-only access to graph configurations.<ul><li>When resources are not assigned to this role, the members and API key will be able to create new graphs.</li><li>If the role is assigned to a namespace instead of a graph, the members and API keys will be able to create new graphs under that namespace.</li></ul></li></ul><hr><h3><strong>Subgraph:</strong></h3><ul><li><strong>Admin</strong>: Provides write access to subgraphs.<ul><li>When resources are not assigned to this role, the members and API key will be able to create and publish any subgraph.</li><li>If the role is assigned to a namespace instead of a subgraph, the members and API keys will be able to create and publish to any subgraph under that namespace.</li></ul></li><li><strong>Publisher</strong>: Provides write access to subgraphs, but, unlike Admin, members and API keys will not be able to create new subgraphs.<ul><li>When resources are not assigned to this role, the members and API key will be able to publish any subgraph.</li><li>If the role is assigned to a namespace instead of a subgraph, the members and API keys will be able to publish to any subgraph under that namespace.</li></ul></li><li><strong>Viewer</strong>: Provides read-only access to subgraphs.</li></ul><p>Each role type can only be added once per group. For example, you can assign the Admin role to multiple resources, but you can’t assign Admin more than once within the same group.</p><p>In <a href="https://cosmo-docs.wundergraph.com/studio/intro">Cosmo Studio</a>, admins use the resource selector UI to assign roles to specific scopes and resources, like namespaces or subgraphs. For example, you can select only the development namespace or a single subgraph within a federated graph. The selector adjusts based on what you pick. If you choose a namespace, it disables individual graph selection. This helps avoid mistakes.</p><p>In the CLI, API keys inherit the roles of their group. This means that if a key is added to a group that can publish to a subgraph, that key can be deployed from CI/CD. But if it doesn’t have access, the operation fails.</p><h3>Rather than just describing it, let me show you how it works:</h3><h2><strong>What We Had to Build</strong></h2><p>We initially went with a hierarchical approach, but it ended up being confusing. If someone had access to a namespace, they also had access to all its graphs. This was not what we wanted. So we removed the hierarchy and made everything explicit.</p><p>We iterated over the resource selector a few times. We wanted it to be simple yet also capable of handling complex cases. For example, we had to make sure the selector could handle selecting all graphs in a namespace or just one. We also had to update the sidebar and features to hide content that users don’t have permission to access.</p><p>Most of the work was in the backend. But the changes are visible across the entire system. The permission checks are enforced everywhere, including introspection APIs.</p><p>We deprecated the separate subgraph member and API key selectors in favor of managing everything through a unified permissions system.</p><h2><strong>Why Groups Make Permissions Easier</strong></h2><p>Now you can:</p><ul><li>Assign production access only to specific users or teams.</li><li>Create groups with scoped roles for CI/CD keys, internal devs, or external collaborators.</li><li>Ensure users only see and interact with resources they’ve been granted access to.</li><li><a href="https://cosmo-docs.wundergraph.com/studio/api-keys/api-key-resources">API Keys</a> with limited resources will continue to work, but we recommend migrating them to groups.</li></ul><p>If you were already using the old system, existing users and keys were automatically placed into groups that reflect their previous permissions. The legacy roles still work, but using Groups makes management and audits much simpler.</p><p>WunderGraph Hub is our new collaborative platform for designing, evolving, and shipping APIs together. It’s a design-first workspace that brings schema design, mocks, and workflows into one place.</p><h2><strong>My Experience</strong></h2><p>This was my first full feature at WunderGraph. Since it involved access control, there was extra attention on getting the security right. Once we had aligned on the requirements and structure, implementation moved smoothly.</p><p>Groups are now available in Cosmo. This new RBAC model gives you more flexibility and security across your federated GraphQL setup. You can manage everything in one place: who can see what, who can publish, and who can create.</p><p><a href="https://cosmo-docs.wundergraph.com/studio/groups">Read the full Cosmo Groups documentation</a></p><p>Let us know how it works for your team. I’m especially interested to hear from users with complex environments.</p><p>Feel free to open a <a href="https://github.com/wundergraph/cosmo/discussions">discussion</a> on GitHub or reach out if your team has unique access needs.</p><hr><hr></article>

managing-permissions-groups-banner.png

Manage access with precision in Cosmo. Groups let you control who can view, publish, or deploy, across users and API keys, in one centralized system.

Managing Permissions in Cosmo Just Got Easier with Groups

<article><p>When I started the interview process at WunderGraph, I didn't think much of the take-home assignment. I assumed it would just be a solid frontend task to showcase how I think. What I <em>did not</em> expect was that it would end up in production with actual users.</p><p>This is the story of how a single interview task became my first open-source contribution to Cosmo—and marked the start of my journey at WunderGraph</p><h2>Designing the Playground Share Feature for Cosmo</h2><p>I applied for a Senior Frontend Engineer role. The assignment was to build a small but thoughtful UI feature inside <a href="https://cosmo.wundergraph.com">Cosmo Studio</a>, WunderGraph's frontend for managing GraphQL federation.</p><p>The task was to allow users to share their current Playground session with a link. It sounded simple on paper, but it involved:</p><ul><li>Selecting which parts of the session to include (operation, variables, headers)</li><li>Compressing and encoding the selected state into a shareable URL</li><li>Restoring the session state cleanly on page load</li><li>Designing a smooth UI flow for it all, and handling edge cases gracefully</li></ul><p>I went all in. I treated it like a real feature, not just a task. I took time to understand the architecture, asked questions when I got stuck, wrote tests, and focused on the full experience. I had no idea it was based on an actual customer request.</p><h2>How It Became a Real Contribution to Cosmo</h2><p>After submitting the assignment, I had a review call with WunderGraph's CTO, <a href="https://de.linkedin.com/in/dustin-deus">Dustin</a>. He liked the approach and suggested that, if I was interested, I could clean it up and raise a PR to the main repo as an open-source contribution.</p><p>That's when it clicked: this thing might actually go live.</p><p>It didn’t ship right away. After joining the team, I spent some time refining the feature before finally pushing it into production.</p><p>I cleaned up the code, polished the user experience, and addressed feedback before submitting <a href="https://github.com/wundergraph/cosmo/pull/1833">this PR</a>. It introduced a modal-based UI for generating shareable session links that are encoded via <code>lz-string</code>. The goal was to make it easier for teams to collaborate, debug, and share reproducible setups.</p><p>Shortly after it shipped, I found out it was already being used in production by companies like <a href="https://soundcloud.com/">SoundCloud</a>. That was a surreal moment.</p><h2>Under the Hood: Decisions, Edge Cases &amp; Trade-offs</h2><p>While implementing the feature, I made a few deliberate technical and UX decisions to keep the experience clean, performant, and intuitive:</p><ul><li><p><strong>Compression Strategy</strong>: I used <a href="https://github.com/pieroxy/lz-string">lz-string</a> to compress the session state. It's widely supported, easy to use in the browser, and produces small payloads that are suitable for URLs without relying on newer APIs like <code>CompressionStream</code>, which don't work in older browsers.</p></li><li><p><strong>State Restoration</strong>: The logic restores the active Playground session on load using a custom query parameter (<code>playgroundUrlState</code>). It handles both optional and required fields carefully, only restoring headers and variables if they were selected and present.</p></li><li><p><strong>UI Edge Cases</strong>: The share modal disables options when no data is available (like headers or variables), shows toast messages when the URL is too long, and handles hydration errors with clear messaging. These patterns ensure that the feature feels stable, even in messy or unexpected scenarios.</p></li><li><p><strong>No Backend Dependence</strong>: The feature is completely stateless, requiring no backend logic or persistence—all state is stored directly in the URL.</p></li><li><p><strong>Navigating a Custom GraphiQL Setup</strong>: WunderGraph's Playground builds on top of <a href="https://github.com/graphql/graphiql">GraphiQL</a> with several extensions and internal overrides. Understanding the internal structure and identifying the right integration points took some effort — but once it clicked, it was a rewarding challenge.</p></li></ul><p>These kinds of product and architectural decisions are what made this fun: solving small but meaningful UX problems while thinking about scale, resilience, and clarity.</p><h2>Writing Developer Docs</h2><p>I also wrote the official developer-facing documentation for the feature: <a href="https://cosmo-docs.wundergraph.com/studio/playground/shared-playground-state">Sharing Playground Sessions</a>. The guide walks users through:</p><ul><li>What's included and not included in the link</li><li>How to generate and share it</li><li>Security considerations</li><li>Use cases for collaboration and debugging</li></ul><p>This was my first time contributing to public OSS docs, and I loved being able to treat documentation as a first-class part of the product.</p><h2>Lessons from My First Open Source Contribution</h2><p>Looking back, this experience taught me a lot:</p><ul><li><strong>Start with ownership</strong>: I treated the assignment like a real feature.</li><li><strong>Communication matters</strong>: Thoughtful PR descriptions and async reasoning helped build trust early on.</li><li><strong>Know what to prioritize</strong>: Done is better than perfect, especially when it solves a real user problem.</li></ul><h2>Joining WunderGraph Through Open Source</h2><p>Finalizing and shipping this feature was more than just a technical milestone—it showed me that WunderGraph genuinely values trust, autonomy, and thoughtful engineering. The team was supportive from the start. They were quick with feedback, open to ideas, and always willing to dive into details together. I got to dive into a complex codebase, shape the user experience, and contribute to something real from day one.</p><p>If that sounds like your kind of work, you might enjoy being part of the team. <a href="https://wundergraph.com/jobs">We're hiring</a>.</p><p>WunderGraph Hub is our new collaborative platform for designing, evolving, and shipping APIs together. It’s a design-first workspace that brings schema design, mocks, and workflows into one place.</p><p>Curious about the feature? Check it out in <a href="https://cosmo.wundergraph.com">Cosmo</a>.</p><p>Questions or feedback? I'd love to connect — you can find me on <a href="https://www.linkedin.com/in/grg-akshay">LinkedIn</a> or drop a note in the <a href="https://discord.gg/Jjmc8TC">WunderGraph Discord</a>.</p></article>

interview_task_to_prod_banner.png

The story of how an interview assignment turned into a real feature and became my first open-source contribution to Cosmo.

From Interview Task to Production Feature

scaling_graphql_observability

GraphQL Observability at Scale

See how Cosmo’s cloud architecture scales GraphQL observability with Kafka, ClickHouse & regional load balancing for enterprise reliability.

Scaling GraphQL Schema Usage to billions of requests per day

<article><p>We officially released the first version of <a href="https://github.com/wundergraph/cosmo/tree/main/aws-lambda-router">Cosmo Serverless GraphQL Federation Router</a> for the AWS Lambda Runtime. This marks a huge milestone for us and the GraphQL community. It is the first time that a vendor officially support AWS Lambda to run a GraphQL Federation Gateway. Let's break down what this means for you and your team. Before we dive into the details, let's first talk about what a GraphQL Federated Gateway is.</p><h2>What is a GraphQL Federation Router / Gateway?</h2><p>A GraphQL gateway, also known as a GraphQL router, acts as a single access point that aggregates data from multiple sources, orchestrating responses to GraphQL queries. It provides a unified GraphQL schema from different microservices, databases, or APIs, streamlining query processing and schema management. By routing requests to the appropriate services and merging the responses, a GraphQL gateway simplifies client-side querying, enabling seamless data retrieval from a variety of back-end services and systems. This architectural design enhances scalability, maintainability, and the ability to evolve API landscapes without impacting client applications.</p><p>In order to scale out a federated Graph, you need more than just a Gateway, though. You should have a Schema Registry, Analytics, Schema Usage Metrics, breaking change detection, and a lot more. For that, we've built Cosmo which is a complete platform to manage your federated Graph. Check out Cosmo on <a href="https://github.com/wundergraph/cosmo">GitHub</a> if you're interested, it's fully open-source.</p><h2>Why AWS Lambda is a good fit for a GraphQL Federation Router?</h2><p>AWS Lambda provides a serverless computing service, offering a hassle-free solution where concerns about server provisioning, scaling, and infrastructure management are handled by AWS. This allows you to concentrate solely on your business logic. Although it might sound like a jargon-filled proposition, the practical benefits for companies are substantial. Operating gateway services is inherently complex and challenging, but AWS Lambda simplifies this process. It stands out as the most popular and mature serverless computing service, available in all regions and featuring an exceptionally generous free tier. These attributes make AWS Lambda an ideal choice for our specific needs.</p><h3>AWS Lambda is Event-driven and requires a small footprint</h3><p>Due to the architecture of Lambda, we had to make our router event-driven. AWS Lambda functions are invoked by events. You can't simply run a http server. Instead, you have to register a handler function that is invoked by an event. This is a huge difference to traditional servers. Luckily, the community already built a lot of adapters to make this process as easy as possible. We decided to use the <a href="https://github.com/akrylysov/algnhsa">akrylysov/algnhsa</a> to handle the event and response handling for us. This allows us to make our router compatible with only a few lines of code <em>&quot;simplified&quot;</em>.</p><pre data-language="go">func main() {
	
    // Create a new router
    r := internal.NewRouter()
	
    // Create a new server
    svr, _ := r.NewServer(ctx)
	
    // Create an adapter for AWS Lambda
    lambdaHandler := algnhsa.New(svr.HttpServer().Handler, nil)
	
    // Start the Lambda handler
    lambda.Start(lambdaHandler)

}
</pre><p>As a side effect, our Router can now be integrated into the existing AWS ecosystem. For example, we can run a GraphQL mutation through an SQS or S3 event. This allows us to build a very flexible and scalable system.</p><p>That's it! Of course not, there is a lot more going on under the hood. We down-shrink the router binary to <code>19MB</code> to advance a small footprint and therefore fast cold starts. We also went a different route to report metrics and traces to our collectors. We will talk about this in the next section.</p><h2>OpenTelemetry support</h2><p>Our Router has built-in support for <a href="https://opentelemetry.io/">OpenTelemetry</a> for tracing and metrics. We also collect schema usage data of your GraphQL operations to give you insights into your schema usage. In that way, we can identify if a schema change is safe to deploy or not.</p><p>In the AWS Lambda handler, you can't do work after the response was sent. If you are thinking in servers, we looked for a way to do arbitrary work in the background and on <code>SIGTERM</code> <a href="https://github.com/aws/aws-lambda-go/issues/318">#Issue</a>. This does not fit into the serverless mindset. Maybe your function will never shutdown, so you have to find another way. We have to flush the metrics and traces synchronously after each request. This is tricky, because it adds latency to the response. It is much better than it sounds, because we optimized the flushing process to be as fast as possible. All metrics and traces are sent in parallel without write acknowledgement to the collectors and traces are sampled to reduce the overhead. This adds less than 75ms of latency to the response. This is a very small price to pay for the benefits you get.</p><h2>Costs / Performance</h2><p>AWS Lambda has a free tier of 1 million requests per month. If you exceed the free tier, you pay $2.28 per 1 million requests. We assume that each request takes 1 second to process. This plan uses the 128MB memory size model which is the cheapest and less powerful one. The CPU power scales proportionally with the memory size.</p><p>Our tests showed that the Router handles subsequent requests with this plan very well but at the cost of much higher cold start times. We recommend using the 1536MB memory size model. This is the sweet spot of good cold starts and low costs. We were able to wake up the Router and making a subgraph request in 641ms. The initialization time of the lambda function took just 300ms.</p><p>Once the Lambda function is initialized <em>warm</em>, it can handle subsequent requests in 80ms. In this scenario, we made a Subgraph request to the <a href="https://employees-api.fly.dev/">Employees API</a> which is hosted on fly.io in the US West (LAX) region. This is a very good result and shows that our Router performs very well on AWS Lambda.</p><h3>Benchmark</h3><p>Additionally, we tested the performance of the router with 100 concurrent users over a period of 60 seconds. We warmed the functions with 2 pre-runs. Due to simplicity, I didn't redeploy the router near my location. Instead, I used the same setup as above. The router was deployed in the US West (Oregon) region while I'm located in Germany. We will definitely see better results if we deploy the router and subgraphs near my location, but I think the results already speak for themselves.</p><p>The router was able to handle all requests with <code>p95</code> of 480ms. This is a very good result and shows that our router scales well on AWS Lambda.</p><pre data-language="bash">          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: bench.js
     output: -

  scenarios: (100.00%) 1 scenario, 100 max VUs, 1m30s max duration (incl. graceful stop):
           * default: 100 looping VUs for 1m0s (gracefulStop: 30s)


     ✓ is status 200

     checks.........................: 100.00% ✓ 11716      ✗ 0    
     data_received..................: 5.7 MB  95 kB/s
     data_sent......................: 2.0 MB  32 kB/s
     http_req_blocked...............: avg=895.92µs min=60ns     med=160ns    max=135.24ms p(90)=240ns    p(95)=270ns   
     http_req_connecting............: avg=332.8µs  min=0s       med=0s       max=58.28ms  p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=212.63ms min=168.82ms med=179.9ms  max=653.23ms p(90)=190.66ms p(95)=480.48ms
       { expected_response:true }...: avg=212.63ms min=168.82ms med=179.9ms  max=653.23ms p(90)=190.66ms p(95)=480.48ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 11716
     http_req_receiving.............: avg=66.29µs  min=5.46µs   med=18.83µs  max=1.15ms   p(90)=204.73µs p(95)=242.36µs
     http_req_sending...............: avg=23.64µs  min=10.05µs  med=22.88µs  max=195.85µs p(90)=31.2µs   p(95)=34.76µs 
     http_req_tls_handshaking.......: avg=294.56µs min=0s       med=0s       max=48.97ms  p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=212.54ms min=168.79ms med=179.81ms max=653.11ms p(90)=190.61ms p(95)=480.41ms
     http_reqs......................: 11716   193.397417/s
     iteration_duration.............: avg=514.09ms min=468.94ms med=480.47ms max=1.02s    p(90)=492.15ms p(95)=782.61ms
     iterations.....................: 11716   193.397417/s
     vus............................: 100     min=100      max=100
     vus_max........................: 100     min=100      max=100
</pre><p>I want to call out <a href="https://twitter.com/codetalkio">@codetalkio</a> who does a fantastic job with their <a href="https://github.com/codetalkio/serverless-federated-graphql">repository</a> to unmistify the landscape of supported GraphQL Federated Gateways on AWS Lambda. It gave us the inspiration to support AWS Lambda as a first-class citizen.</p><h2>Getting started</h2><p>As you might know, we love Open Source! You can find the source code of the Serverless Router on <a href="https://github.com/wundergraph/cosmo/tree/main/aws-lambda-router">GitHub</a>. We manage separate releases for the AWS Lambda router. You can find the releases <a href="https://github.com/wundergraph/cosmo/releases?q=aws-lambda-router&amp;expanded=true">here</a>.</p><p>You don't have to build the router yourself. You can download the latest release and follow the instructions below.</p><ol><li><p>Download the Lambda Router binary from the official <a href="https://github.com/wundergraph/cosmo/releases?q=aws-lambda-router&amp;expanded=true">Router Releases</a> page.</p></li><li><p>Create a .zip archive with the binary and the <code>router.json</code> file. You can download the latest <code>router.json</code> with <a href="https://cosmo-docs.wundergraph.com/cli/federated-graph/fetch"><code>wgc federated-graph fetch</code></a>. The .zip archive should look like this:</p><pre data-language="bash">.
└── myFunction.zip/
    ├── bootstrap # Extracted from the Router release archive
    └── router.json # Downloaded with `wgc federated-graph fetch`
</pre></li><li><p>Deploy the .zip archive to AWS Lambda. You can use SAM CLI or the AWS console. Alternatively, you can use your IaC tool of choice.</p></li></ol><h2>Conclusion</h2><p>Using AWS Lambda, we made it ridiculously simple to deploy, operate and scale a GraphQL Federation Router / Gateway. We are very excited to see what you will build with it. We are looking forward to your feedback and contributions.</p><p>If you are a company and want to use our router in production, please <a href="https://wundergraph.com/contact/sales">reach out to us</a>. We are happy to help you with the setup process and provide you with support.</p></article>

graphql_federation_router_on_aws_lambda

Serverless GraphQL Federation Router for AWS Lambda

Deploy GraphQL Federation on AWS Lambda with Cosmo’s serverless router. Small, fast, OpenTelemetry-ready, and built for real-world performance.

<article><p>WunderGraph Hub is our new collaborative platform for designing, evolving, and shipping APIs together. It’s a design-first workspace that brings schema design, mocks, and workflows into one place.</p><p><a href="https://github.com/wundergraph/graphql-go-tools">graphql-go-tools</a> is the open source GraphQL engine powering WunderGraph. Some of our enterprise customers go even one step further and they use the engine directly to build their GraphQL-based applications: proxies, gateways, caches, routers, etc...</p><p>Recently, one of our customers came back to us with a surprising benchmark: running a mutation against their GraphQL servers directly would take around 1 minute, but if they routed it through their gateway (built on top <a href="https://github.com/wundergraph/graphql-go-tools">graphql-go-tools</a>), it would take around 70 minutes. That's a staggering 70x difference and something we instantly knew we needed to improve.</p><p><a href="https://github.com/wundergraph/graphql-go-tools">graphql-go-tools</a> is built with performance in mind and regularly benchmarked, so we're confident the main request pipeline and common use cases are very fast. This would suggest that our customer was hitting a pathological case which we hadn't yet accounted for.</p><h2>Narrowing the problem down</h2><p>The first thing we did was work closely with our customer to reproduce the problem on our end. It turns out they were running a huge bulk operation consisting of a mutation receiving an array with ~70,000 items as its input without using variables but using a variable definition instead. We created a slightly different test that took 26 minutes to use as the reference for this problem.</p><p>We immediately noticed a potential workaround: Data provided as part of the variables JSON instead of defining the 70k input objects in GraphQL syntax requires less processing by the gateway, allowing us to take faster code paths. We made a quick check in the test case, and we confirmed that moving the data to the variables got rid of the slow-down. We communicated this workaround to the customer, to unblock them as fast as possible, and we started working on the full solution.</p><h2>Our approach to balancing maintainability versus writing fast code</h2><p>Performance matters a lot to us, let's make that clear. Unfortunately, sometimes chasing the best absolute performance gets in the way of maintainability, which is also a very desirable feature because it allows us to move fast and deliver new features quickly.</p><p>So our philosophy at WunderGraph is to keep a fine balance between both of them. This means sometimes we don't initially write our code in the most performant way, instead we choose to make it reasonably fast while still being maintainable. This same maintainability also enables us to quickly debug problems and profile performance problems when they arise.</p><h2>Profile like a pro</h2><p>As you might have guessed, <a href="https://github.com/wundergraph/graphql-go-tools">graphql-go-tools</a> is written in <a href="https://go.dev">Go</a>. We chose Go not just because of its simplicity and the productivity the language enables, but also because of the excellent testing, debugging and profiling tools it comes with. While investigating this problem, the most valuable tool was the CPU profiler.</p><p>There are multiple ways to enable CPU profiling in Go applications, but for our use cases the one that we've found to be the most useful is exposing the profiles using an HTTP handler. Go makes this incredibly easy: all you have to do is import <code>net/http/pprof</code> and, if you're not using the default HTTP server, start it in your <code>main()</code> function via <code>go http.ListenAndServe(&quot;:6060&quot;, nil)</code>.</p><p>This exposes several profiling endpoints. For example, to start collecting a CPU profile, you can run:</p><pre data-language="sh">go tool pprof -http=: 'http://localhost:6060/debug/pprof/profile' 
</pre><p>This starts collecting a CPU profile for 30 seconds and then opens a browser with all the information. This means you should start the profile and then run the operation that you want to measure while the profile is being collected. Additionally, you can collect shorter or longer profiles using a <code>seconds=number</code> parameter in the query string. For this case, we use a smaller test than the initial one that completed in ~7.5 seconds, to help us get faster results, so we collected profiles that lasted 10 seconds.</p><p>Profiles can be inspected in several ways, but we've found the Graph view (available from the <code>VIEW</code> menu) very valuable for getting a general overview. Following the red arrows allows us to quickly notice the code paths taking the most time in the profile and get an idea of what's going on.</p><p>If you'd like to follow along, we've made the image with the <a href="/images/blog/go_profiling/initial_profile_graph.png">full profile</a> available.</p><p>Following the red lines, we can see a lot of the time is spent in manipulating JSON-encoded data. As you might already know GraphQL uses JSON when transported over HTTP, which means a gateway needs to decode a JSON payload, perform some work on the data and then encode it back to JSON to send it upstream.</p><p>The two spots that got our attention were:</p><ul><li>Input value injection (<code>astnormalization (*inputFieldDefaultInjectionVisitor) processObjectOrListInput</code>) taking 61% of the time</li><li>Encoding data back to JSON (<code>ast (*Document) ValueToJSON</code>) taking 23% of the time</li></ul><h3>JSON encoding</h3><p>Our GraphQL engine performs several passes over the input data to normalize it. This is necessary to simplify further processing steps like validation, execution planning, etc... It's much easier to implement an execution planner that can assume that all the input data is following specific rules.</p><p>We started by working on JSON encoding because we were very confident we could gain some time there. The existing approach involved a recursive function creating intermediate JSON objects that were eventually put together into an object representing the whole document. For example, this is how we were handling objects:</p><pre data-language="go">func (d *Document) ValueToJSON(value Value) ([]byte, error) {
  switch value.Kind {
    // Other kinds omitted for clarity
    case ValueKindObject:
      out := []byte(&quot;{}&quot;)
      for i := len(d.ObjectValues[value.Ref].Refs) - 1; i &gt;= 0; i-- {
        ref := d.ObjectValues[value.Ref].Refs[i]
        fieldNameString := d.ObjectFieldNameString(ref)
        fieldValueBytes, err := d.ValueToJSON(d.ObjectFieldValue(ref))
        if err != nil {
          return nil, err
        }
        out, err = sjson.SetRawBytes(out, fieldNameString, fieldValueBytes)
        if err != nil {
          return nil, err
        }
      }
      return out, nil
  }
}
</pre><p>We start with an empty object and populate its fields using a library that manipulates JSON in place. This becomes problematic with big objects because it required a lot of reallocations and copying. Additionally, we were also converting field names to strings, unescaping their JSON representations and then encoding them back to bytes.</p><p>Initially, we thought of using Go maps and slices to build and in-memory representation and feeding that into the JSON serializer in a single pass to encode everything. This would have reduced the number of allocations and made things faster, but then we came up with an even better approach: we read these objects from a JSON payload sent by the client, and keep them in-memory in their escaped form, only decoding them as needed. This means that in order to build that intermediate representation in Go we would be decoding them and then feeding them back to the JSON serializer to escape them again. Instead, we instantiate a <code>bytes.Buffer</code>, we recursively walk the document, and we write the values directly to the buffer without ever decoding and re-encoding any JSON string, only writing necessary characters to reconstruct the objects and arrays.</p><p>We started by changing <code>ValueToJSON()</code> to call instantiate the <code>bytes.Buffer</code> and into a helper function which does the recursion:</p><pre data-language="go">
func (d *Document) ValueToJSON(value Value) ([]byte, error) {
	var buf bytes.Buffer
	if err := d.writeJSONValue(&amp;buf, value); err != nil {
		return nil, err
	}
	return buf.Bytes(), nil
}

</pre><p>And then we implemented a recursive function writing directly to the buffer:</p><pre data-language="go">func (d *Document) writeJSONValue(buf *bytes.Buffer, value Value) error {
  switch value.Kind {
    // Other kinds omitted for clarity
    case ValueKindObject:
    buf.WriteByte(literal.LBRACE_BYTE)
    for ii, ref := range d.ObjectValues[value.Ref].Refs {
      if ii &gt; 0 {
        buf.WriteByte(literal.COMMA_BYTE)
      }
      fieldNameBytes := d.ObjectFieldNameBytes(ref)
      buf.Write(quotes.WrapBytes(fieldNameBytes))
      buf.WriteByte(literal.COLON_BYTE)
      if err := d.writeJSONValue(buf, d.ObjectFieldValue(ref)); err != nil {
        return err
      }
    }
    buf.WriteByte(literal.RBRACE_BYTE)
  }
}
</pre><p>This avoids creating any temporary strings or any kind of intermediate JSON object, we write everything to the buffer in a single pass.</p><p>This made <code>ast (*Document) ValueToJSON</code> disappear entirely from the graph! We had to look at the text list with the top functions to find it. Its execution time had reduced from 1.48s to 9ms, which accounts for a 99.39% reduction.</p><h3>Input value injection</h3><p>After getting JSON encoding out of the way, we turned our sights to default input value injection. Injecting default values is the process of looking at the input data of an operation and injecting default values defined in the GraphQL Schema if no value was provided by the client. Here's an example of a GraphQL Schema that defines a default value:</p><pre data-language="graphql">type Query {
  hello(name: String = &quot;World&quot;): String!
}
</pre><p>If the client sends the following query:</p><pre data-language="graphql">query {
  hello
}
</pre><p>The GraphQL engine will inject the default value for the <code>name</code> argument, which is <code>&quot;World&quot;</code>, and the query will be executed as if the client had sent:</p><pre data-language="graphql">query {
  hello(name: &quot;World&quot;)
}
</pre><p>Interestingly, our test case didn't have to inject any values! It turns out when we wrote that code we made didn't benchmark it with arrays or objects as big as in this case.</p><p>Basically, we were iterating over each element in the input, assigning a variable to its current value, looking if we should override it with a default value and then writing it back naively. The problem was that we didn't check if the value had been actually overridden, performing a lot of unnecessary updates on the JSON payload we had received from the client. For example, this was one of the functions:</p><pre data-language="go">func (v *inputFieldDefaultInjectionVisitor) EnterVariableDefinition(ref int) {
  // Some code omitted for clarity
  // ...
  //
  variableVal, _, _, err := jsonparser.Get(v.operation.Input.Variables, v.variableName)
  // ...
  // More code omitted
  // ...
  //

  // Here v.processObjectOrListInput() might return variableVal unmodified or it
  // might return a new injected value
  newVal, err := v.processObjectOrListInput(typeRef, variableVal, v.operation)
  if err != nil {
    v.StopWithInternalErr(err)
    return
  }
  // This jsonparser.Set() call might be costly if the input variables are big and we
  // perform it unconditionally, even if the value has not changed.
  newVariables, err := jsonparser.Set(v.operation.Input.Variables, newVal, v.variableName)
  if err != nil {
    v.StopWithInternalErr(err)
    return
  }
  v.operation.Input.Variables = newVariables
}
</pre><p>We made this code slightly smarter by modifying <code>processObjectOrListInput()</code> and the functions it calls to return also whether the returned value is the same as it received or something different, allowing us to skip updating the JSON when the value hasn't changed:</p><pre data-language="go">func (v *inputFieldDefaultInjectionVisitor) EnterVariableDefinition(ref int) {
  // Some code omitted for clarity
  // ...
  //
	variableVal, _, _, err := jsonparser.Get(v.operation.Input.Variables, v.variableName)
  // ...
  // More code omitted
  // ...
  //

  newVal, err := v.processObjectOrListInput(typeRef, variableVal, v.operation)
  if err != nil {
    v.StopWithInternalErr(err)
    return
  }
  // Now we know whether v.processObjectOrListInput() returned a different value than
  // what it received
  newVal, replaced, err := v.processObjectOrListInput(typeRef, variableVal, v.operation)
  if err != nil {
    v.StopWithInternalErr(err)
    return
  }
  // If the value has not changed, we skip the potentially expensive jsonparser.Set() call
  if replaced {
    newVariables, err := jsonparser.Set(v.operation.Input.Variables, newVal, v.variableName)
    if err != nil {
      v.StopWithInternalErr(err)
      return
    }
    v.operation.Input.Variables = newVariables
  }
}
</pre><p>Once we updated our code to keep track of whether the value had changed and updated the payload only when necessary we saw <code>astnormalization (*inputFieldDefaultInjectionVisitor) processObjectOrListInput</code> go down from 4.30s to 10ms, or a 99.76% reduction.</p><h2>Profiling again</h2><p>Once we implemented those two optimizations, we ran the test again and measured its full execution time at 150ms, with 80ms accounting for sending the request upstream and awaiting the GraphQL server to send it back.</p><p>We also collected and examined a <a href="/images/blog/go_profiling/final_profile_graph.png">new profile</a> in which we determined that there was no more low-hanging fruit for us to optimize at this time.</p><h2>Notifying our customer</h2><p>Very happy with the results, we prepared a <a href="https://github.com/wundergraph/graphql-go-tools/pull/547">pull request</a>, merged it and got back to the customer in less than 48 hours to deliver the improvements.</p><p>They ran their test again, and it took 20 seconds, down from 26 minutes (or a 98.71% reduction). They thought it was so fast that they had to double-check the results to verify that all the data was sent back and forth!</p><h2>Surprisingly, the Operation is now faster when using the Gateway!</h2><p>What's even more interesting is that the operation is now faster when using the Gateway than when using the GraphQL server directly. How is that possible?</p><p>The answer is that we've built a highly optimized parser and normalization engine. Our gateway parses the GraphQL Operation and normalizes the input data out of the GraphQL Operation and rewrites it into a much smaller GraphQL Operation with the input data as part of the variables JSON. The GraphQL framework of the customer isn't optimized for this use case. By using the Gateway in front of their GraphQL server, we're able to &quot;move&quot; the input from the GraphQL Operation into the variables JSON, so the GraphQL server only has to parse a reasonably small GraphQL Operation, and JSON parsing libraries are quite fast and optimized compared to GraphQL parsing libraries.</p><h2>Conclusion</h2><p>At WunderGraph we like to use the best tools for the job, and we're incredibly confident that Go is a great choice for networked services. Not just because of the simplicity of the language, but also because of how fast we're able to iterate, test, debug and profile with its amazing built-in tools.</p></article>

78x_improvement_with_pprof.png

78x improvement with pprof

How we reduced the executing time of a huge GraphQL operation using Golang's profiling tools

Router / Gateway

Connect

MCP Gateway

ConnectRPC

Documentation

Zero to Production

GitHub

Community

Blog

Managing Permissions in Cosmo Just Got Easier with Groups

From Interview Task to Production Feature

Scaling GraphQL Schema Usage to billions of requests per day

Serverless GraphQL Federation Router for AWS Lambda

From 26 Minutes to 20 Seconds: Using pprof to optimize large GraphQL Operations in Go