<article><p>WunderGraph Hub is our new collaborative platform for designing, evolving, and shipping APIs together. It’s a design-first workspace that brings schema design, mocks, and workflows into one place.</p><p>Everything is a key-value store if you try hard enough. 🤓</p><p>We're excited to announce that the <a href="https://www.openpreviews.com/">Open Previews</a> beta is now available. In this blog post we will explain how we built Open Previews and how we use divs in GitHub discussions as a key-value store. You can try it out live on <a href="https://www.openpreviews.com/">https://www.openpreviews.com/</a>, the <a href="https://github.com/wundergraph/open-previews">source code is available on Github</a>.</p><h2>What is Open Previews?</h2><p>Open Previews allows you to add commenting functionality to your previews/staging environments, or any other website that you want to collect feedback on, like documentation pages. It's a great way to collect feedback from your non technical team members, customers or other stakeholders. Preview comments is not a novel idea, services like Vercel and Netlify have supported this for a while now, but we wanted to build something that is open source and can be self hosted.</p><h2>How we built Open Previews</h2><p>Before building Open Previews we had a few requirements:</p><ul><li>No database</li><li>The backend had to be stateless</li><li>Utilize WunderGraph as much as possible</li><li>Embeddable using a single script tag</li></ul><p>The solution we came up with is inspired by <a href="https://giscus.app/">Giscus</a>, which uses Github discussions to store comments. This is a great solution and allowed us to build Open Previews without a database.</p><h2>The Problem of storing meta data in Github Discussions</h2><p>There was one problem though, we had to store additional information with the comments, like the position, selection, and other meta data. How do we store this information in Github discussions? But not only that, the meta data should also not be visible to the user to not distract them, and we need to be able to retrieve it from the discussion when we render the comments.</p><h2>The Solution: Using divs in HTML comments as a key-value store</h2><p>Github comments <a href="https://github.github.com/gfm/#html-block">support basic HTML</a>, so what if we can simply embed JSON inside a data attribute in an empty DIV? After a simple experiment we found out that this works great.</p><p>Adding JSON directly in a data attribute didn't work though because of the quotes and some other escaping issues, so we encode the JSON using <code>encodeURIComponent</code>. Now we had a functional key-value store using divs and could store all the information that we needed. 🥳</p><p>Here's how it looks like in the code:</p><pre data-language="ts">export const constructComment = ({
  body,
  meta,
}: {
  meta: {
    timestamp: number
    x: number
    y: number
    path: string
    href: string
    resolved?: boolean
    selection?: string
  }
  body: string
}) =&gt; {
  const jsonMeta = JSON.stringify(meta)

  const encodedJsonMeta = encodeURIComponent(jsonMeta)

  return `${body}

  &lt;div data-comment-meta=&quot;${encodedJsonMeta}&quot; /&gt;`
}
</pre><p>Now when we retrieve the comments we can simply get the encoded data from the div, decode it, parse the JSON and render the comments on the page. The possibilities are endless 😎</p><h2>Why GitHub Discussions is a great key-value store</h2><p>GitHub Discussions as a key-value store is amazing because it comes with a lot of powerful features out of the box:</p><ul><li>our application is stateless</li><li>GitHub Discussions are virtually free and scale infinitely</li><li>built-in authentication: We can use GitHubs authentication to authenticate users</li><li>build-in authorization: You can only comment on a discussion if you have access to the repository</li><li>built-in notifications: Users get notified when someone replies to their comment</li><li>built-in versioning: You can see the history of a discussion</li><li>built-in spam protection: GitHub Discussions has spam protection built-in</li></ul><h3>Stateless authentication</h3><p>The next challenge was building stateless authentication. Adding authentication with WunderGraph is easy using the <a href="https://bff-docs.wundergraph.com/docs/auth/cookie-based-auth/github">built-in Github Oauth provider</a>, but we had to store the access tokens somewhere safely. Since previews are typically hosted on a different domain than the WunderGraph server, we can't use secure cookies either.</p><p>You might think that we could store the access tokens in our new Github Discusions KV, but that would be a very bad idea. Instead we came up with a solution that uses <a href="https://datatracker.ietf.org/doc/html/rfc7516">JSON Web Encryption</a> to store the access tokens in the browser. The JWT is encrypted using a secret that is only known to the backend. This allows us to verify the JWT and extract the access token from the JWT payload. The access token is then used to authenticate the user with Github.</p><p>After logging in, the encrypted JWT is exchanged between the WunderGraph server and the preview. We do this by starting the authentication flow in a popup that allows us to pass the JWT to the preview using the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage">postMessage API</a>. Alternatively a compact token could also be appended to the redirect uri. The JWT is stored in the browser using the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage">localStorage API</a>.</p><p>The only limitation of this approach is that the JWT is only valid for a limited time and the user needs to log in again after the JWT has expired.</p><h2>What's next?</h2><p>We're still working on Open Previews and we have a lot of ideas for new features. Some feature we'd love to add:</p><ul><li>Support PR reviews and Github Checks</li><li>Optimistic updates (make the UI snappier)</li><li>Like &amp; repy to comments</li><li>Markdown / Emoji support</li><li>Upload images</li></ul><p>If you have any ideas, feedback, or want to contribute, please let us know on <a href="https://twitter.com/wundergraphcom">Twitter</a>, <a href="https://github.com/wundergraph/open-previews">Github</a> or <a href="https://wundergraph.com/discord">Discord</a>.</p></article>

using_divs_in_github_discussions_as_a_key_value_store

Everything is a key-value store if you try hard enough. 🤓

Introducing Open Previews and how we built it without a database.

Using invisible divs in GitHub Discussions as a key-value store

<article><p>WunderGraph Hub is our new collaborative platform for designing, evolving, and shipping APIs together. It’s a design-first workspace that brings schema design, mocks, and workflows into one place.</p><p>Today we're happy to announce the open source release of WunderBase, a Serverless Database with a GraphQL API on top of SQLite, Firecracker/Fly machines, and Prisma. It's embarrassingly simple, but powerful, as the codebase is less than 400 lines of Go.</p><p>WunderBase is built on top of Fly machines, which is a REST API that allows you to run virtual machines in seconds. What's special about machines is that they can sleep when an application exits with a zero exit code.</p><p>When you send a request to WunderBase, the virtual machine wakes up in about 300-500ms and executes the request. Ten seconds (configurable) after the last request was processed, we send the machine to sleep again. This means that you really only pay for storage and the CPU time you really use, hence the name &quot;Serverless Database&quot;.</p><p>If you'd like to jump right into the code and try WunderBase out for yourself, you can find the <a href="https://github.com/wundergraph/wunderbase">source code on GitHub</a>.</p><h2>See WunderBase in action</h2><p>If you're less of a reader, you can also watch the video below to see WunderBase in action.</p><h2>Why we've built WunderBase</h2><p>We're in the works of building WunderGraph Cloud, applying the principles of Vercel to Backend/API development. We take the most important backend primitives like Authentication &amp; Authorization, Database, Queue, PubSub, Key-Value Store, Cache, etc. and make them available as a single, unified SDK.</p><p>Git push and 30 seconds later, you've got a fully functional Serverless backend without touching any infrastructure. Make some code changes, open a PR, and get a preview environment with your changes.</p><p>For this to work, we need a fast and inexpensive way to create and destroy databases. But as you'll see, there are many other use cases for WunderBase than just preview environments.</p><h2>How does WunderBase work?</h2><p>Here's an overview of how WunderBase works:</p><ol><li>You deploy WunderBase to a Fly machine with a volume attached</li><li>The Fly machine boots up and starts WunderBase</li><li>We use the the <code>prisma.schema</code> file to migrate the database using the Prisma Migration Engine</li><li>Once the migration is done, we create a lock file to prevent future migrations if the <code>prisma.schema</code> file didn't change</li><li>Finally, we start the Prisma Query Engine to serve the GraphQL API</li><li>Ten seconds (configurable) after the last request was processed, we exit WunderBase with a zero exit code to indicate to fly that this Machine should go to sleep</li></ol><p>Request flow:</p><ol><li>You send a request to WunderBase</li><li>The Fly machine wakes up in about 300-500ms</li><li>The fly proxy forwards the request to WunderBase</li><li>WunderBase sanitizes the request and proxies it to the Prisma Query Engine</li><li>The Prisma Query Engine executes the request and returns the result</li><li>WunderBase sends the result back to the client</li><li>Again, after ten seconds of inactivity, we shut down the Machine</li></ol><h2>Rethinking Serverless with Fly machines</h2><p>When using AWS Lambda, you have to follow certain rules to make your application compatible with the environment. E.g. when you're using Golang, you have to export a <code>Handler</code> function that takes a <code>context.Context</code> and the name of the event. Also, you don't really have much control over the environment your application runs in. At some point, the Lambda runtime will shut down your application.</p><p>With Fly machines, we can operate at a much lower level. We create a Docker container that listens on a specific port. We tell fly to run this container and send all requests to this port. If we think the Machine should sleep, we exit the container with a zero exit code. Fly will keep our volume around and start the container again when we send a request.</p><p>A lot of applications are written in a Serverful single-tenant way would usually be quite expensive to run as a multi-tenant service. With Fly machines, we can run these applications in a very cost-efficient way. However, there's a catch. We have to optimize the application for fast startup and shutdown. I know that fly is working on allowing machines to &quot;sleep&quot; and &quot;resume&quot;, just like closing and re-opening a laptop. This would allow us to run applications that are not optimized for fast startup and shutdown. In the meantime, we've built WunderBase in a way to optimize for fast startup and shutdown times.</p><h2>Optimizations to reduce cold start and shutdown times</h2><p>When a request is waiting to be served, we need to make sure that the Machine starts up as fast as possible. At the same time, we always want to keep the database schema in sync with the <code>prisma.schema</code> file. If we ran a migration every time the Machine starts up, we would have to wait for the migration to finish before we can serve the request. That's not ideal, so we've optimized this path.</p><p>When the Machine starts up, we hash the <code>prisma.schema</code> file and check if there's a lock file with the same hash. If there's a lock file, we compare the content of the lock file with the hash of the <code>prisma.schema</code> file. If they are the same, we know that the migration for this schema has already been executed. If there's no lock file, we run a migration and create a lock file with the hash of the <code>prisma.schema</code> file.</p><p>We store the lock file in the volume, so it's available even if the Machine was sleeping.</p><p>Next, we needed to make sure that the Prisma Query Engine shuts down properly. We start the Prisma Query Engine as a separate process. Before we kill the main process, we send a <code>SIGTERM</code> signal to the Prisma Query Engine. If we would immediately kill the main process, the sub-process would keep the Machine from shutting down for a few more seconds. Instead, we're using a <code>sync.WaitGroup</code> to wait for the Prisma Query Engine to shut down before we exit the main process. This way, we've reduced the shutdown time from 5-10 seconds to 1-2 seconds.</p><p>If we didn't do this, you'd have to wait up to 10 seconds when sending a request to WunderBase while it's shutting down.</p><h2>How does WunderBase compare to other Serverless Databases like DynamoDB, CockroachDB, MongoDB, FaunaDB, Planetscale or Neon?</h2><p>We can have long debates about what Serverless really is or if the term actually makes sense. After all, there are still servers involved. To me, Serverless means that you don't have to worry about the infrastructure and that you only pay for what you really use.</p><p>For a long time, Serverless was mostly about functions. You write a function, deploy it, and you're done. Someone else takes care of the infrastructure and you only pay per request.</p><p>Then came a new wave of Serverless offerings that give you a &quot;Serverless Database&quot;. You write a schema, deploy it, and you're done. But are these really Serverless?</p><p>From a user perspective, yes. From a technical perspective, no. Most databases still are &quot;Serverful&quot; in the sense that the database server is always running.</p><p>Some solutions, like Neon, try to solve this problem by separating the compute and storage layer. Others, like CockroachDB or Mongo put a proxy in front of the database so that you can &quot;imitate&quot; a Serverless Database.</p><p>In contrast, the storage layer of WunderBase is always &quot;sleeping&quot;, because it's just a file. SQLite is probably the only real Serverless database because it's just a file. The Serverful part of WunderBase is the proxy that runs the Prisma Query Engine and translates between GraphQL and SQL. But as we've discussed earlier, we can send this proxy to sleep and wake it up again when we need it.</p><p>Another important aspect of WunderBase is that it's actually quite simple and very transparent what's going on. We've got a proxy that translates between GraphQL and SQL, and we have a SQLite database / file on a volume.</p><p>I've told one of my Co-Founders that I'm a bit embarrassed to release WunderBase because it's so simple and just a few hundred lines of code. He answered that it's not embarrassing at all, because it's actually quite impressive that we've managed to build something that's so simple and yet so powerful. And he's right! Sometimes you combine the right ingredients in the right way and achieve something that took others years to build. It took me just a few hours to build WunderBase. The most time-consuming part was to write proper tests and this blog post.</p><h2>What are the use cases for WunderBase?</h2><p>You might be thinking that WunderBase is just a toy project. We're definitely not going to compete with the big players in the Serverless Database space. Instead, we're looking at serving use cases that others are unable to serve.</p><p>I'll give you a few examples:</p><ul><li>It takes seconds to create a new WunderBase instance. You can use it for a quick prototype or to test something out.</li><li>For each branch you deploy, you can have a separate WunderBase instance that's isolated from the main branch/database.</li><li>A lot of applications don't ever store more than a few gigabytes of data and have very little traffic. WunderBase is perfect for these use cases.</li></ul><p>One use case that excites me the most is that you can &quot;shard&quot; at the database layer. What this means is that you can have a single database model that's shared across multiple databases. You can have one database per user, tenant, or any other key and route traffic to the correct database based on the user's (tenant's) ID. If you have a customer with a lot of data, you can easily put them on a separate database. If each customer has their own database, you can do point in time recovery for each customer individually.</p><p>Another interesting use case is OLAP. Let's say we'd like to analyze terabytes of data in seconds. We can shard the data across multiple databases and run a query on each database in parallel. We can then aggregate the results and return them to the user. While we're not serving any requests, we can shut down the databases. This way, we're only really paying for the storage and compute that we're using. This could be an Open Source alternative to BigQuery.</p><h2>How fast is WunderBase?</h2><p>I've done some benchmarking and was able to achieve 2k write requests per second and 10k read requests per second. There's a benchmark script in the WunderBase repository that you can use to run your own benchmarks. Make sure to set the rate limit environment variables properly to not get rate-limited. During my testing I've realized that 2k/10k is the maximum that I can achieve before getting Timeout errors, so I've added some rate limiting to the proxy to keep everything stable.</p><h2>How can you scale WunderBase?</h2><p>There are multiple ways to scale WunderBase. We can add read replicas to scale reads. There's tooling to replicate SQLite databases, both locally or even remotely. So we could have a master database and read replicas in different regions.</p><p>Another way to scale WunderBase is to have multiple master databases. With this approach, we can scale writes based on a key like the user ID or tenant ID.</p><p>If you've got users all over the world, you can combine both approaches to optimize for latency.</p><h2>Can you easily migrate from WunderBase to a different database?</h2><p>Another question you might ask is if you're able to migrate from SQLite to e.g. PostgreSQL or MySQL. The answer is yes and it's actually quite easy. As we're using Prisma, we can just change the <code>provider</code> in the <code>datasource</code> block in the <code>prisma.schema</code> file. If we're using the same Schema, Prisma will give us the same GraphQL API, even if we're using a different database.</p><p>So it's possible to switch from SQLite to MySQL, PostgreSQL, SQLServer or even Planetscale.</p><h2>How can you backup WunderBase?</h2><p>Backups can be implemented e.g. by leveraging LiteStream, a tool that streams the changes from a SQLite database to S3.</p><h2>Caveats</h2><p>There's one caveat that you should be aware of. The GraphQL API that WunderBase (Prisma Query Engine) exposes is not intended to be publicly exposed. You should always put a <a href="https://github.com/wundergraph/wundergraph">GraphQL API Gateway like WunderGraph</a> in front of it. It's perfectly fine to use GraphQL as the ORM layer, but this API is not intended to be consumed by clients in the browser directly.</p><p>Additionally, Prisma Query Engine is only exposing a subset of GraphQL, e.g. you cannot use variable definitions. We've done some extra steps in WunderGraph to make it compatible with GraphQL, like writing a &quot;variable definition inliner&quot; to automatically inline variable definitions and make the GraphQL Operation compatible with the Prisma Engine. I'll follow up with a blog post about this topic as it was actually quite interesting to reverse engineer the Prisma Query Engine and make all of this work together.</p><h2>Thank you Prisma!</h2><p>All of this wouldn't have been possible without the amazing work that the Prisma team has done. Besides the 400 lines of glue code that I've written, the rest is just Prisma.</p><p>I also know that this is not the intended use case of Prisma. Nikolas Burk keeps reminding me that Prisma is an ORM and the GraphQL layer of the Prisma Query Engine is an internal detail. Prisma generates a client library on top of this Engine which uses GraphQL internally.</p><p>I personally think that it's much more powerful to expose the GraphQL API directly. This way, I'm able to <a href="https://bff-docs.wundergraph.com/docs/features/cross-api-joins-to-compose-apis">join</a> multiple APIs together easily and talk to all my services with a single language.</p><p>So, thank you and sorry for abusing your product! Open Source is awesome! =)</p><h2>The future of Firecracker-based Applications</h2><p>I'm very excited about the future of Firecracker-based applications. I'm pretty sure that this paradigm will allow us to build or re-build a lot of applications in a more efficient way, like Serverless Databases, Serverless Caches, Serverless Queues, Serverless Search Engines, etc.</p><h2>What's next?</h2><p>Our goal for WunderGraph is to create a TypeScript Framework that allows you to build Serverless Applications in a very simple way. From databases to file storage, queues, pub/sub, key-value, and caching. We want to provide a unified SDK that allows you to focus on the code, not on the infrastructure.</p><p>Today, we're starting with a Database, but there's alot more to come.</p><p>Be amongst the first to try WunderGraph Cloud &amp; WunderBase for free. With WunderGraph Cloud, you'll be able to git push your application and get a fully functional Serverless API alongside your Database in seconds.</p><p>Make sure to follow us on twitter or linkedin to stay up to date on our progress.</p><h2>What would you build with WunderBase?</h2><p>Finally, I'd love to hear what you'd build with WunderBase. I've got a few ideas (see above), but I'm sure that there are many more use cases that I haven't thought of yet.</p><p>In case you've missed it above, here's the link to the <a href="https://github.com/wundergraph/wunderbase">source code of WunderBase on GitHub</a>. Give it a star if you like the project and go tinkering with it. =)</p></article>

WunderBase is an open source Serverless GraphQL database built on SQLite, Firecracker, and Prisma. Sleep mode saves cost—perfect for fast, lightweight apps.

WunderBase – Serverless GraphQL DB using SQLite, Firecracker & Prisma

Learn more about how WunderHub, the first Package Manager for APIs, creates a whole new Experience of sharing and integrating APIs. We are currently in a public beta and are looking for feedback and ideas.

Blog

Using invisible divs in GitHub Discussions as a key-value store

WunderBase – Serverless GraphQL DB using SQLite, Firecracker & Prisma

WunderGraph - The Next-Gen API Developer Platform - is Open Source!