Michael Friis' Blog

Podnanza is a simple screen-scraper/feed-generator that I built for my own amusement to podcast shows from Danish Radio’s (DR) Bonanza archive. Check out the Podnanza announcement post for details. This post describes how Podnanza was built using ASP.NET Core running on AWS Lambda. The Podnanza source code is on GitHub.

I’ll start by admitting that AWS Lambda is the wrong technical architecture for Podnanza. Nothing’s wrong with Lambda, but Podnanza is a set of very static RSS feeds: The shows are from DR’s archive and don’t change or get new episodes. A simpler Podnanza implementation would have been a static-site generator that scraped the archive and put the XML RSS feed files in AWS S3.

I opted for Lambda for the very bad reason that I wanted to learn about serverless/function-based development by implementing a “real” project, and Podnanza was the realest small-size idea on my mind at the time. At least it’ll only be me that has to deal with maintenance of the over-complicated setup.

FaaS and HTTP Apps

Working (as I do) on PaaS/FaaS/Serverless products one might encounter arguments like:

FaaS is event-based programming and HTTP requests can be thought of as events. If a PaaS platform has good autoscaling and scale-to-zero (idling) then separate FaaS-features are not needed—people should just build FaaS apps as normal HTTP services.

Or the other way around:

If we have FaaS and event-based programming, why would we also support long-running processes for serving HTTP requests? People should just build HTTP apps from FaaS features since dealing with HTTP requests is an example of handling events

In the abstract, both of these of these statements are correct but they also obscure a lot of useful nuances. For example, even the slickest HTTP app platform pushes some HTTP handling overhead onto developers. Programs that only have to accept events through an interface defined in an SDK maintained by the FaaS platform can be a lot simpler than programs dealing with HTTP, even when a HTTP endpoint is only required for ingesting events. And because event-handling is a more constrained problem than general HTTP, platform-provided tooling such as SDKs and test-mocks can be more targeted and effective.

Similarly, forcing all HTTP apps to be built by handling events coming through a FaaS platform event interface is not ideal either:

• Lots of apps have already been built using HTTP frameworks like Node.js Express, and those apps would have to be rewritten to conform to the event interface
• Many developers are very experienced and productive building HTTP apps using existing HTTP frameworks and it’s not worth it for them to ditch those frameworks for an event-based HTTP model, even if it comes with slightly reduced management overhead
• FaaS interfaces are still largely proprietary and platform-specific, causing lock-in (although middleware like the Serverless Framework can help mitigate that). HTTP apps, on the other hand, can run anywhere

ASP.NET Core on AWS Lambda

With all that out of the way, let’s look at how AWS made ASP.NET Core respond to HTTP requests on Lambda. Spoiler alert: It’s a pretty clever blend of the two dogmas outlined above.

Generally serverless “web apps” or APIs are built with Lambda by using an AWS API Gateway (optionally combined with CloudFront for CDN and S3 for static assets) that sends API Gateway Message Events to a Lambda function. The events are basically JSON-formatted HTTP requests, and the HTTP “response” emitted by the function is also JSON formatted. Building a serverless .NET web app on top of that would be pretty frustrating for anyone familiar with ASP.NET because all of the HTTP, MVC, routing and other tooling in ASP.NET would not work.

But here’s the genius: Because the ASP.NET Core framework is fairly well-factored AWS was able to build a HTTP request pipeline frontend (Amazon.Lambda.AspNetCoreServer) that marshals API Gateway Message Events and feeds them into the rest of ASP.NET Core as if they were normal HTTP requests (which, of course, they were before the AWS API Gateway messed them up and turned them into JSON). The AWS blog post has more details and also diagrams (reproduced below) showing the two execution models.

The result is that ASP.NET Core web apps can be debugged and tested locally using the “standard” IIS/Kestrel-based pipeline and then built and deployed using the Amazon.Lambda.AspNetCoreServer based pipeline for production deploys to AWS Lambda. AWS even ships Visual Studio plugins and dotnet new templates that make getting started simple.

While neat, the Lambda approach completely ignores the ideal of dev/prod parity and the execution framework during local testing (with IIS/Kestrel) is very different from the production environment. Somewhat to my surprise I encountered zero problems or abstraction-leaks with the exotic HTTP setup when building and evolving Podnanza, but I suspect that more complex apps that make fuller use of HTTP semantics might see oddities.

Summary

Podnanza has been running without a hitch on AWS Lambda for more than 6 months at the time this post was written, typically costing around $0.20/month including CloudFront and API Gateway use. I’ve pushed multiple tweaks and improvements without issue during that time, always using the dotnet lambda package. On a side-note I admire the AWS .NET team’s zeal in building the Lambda deploy flow into the dotnet tool, but I wonder if it would have made more sense to just add it to the aws CLI that developers use to complete other AWS tasks. Also note that I haven’t built any CI/CD or GitHub-based deployment flow since it’s just me working on and deploying Podnanza. Maybe improving that would be a good way to learn about GitHub Actions…

AWS recently launched improvements to the CloudFront CDN service. The most important change is the option to specify more HTTP headers be part of the cache key for cached content. This lets you run CloudFront in front of apps that do sophisticated Content Negotiation. In this post, we demonstrate how to use this new power to selectively serve WebP or JPEG encoded images through CloudFront depending on client support. It is a follow-up to my previous post on Serving WebP images with ASP.NET. Combining WebP and CloudFront CDN makes your app faster because users with supported browsers get highly compressed WebP images from AWS’ fast and geo-distributed CDN.

The objective of WebP/JPEG content negotiation is to send WebP encoded images to newer browsers that support it, while falling back to JPEG for older browser without WebP support. Browsers communicate WebP support through the Accept HTTP header. If the Accept header value for a request includes image/webp, we can send WebP encoded images in responses.

Prior to the CloudFront improvements, doing WebP/JPEG content negotiation for images fronted with CloudFront CDN was not possible. That’s because CloudFront would only vary response content based on the Accept-Encoding header (and some other unrelated properties). With the new improvements, we can configure arbitrary headers for CloudFront to cache on. For WebP/JPEG content negotiation, we specify Accept in list list of headers to whitelist in the AWS CloudFront console.

Once the rest of the origin and behavior configuration is complete, we can use cURL to test that CloudFront correctly caches and serves responses based on the Accept request header:

curl -v http://abcd1234.cloudfront.net/myimageurl > /dev/null
...
< HTTP/1.1 200 OK
< Content-Type: image/jpeg
...

And simulating a client that supports WebP:

curl -v -H "Accept: image/webp" http://abcd1234.cloudfront.net/myimageurl > /dev/null
...
< HTTP/1.1 200 OK
< Content-Type: image/webp
...

Your origin server should set the Vary header to let any caches in the response-path know how they can cache, although I don’t believe it’s required to make CloudFront work. In our case, the correct Vary header value is Accept,Accept-Encoding.

That’s it! Site visitors will now be receiving compact WebP images (if their browsers support them), served by AWS CloudFront CDN.