Running .NET apps on Docker

Posted on | | 3 Comments on Running .NET apps on Docker

This blog post covers running simple .NET apps in Docker lightweight containers using Mono. I run Docker in a Vagrant/VirtualBox VM on Windows. This works great and is fast. Installation instructions are available on the Docker site.

Building the base image

First order of business is to create a Docker image that has Mono installed. We will use this as the base image for containers that actually run apps. To get the most recent Mono version (3.2.6 at the time of writing) I use packages created by Timotheus Pokorra installed on a Ubuntu 12.04 LTS Docker image. Here’s the Dockerfile for that:

FROM ubuntu:12.04
MAINTAINER friism

RUN apt-get -y -q install wget
RUN wget -q http://download.opensuse.org/repositories/home:tpokorra:mono/xUbuntu_12.04/Release.key -O- | apt-key add -
RUN apt-get remove -y --auto-remove wget
RUN sh -c "echo 'deb http://download.opensuse.org/repositories/home:/tpokorra:/mono/xUbuntu_12.04/ /' >> /etc/apt/sources.list.d/mono-opt.list"
RUN apt-get -q update
RUN apt-get -y -q install mono-opt

Here’s what’s going on:

  1. Install wget
  2. Add repository key to apt-get
  3. remove wget
  4. Add openSUSE repository to sources list
  5. Install Mono from there

At first I did all of the above in one command since these steps represent the single logical step of installing Mono and it seems like they should be just one commit. Nobody will be interested in the commit after wget was installed, for example. I ended up splitting it up into separate RUN commands since that’s what other people seem to do.
With that Dockerfile, we can build an image:

$ docker build -t friism/mono .

We can then run a container using the generated image and check our Mono installation:

vagrant@precise64:~/mono$ docker run -i -t friism/mono bash
root@0bdca65e6e8e:/# /opt/mono/bin/mono --version
Mono JIT compiler version 3.2.6 (tarball Sat Jan 18 16:48:05 UTC 2014)

Note that Mono is installed in /opt and that it works!

Running a console app

First, we’ll deploy a very simple console app:

using System;

namespace HelloWorld
{
	public class Program
	{
		static void Main(string[] args)
		{
			Console.WriteLine("Hello World");
		}
	}
}

For the purpose of this example, we’ll pre-build apps using the VS command prompt and msbuild and then add the output to the container:

msbuild /property:OutDir=C:\tmp\helloworld HelloWorld.sln

Alternatively, we could have invoked xbuild or gmcs from within the container.

The Dockerfile for the container to run the app is extremely simple:

FROM friism/mono
MAINTAINER friism

ADD app/ .
CMD /opt/mono/bin/mono `ls *.exe | head -1`

Note that it relies on the friism/mono image created above. It also expects the compiled app to be in the /app folder, so:

$ ls app
HelloWorld.exe

The CMD will simply use mono to run the first executable found in the build output. Let’s build and run it:

$ docker build -t friism/helloworld-container .
...
$ docker run friism/helloworld-container
Hello World

It worked!

Web app

Running a self-hosting OWIN web app is only slightly more work. For this example I used the sample code from the OWIN/Katana-on-Heroku post.

$ ls app/
HelloWorldWeb.exe  Microsoft.Owin.Diagnostics.dll  Microsoft.Owin.dll  Microsoft.Owin.Host.HttpListener.dll  Microsoft.Owin.Hosting.dll  Owin.dll

The Dockerfile for this exposes port 5000 from the container, and CMD is used to start the web app and specify port 5000 to listen on:

FROM friism/mono
MAINTAINER friism

ADD app/ .
EXPOSE 5000
CMD ["/opt/mono/bin/mono", "HelloWorldWeb.exe", "5000"]

We start the container and map port 5000 to port 80 on the machine running Docker:

$ docker run -p 80:5000 -t friism/mono-hello-world-web

And with that we can visit the OWIN sample site on http://localhost/.

If you’re a .NET developer this post will hopefully have helped you place Docker in the context of your everyday work. If you’re interested in more ideas on how to use Docker to deploy apps, check out this Automated deployment with Docker – lessons learnt post.

Serving WebP images with ASP.NET MVC

Posted on | | 3 Comments on Serving WebP images with ASP.NET MVC

Speed is a feature and one thing that can slow down web apps is clients waiting to download images. To speed up image downloads and conserve bandwidth, the good folks of Google have come up with a new image format called WebP (“weppy”). WebP images are around 25% smaller in size than equivalent images encoded with JPEG and PNG (WebP supports both lossy and lossless compression) with no worse perceived quality.

This blog post shows how to dynamically serve WebP-encoded images from ASP.NET to clients that support the new format.

One not-so-great way of doing this is to serve different HTML depending on whether clients supports WebP or not, as described in this blog post. As an example, clients supporting WebP would get HTML with <img src="image.webp"/> while other clients would get <img src="image.jpeg"/>. The reason this sucks is that the same HTML cannot be served to all clients, making caching harder. It will also tend to pollute your view code with concerns about what image formats are supported by browser we’re rendering for right now.

Instead, images in our solution will only ever have one url and the content-type of responses depend on the capabilities of the client sending the request: Browsers that support WebP get image/webp and the rest get image/jpeg.

I’ll first go through creating WebP-encoded images in C#, then tackle the challenge of detecting browser image support and round out the post by discussing implications for CDN use.

Serving WebP with ASP.NET MVC

For the purposes of this article, we’ll assume that we want to serve images from an URI like /images/:id where :id is some unique id of the image requested. The id can be used to fetch a canonical encoding of the image, either from a file system, a database or some other backing store. In the code I wrote to use this, the images are stored in a database. Once fetched from the backing store, the image is re-sized as desired, re-encoded and served to the client.

At this point, some readers are probably in uproar: “Doing on-the-fly image fetching and manipulation is wasteful and slow” they scream. That’s not really the case though, and even if it were, the results can be cached on first request and then served quickly.

Assume we have an Image class and method GetImage(int id) to retrieve images:

private class Image
{
	public int Id { get; set; }
	public DateTime UpdateAt { get; set; }
	public byte[] ImageBytes { get; set; }
}

We’ll now use the managed API from ImageResizer to resize the image to the desired size and then re-encode the result to WebP using Noesis.Drawing.Imaging.WebP.dll (no NuGet package, unfortunately).

public ActionResult Show(int imageId)
{
	var image = GetImage(imageId);

	var resizedImageStream = new MemoryStream();
	ImageBuilder.Current.Build(image.ImageBytes, resizedImageStream, new ResizeSettings
	{
		Width = 500,
		Height = 500,
		Mode = FitMode.Crop,
		Anchor = System.Drawing.ContentAlignment.MiddleCenter,
		Scale = ScaleMode.Both,
	});

	var resultStream = new MemoryStream();
	WebPFormat.SaveToStream(resultStream, new SD.Bitmap(resizedImageStream));
	resultStream.Seek(0, SeekOrigin.Begin);

	return new FileContentResult(resultStream.ToArray(), "image/webp");
}

System.Drawing is referenced using using SD = System.Drawing;. The controller action above is fully functional and can serve up sparkling new WebP-formatted images.

Browser support

Next up is figuring out whether the browser requesting an image actually supports WebP, and if it doesn’t, respond with JPEG. Luckily, this doesn’t involve going back to the bad old days of user-agent sniffing. Modern browsers that support WebP (such as Chrome and Opera) send image/webp in the accept header to indicate support. Ironically given that Google came up with WebP, the Chrome developers took a lot of convincing to set that header in requests, fearing request size bloat. Even now, Chrome only advertises webp support for requests that it thinks is for images. In fact, this is another reason the “different-HTML” approach mentioned in the intro won’t work: Chrome doesn’t advertise WebP support for requests for HTML.

To determine what content encoding to use, we inspect Request.AcceptTypes. The resizing code is unchanged, while the response is generated like this:

	var resultStream = new MemoryStream();
	var webPSupported = Request.AcceptTypes.Contains("image/webp");
	if (webPSupported)
	{
		WebPFormat.SaveToStream(resultStream, new SD.Bitmap(resizedImageStream));
	}
	else
	{
		new SD.Bitmap(resizedImageStream).Save(resultStream, ImageFormat.Jpeg);
	}

	resultStream.Seek(0, SeekOrigin.Begin);
	return new FileContentResult(resultStream.ToArray(), webPSupported ? "image/webp" : "image/jpeg");

That’s it! We now have a functional controller action that responds correctly depending on request accept headers. You gotta love HTTP. You can read more about content negotiation and WebP on lya Grigorik’s blog.

Client Caching and CDNs

Since it does take a little while to perform the resizing and encoding, I recommend storing the output of the transformation in HttpRuntime.Cache and fetching from there in subsequent requests. The details are trivial and omitted from this post.

There is also a bunch of ASP.NET cache configuration we should do to let clients cache images locally:

	Response.Cache.SetExpires(DateTime.Now.AddDays(365));
	Response.Cache.SetCacheability(HttpCacheability.Public);
	Response.Cache.SetMaxAge(TimeSpan.FromDays(365));
	Response.Cache.SetSlidingExpiration(true);
	Response.Cache.SetOmitVaryStar(true);
	Response.Headers.Set("Vary",
		string.Join(",", new string[] { "Accept", "Accept-Encoding" } ));
	Response.Cache.SetLastModified(image.UpdatedAt.ToLocalTime());

Notice that we set the Vary header value to include “Accept” (as well as “Accept-Encoding”). This tells CDNs and other intermediary caching proxies that if they try to cache this response, they must vary the cached value based on value of the “Accept” header of the request. This works for “Accept-Encoding”, so that different values can cached based on whether the response is compressed with gzip, deflate or not at all, and all major CDNs support it. Unfortunately, the mainstream CDNs I experimented with (CloudFront and Azure CDN) don’t support other Vary values than “Accept-Encoding”. This is really frustrating, but also somewhat understandable from the standpoint of the CDN folks: If all Vary values are honored, the number of artifacts they have to cache would increase at a combinatorial rate as browsers and servers make use of cleverer caching. Unless you find a CDN that specifically support non-Accept-Encoding Vary values, don’t use a CDN when doing this kind of content negotiation.

That’s it! I hope this post will help you build ASP.NET web apps that serve up WebP images really quickly.

Heroku .NET buildpack update to Mono 3.2 and more

Posted on | | 1 Comment on Heroku .NET buildpack update to Mono 3.2 and more

Hot on the heels of posts on Running .NET on Heroku and Running OWIN/Katana apps on Heroku, I’m happy to announce a couple of updates to the buildpack:

  1. Now uses the newly released Mono 3.2 runtime. I need to figure out some way for users to select what version they want the same way the Python buildpack uses runtime.txt.
  2. Adds symbolic links from NuGet.targets files to nuget.targets files to account for inconsistent casing practices in NuGet.
  3. Fetches Nuget.targets file that is not broken when doing package restore on Linux. I’m still trying to get Microsoft to accept a pull request to NuGet that fixes this.

I’ve also spent some time trying to switch from xsp4 to nginx and fastcgi-mono-server4, but am currently stuck.

Running OWIN/Katana apps on Heroku

Posted on | | 7 Comments on Running OWIN/Katana apps on Heroku

This post demonstrates how to get a an OWIN/Katana sample app running on Heroku. It uses the Mono buildpack that I’ve already covered on the blog. As far as I can determine, Heroku is the only public platform where you can conveniently prototype and share OWIN apps running on Katana.

OWIN and Katana

OWIN is the Open Web Interface for .NET, a specification for how apps and frameworks should interact with servers that host them. It’s very much like Ruby Rack, only for .NET. By decoupling apps from web servers, it should become much easier to write portable middleware that can be reused between apps and  frameworks. There’ already a ton of good OWIN stuff on NuGet.

Katana is a set of components (including a web server) built by Microsoft and others that conforms to the OWIN spec. Here’s a great overview write-up about Katana. There are other OWIN compliant servers (like Kayak) and hopefully more will show up.

OWIN and Microsoft adopting OWIN is pretty exciting: It will make writing good .NET web apps easier and it’s going to free .NET web developers from their historical over-reliance on the IIS web server. With IIS out of the picture, hosting apps on Mono and on open platforms like Heroku will be much easier.

Getting started

The sample demoed here is an expanded version K. Scott Allen’s sample. Here’s how to get started:

  1. Open Visual Studio and Create a C# Console Application and enable NuGet package restore. I used Visual Studio 2012 and named the app KatanaTest.
  2. Install OWIN NuGet packages
    install-package Microsoft.Owin.Hosting -IncludePreRelease
    install-package Microsoft.Owin.Host.HttpListener –IncludePreRelease
    install-package Microsoft.Owin.Diagnostics –IncludePreRelease
    install-package Owin.Extensions -IncludePrerelease
  3. Add a class that configures the app to just display the Katana sample page.
  4. Add a Main entry point that listens for HTTP on a port specified as a command line argument and waits for ctrl+c to exit.
  5. Add a Procfile that tells Heroku how to start the app, passing in the $PORT as an argument.
  6. Fix casing bug and restore command to make project build on Heroku (these will hopefully soon be fixed in NuGet). (Fixed in latest buildpack version)
  7. Deploy to Heroku:
        $ heroku create
        $ heroku config:add BUILDPACK_URL=https://github.com/friism/heroku-buildpack-mono/
        $ git push heroku master

Check out the sample running live on Heroku. The sample project is available on GitHub.

Running .NET on Heroku

Posted on | | 7 Comments on Running .NET on Heroku

Since joining Heroku I’ve been wanting to get .NET running on the platform and I’m happy to report that we now have a reasonably workable Mono buildpack. My goal was to be able to take an ASP.NET MVC solution created with Visual Studio on Windows, create a Heroku app, run git push heroku master and have Heroku build, deploy and run the app using Mono and the XSP web server.

The result is based heavily on previous work by my colleague Brandur.

Getting started

To use the .NET buildpack, create an ASP.NET MVC 4 web site and enable NuGet package restore. There are then a few tweaks required to make the solution palatable to Mono and xbuild (striked-out issues have been fixed in buildpack and are not necessary anymore):

Hopefully, we can get these obstacles eliminated through improvements to either Mono, NuGet or the buildpack.

Now, deploy to Heroku:

    $ heroku create
    $ heroku config:add BUILDPACK_URL=https://github.com/friism/heroku-buildpack-mono/
    $ git push heroku master

I’ve created a few samples that are known to work. TestingMono is an extremely simple MVC site with a background worker that logs a message once every second. To run the background worker, add a Procfile that defines the command worker: mono WorkerTest.exe and scale it to 1 with heroku scale worker=1. The other sample is forked from an AppHarbor sample and demonstrates simple use of a Heroku PostgreSQL database. Note that the connectionstring has to be read from the environment, not Web.config as is usual in .NET. You can find the sample running here.

Overview

Here’s what works:

  • Running ASP.NET MVC 4 on top of Mono 3.0.11 and XSP 3.0.11
  • NuGet package restore so you don’t have to include library dependencies in your repo
  • Caching of build output and incremental builds, including caching of already-downloaded NuGet packages
  • Running executables in background worker dynos

Here’s what needs love:

  • Insertion of config into appSettings in Web.config
  • Make more of the default Visual Studio templates work out of the box
  • Look for XSP replacement (likely nginx)

Also see the TODO in the README. Contributions are very welcome. I hope to cover how buildpack dependencies (Mono and XSP in this case) are generated in a future blog post.

And yes, I’m working on getting Visual Basic running.

Compressed string storage with NHibernate

Posted on | | 1 Comment on Compressed string storage with NHibernate

This blog post demonstrates how to use a IUserType to make NHibernate compress strings before storing them. It also shows how to use an AttributeConvention to configure the relevant type mapping.

By compressing strings before storing them you can save storage space and potentially speed up your app because fewer bits are moved on and off physical storage. In this example, compression is done using the extremely fast LZ4 algorithm so as to not slow data storage and retrieval.

The downside to compressing string stored in the database is that running ad-hoc SQL queries (such is mystring like '%foo%') is not possible.

Background

I was building an app that was downloading and storing lots HTML and for convenience I was storing the HTML in a SQL Server database. SQL Server has no good way to compress stored data so the database files grew very quickly. This bugged me because most of the content would compress well. I was using Entity Framework and started throwing around for ways to hook into how EF serializes data or for a way to create a custom string type that could handle the compression. Even with the EF6 pre-releases, I couldn’t find any such hooks.

NHibernate IUserType

So I migrated to NHibernate which lets you define custom datatypes and control how they’re stored in the database by implementing the IUserType interface. The best tutorial I’ve found for implementing IUserType is this one by Jacob Andersen. You can check out my full implementation of a compressed string IUserType on GitHub. The two most interesting methods are NullSafeGet() and NullSafeSet():

	public object NullSafeGet(IDataReader rs, string[] names, object owner)
	{
		var value = rs[names[0]] as byte[];
		if (value != null)
		{
			var deCompressor = LZ4DecompressorFactory.CreateNew();
			return Encoding.UTF8.GetString(deCompressor.Decompress(value));
		}

		return null;
	}

	public void NullSafeSet(IDbCommand cmd, object value, int index)
	{
		var parameter = (DbParameter)cmd.Parameters[index];

		if (value == null)
		{
			parameter.Value = DBNull.Value;
			return;
		}

		var compressor = LZ4CompressorFactory.CreateNew();
		parameter.Value = compressor.Compress(Encoding.UTF8.GetBytes(value as string));
	}

The actual compression is done by LZ4Sharp which is a .NET implementation of the LZ4 compression algorithm. LZ4 is notable, not for compressing data a lot, but for compressing and uncompressing data extremely quickly. A single modern CPU core can LZ4-compress at up to 300 MB/s and uncompress much faster. This should minimize the overhead of compressing and uncompressing data as it enters and leaves the database.

For SqlTypes we use BinarySqlType(int.MaxValue):

	public SqlType[] SqlTypes
	{
		get { return new[] { new BinarySqlType(int.MaxValue) }; }
	}

This causes the type to be mapped to a varbinary(max) column in the database.

Mapping

To facilitate mapping, we’ll use an Attribute:

	[AttributeUsage(AttributeTargets.Property)]
	public class CompressedAttribute : Attribute
	{
	}

And an AttributeConvention for FluentNHibernate to use:

	public class CompressedAttributeConvention : AttributePropertyConvention
	{
		protected override void Apply(CompressedAttribute attribute, IPropertyInstance instance)
		{
			if (instance.Property.PropertyType != typeof(string))
			{
				throw new ArgumentException();
			}

			instance.CustomType(typeof(CompressedString));
		}
	}

Here’s how to use the convention with AutoMap:

	var autoMap = AutoMap.AssemblyOf()
		.Where(x => typeof(Entity).IsAssignableFrom(x))
		.Conventions.Add(new CompressedAttributeConvention());

The full SessionFactory is on GitHub.

With this, we get nice, clean entity classes with strings that are automatically compressed when stored:

	public class Document : Entity
	{
		[Compressed]
		public virtual string Text { get; set; }
	}

Limitations

As mentioned in the introduction you can’t do ad-hoc SQL queries because compressed strings are stored in the database as binary blobs. Querying with NHibernate as also somewhat limited. Doing document.Text == "foo" actually works because NHibernate runs “Foo” through the compression. Queries that involve Contains() will (silently) not work, unfortunately. This is because NHibernate translates this to a like query, which won’t work with the compressed binary blob. I haven’t looked into hooking into the query engine to fix this.

ASP.Net MVC Layar layer, ghetto-style

Posted on | | Leave a Comment on ASP.Net MVC Layar layer, ghetto-style

Layar is a really great meta-app for iPhone and Android that lets you see a lot of third-party geo-based augmented reality layers on your phone. A “layar” consists of a JSON webservice that provides Points of Interest to users. There is HttpHandler implementation available for .Net, but the Layar specification is so simple (in the good sense of the word) that I decided to just whip up my own in a MVC controller. Computing distances is pretty akward using LinqtoSQL and SQL Server, I use the DistanceBetween function described here. It is used by the FindNearestEvents stored procedure in the code below.

public class LayarController : Controller
{
    public ActionResult GetPOIs(string lat, string lon, 
        string requestedPoiId, string pageKey)
    {
        var db = new DatabaseDataContext();

        int? page = null;
        if (!string.IsNullOrEmpty(pageKey))
        {
            page = int.Parse(pageKey);
        }

        var eventssp = db.FindNearestEvents(
            float.Parse(lat, NumberStyles.Float, CultureInfo.InvariantCulture),
            float.Parse(lon, NumberStyles.Float, CultureInfo.InvariantCulture),
            20, page ?? 0);

        var events = eventssp.Select(e => new POI()
        {
            lat = e.Lat.Value.ToLayarCoord(),
            lon = e.Lng.Value.ToLayarCoord(),
            distance = e.Distance.Value,
            id = e.PermId,
            title = e.Title,
            line2 = e.BodyText,
            attribution = "Ekstra Bladet Krimikort"
        }).ToList();

        return this.Json(
            new Response
            {
                radius = (int)(events.Max(e => e.distance) * 1000),
                nextPageKey = page != null ? page + 1 : 1,
                morePages = events.Count() == 20,
                hotspots = events,
            }, JsonRequestBehavior.AllowGet
            );
    }
}

public class Response
{
    public string layer { get { return "krimikort"; } }
    public int errorCode { get { return 0; } }
    public string errorString { get { return "ok"; } }
    public IEnumerable hotspots { get; set; }
    public int radius { get; set; }
    public int? nextPageKey { get; set; }
    public bool morePages { get; set; }
}

public class POI
{
    public object[] actions { get { return new object[] { }; } }
    public string attribution { get; set; }
    public float distance { get; set; }
    public int id { get; set; }
    public string imageUrl { get; set; }
    public int lat { get; set; }
    public int lon { get; set; }
    public string line2 { get; set; }
    public string line3 { get; set; }
    public string line4 { get; set; }
    public string title { get; set; }
    public int type { get; set; }
}

public static class Extensions
{
    public static int ToLayarCoord(this double coord)
    {
        return (int)(coord * 1000000);
    }
}

Dynamic Sitemap with ASP.Net MVC (incl. geo)

Posted on | | Leave a Comment on Dynamic Sitemap with ASP.Net MVC (incl. geo)

Here is how I generate sitemaps using the XDocument API and a ContentResult. The entries are events that come out of the EventRepository, please substitute as needed. Note that it would be vastly more elegant to use ActionLinks in some way. Note also that the first entry is a link to a Google Earth KMZ file (more here).

[OutputCache(Duration = 12 * 3600, VaryByParam = "*")]
public ContentResult Sitemap()
{
    string smdatetimeformat = "yyyy-MM-dd";

    var erep = new EventRepository();
    var events = (from e in erep.GetGeocodedEvents()
                    where e.IncidentTime.HasValue
                select new {e.Title, e.PermId, e.IncidentTime}).ToList();

    XNamespace sm = "http://www.sitemaps.org/schemas/sitemap/0.9";
    XNamespace geo = "http://www.google.com/geo/schemas/sitemap/1.0";
            
    XDocument doc = new XDocument(
        new XElement(sm + "urlset",
            new XAttribute("xmlns", "http://www.sitemaps.org/schemas/sitemap/0.9"),
            new XAttribute(XNamespace.Xmlns + "geo", 
                "http://www.google.com/geo/schemas/sitemap/1.0"),
            new XElement(sm + "url",
                new XElement(sm + "loc", "http://krimikort.ekstrabladet.dk/gearth.kmz"),
                new XElement(sm + "lastmod", DateTime.Now.ToString(smdatetimeformat)),
                new XElement(sm + "changefreq", "daily"),
                new XElement(sm + "priority", "1.0"),
                new XElement(geo + "geo",
                    new XElement(geo + "format", "kmz")
                )
            )
            ,
            events.Select(e => 
                new XElement(sm + "url",
                    new XElement(sm + "loc", EventExtensions.AbsUrl(e.Title, e.PermId)),
                    new XElement(sm + "lastmod", e.IncidentTime.Value.ToString(smdatetimeformat)),
                    new XElement(sm + "changefreq", "monthly"),
                    new XElement(sm + "priority", "0.5")
                )
            )
        )
    );

    return Content(doc.ToString(), "text/xml");
}

Linq-to-SQL, group-by, subqueries and performance

Posted on | | 2 Comments on Linq-to-SQL, group-by, subqueries and performance

If you’re using Linq-to-SQL, doing group-by and selecting other columns than those in the grouping-key, performance might suffer. This is because there is no good translation of such queries to SQL and Linq-to-SQL has to resort to doing multiple subqueries. Matt Warren explains here. I experienced this firsthand when grouping a lot of geocoded events by latitude and longitude and selecting a few more columns (EventId and CategoryId in the example below):

from e in db.Events
group e by new { e.Lat, e.Lng } into g
select new
{
    g.Key.Lat,
    g.Key.Lng,
    es = g.Select(_ => new { _.EventId, _.CategoryId })
};

One possible solution is to fetch all events, to a ToList() and do the grouping in-memory.

var foo =
    from e in db.Events
    select new { e.Lat, e.Lng, e.EventId, e.CategoryId };

var bar = from e in foo.ToList()
            group e by new { e.Lat, e.Lng } into g
            select new
            {
                g.Key.Lat,
                g.Key.Lng,
                es = g.Select(_ => new { _.EventId, _.CategoryId })
            };

C# and Google Geocoding Web Service v3

Posted on | | 9 Comments on C# and Google Geocoding Web Service v3

Need to geocode addresses using the v3 Google Geocoding Web Service? There are some good reasons to choose the new v3 edition — most importantly, you don’t need an API key. You could use geocoding.net which — at the time of writing —  has some support for v3. I decided to hack up my own wrapper though, and using Windows Communication Foundation, it turned out to be really simple! Note that if you need more of the attributes returned by the Web Service, you should add them to the DataContract classes.

using System;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Json;
using System.Net;
using System.Web;

.
.
.

private static GeoResponse CallGeoWS(string address)
{
	string url = string.Format(
		"http://maps.google.com/maps/api/geocode/json?address={0}&region=dk&sensor=false",
		HttpUtility.UrlEncode(address)
		);
	var request = (HttpWebRequest)HttpWebRequest.Create(url);
	request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
	request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
	DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(GeoResponse));
	var res = (GeoResponse)serializer.ReadObject(request.GetResponse().GetResponseStream());
	return res;
}

[DataContract]
class GeoResponse
{
	[DataMember(Name="status")]
	public string Status { get; set; }
	[DataMember(Name="results")]
	public CResult[] Results { get; set; }

	[DataContract]
	public class CResult
	{
		[DataMember(Name="geometry")]
		public CGeometry Geometry { get; set; }

		[DataContract]
		public class CGeometry
		{
			[DataMember(Name="location")]
			public CLocation Location { get; set; }

			[DataContract]
			public class CLocation
			{
				[DataMember(Name="lat")]
				public double Lat { get; set; }
				[DataMember(Name = "lng")]
				public double Lng { get; set; }
			}
		}
	}
}

If you need to geocode a lot of addresses, you need to manage your request rate. Google will help you throttle requests by returning OVER_QUERY_LIMIT statuses if you are going too fast. I use the method below to manage this. It’s decidedly unelegant, please post a reply if you come up with something better.

private static int sleepinterval = 200;

private static GeoResponse CallWSCount(string address, int badtries)
{
	Thread.Sleep(sleepinterval);
	GeoResponse res;
	try
	{
		res = CallGeoWS(address);
	}
	catch (Exception e)
	{
		Console.WriteLine("Caught exception: " + e);
		res = null;
	}
	if (res == null || res.Status == "OVER_QUERY_LIMIT")
	{
		// we're hitting Google too fast, increase interval
		sleepinterval = Math.Min(sleepinterval + ++badtries * 1000, 60000);

		Console.WriteLine("Interval:" + sleepinterval + "                           \r");
		return CallWSCount(address, badtries);
	}
	else
	{
		// no throttling, go a little bit faster
		if (sleepinterval > 10000)
			sleepinterval = 200;
		else
			sleepinterval = Math.Max(sleepinterval / 2, 50);

		Console.WriteLine("Interval:" + sleepinterval);
		return res;
	}
}

Older Posts