Michael Friis' Blog

Here is how I generate sitemaps using the XDocument API and a ContentResult. The entries are events that come out of the EventRepository, please substitute as needed. Note that it would be vastly more elegant to use ActionLinks in some way. Note also that the first entry is a link to a Google Earth KMZ file (more here).

[OutputCache(Duration = 12 * 3600, VaryByParam = "*")]
public ContentResult Sitemap()
{
    string smdatetimeformat = "yyyy-MM-dd";

    var erep = new EventRepository();
    var events = (from e in erep.GetGeocodedEvents()
                    where e.IncidentTime.HasValue
                select new {e.Title, e.PermId, e.IncidentTime}).ToList();

    XNamespace sm = "http://www.sitemaps.org/schemas/sitemap/0.9";
    XNamespace geo = "http://www.google.com/geo/schemas/sitemap/1.0";
            
    XDocument doc = new XDocument(
        new XElement(sm + "urlset",
            new XAttribute("xmlns", "http://www.sitemaps.org/schemas/sitemap/0.9"),
            new XAttribute(XNamespace.Xmlns + "geo", 
                "http://www.google.com/geo/schemas/sitemap/1.0"),
            new XElement(sm + "url",
                new XElement(sm + "loc", "http://krimikort.ekstrabladet.dk/gearth.kmz"),
                new XElement(sm + "lastmod", DateTime.Now.ToString(smdatetimeformat)),
                new XElement(sm + "changefreq", "daily"),
                new XElement(sm + "priority", "1.0"),
                new XElement(geo + "geo",
                    new XElement(geo + "format", "kmz")
                )
            )
            ,
            events.Select(e => 
                new XElement(sm + "url",
                    new XElement(sm + "loc", EventExtensions.AbsUrl(e.Title, e.PermId)),
                    new XElement(sm + "lastmod", e.IncidentTime.Value.ToString(smdatetimeformat)),
                    new XElement(sm + "changefreq", "monthly"),
                    new XElement(sm + "priority", "0.5")
                )
            )
        )
    );

    return Content(doc.ToString(), "text/xml");
}

Shan McArthur put up a notice that the latest version (4.0.12) of the Microsoft CRM SDK includes Linq querying support. The CRM Team have a couple of blog posts describing the new features. I haven’t tested the new SDK, but I definitely recommend you try it out before using LinqtoCRM and I’ve put a notice to that effect on the LinqtoCRM front page.

It’s a little bit sad that LinqtoCRM probably won’t be used much anymore, but I also think it’s great that Microsoft is now providing what looks to be a solid Linq implementation for Dynamics CRM (especially considering the fact that we haven’t released new versions for more than a year).

Anyway, thanks to everyone who have contributed (esp. Mel Gerats and Petteri Räty) and to all the people who have used LinqtoCRM over the years! Now go get the new SDK and write some queries.

If you’re using Linq-to-SQL, doing group-by and selecting other columns than those in the grouping-key, performance might suffer. This is because there is no good translation of such queries to SQL and Linq-to-SQL has to resort to doing multiple subqueries. Matt Warren explains here. I experienced this firsthand when grouping a lot of geocoded events by latitude and longitude and selecting a few more columns (EventId and CategoryId in the example below):

from e in db.Events
group e by new { e.Lat, e.Lng } into g
select new
{
    g.Key.Lat,
    g.Key.Lng,
    es = g.Select(_ => new { _.EventId, _.CategoryId })
};

One possible solution is to fetch all events, to a ToList() and do the grouping in-memory.

var foo =
    from e in db.Events
    select new { e.Lat, e.Lng, e.EventId, e.CategoryId };

var bar = from e in foo.ToList()
            group e by new { e.Lat, e.Lng } into g
            select new
            {
                g.Key.Lat,
                g.Key.Lng,
                es = g.Select(_ => new { _.EventId, _.CategoryId })
            };

Need to geocode addresses using the v3 Google Geocoding Web Service? There are some good reasons to choose the new v3 edition — most importantly, you don’t need an API key. You could use geocoding.net which — at the time of writing —  has some support for v3. I decided to hack up my own wrapper though, and using Windows Communication Foundation, it turned out to be really simple! Note that if you need more of the attributes returned by the Web Service, you should add them to the DataContract classes.

using System;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Json;
using System.Net;
using System.Web;

.
.
.

private static GeoResponse CallGeoWS(string address)
{
	string url = string.Format(
		"http://maps.google.com/maps/api/geocode/json?address={0}&region=dk&sensor=false",
		HttpUtility.UrlEncode(address)
		);
	var request = (HttpWebRequest)HttpWebRequest.Create(url);
	request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
	request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
	DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(GeoResponse));
	var res = (GeoResponse)serializer.ReadObject(request.GetResponse().GetResponseStream());
	return res;
}

[DataContract]
class GeoResponse
{
	[DataMember(Name="status")]
	public string Status { get; set; }
	[DataMember(Name="results")]
	public CResult[] Results { get; set; }

	[DataContract]
	public class CResult
	{
		[DataMember(Name="geometry")]
		public CGeometry Geometry { get; set; }

		[DataContract]
		public class CGeometry
		{
			[DataMember(Name="location")]
			public CLocation Location { get; set; }

			[DataContract]
			public class CLocation
			{
				[DataMember(Name="lat")]
				public double Lat { get; set; }
				[DataMember(Name = "lng")]
				public double Lng { get; set; }
			}
		}
	}
}

If you need to geocode a lot of addresses, you need to manage your request rate. Google will help you throttle requests by returning OVER_QUERY_LIMIT statuses if you are going too fast. I use the method below to manage this. It’s decidedly unelegant, please post a reply if you come up with something better.

private static int sleepinterval = 200;

private static GeoResponse CallWSCount(string address, int badtries)
{
	Thread.Sleep(sleepinterval);
	GeoResponse res;
	try
	{
		res = CallGeoWS(address);
	}
	catch (Exception e)
	{
		Console.WriteLine("Caught exception: " + e);
		res = null;
	}
	if (res == null || res.Status == "OVER_QUERY_LIMIT")
	{
		// we're hitting Google too fast, increase interval
		sleepinterval = Math.Min(sleepinterval + ++badtries * 1000, 60000);

		Console.WriteLine("Interval:" + sleepinterval + "                           \r");
		return CallWSCount(address, badtries);
	}
	else
	{
		// no throttling, go a little bit faster
		if (sleepinterval > 10000)
			sleepinterval = 200;
		else
			sleepinterval = Math.Max(sleepinterval / 2, 50);

		Console.WriteLine("Interval:" + sleepinterval);
		return res;
	}
}

This post will demonstrate how to draw polygons on Google Maps v3 using geojson-encoded data from GeoDjango. The most common method for displaying polygons on Google Maps seems to be by using KML. Google Maps requires the KML-file to be available on a public website though and that is kind of a bore for debugging. This approach uses only json and the standard maps drawing API. To get the display the polygons, you have to loop over them and do a setMap() with your map.

I’m assuming the browser is getting geojson from from a GeoDjango model object like this: model.area.geojson, but it should work for data from other sources. Note that I’m using the very excellent Underscore.js Javascript library to do really terse functional programming. Also note that the function takes an optional bounds object which gets expanded as polygon points are added.

The code seems to perform very well in modern browsers, even for fairly large and complex polygons. Unfortunately there is no live demo, but the site I’m working on should go up soon.

function createPolygons(areajson, bounds){
  var coords = areajson.coordinates;
  var polygons = _(coords).reduce([], function(memo_n, n) {
    var polygonpaths = _(n).reduce(new google.maps.MVCArray(), function(memo_o, o) {
      var polygoncords = _(o).reduce(new google.maps.MVCArray(), function(memo_p, p) {
        var mylatlng = new google.maps.LatLng(p[1], p[0]);
        if(bounds){
          bounds.extend(mylatlng);
        }
        memo_p.push(mylatlng);
        return memo_p;
      });
      memo_o.push(polygoncords);
      return memo_o;
    });
    var polygon = new google.maps.Polygon({
      paths: polygonpaths,
      strokeColor: "#808080",
      strokeOpacity: 0.8,
      strokeWeight: 2,
      fillColor: "#C0C0C0",
      fillOpacity: 0.35
    });
    memo_n.push(polygon);
    return memo_n;
  });
  return polygons;
}

I really like the way Javascript is moving from being this annoying thing you have to deal with when doing web-development to becoming a proper server-side programming languages with standard libraries and fast VMs. Yesterday I cloned the Narwhal git repo and tried to create my own package with a few simple collection types. Narwhal is done by the guys at 280north and conforms to the CommonJS standard, an attempt at a cross-platform standard by the various Javascript platform implementors.

I haven’t quite figured out what goes into creating a proper package, but if you check out and configure Narwhal, you can dump the collection implementation in /lib and the tests in /tests and see it all work. First the collections:

// -- friism Michael Friis

exports.Stack = function() {
  return new stack();
};

exports.Queue = function() {
  return new queue();
};

/**
 * Stack implementation, using native Javascript array
 */

function stack() {
  this.data = [];
};

stack.prototype.pop = function() {
  return this.data.pop();
};

stack.prototype.push = function(o) {
  this.data.push(o);
};

/**
 * Queue implementation. Mechanics lifted from Wikipedia
 * Could be optimised to do fewer slices, see here: 
 * http://safalra.com/web-design/javascript/queues/Queue.js
 */

function queue() {
  this.data = [];
  this.length = 0;
}

queue.prototype.isEmpty = function() {
  return (this.data.length == 0);
};

queue.prototype.enqueue = function(obj) {
  this.data.push(obj);
  this.length = this.data.length;
}

queue.prototype.dequeue = function() {
  var ret = this.data[0];
  this.data.splice(0,1);
  this.length = this.data.length;
  return ret;
}

queue.prototype.peek = function() {
  return this.data[0];
}

queue.prototype.clear = function() {
  this.length = 0;
  this.data = [];
}

… and the tests:

var assert = require("test/assert");

var js5 = require("js5");

exports.testStackSimple = function() {
  var mystack = new js5.Stack();
  var myobject = "1";
  mystack.push(myobject);
  var popped = mystack.pop();
  assert.isEqual(myobject, popped);
};

exports.testStack = function() {
  var mystack = new js5.Stack();
  for(var i = 99; i >= 0; i--) {
    mystack.push(i.toString());
  }

  for(var j = 0; j < 100; j++) {
    assert.isEqual(mystack.pop(), j.toString());
  }
}

exports.testQueue = function() {
  var myqueue = new js5.Queue();
  for(var i = 0; i < 100; i++) {
    myqueue.enqueue(i.toString());
  }

  for(var j = 0; j < 100; j++) {
    assert.isEqual(myqueue.dequeue(), j.toString());
  }
}

if (module == require.main) {
    require("os").exit(require("test").run(exports));
}

I asked for help on the Narwhal IRC channel and Kris Kowal pointed me to Chiron, a module library he's working on that already contains set and dictionary implementations. I recommend checking out the code, it highligts some of the interesting challenges of implementing collections in Javascript.

Also, this "Javascript, the Good Parts" talk by Doug Crockford (author of book of the same name) is really good:

Last fall, I wrote a chapter for a book titled “State of the eUnion”. My chapter is called “Democracy 2.0” and is about how sites like Folkets Ting, OpenCongress and TheyWorkForYou get built and what features should go into them. The other chapters are about the challenges and possibilities of governments and the Internet in general. They are written by people like Tim O’Reilly, Lawrence Lessig and David Weinberger — very humbling company. You can download a pdf or buy a copy on Amazon.

The whole Folkets Ting business has turned out rather well (even though the site is not currently updated — we’re working on it!) and I’ve been invited to speak on a few occations. Some of the talks were recorded, and in the interest of self-agrandissement they are included below in chronological order (except for the last one).

Short interview at Halvandet, the day before Reboot11 started:

Talk at Reboot11:

Talk (in Danish) at HeadStart morning inspiration-session in Århus:

Short blurb (in Danish) on what I think about the usefulness of public data at the ODIS conference:

Speech on “Political Data API” after a project of mine won a competition promoting reuse of public data (winners were announced at the conference mentioned above):

You can watch the same video with slides here

And finally, a non Folkets Ting video where I talk about TEDBot, recorded at the “Berlin in October” un-conference:

Denmark somehow seems to have hatched more programmers and language designers of note than one would expect of a country of 6 million. Since almost none of them live in Denmark, it is kind of easy to forget. Here’s a partial list (alphabetical, inclusion determined by my completely whimsical notions of famousness, reasons for inclusion may be somewhat exaggerated):

• Anders Hejlsberg (came up with Turbo Pascal, lead architect for C#)
• Bjarne Stoustrup (came up with C++)
• David Heinemeier Hansson (created Ruby on Rails)
• Lars and Jens Rasmussen (came up with Google Maps and Google Wave)
• Lars Bak (created V8 Javascript VM for Google Chrome)
• Peter Naur (Turing Award winner, contributed to ALGOL 60)
• Rasmus Lerdorf (created PHP, born on/in Greenland)

(This summer I applied for the “Coders Wanted” Knight Foundation Scholarship at the Medill School of Journalism. In case anyone’s interested, I’m uploading the essays I wrote for my application.)

Question: In the new media landscape, it’s possible for anyone to do the things that professional journalists do: for instance, dig up information other people are interested in, shoot photos or video of newsworthy events, and publish their work for others to see. What is the role of the professional journalist in this world where anyone can publish? What should be the relationship between the professionals and “citizen journalists”?

The relationship between citizen and professional journalist was put to the test by media coverage of the protest and disturbances that followed in the wake the recent Iranian election. Most professional journalists and photographers from western media had been ejected from Iran or were in other ways prevented from filing stories from the country. A lot of media coverage ended up being built on information from Iranian bloggers and Twitter-users and from videos posted to YouTube and similar online services.

While Iran is certainly an extreme case, events there underscore the trend that breaking news coverage is increasingly handled by so called “citizen journalists”. There are several reasons for professional journalists being less around when stuff happens. First, repressive governments, aware of the explosive role of media, may simply ban journalists. This was what happened in Iran and also — arguably — in Gaza in 2008-2009. Second, news organizations today do not have the resources to support a large and dispersed network of journalists deployed around the world. Last, the probability that a citizen will be on the spot with an Internet-connected videophone when something newsworthy happens, is just much greater than a news-team being nearby.

Because “breaking news” and setting the agenda with “exclusive” stories has traditionally been a point of competition among professional journalists, this development seems to be causing cases of Twitter-envy at some news-organizations. The result can be thinly sourced news based on random tweets and un-dated YouTube videos. Indeed, the news-hounds on Twitter and other places demand that journalists pick up these stories, as coverage on, say, CNN is considered a validation of the seriousness of what is going on. Journalists and editors at CNN caught an earful from Twitter users and bloggers for a perceived lack of Iran-coverage immediately after the election, in spite of there being preciously little verifiable information to report at the time.

In my opinion, it would behoove journalists and editors to refrain from propagating largely unsubstantiated news found on social media platforms, even when — as is typically done now — they are presented with large disclaimers. The trouble with these stories is that they add value for no one:  The news-junkie with an interest in the topic at hand will invariably already be well informed, while the casual observer will only understand that, apparently someone on Twitter is writing about an event alleged to have happened just now. Worse, even with disclaimers (and partly because of them), the credibility of professional news organisations suffer when some stories turn out to be false or outright scams.

This is not to say that professionals cannot draw on citizen journalists when piecing together stories. A foreign correspondent analyzing the situation in Iran could very well discuss third-hand reports from citizens, but in a critical manner and not, on its own, as the primary source. Reports could also be carefully augmented with video and pictures shot by citizens. This critical and cautious approach may sometimes be construed by opinionated citizens as professional journalists’ arrogance and aloofness. To avoid this, professionals should reach out and educate about their need for credibility and their commitment to fair and balanced reporting.

The optimal relationship would have professional journalists that are continually being kept to task by engaged citizens who, on the other hand, are encouraged by the same journalists to file credible and (if possible) verifiable photos, videos and eyewitness accounts. This will give media users access to a range of news-sources, from reasoned analysis by journalists, corroborated and augmented by citizen reports, to drinking straight from the pipe of raw and opinionated coverage flowing out of Twitter, YouTube or whatever other platform that is in vogue. Some professional journalists already embrace this development and I think Rick Sanchez of CNN says it particularly well when defending that channels Iran-coverage in the last 30 seconds of this clip.

An excellent example of citizen and professional journalists working together is found in a recent unravelling of a string of cases of medical malpractice in Denmark. Two journalists were contacted by a couple whose infant child had died some time after swallowing a battery. The parents had pleaded with doctors to examine their child, to no avail. Their complaint about the lack of treatment had also been turned down (Denmark has a single-provider health care system which is sometimes not very receptive to criticism). The journalists saw they had a powerful and emotional story, but wanted to find out if it was part of a trend or just a lone case. To that end, they created a Facebook group where people could volunteer similar stories. This unearthed a string of malpractice cases where complaints had also fallen on deaf ears. These were duly investigated and yielded a series of articles on medical negligence and ignored complaints. The journalists continued to use the Facebook group as a sounding board for new article angles and ideas and for soliciting feedback. Investigating and building this sort of story, while not impossible, would certainly have been very time consuming without the active participation of involved citizens.

While labeling people volunteering stories on a Facebook group “citizen journalists” may be a bit thick, they do form part of a continuum that extends over Twitter-users and YouTubers to bloggers. In the end, the professional journalists could write a string of explosive articles, citizens got their previously ignored stories told and all Danes will hopefully get better health care as a result.

What is the role then of the professional journalist confronted with wired, media-savvy and outspoken citizens? Journalists should insist on their commitment to provide fair and balanced reporting with integrity, even in the face of demands for speedy coverage of events that may or not be breaking right now. They should also reach out and tab into the wealth of information and opinion provided by citizen journalists and use it to augment and improve the stories they create.