Tag projects (76)

After vacation and traveling and such, I've been working on the bridge program again. I'm at the point in the webOS client where I can now play a complete hand, bidding and all. It even looks relatively nice with cards sliding around, things fading into view, etc. Ask me to show it off if you're interested sometime!

So the next task is adding the AI for playing cards. I've decided to go with a rule-based system, where there are a bunch of rules that can decide if they apply. So I have "second hand low" and "third hand high" rules that go near the end (since they're fairly generic).

Last night I tried to sit down and really hammer out the rule that applies if you can determine exactly what will be played. If you're the last player on the trick, this is relatively easy - see if your partner is winning, if not win as cheaply as you can, if so (or if you can't win) throw some trash.

Even this is a little vague - what card should I throw away? Well, if there's only one suit I can play it's easy, but if not then it's hard. So I put this off with a TODO.

The tricky part is if you're the third player on the trick and the fourth player is the dummy. Then it's something like: look at all dummy's cards and the declarer's card that was played (if it's currently winning), then see if I can beat all these. If so, play the cheapest. If not....well, we are in the third hand and we'd like to force dummy's best card if we can. So find dummy's second best card and see if we can at least beat that. If not...well, at least we should try to beat the current high card played by declarer. If we can't do that, then just toss something.

--

This took a while to code up, and I need to test it thoroughly because I'm very much not confident in it. And it still leaves a lot of details out - what if dummy has AK6 and I have the 57 - under the algorithm I'd play the 5 which is pretty clearly wrong. It seems like to handle this case I should have been keeping track of which cards are winners in their suit, which is another layer of complexity!

And all this is for a (all things considered) pretty simple case where I can see all the cards. I'd also like to keep track of how many cards of each suit each person has (based on the bidding), which would help but would also add enormous complexity.

Maybe the AI should cheat and see all the cards? Is there some better thing to keep track of?

It could be a long time before the game seems to play intelligently...

I went to Minnesota last week on a recruiting trip and took a few pictures:

Civilization 5 is releasing tomorrow! The full manual is available online (warning: 233 page .pdf) and it look spretty good. I do miss having a paper manual, though.

I've been working on the bridge program for webOS. It's coming along nicely - the framework is all in place to bid and play, and I've done some of the graphical stuff so you can almost actually play a hand on the webOS emulator. The next big thing to do is the AI for bidding and playing, which will probably be big tasks. We're going to be fairly busy over the next month or two so I'm not expecting much progress for a while. (also, Civ 5 coming out certainly doesn't help :-) )

Happily, since webOS is Javascript-based it was pretty easy to make a webpage version of the game, which will make testing out AI algorithms, etc. much easier.

Eric Fischer took census data on race and mapped where people live in major cities. (here's Houston, Austin, and San Antonio) Some patterns definitely jump out at you, but if you zoom in you can see there's nowhere that's exclusively one race, which is a good thing.

After some thought and some more research about what webOS app to work on next, it looks like the mystery option is going to be somewhere between difficult and impossible, and it's going to be a while before I can tell which.

So, bridge it is! It's going to be pretty tough, but a few people have volunteered to help, and it's better to aim high, right? So far it can deal out hands and count points...

I can't decide whether the hardest part will be the AI for bidding, playing, or just making the graphics look nice (with pictures of cards, etc). I guess we'll see!

Having finished We the People and done some small changes to earlier apps, I'm raring to work on a new webOS app for my Palm Pre. (partially feeling invigorated by the announced webOS 2.0 features)

So here are my ideas:

A client to easily browse Reddit.
- Pro: I've played around with it a little and gotten some stuff to work, which is promising.
- Pro: There's a real API which looks pretty easy to interact with.
- Pro/Con: There are already a few existing Reddit clients, although none of them are in the full App Catalog (one's in beta, I believe?)
- Con: It would be a lot of work to make pages look attractive, especially since I suck at it.
- Con: I'm not sure how much more useful it is than just going to the Reddit site in the browser.
- Con: I couldn't see charging more than $1.99 for it, and I'm not sure how many people would be interested in buying it.

A bridge game (probably single-player only, at least at first)
- Pro: There are no existing bridge games in the Catalog. Even in Apple's I only see two.
- Con: That's probably because it's a huge pain to write AI that bids well. And if it doesn't bid well, it's almost useless.
- Pro: I could see charging $5-$10 for it if I spent the time to do it well.
- Con: Bidding aside, it's still a lot of work to put in correct play, proper scoring, fancy card graphics, etc. I'm not convinced I won't give up or lose interest before I'm done.

Mystery option #3, which I just thought of
- Pro: Uses some exciting new features in webOS 2.0, like <redacted>!
- Con: It's not a very original idea, and I bet someone can beat me to it.
- Pro: But it would be kinda fun to write and play around with...and I would use it...
- Con: But I can't start working on it until webOS 2.0 releases, whenever that is.
- Pro/Con: Probably a 99 cent app, although a fairly wide audience.

What do you think? (open to other ideas!)
[ Fill out Poll ] [ View Poll Results ] [ Discuss Results ] [ Close Poll ]

Poll #1614363 webOS app plans
Open to: All, detailed results viewable to: All, participants: 6

View Answers

Reddit client
0 (0.0%)

Bridge game
1 (16.7%)

Mystery option #3
5 (83.3%)

The summer musical went well this weekend, although I was losing my voice by the end on account of doing the pirate accent. Happily, today begins 3 days of no rehearsals/performances, which hasn't happened since we started rehearsals just under a month ago. So, I've been catching up on a few things:

- Pictures from Jonathan and Sarah's wedding are finally up:

- My PasswordHash thing now has a Google Chrome version. Unfortunately right now there's no support for adding things to the right-click context menu, so it just copies the password to the clipboard.

Palm just posted the official results of the Hot Apps competition, and FlightPredictor made the list! Really looking forward to improving it and making some new apps once I have some free time...

Speaking of which, a new life rule: no sushi close to shows. Of course, the one time I get sick from it is during a rehearsal. (nothing overly dramatic, but I did feel pretty crappy and sat down a lot) Hopefully tonight is better - only 2 days until we open!

I've been wanting to play around with Tableau Public, so I decided to revamp my home energy usage graph.

Anyway, the new version is here, and I think it does a good job of showing the difference that having someone upstairs during the day makes, as well as the new A/C unit we bought a few months ago. I used a quadratic fit which seems to fit the data much better than the linear one I used before (you can mouseover the trend lines to see the equations!). Now I just need to wait to get some more data points!

You can click on points in either graph to see the corresponding one in the other, and Tableau makes it easy to download the raw data I used. If you select "True" for "Upstairs during day?" you can really see the effect of the new A/C. (green points vs. orange)

Thanks to Robert Morton for his help in getting it up and running!

We went down to Houston this weekend to see the Young Frankenstein musical (verdict: entertaining but not as good as the movie), and ended up seeing a nice backyard with some bees. Also: back in Austin we saw the original Batman movie at the Paramount with Adam West. Pictures of all the above:

The Palm Hot Apps competition will award cash prizes to the 221 most popular free and paid apps on June 30. I think FlightPredictor has a (very) outside shot at winning, so I bought an ad for it on webOSroundup. (it's one of the boxes on the right - doesn't show up every time so you might have to reload) Thanks to destroyerj and skimmerduk for design help!

One year and one day ago, the Palm Pre went on sale - PreCentral has a a good retrospective on what went wrong and what went right.

The good news: Since my last update, I'm up to 132 copies sold. I've added some new features, including the heavily-requested thumbnails in posts, and I made a video walkthrough of the app that's linked to from the App Catalog. (which is a cool feature - thanks Palm!)

The bad news: The rate of sales is really slowing down - this last week I had three days in a row where no copies were sold, which is the first time that's happened since I've been tracking the daily numbers. Pushing the update helped a little, but I'm getting a feeling that the market may be somewhat saturated - the intersection of "people who own a Pre" and "people who use LiveJournal enough that they're willing to pay a few bucks for a good client" probably isn't that big to begin with. Or maybe it's just a blip; I suppose time will tell.

I was hoping that I might squeak in to the Palm Hot Apps competition, but looking at the leaderboard I'm a ways out from the #200 slot. Right now I'd have to sell 51 more copies to get on the list, and that's only going to go up over time. Oh well!

In any case, I've had a lot of fun working on LJ for WebOS and I'm glad that people seem to generally find it useful. I'm hoping to publish my next app (the FlightCaster-based one) within a few weeks - it's mostly ready to go but I'm waiting on some API changes before I call it done, and then I have to make a video, etc.

A post by someone on Palm's developer relations team sums up well why I like WebOS so much and why I'm going to be a sad panda if it goes away.

My backup strategy as of last week was to copy the really important stuff onto my 16GB USB thumbdrive relatively frequently, and more infrequently copy everything onto an external hard drive. Feeling rather clever at the time, I bought an enclosure like this, so I could buy a cheaper SATA hard drive and stick it in there and plug it into my computer via USB. Then when I wanted to upgrade drives I could just buy another SATA drive.

This sounded good in theory, and indeed kinda worked, but it was very slow for some reason - copying 200GB over to the drive and then untarring the 200GB file took around 3 days. Because of this I backed up way less often than I should have.

As luck would have it, the SATA drive seemed to die last week, and so after doing a little research I bought a Western Digital Elements USB Hard Drive with 1TB of space for $90. It was easy to get it to mount on Linux, and took less than 24 hours to do the same copying/untarring procedure. And it doesn't look totally cobbled together! So I'd recommended it if you're looking for an easy backup solution. You can also get a version with fancy software and an e-ink label on the outside that was actually kind of tempting.

The health care bill passed the House! The plan is for the Senate to pass it this week and then Obama will sign it into law.

The bill: Here's some information on the final bill - it was confusing for a while with the House bill and the Senate bill, but this is the final one. It expands coverage to 32 million uninsured Americans. It prohibits lifetime limits on insurance coverage and denial of coverage due to preexisting conditions. Yes, it's not a perfect bill, and it's hard to predict how well the cost control measures will work, and there's no public option, but there's a lot of good stuff in the bill. And it will be easier to adjust this bill in the future than if we had started over or given up because it wasn't "good enough". As James Fallows said:

LJ for WebOS is currently being featured on the Palm homepage! (it's the lower right of the 3x3 grid)

Screenshot:

My Board Game Geek app for WebOS is now available in the Palm App Catalog!

What If Everybody in Canada Flushed At Once? - or, Canadians really like hockey.

A map of the most common fast food restaurants across the US, made by calculating the "burger force" (proportional to one over distance squared). McDonalds obviously reigns supreme, but Sonic makes a pretty good run in Texas!

LOST: Baywatch intro

The subscriber rates cable companies pay to carry each channel. ESPN and Fox Sports Net are by far the most expensive, and even the FOX Soccer Channel (which apparently exists) costs more than Comedy Central.

Florida (like many places) offers tax credits if you film a movie or TV show there. Now they're considering a bill that would deny these credits if the show has a gay character. (among other things)

As a Michigan fan, I'm not sure how to take the fact that Jim Tressel (the Ohio State football coach) answered questions from a local gay publication. (and said some very nice things) He's probably the first major coach to do so.

An interview with David Boies and Ted Olson about their Prop 8 case. Nothing earthshaking, but it's interesting to hear them talk about the trial and what they think their chances are. They're expecting a ruling by late April/early May.

Yesterday, we were driving home from work as usual on the onramp to North Mopac just south of Duval when the car in front of me started slowing down and then came to a complete stop. On the onramp.

Luckily, I was watching and was able to stop in time. Even more luckily, the car behind me was barely able to stop in time as well. (David says he could hear them slam on their brakes, but I was focused on stuff in front of me) Then the crazy honking began, both from the car behind me and me, and the car pulled off into the shoulder. I'm guessing they stopped because the sun was just at the right angle so that you couldn't see anything in the side view mirror, which made getting on the freeway from a dead stop rather challenging. Luckily no one hit me.

I was pretty amped up from this and my heart was definitely beating faster than normal. Unfortunately, this continued for quite a while. Apparently reflux can present this way, and taking some antacids seemed to help for a while, but it made me jumpy all evening and made it hard to sleep...

I worked on a new WebOS app last night - this one's a quick interface to BoardGameGeek. To submit it, I had to make the decision again about where to put it. To review:
- To put it on the App Catalog costs $50 (one time fee)
- Not doing that is free, but then it's a lot harder to find without a direct link
I prefer putting my apps on the App Catalog for the widest exposure, but then I feel like I have to charge something to make my $50 back. (and it's not that I really begrudge the $50, I understand it takes Palm time to review the apps, etc.) So I did that and am charging .99, the lowest allowable. We'll see what happens1

Good lost episode last night: At this point my favorite episodes are Locke, Ben, and Sawyer ones. Michael Emerson (actor who plays Ben) is amazing and really sold me on the whole Machiavellian teacher bit. Lots of sweetness when he says "Because no one else will have me" and Ilana replies "I'll have you". Awww!

The excessive "winking" at the audience is getting to be a bit much. I'm fine with Hurley acting as a fan of the show and asking a question about something every once and a while, but asking if Richard was a cyborg seemed kinda silly.

A Lost encyclopedia is coming out in August! I know it makes sense to release it after the end of the show since then it can be "complete", but it would have been helpful earlier. Still, I'll probably pick it up.

Random note: Like Greek food? Try Pars Deli at the corner of 183 and Burnet. Delicious gyros, and it's clearly a family run place. They left the door open today and it was soooo nice :-)

House stuff: We had solar screens installed by Josh Hobbs a few weeks ago. The process was pretty easy (we didn't have to be there any of the time), the prices were reasonable and the screens look nice. (need to take a picture at some point) Haven't really been able to measure their effectiveness yet, but it's definitely darker, so presumably that will keep out heat too.

Next week we're getting a high efficiency A/C unit put in, as well as making it "multi-zone" so we can control the upstairs and downstairs temperature independently. After crunching the numbers it's not going to quite pay for itself in reduced electricity bills, but it will get pretty close and we get a new unit out of the deal. (we spend a fortune on cooling in the summer!) The $1500 tax credit helps, and Austin Energy has some nice rebates too.

Bought the Heavy Rain soundtrack from iTunes. (yay for DRM-free music!) I might try to transcribe the really pretty piano music (Painful Memories), if I feel inspired enough.

Final Fantasy 13 was bought and played yesterday. Even after just playing Heavy Rain, whose visuals were pretty impressive, it looks fantastic - definitely using the PS3 to its full potential. Also, does anyone know if the characters have such ridiculous names in Japanese too, or is that just a translation thing?

I worry about Palm sometimes. They recently lowered their guidance for this quarter, analysts don't seem too upbeat, and their stock price for the last year looked promising when they released the Pre in July, but has dropped dramatically since then.

More concerning is the fact that, 9 months after releasing the first WebOS phone, Gartner estimates that 0.7% of smartphones are running WebOS. Hopefully this will improve now that they're on Verizon (and rumor is they'll be on AT&T sometime this year) and once they launch in more countries.

The good news is that the mobile phone market isn't quite like, say, the social networking website market, which has a very strong network effect. If all your friends are leaving Friendster for Facebook, then Friendster is less valuable to you, and you'll probably switch to Facebook. But I can still use the web just fine from my Palm Pre even if the rest of the world switches to iPhones and Droids and Nexus Ones. There is somewhat of a problem that if fewer people use WebOS, fewer people will write apps for it, but this is more of a slow process. Also, at least in the US, most people are under contract for their phones and so they only have an opportunity to switch cheaply every one or two years. I'm really hoping Palm can keep turning things around - just yesterday they released an update to the Facebook app that makes it much nicer.

Speaking of apps...

LJ for WebOS has been doing pretty well since my last update - as of this very moment I've sold 72 copies. It seems fairly random how many copies are sold a day - thanks to the new app My WebOS Apps I have a nifty graph on my phone with these totals for the last week: 5, 1, 1, 2, 4, 2, 0. So...who knows?

One of the frustrating parts has been seeing bad reviews indicating that it just isn't working for a few people. Most of these reviews came early and I'm pretty sure I've fixed the bugs since then, but most people don't go back and edit their review when problems get fixed, and I have no way of contacting them to ask them if it's working for them and to try to diagnose their problem if not. I've been trying to make it more and more obvious how to contact me to the point that if you can't load the posts a dialog comes up with a button to email me the problem. We'll see if this helps at all. Encouragingly, more of the recent reviews have been good than bad, bringing the average back up to 3.5/5 stars.

I spent a lot of last week working on a new feature that I really wanted to add: the ability to browse other people's journals. I even wrote the parsing code before I got stuck on a problem that I've gotten stuck on before - the inability to properly authenticate so that the client can load friends-only posts. The API way to do this is to call the sessiongenerate method - unfortunately LiveJournal's cookie scheme has changed because of some security holes, and no one's gone back and updated or added a new API.

Every time I run into this problem, I spent some time trying to "fake it" by POSTing the right thing to the login page, essentially pretending I'm a regular user signing in from a browser. As in the past, I can't get this to work and I'm not sure why, and it's really frustrating to try to debug because it's all guesswork.

So I was a little down about that, and the last release (made it to the App Catalog yesterday) only had a few small features like deleting posts. This week I took a stab at some random not-quite-featurey things that have been on my list for a while, and everything just kinda worked. The next release will use a lot less bandwidth on the initial request (66% less in my test case!), and it fixes a bug with comments not posting by showing the CAPTCHA dialog that LiveJournal was returning. I was amazed that both of these basically worked the first time, so that was a nice pick me up :-) I'll probably submit it to the App Catalog tonight after a bit more testing.

I'm running out of features to work on that are actually possible to do, so I'm very open to suggestions!

It's been 2.5 weeks since LJ for WebOS was published on the Palm App Catalog, so I thought it would be nice to talk about how many copies have sold and whatnot.

As of this very moment (according to Palm's magic page that I reload far more often than I should), I've sold 44 copies. You can see a rough graph of the copies per day here, although this tracks downloads and not purchases, so for example when an update got published yesterday the download numbers spiked but the purchases did not.

I was actually expecting a little more of a bump from my app being near the top of the "Recent" list in the App Catalog, but I only sold 2 copies yesterday, which is around the same as most days. Which in retrospect makes sense, because already my app's appeal is fairly limited: only WebOS users who also use LiveJournal and use both enough that they'd be willing to pay to get a better interface to it. An impulse buy this is not.

This is part of the reason I priced it as $2.99, which is "high" in App Catalog terms - the number of people who would be interested in it is so artificially limited in the first place. Of course the other reason is that I think it provides at least that much value - it's much more pleasant to use than LiveJournal's mobile site, which is really the only alternative.

So how much money have I made? Well, let's do the math: 44 copies at $3 each is $132. My cut of that is 70%, which is $92.40. I had to pay $50 to get it on the App Catalog in the first place, so that's down to $42.40, and after taxes I end up with around $32.

On the one hand, hey, free money, right? Except I've spent a ton of time on this project. I've been working on it in my spare time since August, and I've written over 7000 lines of JavaScript code. I've definitely put at least 50 hours into it and probably closer to 100, so calculating the hourly rate is a little depressing. This is probably the project I've spent the second-most amount of time on. (I'm guessing top is whereslunch.org)

I think I put myself in a bad place here - when I work on projects for fun and release them "to the world" open source and all (see: almost everything on gregstoll.com) then I get the satisfaction of completing a project and the satisfaction whenever I see anyone use it, which is a pretty low bar. When I work on stuff for money, then, well, I get money for it, and the idea that someone cares enough about what I'm working on to pay for it.

But this model where I work on stuff in my spare time for fun and then try to make a little money off of it puts me in the mindset of working for the money, and then when the money fails to materialize I get depressed about it. Not to mention I've spent so much time on this in the last few weeks that I could feel myself burning out last night. So I think I'm going to back off a bit on new features and work on things that seem interesting or useful to me, not necessarily other people.

Anyway, this has been a bit meandering, so thanks for reading :-)

I'm happy to announce that my LiveJournal client for WebOS devices has just been released on the Palm App Catalog! It includes support for:
- viewing your friends post and own posts
- making posts with custom security, userpics, and tags
- uploading photos to imgur.com and including them in posts
- viewing cached posts offline

Here's the official home page.

I signed up for the Houston Turkey Trot 5K with my family. Unfortunately, it starts at 8 AM (which is very early for me these days) and I'm not in great shape and it's going to be cold, which doesn't do so great on my lungs. Hopefully I survive!

Been working on WebOSJournal (the LiveJournal client for the Palm Pre/Pixi) - it's coming along decently but I'm running into some frustration trying to allow replying to posts/comments. The authorization scheme is tricky and of course you don't get useful feedback. I actually downloaded the LiveJournal source code to try to figure out what I'm doing wrong, but it's hard to find my way around...

Obama kicks off massive science education effort - yay!

I couldn't sleep last night so I whipped up this analysis of our home energy usage vs. temperature. It turns out that (spoiler!) we use more energy when it's hot outside. But I got to run my first linear regression!

People who know about statistics: feel free to criticize/suggest improvements. (I'm not even sure I was looking at the right R-squared value...)

How many times have you had a log file with lots of pointer values and wanted to quickly see which values were equal, and even rename them with a meaningful name?

Wow. Really? Never? We lead very different lives.

Anyway, I wrote a little log colorer to do that, which has been helpful trying to track down a race condition. (and inserting a breakpoint tends to make it not happen) Also, colors!

Ben & Jerry's is celebrating that gay marriage is legal in Vermont effective yesterday (yay!) with Hubby Hubby ice cream.

In Maine, gay marriage is legal but it will be on the ballot in November. The Catholic Church, in a disappointing but not too surprising mood, is contributing $100,000 to try to repeal it even as they have to close local parishes.

Did a quick project so that GLSlideshow (my screensaver), which currently shows random pictures from my gallery, would show the comment on the picture as well. Wrote it up here but there are no demos or anything so it's not very exciting :-)

Played around with the Palm Pre API last night, and I feel like I have my legs under me and can start making things that are actually neat rather than just toys. Exciting!

The second in an occasional series
Idea: Now I'm focusing on improving the time to insert the iTunes data into database. Where we left off last time, our script took 71 seconds to run, ~50 seconds of which was database operations. The idea I had to speed this up was to batch a bunch of queries together and thus make fewer calls to the database. It turns out this actually slowed things down.

So I did a little research and it turns out if you insert data with the same query structure over and over again (but with different bind variables), the database doesn't have to reparse the query which speeds things up a lot. I tried doing this with pyPgSql but couldn't find any documentation how it was supposed to work, so I switched to using psycopg2 and changed the query for inserting the playlist data. Just switching to a psycopg2 sped things up a lot, it seems. I tried switching to a similar sort of query for inserting track data, but that actually slowed things down.

Anyway, the new script runs in 25 seconds, and it looks like only around 9 seconds for database operations. This is a 400% speedup in the database time! Overall, this step improved performance by ~180%, and since we started at 114 seconds we've improved ~350%.

Conclusion: Another big success, and I'm not sure how much more I can squeeze out of the iTunesInfo.py script. Next time I'll focus on the analyzedb.py script, which does the analysis from the database - right now it's taking between 5 and 12 minutes to run on my library of 6400 tracks.

Source files:
- old script
- new script

The first in an occasional series
Intro: A while back I wrote a script to analyze an iTunes library and find your favorite artists, albums, etc. It works pretty well and I regularly use it to update my own analysis. Unfortunately, it generally takes a long time to run, which is sort of OK for me (because I just start it running and go do something else) but less good for people who are running the analysis through the web site.

So I'd like to make it run faster, and I have a number of ideas to do so.

Idea: There are two main parts to the system - parsing the iTunes Music Library.xml file into a database, and running the analysis on the database. First I'm focusing on the parsing part.

The first version of the parsing script uses Python's xml.dom.minidom package to completely parse the library file.

After profiling the first version by running python -m cProfile -o profiledata.oldway iTunesInfo.py "iTunes Music Library.xml", I see that the whole parsing process takes 114 seconds. The major parts of this are 60 seconds for the xml.dom.minidom.parse method and 46 seconds for the database operations. Note that this only leaves ~8 seconds for figuring out the track information - clearly this is not the bottleneck!

So I'd like to improve parsing speed. There are two basic kinds of XML parsers - what we're using now is a DOM or Document Object Model-style parser, which means that the parser reads the entire file in and returns a parsed structure containing all the data. (I remember writing a simple XML parser that did this as a project in COMP 314. Ah, memories...) The advantage to this method is that after the parsing is done, it's easy to traverse the DOM tree and find the data that you're interested in. The downside is that, well, it's slow. Also, the entire document has to be read into memory which means that your memory usage is proportional to the size of the file you're processing, which adds to the slowness and can lead to out of memory problems on huge files (although we weren't seeing that here).

The other basic kind of XML parser is known as SAX, or Simple API for XML. You provide callback functions that are called whenever the parser runs across the start of a tag, end of a tag, character data, and...that's it. Whatever processing you want to do you have to do in those callback functions. So if you're just, say, counting the number of <key> tags in a document this works really well. It's also much faster than the DOM-style parser, since it doesn't have to generate a giant tree structure. But doing the sorts of processing we're doing on the library file seems a bit more tricky.

Anyway, I take a stab at it, and after a bit end up with version 2 of the script. Notice that the logic in the Handler class is a bit twisted - we have to keep track of where we are in the document (so if things get out of order we'll have problems) and use a state-based system which is a bit brittle and unclear.

But how does it perform? The old version of the script ran in 114 seconds, and this version runs in 71 seconds for a ~60% increase in speed. But really, it's better than that, because the database operations still take around 50 seconds - if we subtract that from both we get 64 seconds versus 21 seconds which is a ~200% increase in the speed of the parsing.

Conclusion: This was a big success! Most of the time is now in the database layer, which I have some ideas for speeding up next time.

Source files:
- old script
- new script

Just got done working on a database of the games we own. Not everything we own is in there, but most of it is, and I'll try to get the rest added soon. You can search based on number of players and the length of the game. Give it a shot and let me know what you think!

My same-sex marriage map got some link love from Metafilter (thanks kernelm!) and somethingawful, oddly enough. And twitter, come to think of it. This added up to 225 hits over the past 3 days, which is a lot for me.

I'm not sure why it makes me so excited to see people linking to my stuff - there are no ads on the page, and presumably most people click on it, take a look, think "Neat!" (or "What a piece of crap!") and move on with their day. I guess it's a measure of fame or prominence, however slight.

The government just released data.gov which is a collection of parseable data from various government agencies. Unfortunately, there isn't a lot of too interesting stuff there yet, but there is a a contest to write a cool app. (the first entrant is FBI Fugitive Concentration!)

The Dallas Cowboy's new stadium has the world's largest HD video screen.

Created with Pretty Pictures with genotype:

(div-clip (colorperlin 1362710210 278847481 1402125762 (add-wrap (abs (num 0.031681362818044456)) (sub-clip (cos (ccrgb (sin (atan-clip (rd (add-clip (add-wrap x (num 0.578092059228798)) x)))) y (num 0.23646468718646263))) (colorperlin 1502179640 1316738189 454935563 (neg (div-wrap x y)) (num 0.5078184951973896)))) x) (bwperlin 880129745 (abs (add-clip (sin x) (ccrgb y (sub-wrap (cos x) (neg (log-clip (num 0.05318551299931629)))) (num 0.751250719855975)))) y))

I updated Pretty Pictures to include perlin noise, as copperwolf suggested, and added a tuning option. The pictures, they are prettier now.

On the tooth front, I only took one Advil yesterday...maaaaaybe it's getting better by itself?

I'm getting awfully tired of reading National Review editorials in my morning Austin American-Statesman, like this one saying torture really isn't so bad.

Just finished rewriting the backend to Pretty Pictures from Genetic Algorithms in C++ (it used to be in Python). It's easy to forget how much faster C++ can be than Python - I even turned on an option by default to make all images color, which made things unacceptably slow in Python. It's fun to play with!

It's true! You can click on the legend to make states that have that status blink.

Note to self: using fractional RGB values makes things silently not work. Don't do that.

The latest project I've been working on is this same-sex marriage map which shows the status of same-sex marriage and civil unions in the United States. It has pretty colors, and you can animate it to see how things have gone over time, and click on states to see their specific history.

Did you know that same-sex marriage isn't against the constitution in Wyoming? Or that in Virginia and Michigan, not only is same-sex marriage or civil unions against the constitution, but private contracts between people that do the same thing are too?

This weekend I was proud of myself as my weight hit its lowest level since I started tracking it this time. (since then it's bounced back up a little, but I'm going to ignore that for now) So I thought it would be a good time to look back at my previous attempts at weight loss to see how well I did.

And I had forgotten that (last very short attempt notwithstanding) I've done much better than this in the past. Right after college I was a full 15 pounds lighter than I am now, and even an attempt a few years later I got down to 5 pounds below my current weight. I really can't decide if this should be encouraging (it is possible for me to weigh less!) or discouraging (I've been trying for 2.5 months and only lost 7 pounds!).

On Friday I discovered that GnuCash (which we use to do our finances) stores its data in an XML format! So I wrote a script to generate a report on how much we spend on various categories each month (GnuCash has some builtin facilities to do this, but I couldn't quite get it to do what I wanted). The report looks pretty but I don't feel comfortable posting it to the web, so you'll have to take my word for it.

Related: I found this awesome ColorBrewer which helps you pick a set of colors for maps or graphs or whatever. This is great news since I suck at picking colors!

The British military put some of their troops on LSD to see how it would affect them. The results are amusing but not really surprising.

adieu genetic Mona Lisa
Mood: chipper
Music: The Lonely Island - "I'm on a Boat"
Posted on 2009-03-09 10:58:00
Tags: projects
Words: 260

That project I was working on? Yeah, not so much. After running it overnight, here's what I got:
The target image

Um, so it seems to have figured out the image is mostly pink. Hooray?

Things I did learn:
- I originally tried crossover for breeding (specifically cut and splice) and that was just terrible. I think the algorithm really needs lots and lots of new polygons to be tried, and the weakness of crossover is that you're just combining existing ones.
- Originally I had a generation size of 500, and I implemented the image difference calculation in Python. This took around 500 seconds, which is way way way too long. After some optimizing, I got it down to around 280 seconds.

I had planned to wait until I had things working to write a C module for Python to do the difference, but I figured I couldn't really make progress until I could actually do the difference in a reasonable amount of time. After writing it in C (not even with MMX or anything) the total time went down to 1 second. Win!
- The best results seem to come from having a very small generation size and randomly mutating, which is pretty much what the original guy did.

Wanna see something else cool? Virgil did these: Books that make you dumb and Music that makes you dumb by correlating popular books and music at universities (found on Facebook) with average SAT scores. Obviously correlation is not causation but it's still pretty interesting. Guster, Ben Folds, and Radiohead fans should feel good, I suppose :-)

the advantages of living in a medium-sized city
Mood: determined
Music: Foo Fighters - "Let It Die"
Posted on 2009-03-06 14:15:00
Tags: car projects
Words: 92

2.5 years ago: My car's check engine light turns on, I have to drive 40 miles early Saturday morning to get it checked out.

Today: My car's check engine light turns on, I drive 3 miles to drop it off during the middle of the day. (and hitch a ride back with destroyerj) Life is good!

My next project: This guy genetic algorithmed his way to a Mona Lisa with translucent polygons, and then this guy did something similar. I like images and especially genetic algorithms. Hopefully will have pretty results soon!

So I won the code bounty! I've really been enjoying writing Firefox extensions - it's easy to get started and fast to see results, and I'm starting to understand XUL better. Anyway, Kate (who offered the bounty) is going to polish it up and release it, at which point I might consider using it - it's convenient and is a better solution to passwords than a tiered password scheme.

I also got the $100 Amazon gift card, which I'm not sure what to do with. Is this good because I'm less materialistic or bad because I'm not excited about anything in particular at the moment?

Links!

- Happy Texas Independence Day!

- If you like crazy Obama theories, you'll like this Daily Show segment! Sometimes I wonder where they find these people...

- The government is bailing out AIG some more, but the real story is that AIG lost $62 billion in 3 months!

- In a recent study, only 29% of people supported gay marriage, but this number went up to 43% if assurances were made that no church would be required to perform gay marriages. Which is kind of weird, because no church is required to perform particular kinds of marriages today.

- Obama frames things as people versus corporations and industries rather than Republican versus Democrat.

So I wrote this benchmark analyzer to analyze benchmarks that have many parameters and try to figure out what the important ones are. To that end, it creates a decision tree in R, and also generates lots of boxplots to visualize the different parameters. Here's a sample report (with boring data). It also (somewhat cleverly) saves the original .csv file with the web page it generates, so you can save the page and send it to a colleague who can play with the options and rerun the analysis.

It occurred to me as I was prettying this up that most benchmarks don't involve lots of parameters, and so the probability that anyone else is going to find this useful is pretty low. I'm OK with that because
- it's useful to me right this very moment for work stuff
- I had fun writing it and learned some more about R
- pretty graphs!
- it was good to work on something other than whereslunch.org for a while

So I wrote this ThumbnailCopy extension (see Kurt's post) and in order for it to go public people have to register for an account there and review it. So...anyone mind doing that? :-)

Also, Texas Bowl pictures are up.

So I started the week of happy a week or so ago, and I'm pretty...happy with the results. It turned out to be kind of a tough week, what with the root canal and the painful aftermath, but focusing on the little things that make you happy during the day is good. I'm pretty sure I read somewhere that big changes in lifestyle (new bigger house, etc.) don't generally affect one's happiness level in the long run, but little things can if you focus on them. Or something like that.

Dentistry - I gave in and called this morning to make sure it was normal for the pain to last all week. The person I talked to told me to come in, and they took an X-ray and thank goodness everything looks fine. (if I had had to have like another root canal or something I would have been seriously sad) She said the pain can last a while, and as long as it's getting better (it is!) it's OK, and she gave me another prescription for Zipac(sp?) to take in case there's a lingering infection or something. (and she told me to switch to Advil instead of Tylenol for the inflammation) So, yayish!

Last night when I got home quijax was watching Mythbusters, which I don't get to see much but I like! One of the myths they were busting was that buttered toast tends to land buttered side down, and long story short they ended up dropping toast off of a building and observing. Anyway, the results were slightly more in favor of the toast landing buttered side up, and Jamie had a physics explanation (when you butter toast it gets indented) but their numbers weren't convincing. Long story short, I created ismydatasignificant.com for an easy chi-squared test for significance. If only I understood more statistics...

Finished the adding of restaurants last night! Now I just need to add an editing interface and do some more testing...

A long lipdub of Thriller - impressive camerawork! I wish I could keep the camera that steady while I walked. (obviously I should have combined all my Thriller links into one post, but oh well)

A federally funded group is sponsoring a contest to find an engaged couple who hasn't had sex yet. No takers yet...

Election data: lots of people have already voted early. Deadline in TX is Friday, and I'd recommend doing it today or tomorrow because I bet Thursday and Friday will be quite busy...

Thanks for responding to my where's lunch poll! As I had hoped, it motivated me to code some more this weekend. I implemented the suggested restaurants (although it's pretty weak right now since there's not a lot of data) and am about halfway done with the adding new restaurants thing. Once I get that done and put it through another round of testing (haven't tested on IE for a while...*shudder*) I'll post it for real!

We voted this weekend, and while it was a little anticlimactic at the time, I'm still pretty excited by it. We went to the Randall's at 183 and Braker - happily it was during the UT game so there was no line. Voted against Prop 2, for Obama and Noriega and Larry Joe and most of (but not all) the other Democrats. Election Day is only 8 days away!

As a followup to Austin Mayor Will Wynn dancing to Thriller, this weekend Austinites shattered the Guinness World record for the largest synchronized Thriller dance. The previous record was 140 (only 140?) and we got 881(!). Here's a video of the whole thing although it's of disappointingly low quality.

Politics:
Will Ferrell was on SNL last night as George W. Bush and it was pretty good (although maybe not "one of the best skits ever"). It still looks like Obama is way ahead. It warms my heart that someone who wants to investigate all members of Congress to see whether they're pro-American or not (hello, Joe McCarthy!) can suddenly find that most people won't stand for that crap. Obama makes a good fantasy football partner. Also, politics has always been sleazy - this is not a recent phenomenon.

Where's lunch:
This is my project for a google map of lunch places in Austin. It's kinda stalled out, because I haven't had a lot of free time and I can't decide what to do next. Right now you can view restaurants, rate them, leave comments, color the markers based on rating and some other things, and filter which markers are shown. (if you're interested in trying it out, drop me a line and I'll hook you up) Things I want to add at some point:
- letting people add restaurants themselves. This is kind of a pain to do, and raises some security issues and means I have to police the data to some extent.
- suggesting restaurants you might like based on other people's ratings. This isn't too hard to do but since there are few ratings in the system it won't be interesting for a while.
- add user profiles where you store where you work and you can limit lunch places by their estimated time for lunch (2 * travel time + time it takes to get food).
- putting ads on the site to make $$$
So, how interested would you theoretically be in these features?
[ Fill out Poll ] [ View Poll Results ] [ Discuss Results ] [ Close Poll ]

Poll #1284500 Where's lunch features
Open to: All, detailed results viewable to: All, participants: 7

View Answers
Mean: 6.29 Median: 8 Std. Dev 3.10

1	1 (14.3%)
2	0 (0.0%)
3	1 (14.3%)
4	0 (0.0%)
5	1 (14.3%)
6	0 (0.0%)
7	0 (0.0%)
8	2 (28.6%)
9	1 (14.3%)
10	1 (14.3%)

View Answers
Mean: 7.14 Median: 7 Std. Dev 1.96

1	0 (0.0%)
2	0 (0.0%)
3	1 (14.3%)
4	0 (0.0%)
5	0 (0.0%)
6	0 (0.0%)
7	3 (42.9%)
8	2 (28.6%)
9	0 (0.0%)
10	1 (14.3%)

View Answers
Mean: 3.86 Median: 4 Std. Dev 2.70

1	3 (42.9%)
2	0 (0.0%)
3	0 (0.0%)
4	1 (14.3%)
5	0 (0.0%)
6	2 (28.6%)
7	0 (0.0%)
8	1 (14.3%)
9	0 (0.0%)
10	0 (0.0%)

View Answers
Mean: 5.71 Median: 5 Std. Dev 1.28

1	0 (0.0%)
2	0 (0.0%)
3	0 (0.0%)
4	1 (14.3%)
5	3 (42.9%)
6	1 (14.3%)
7	1 (14.3%)
8	1 (14.3%)
9	0 (0.0%)
10	0 (0.0%)

Poll #1284501 Time or Newsweek?
Open to: All, detailed results viewable to: All, participants: 5

View Answers

Time
3 (60.0%)

Newsweek
2 (40.0%)

Check it out! Or, if you're in a hurry, at least check out the awesome graph! (I spent a while coming up with a decent algorithm for label placement...)

Onward to...something else!

According to a new Gallup poll, 67% of Americans believe the president should meet with the leaders of enemy countries, and 59% of Americans believe the president should meet with the president of Iran. (I guess that means 8% of Americans don't think Iran is an enemy?) That seems like a pretty firm rebuttal of McCain's attacks on Obama for wanting to do just that.

The Target probability page is really filling out now. I think I'm going to give up on justifying the counts for a mixed straight (a 4 card straight including cards of all 4 suits) which seems really really tough to count, and maybe the 3 card and 4 card straight flush of any suit (which are only really tough to count, but still more than I'm up for). After I calculate the numbers I'll do a graph and try to figure out what the dividing line is of which targets are worth which points, and see if any are "misclassified".

Lately I've been feeling a lack of...something. A desire to build something great, some awesome website or something. I don't know where it's coming from and maybe it'll just fade away. Drive isn't bad, but it's kinda inconvenient. I just don't have a whole lot of time to devote to stuff like that.

Although, keep your eyes open for whereslunch.org after I finish my current project!

Speaking of which, last night in bed I finally figured out how to use inclusion-exclusion to calculate the number of 3 card straights, etc. The key is to make your sets "hands that don't include a green 0", "hands that don't include a green 1", etc., figure out the number of hands in the union of those sets and then take the complement. I guess it's been a while since I've done this combinatorics stuff. I also filled out some more entries in the table with explanations.

xkcd story in the NY Times!

I couldn't sleep last night, so I worked on the Target probabilities project. It was annoying me that the old version of the generated HTML (which was generated by LaTeX2HTML) looked kinda ugly - look how jagged the math graphics are! (especially compared with the beauty that is the generated .dvi file) So I tried a few different packages and settled on TeX4ht, which produced this much nicer looking page. Ah, that's better. (although why is that table floating to the right? grr)

I've been making slow progress on actually figuring out the numbers (3 card straights and straight flushes are the worst!) but I'm getting there. Getting the right numbers is a little more challenging.

Before that, we watched Citizen Kane (still good even though we weren't paying full attention) and I played some GTA IV, including a little multiplayer with destroyerj. It was tough without a headset and sucking at the game :-) I managed to flip the cop car we were driving in, and another time destroyerj was driving and I was shooting at something and I saw him jump out of the car. Had just enough time to wonder "Huh?" before we slammed into a stopped car and caught on fire :-)

Finally downloaded The Slip by Nine Inch Nails. So far: very good.

Game night tonight!

seem nonexistent. (anyone know a good antonym for "doldrums"?) Last night we bought some exciting new books, I bought GTA IV, got a graduation present for my sister, and committed to working on my new project. (calculating the probability of satisfying target cards in the game of Target)

It seems that my computer is under attack from a botnet. I pretty frequently get ssh requests coming from wildly different IP addresses trying out a dictionary list of user names. I'm not too worried about them guessing a username/password combo but the fact that they're hitting my box in general is a little scary. I've started turning off ssh at night but I do use it occasionally during the day...

things younger than mccain

If you're in to electoral predictions (for the general election), fivethirtyeight.com is a godsend. Probabilities that each candidate will win each state, cartograms, graphs...

rattling around in my brain
Mood: pleased
Music: Run Lola Run Soundtrack - "Running One"
Posted on 2008-04-15 13:54:00
Tags: projects programming
Words: 54

I like being around people. I like having people over/visiting people. I'm sorry if I don't give out that impression.

watchdog.net launched today and it looks pretty neat - I volunteered my services. One of the founders wrote this essay on how to be more productive.

jQuery is now my default choice of JavaScript library.

This weekend we ripped through Zul'Aman - first time for downing the bear boss in 20 minutes, downing the dragonhawk boss, downing the gauntlet and eagle boss (one-shot!), and downing the lynx boss. (I got a nice new chestpiece - now my frostbolts cast in 2.45 seconds!)

I've also been working on a new design for my frost mage DPS calculator - here's the new version versus the old version.

We also went to a few birthday parties and bought a new bed from Austin Discount Mattress, and took a very few pictures of said events. I'm going to take my camera places from now on instead of relying on my cameraphone - while it works in a pinch, the pictures are blurry and the color is always a bit off. I also learned that alcohol + hot tub is bad news for me, even after just one glass of wine. I got extremely woozy and lightheaded after a little while, and it all happened very fast. Anyway.

We need a Wikipedia for data - as an avid consumer of data, I heartily agree!

Strong Bad's Cool Game for Attractive People

So I've been working on this startup in my spare time with some people, and it died Saturday night for reasons I'm not going to go into. The consequences are I'll have a little more free time and I'm looking for a new project! More work on the WoW Frost Mage DPS calculator is a possibility, but it doesn't honestly sound that interesting. I'm working on rewriting my gift list site in Django, but that's not really catching my fancy either.

We watched a lot of Battlestar Galactica this weekend, including the Season 4 premiere. Here's an primer to catch you up, or alternatively a video version called "What the frak?". Here's an old feature on what the heck Donnie Darko was about, which I found interesting.

And if you like people talking fast (long live ze frank!), check out Zero Punctuation, which is a 4-5 minute weekly review of video games. The gay jokes get a little annoying, but honestly it's worth it because it's pretty hilarious.

One of the things Obama mentioned last night was the bill he and Tom Coburn passed to provide a "Google for government", a searchable database of government spending. Well, it exists! Check out usaspending.gov - and it even has an API! I bet there's a project in this...

I took pictures of the latest puzzles we've completed, whereby "we" I mean "David, Miriam, and sometimes me". The last two were taken with my new phone, which is why the color is pretty messed up. (I tried to fix it in iPhoto but I may have just made things worse) So: having a camera with me at all times: pretty neat, but I'll stick to the real camera when it's available.

Speaking of the new phone, it's pretty neat! The screen is much nicer than my old one, and it's my first clamshell phone, which I think I like. It has pretty good web access, although my one big complaint is that it doesn't let apps access the network, which means that I have to use the kinda crappy web version of Google Maps instead of the super awesome one(*) that runs on the phone.

I got my car's oil changed today, which revealed that the rat that used to live(**) in our garage was gnawing on my air filter. Luckily it didn't gnaw all the way through; I don't know what implications that would have, but I can't imagine they're good.

Work is going really well today - between that and a new project at home(***), I am really peppy!

(*) - I've never seen or used said version so it may not be super awesome.
(**) - "used to" because we haven't seen any signs of it in a while, so it seems likely it's gone. We're moving out soon anyway. Whee!
(***) - which I will post of the results of hopefully tomorrow. Thanks to destroyerj for the idea...

This week I've been in the odd position of being really excited about work and not so excited about non-work projects. Work has been going really well - a lot of the setup for my big project is done so I'm getting into the fun part. The non-work project of pulling down Netflix ratings and analyzing them is pretty similar to what I've done in the past, but has enough different parts that just aren't interesting to leave me totally unenthusiastic about working on it. But I've already done enough that I feel like I should finish. I'll put it on hiatus for a while.

We've been catching up on Simpsons episodes from the last few months with people, and as a result I've seen "Eternal Moonshine of the Simpson Mind" twice in the last week or so. It's a pretty good episode, but the part that struck me both times was the montage-like scene where it shows lots of pictures of Homer throughout his life to some hauntingly beautiful piano music. Anyway, djedi found the song, which is called "Everyday" by Carly Comando and I bought it last night - yay for Amazon MP3! The scene itself is based on Noah takes a photo of himself every day for 6 years which is pretty creepy in itself, although the music really amplifies that. Good stuff all around, and I feel bad I missed this all the way back in 2006.

Yesterday at lunch I was really happy because of a problem I had just solved, so I went out to Pars Deli, a greek place at 183 and Burnet. Unfortunately, the parking spots are deceptively thin, and so as I was pulling in I heard a noise and realized I hit the car parked next to me. After I parked and got out, the car looked like a mess, but I could brush off most of it. (dust from my bumper or something?) There was still a scratch that was very narrow but kind of long, and I wasn't quite sure what to do, and I was mad for having done it, so I just walked away to the deli. It didn't take long for my conscience to get the better of me and realize that that was a pretty stupid/selfish/horrible thing to do, so after I ordered I went back out there and left a short note with some paper I found in the car. Not my finest hour - "doing the right thing...eventually" isn't exactly high praise.

I have a few Best Buy gift cards and kinda prompted by that I'm looking for some new music. Definitely getting the new Radiohead album, other options include "Year Zero" by Nine Inch Nails, "Distortion" by The Magnetic Fields, maybe something by LCD Soundsystem ("Sound of Silver"?), maybe something by Arcade Fire. I am open to suggestions! (surprisingly hip suggestions stolen from the AV Club's best of 2007)

So I spent some time updating this script to pull your Netflix ratings and after redesigning the database tables a bit (which was fun since it isn't something I usually do) I think it's about done. Now I need to add in some analysis or something :-)

We had our release party for the new NI smart camera (get yours today!) yesterday at 300, a bowling place that feels clubby. (there were music videos playing on screens in front of lanes and such) It was fun.

Phones: Like before, my cell phone's alarm isn't going on again, except this time I can't figure out how to fix it. I was planning on getting a new phone pretty soon anyway, but this may expedite the plans. My current top choices are the Samsung t439, Motorola RAZR V3, and maybe the Nokia 6133 - anyone have any opinions on these?

I found this awesome script to get your Netflix ratings last night and it's pretty cool. I may improve it some, throw in some analysis and make it my next project.

Speaking of backing up your personal data stored at an online site, LiveJournal was bought by SUP. It sounds like nothing big's going to change, but it might be a good chance to backup your LiveJournal with LJBackup. Shameless plug transmission over and out!

Friday started pretty crappily (as I noted), then we went to see Beowulf 3D with people. It was better than I expected - I guess 3D technology has come a ways since I last saw it. The movie was pretty typical action fare, except it was all motion capture CGI-y. It was pretty well done - you could tell most of the time that the characters weren't real actors, and some of the emotions didn't come through as well as they could have, but not bad.

The music was also surprisingly good - the few themes they used are pretty catchy, and the "main" song was sung during the end credits by Idina Menzel! (the original Elphaba in Wicked)

Saturday was spent being cranky for a large part of the day for various reasons. We did get to start Super Mario Galaxy which is a lot of fun, and hang out with wildrice13 in the evening. I worked on my arena rating graph which is coming along nicely - got some statistics to show. (did you know we are much better against teams with 0 mages than teams with 1 mage?) Next step is making it so you can choose criteria (say, matches where the opponents had at least one hunter and paladin that we played in Blade's Edge) and show statistics based on that. Should be fun.

Today was church and then house-hunting. After talking for a while about the way things work (helpful if a bit long), we looked at 6 houses. Unsurprisingly, our top choice was one that looked kinda OK on paper, and our #1 choice on paper was OK but not great (and is under contract anyway so we can't buy it). I took a ton of pictures that hopefully will help us remember the one we liked. It was pretty good but not super amazing or anything. I guess in the future we'll just get more listings from our agent and periodically look at more houses. It's a bit anticlimactic.

I'm "officially" releasing the iTunes Rating Analysis after making a few last minute tweaks. Unfortunately you have to be at your home computer (or whatever computer your iPod is synced to) to use it, but you can at least see see my analysis, complete with fancy charts! Not being able to use it at work means it'll probably be significantly less popular than the WoW Frost Mage DPS calculator (currently the most popular page on my site by a factor of 2 or 3 or so), but that is OK.

We bought and played through Portal this weekend, which is a first person puzzle game and pretty darn fun (and pretty short). The end credits song is extremely cute and catchy, and it's sung by the passive-aggressive computer who's talking to you for the whole game. The song was written by Jonathan Coulton (here's his blog post about writing the song) who happens to have lots of music available for download! So I'll probably check that out this week. Edit: live version of Jonathan Coulton performing "Still Alive" - there's cake!

wildrice13, abstractseaweed, quijax, djedi and I went to the Renaissance Festival on Saturday. David and I were going to see the Bolton's renewal of vows (here are the few pictures that I took - we participated in the ceremony so I didn't want to interrupt too much :-) ) which went well except for their not letting us do it at the festival proper (the words used were "unauthorized wedding" which I found amusing), so we went out to the parking lot and had a nice ceremony. The trip back wasn't so great - we got stuck in the parking lot as everyone else was leaving and then there was an accident 10-15 cars in front of us just as we were about to get to 290, so we didn't moved for half an hour or so. Ugh.

I slept on my neck funny. It hurts.

Went out to lunch today with WoW people and we talked about this neat idea for an Arena rating graph. So you start at a 1500 rating and every match you play you gain or lose 1-29 points, and it would be neat to see the progression of that as the season goes on. Also nice to see what composition of teams we're good against and which we're not, and the ratings of the teams we win or lose to. I'm having to try really hard not to just start writing down ideas here since it's not work-related :-)

My first thought was to do the graph in Flex, but I forgot that Flex Charting is a separate package that costs money. So I did some research and there are lots of options for this sort of thing (found this excellent article on interactive charts and graphs via del.icio.us). AmCharts looks like the best option, with FusionCharts Free close behind. I'll have to download and try them out at home; of course the hard part will be figuring out how to display all that data in a useful way.

Here's an excellent article on how Simon Fink's team got into the top 10 for the Netflix Prize - lots of juicy juicy math. Makes me want to get back into that sort of thing, which would make three active projects at once, which is too much.

Poll #1083910 Pimp my iTunes Ratings Analysis!
Open to: All, detailed results viewable to: All, participants: 8

Version 1 has alternating background colors on tables, while version 2 does not. Thoughts?

View Answers

Version 1 (alternating colors) looks nicer
5 (62.5%)

Version 2 (no background colors) looks nicer
0 (0.0%)

I like the idea of alternating colors, but not those colors in particular (please explain!)
3 (37.5%)

Version 1 has black table headers while version 2 has more grayish ones - opinions?

View Answers

Version 1 (black headers) looks nicer
6 (85.7%)

Version 2 (gray headers) looks nicer
1 (14.3%)

They both suck (please explain!)
0 (0.0%)

View Answers

Yes
2 (40.0%)

No
3 (60.0%)

So we played another game of Clue yesterday with wildrice13 and abstractseaweed and despite the fact that the clue solver declares it inconsistent, I'm going to be optimistic and assume I wrote something down. Also fixed a few bugs and added a special case for a weird situation that came up. (I knew David had Conservatory or Ms. White, and Andrew had Conservatory or Ms. White, so no one else can have either of those cards) Thinks like that would have made using first-order logic or propositional logic nicer (since it would have presumably taken care of them automatically), but oh well.

Thus, I am declaring the Clue Solver done! Put a few example games up so you can load them and see how they work. (and the pretty colors on the simulation tab!) Been working on it for a while so it's nice to have it done. I'll take a little while off (what with starting work and all) and see what I feel like doing next. I do feel like I got a decent handle on the Google Web Toolkit, so that's nice.

When I was growing up¹, I didn't play Clue much. There was that game where we drew three random cards for the solution and ended up with two weapons or something like that. We never played with the rule that if you suggested someone they moved into your room, or that your suggestion had to include the room you were in (I think). And I remember at least one game where everyone showed you a card instead of just one person. My strategy was always to simply mark off cards I had seen, which was not terribly successful when I started playing with friends who played with the "real" rules.

I finally figured out (or refigured out? maybe I knew this before? my memory - unreliable...) a better strategy on this last visit to Austin, which is to use the information about if other people have certain cards or not. So for every suggestion that someone makes, when someone passes I write down on each of those three cards that that person doesn't have them, and when someone shows a card I write down that they have one of those three cards, and using that information you can learn a lot. The last game I played I didn't win, but I was pretty close to knowing what the solution was.

So, I thought it would be a fun project to write a Clue solver - that is, something on which you indicate the information you see (Mary made the following suggestion and it got passed around until John showed a card) and it deduces everything it can from that.

My current approach is to specify rules in first-order logic and use some sample code from my old AI book (AIMA) to figure things out. Unfortunately, the rules don't lead to definite clauses, so I think forward chaining is out. I've been trying to use resolution, but it doesn't work right and is slow. Maybe I'll try backward-chaining instead. If I have to I'll break down and write Clue-specific code to figure things out but it would be really awesome if I can get it working just by specifying the correct rules.

If you're following along at home in your copy of AIMA, I've been reading a lot of Chapter 9. :-)

I found this class project to do something similar (here's the project description (pdf)) which has a good explanation of resolution and the rule-based approach, although they seem to be using propositional logic and not first-order logic. Which might be what it takes to make it work in a reasonable amount of time.

----
¹ - I fully expect wonderjess to correct me here since my memory is pretty horrible. (back)

but first, math!

From last time - the answer is that they're not symmetric. If a=10, b=11, c=20, then C enters A's stall at time 10 and you get to enter B's stall at time 11. However, if a=20, b=11, c=10, then C enters B's stall at time 11 and you have to hold it until time 20. More formally, the time you have to wait is

min(min(a,b) + c, max(a,b))

Using the formulas min(a,b) = (a+b-|a-b|)/2 and max(a,b) = (a+b+|a-b|)/2, this expression simplifies down to (a+b+c-|c-|a-b||)/2, which is clearly not symmetric in a and c. I'd be interested to see how this formula works for more people and stalls - it seems to blow up pretty quickly, although it would be easy to write a program to calculate it. I have a gut feeling that you want the fastest people earlier in the line (ideally if there are n stalls the n-1 slowest people will be all in their stalls when you get to go in, so you don't have to wait for them) but proving that seems rather tricky.

When will LJ get inline LaTeX?? :-)

Good article about risk management and terrorism (via schneier)

I'm making good progress rewriting the Pretty Pictures picture generator in Haskell - it spits out valid PPM images that look correct for simple cases, but it needs more testing. (and I need to see if browsers support PPM natively or if I have to convert to PNG which would be a pain) I'm not enjoying it a whole lot, though. The idea of learning Haskell was to expand my mind and change the way I programmed, and I guess it has done that some, but I always feel handcuffed when I sit down and try to do seemingly easy things, like write out a raw PNG file. (got about halfway there and said forget it)

To antidote this (yes, it's a verb!) I think my next project will be less challenging technically and more interesting, like involving lots of neat data (census? past wars?) and some sort of neat visualization. (R?) We'll see.

A coworker of mine is leaving to Google, which has the unintended consequence of making me feel jealous. And I hate feeling jealous.

So I guess what that means is I want a new project. I could rewrite the image-generating part of Pretty Pictures in something faster than Python (C++? D?), but that's not really exciting. Things with collaborative filtering or genetic algorithms are neat. I have a list of ideas at home that I guess I'll look at.

Or I could deal with my emotions, but that sounds a lot harder.

Also, did everyone give up on twitter?

So my next project is fun and neat. I can't remember how exactly I got here, but I noticed that all roll-call votes in congress are accessable in a handy XML format (Here's an example vote - note that the page you see is that XML file after being processed with an XSLT stylesheet, so you'll have to hit View Source to see the raw data). My idea (a little vague at this point) is to take all of the votes, and then figure out which issues I care about and which way I would have voted, and then "rate" representatives as to how closely their votes align with mine. This is a little grand in scope.

Anyway, it took me a while to get HaXml (a Haskell XML parser) installed, because I kept not being able to compile it from source for various stupid reasons. Anyway, I finally figured out that it was in fact in Debian in the libghc6-haxml-dev package, which made my life about 5 times easier.

So I saved a sample vote and have it parsing and I'm extracting simple data from it, which is exciting! I have a few questions, though: does anyone know the answer to these?

- I have these two functions:



nothing :: Maybe a -> Bool

nothing Nothing = True

nothing (Just _) = False



findElementContent :: String -> Content -> Maybe Element

findElementContent target (CElem el) = findElement target el

findElementContent target _ = Nothing



findElementContents :: String -> [Content] -> Maybe Element

findElementContents _ [] = Nothing

findElementContents target (c:cs) = if (nothing (findElementContent target c))

                                      then findElementContents target cs

                                      else findElementContent target c

findElementContent takes in a target tag and some data (Content), and returns the element that has that tag name if it exists, and Nothing otherwise. (findElementContents is just a helper function to do the same thing with a list of Content) But findElementContents looks pretty ugly to me - what I want it to do is return findElementContent target c if that isn't Nothing, and otherwise recur on the rest of the list. The code is correct, but is it inefficient since I'm calling findElementContent target c twice? My limited understanding says no, since findElementContent is referentially transparent since it doesn't use monads (i.e. if you call it again with the same inputs it will return the same thing, always), but I'm not entirely clear on this.

- As I mentioned, findElementContents seems a little inelegant - is there a better way to do this? Is there some builtin nothing that I couldn't find?

Resources I've been using:
- HaXml reference
- standard library reference, including the Prelude

Vote today!!

Thanks a lot for y'all's comments on my post yesterday! I've decided to work on the LJ backup thing. For some reason I sorta half-started on it a while ago and was convinced that it was going to be really hard. Last night I spent most of the evening coding (djedi wanted to make sure I was actually enjoying myself and not just doing it because I felt I needed to be working, but I really do enjoy coding, especially at the beginning of a project!) and got to the point where I can fetch all comments to all of my posts and I can fetch my posts, so that's a lot of the hard part. I'm even displaying it decently, although I need to work on comment threading and prettification and such. I can even do neat statistics - 43 people have ever commented on my journal! (42 real users, 1 generic anonymous user)

Since I have to wait at home this afternoon for the cable people to come, I'm hoping on getting more stuff done as well :-)

Vote tomorrow!

(I've been kinda unhappy and antsy the last few days, so the below is as much for myself as for anyone else...)
So I'm generally happiest when I have a side project that I'm working on outside of work, especially when work stuff is boring (i.e. now). I have been working on the Netflix Prize, but as of this evening (when my last run finishes and produces the same unimpressive results) I'll be done. Things that made it a good project:

- I used C++, which I haven't used in a little while
- The whole contest framework gave me a way to quantify how well the project was going
- Using real data with real movies is always neat

Not so good things:
- Because there was so much data, the nature of the project was to work on something for a few minutes or an hour and then leave it running for days. This meant at any given point in time I probably couldn't work on the project, and my computer was tied up so doing other things was painful.
- I like projects that other people can use (vanity?), but there's not much one can use about this one. I'm thinking of putting up a related movies finder but that might violate the TOS of the contest.
- Because people are so good it became pretty darn hard to get on the leaderboard.
- It was fun to play around with algorithms, but apparently you need a good idea or to do a lot of research to find a good one, which I didn't have/do.

Anyway, so like I said I'm putting this project to bed this evening. My question is, what will I work on next? I have some ideas stored up but I'm not thrilled with any of them.

LiveJournal backup - provide some way to backup all LJ posts and comments to those posts.
Minus: There's already a decent way to do this, except for comments.
Plus: This would be potentially useful for myself and others.
Minus: I have a hard time figuring what outputs to produce: one giant page with tons of entries and comments? A zip file with pages for each month?
Minus: Because of friends-locked posts, you would only want to view it on your own computer (and not publish it or anything) unless I somehow tied LJ users in with viewers of the page, which is not going to happen.
Plus: Ooh, I could provide some interesting statistics on moods and such. Or even some kind of randomish text from your LJ posts. Maybe.

Some kind of World of Warcraft mod
Plus: I've already done this, so I have the basics down at least.
Minus: I've already done this so it wouldn't be as interesting.
Minus: I don't have an idea of what kind of mod would be useful - there are lots of them out there already.
Minus: Developing for WoW is kind of a pain, since you have to open WoW a lot and develop at the same time, which is kinda slow.
Minus: I'm already spending a lot of free time in WoW...on the one hand this means I might be using whatever mod I make a lot, but I'd also prefer a non-WoW-related project.

Adding annotations to the baseball Win Expectancy Finder graph.
Minus: This project already exists, so it's less interesting to add little features to it.
Minus: It's not baseball season anymore, making this seem less appealing.
Minus: The main problem is coming up with a placement algorithm for annotations so they don't run into each other or the graph, which sounds pretty hard.

Add some kind of Getting Things Done style tickler list to my todo list
Plus: I kinda use the todo list, and working on it might be more incentive for me to use it again.
Minus: ...but I don't really use Getting Things Done stuff anymore. Without that, the todo list is pretty good as it is.
Minus: Again, not a new project.