Tag programming (69)

Windows Phone 7 App Hub - my complaints
Mood: irritated
Posted on 2012-05-16 19:52:00
Tags: rant essay windowsphone programming
Words: 479

In the wake of complaining about background tasks in Win 8 (that was later rescinded), here are my complaints about the Windows Phone 7 App Hub, where you submit new apps and look at download numbers, etc.

Before I begin: I rant because I care :-) I'm still very happy with Windows Phone 7 (and Windows 8!) development, and plan on sticking around it for a while, unless of course Microsoft pulls a webOS like HP did. (dear Microsoft: please don't do this!) But there is some serious need for improvement here.

(Had I been writing this a few months ago, my number one complaint would be that the page itself was dog slow, but Microsoft seems to have fixed that. Yay!)


When I load the App Hub page, 95% of the time I have a simple question: How many copies did my app sell today? For webOS and Android, their consoles answer an equivalent question: how many copies has the app sold over its lifetime? Let's count the number of clicks it takes to answer that question:

First I load the page (I won't count this as a click), and I see something like this:

You might think the answer to my question is right by the word "FlightPredictor": 101. Or at least you might think that's the total number of sales. Sadly, it is the number of downloads in the last 30 days. This is interesting information (as is the graph at the bottom), but it doesn't answer my question. So click 1 is to click on the "101" number, which by the way does not do the same thing as clicking "FlightPredictor".

That takes us to this screen:

Here we can see a nice pretty graph, which is again interesting, but it only covers the last thirty days. So clicks 2-4 are setting the Start date to March 1, which is before FlightPredictor was released. (for bonus fun: this will take one more click every month!) Then click 5 is clicking Refresh Report, which shows me the full graph. But the numbers include trial downloads, which I am interested in, but I don't get paid for. To see the paid numbers, click 6 is clicking on Details, which takes me to this screen:

and then click 7 is advancing to page 2, where I can see the paid downloads for the US. If other countries had any paid downloads, I guess I'd have to add those up.


So...yeah. In theory I could get by with not resetting the date range and use the "export details" button to look at an Excel spreadsheet and look at just the most recent daily numbers that way, but most of the time I'd much rather be able to see the numbers in a browser, rather than firing up Excel. It would be really nice if this information was more easily accessable!


Windows 8 Metro: everything is fine re background tasks, nothing to see here
Mood: satisfied
Posted on 2012-05-14 21:17:00
Tags: windows essay windowsphone programming
Words: 137

After kvetching a bit about Windows 8 Metro background tasks, Jared Bienz set me straight.

The right way to do this is to either use push notifications to update the tile, or a TileUpdateManager to have it poll for tile updates. In both cases the difference is that you're not actually scheduling a background task - you're just providing a way to get a tile notification, which is a bit of XML that describes how to change the tile.

Now, this isn't ideal for me the programmer, since I need to set up an extra server, but it does scale better with lots of different apps and preserves battery life, etc. So it's all good, and hopefully lots of Win 8 apps will take advantage of one of these two ways to get power-efficient live tiles that update!


Windows 8 Metro background task restrictions are way too strict (even versus Windows Phone 7!)
Mood: confused
Posted on 2012-05-08 13:43:00
Tags: windows rant essay windowsphone programming
Words: 548

Edit: See the followup post on why this isn't a big deal, and the right way to do updating live tiles in Windows 8 Metro.

I've been working on porting FlightPredictor to the shininess of Windows 8 Metro. One of the key features of FlightPredictor is the ability to pin a live tile for a flight, and have that live tile update periodically in the background.

On Windows Phone 7, there are a lot of restrictions on background agents like the one in FlightPredictor that updates the live tiles. The agent can't run more frequently than every 30 minutes (and when it actually gets called is not that regular), if it runs longer than 25 seconds it gets killed, and most importantly there's a limit to how many you can have active on the phone - Microsoft says phones must allow 6, but my Lumia 900 allows a maximum of 9. I've actually hit this limit, because live tiles that are always up to date are cool.

So, I've started to work on this for Windows 8 Metro, and I found this whitepaper about Metro background tasks. There are a bunch of new ways to trigger running tasks (when you have internet access, for example, or when a user logs in), but you can also trigger them every 15 minutes, which is nice. There are also CPU requirements (now they're measured in CPU-seconds instead of wall time that the task runs) as I would expect.

Unfortunately, you can't use the TimeTrigger unless you display information on the lock screen. This requires extra user permission, which is OK enough, I guess. I wasn't planning on looking into showing flight info on the lock screen, but if that's necessary for my app to work the way I want it to I'm fine with that.

But here's the kicker: a user can only have seven apps show information on the lock screen. And three of these (Mail, Calendar, Messaging) are builtin! So this leaves four slots for apps to run based on a TimeTrigger in the background.

Now for the rant-y part: this is crazy. I have a bunch of apps that run in the background on my phone - as I said, having auto-updating live tiles is one of my favorite features of Windows Phone. To drastically cut the number of apps that can do this on an OS that's designed for a tablet (so presumably better battery life, etc. than on a phone) is way too strict and a step backwards.

It is possible I've missed something here, and I'd love to hear feedback on it. This is just a policy decision - what I would really like is the ability to run based on a TimeTrigger even if my app isn't on the lock screen, with reasonable limits.

Addendum: one way around the limit on Windows Phone 7 is to use push notifications - then your app doesn't have to run in the background and doesn't count against the limit. But it looks to me from reading the Metro whitepaper is that even apps that want to run based on a push notification have to be on the lock screen. Or is there a way to use a push notification to update a tile without having an app run? I'm not sure...


playing with Windows 8, 80k steps last week!, cold fusion
Mood: chipper
Posted on 2012-04-23 10:38:00
Tags: windowsphone projects programming
Words: 374


programming: checking that downloaded maps are up-to-date
Mood: thoughtful
Posted on 2012-03-27 15:18:00
Tags: projects programming
Words: 531

I came across an interesting problem when working on FlightPredictor for Windows Phone: ensuring that downloaded airport maps are up-to-date.

This is the first time I've had to deal with this problem when porting FlightPredictor:

But neither of these options were viable for WP7 - there are limits on app sizes (if I included all the maps, it could only be downloaded over WiFi), and there's no Content Provider-type mechanism. So I decided to let the app download the maps from inside the app itself, if the user chooses to do so.

This created a new problem: how could I tell when new maps were available? Here were my criteria:
So, here's what I came up with:
And my process for updating maps is to publish them to S3 first, then submit a new version of the app with a new maps index hash. If old users manage to get the new version of the maps, no problem - they won't report as out of date (since the hash won't match anything), and when they get the new version of the app they won't report as out of date either.

1 comment

Debugging Windows Phone scheduled tasks
Mood: accomplished
Posted on 2012-02-26 14:43:00
Tags: essay windowsphone projects programming
Words: 641

I was about ready to release a beta for FlightPredictor, and then I realized that the scheduled task that updated the flights that are pinned to live tiles (a key feature of the app) was horribly, horribly broken. The reasons I'd been putting off dealing with this are:
- I have a WiFi only device, and so whenever it wakes up from sleep it has to reconnect to the WiFi, and I assumed this was why the tiles didn't update very often. But this is a pretty poor excuse, and turned out to be a little true but didn't account for many of the problems.
- I had trouble finding good ways to debug the scheduled task to figure out what was going on.

So, here are the three things I did to make debugging scheduled tasks a snap! (or, at least, possible)

1. ScheduledActionService.LaunchForTest: One thing that makes dealing with scheduled tasks difficult is that they have pretty tight restrictions - the PeriodicTask that I'm using can't use more than 6 MB of memory, or take more time than 25 seconds, and it only runs every 30 minutes. This last point would make debugging unbelievably annoying - luckily, you can call ScheduledActionService.LaunchForTest in a debug build to make it launch in a minute, or however much time you want.

For some reason, after a while of debugging on an emulator instance this call seems to stop working for me, and I had kinda forgotten that it ever worked at all, so I used it early in development but not since then. But it really does work, and if it stops working, just close and relaunch the emulator! This is a great way when you absolutely, positively have to step through code and see what's going on.

2. Little Watson - Andy Pennell blogged about this, and I had had it recommended to me but hadn't bothered to include it until a week ago. If your application crashes, you can capture the call stack, and then the next time the app is launched you can give the option to email you the stack trace. This is awesome for hard-to-track-down crashes, although I wish the call stack had a little more information. What Andy didn't talk about is the fact you can use this to capture call stacks if your scheduled task crashes! I turned this on yesterday and have already tracked down a few crashes I didn't know were happening - I was hitting the memory limit and not even realizing it...

3. ScheduledTaskLogger - This is my own creation - it's basically a way to log a bunch of data while your scheduled task is running, and then view it later in the main app. This helped me to find a glaring error in my update code (note to self: always look up the return value of CompareTo(), as I've gotten it wrong at least five times now...), and I'm hoping it can help me catch any other issues that arise.

Here are the code files for ScheduledTaskLogger:
- ScheduledTaskLogger.cs - This logs data while the scheduled task is running. You can set how many logs to keep, and also whether it writes out to file after every log message - probably not a good idea for release, but useful if there's a problem that's causing a crash. I also try to save the log in ScheduledAgent_UnhandledException.
- ScheduledTaskLog.xaml and ScheduledTaskLog.xaml.cs - these are the views in the main app that you can use to view the logs. There's also a button to email a particular log to support, although I need to add one that emails of all logs.

Edit: I added this code to the Nokia Developer Wiki.


Anyhow, I've squished a bunch of bugs and am on the lookout for more. I was kind of dreading trying to figure out what was going wrong, but good tools make everything easier!

1 comment

two work vignettes; or, debugging is fun!
Mood: tired
Posted on 2012-01-19 14:20:00
Tags: essay work programming
Words: 414

1. I had a bug filed to me about a hang happening. It started happening after some code I submitted in September that seemed pretty unrelated - I added a flag to let some calls be made in a certain way that _could_ cause a hang, but I only made one call behave that way, and I was pretty darn sure that wasn't happening in this particular example.

So I tried to debug what was happening with the hang, but I don't have a lot of experience figuring such things out, so I asked someone over to help. It took a few hours and we didn't learn too much except it did seem to be related to the flag I added.

After stepping away from the keyboard a bit, I did some thinking, and realized that the only thing that made any sense is that my flag must be set on the call that was happening right before the hang. I couldn't see how that would have happened, but I had an outlandish and unlikely theory.

The next day, I was eager to check it out, and lo and behold the flag was being set! When my breakpoint got hit, I was giddy with happiness because I had reasoned out the problem. Although my outlandish theory was totally wrong (the real culprit was a 64-bit int being silently coerced to a 32-bit int...grrr), I was happy I had deduced what must be happening.

Lesson: I'm often eager to jump in and start debugging, but thinking about the problem (i.e. playing "What Do We Know?") can be valuable too.

2. I was debugging a different problem, and I knew some part of a data structure was returning some sort of error code from a particular operation. Unfortunately, I was debugging a release build, and so I didn't have an easy way of figuring out which one since there was very little debug information about local variables, etc.

So, I dropped down to disassembly (which never lies!) and stepped through one statement at a time. When I found that a particular call returned an error, I would "Set Next Statement" to before the assembly was setting up the call and then step into it. It took a little while, but I was comforted that I was guaranteed to find which part of the structure was returning the error.

Anyway, it was neat because I felt like I was totally in control of the executing code. Fun times!


taking joy in fixing a badly broken LJ app
Mood: cheerful
Posted on 2011-12-29 17:10:00
Tags: lj for webos essay palmpre projects programming
Words: 211

After the latest LJ release, I got a few emails saying LJ for WebOS was badly broken, and lo and behold they were right.

I was not excited about fixing the app, since I knew the code for parsing posts, which is most of what broke, was pretty brittle and terrible. (c.f. "don't parse HTML pages with hand-written state machines") And I don't even use the app much anymore, and it's certainly not going to sell many more copies since it only works on webOS phones, which are not exactly flying off the shelves, and new ones might not exist. So I toyed with the idea of dropping support entirely, but that just felt wrong, even though I'd rather be working on new shiny apps for Windows Phone 7.

Last night I took the first serious stab at fixing things, and it turned out to be much more fun than I had hoped. The app was so nonfunctional it felt like writing a whole new one, and it turns out the new page format is a bit nicer to parse to boot. So I've fixed maybe 60% of the issues already, and hopefully I can fix the rest by next week sometime (pending New Year's festivities) and get back to WP7.


Setting values on a view: webOS vs. Android vs. Windows Phone 7
Mood: curious
Posted on 2011-12-11 13:00:00
Tags: essay palmpre windowsphone programming android
Words: 574

Now that I'm working on porting FlightPredictor to a third platform (Windows Phone), I have a better idea of the differences between the platforms. This post is about one of the more common things to do: taking values in code and making them show up in a view.

For reference, here's a screenshot of the flight info page of FlightPredictor for Android:

This view is one of the most important ones in the app, and there's a ton of information on it. Let's focus on just one part: how to make the flight times (the "4:35 PM - 5:50 PM" in the upper-right corner) show up on screen.

For a blast to the past, let's look at Mojo first. This is the old application framework that is only used for older phones today.

In Mojo, each view is an HTML page, so in our HTML page for the flightInfo view, we include the following:

#{-publishedDepartureTimeStr} - #{-publishedArrivalTimeStr}
(I'm deliberately omitting the layout stuff...maybe I'll cover that in a future post!)
The #{} syntax means to use the property from the templateModel of the scene. (the - means don't escape any HTML in the property). So, my Flight object has these properties, and when pushing the scene I call
this.controller.stageController.pushScene({name: 'flightInfo', templateModel: event.item}, event.item);
The event.item is the Flight object, and since I'm passing it as the templateModel, that's it! All in all, this is pretty simple - as long as the property's already defined in the Flight object, I only have to change one thing (the HTML source of the view) to get it to show up.


Now, let's look at Enyo, the new application framework that is used on the TouchPad (and some newer phones). In Enyo, views are defined by JavaScript that generates HTML. (for a real introduction, see Enyo Basics - From the Bottom Up) Here, the FlightInfo kind includes this for the view:
{name: 'currentFlightTimes', allowHtml: true, content: ''}
and then in the create code for the scene, we have:
this.$.currentFlightTimes.setContent( flight.publishedDepartureTimeStr + ' - ' + flight.publishedArrivalTimeStr);
Here we have two things to do, but it's still fairly straightforward to make new properties show up.


Things are considerably less happy here. First, we have to define the TextView in the layout's .xml file:
<TextView android:id="@+id/flightInfoCurrentFlightTimes">
Then we have to create a variable for this TextView in the Activity class:
private TextView currentFlightTimesView;
Then, in the constructor for the Activity class, we have to get the view:
currentFlightTimesView = (TextView) findViewById(R.id.flightInfoCurrentFlightTimes);
And finally, when we set a flight, we have to set the text on that view:
currentFlightTimesView.setText( flight.getOriginalDepartureTime().getTimeStr() + " - " + flight.getOriginalArrivalTime().getTimeStr());
So that's a total of 4 changes to make, in two different files. This is significantly more irritating than either webOS version, and it really shows up when you have ~20-30 of these to add to the view.

Windows Phone 7:
Back to a world of happiness: with data binding, we can go back to a Mojo-like model and just add this to the XAML:
<TextBlock Text="{Binding PublishedFlightTimes}">
Admittedly, in this case we'd have to add an extra property for this (or make two separate TextBlocks), but there are plenty of cases where I just want a property that already exists. In any event, it's much simpler than Android. Hooray!

So, the final count:

webOS - Mojo1
webOS - Enyo2
Windows Phone1 (maybe 2)

I would be very curious to see what the corresponding number is for iOS - is there a way to do simple templating?


FlightPredictor progress, a few links
Mood: determined
Posted on 2011-12-02 13:41:00
Tags: projects programming links
Words: 407

The sales of FlightPredictor for Android have still been pretty flat, but it did at least get 2 5 star ratings! And it looks like more people are downloading FlightPredictor Lite, which will hopefully translate into more sales down the line. (as for the ads, I have made a whopping 3 cents so far...so that's not looking too promising)

Having said that, I'm excited to jump into the Windows Phone version. Having just finished a port makes it easier to know how to start, although I'm trying to be careful to port it intelligently, using the nice features of C# and XAML, rather than just making it a direct port. I plan on working in Windows Phone for the foreseeable future so I'd like to take the time to learn how to do things right.

One of the big wins from the Android port was being able to test the updating in the background functionality relatively early. This is one of the riskiest parts, as it's done almost entirely differently on every platform, and it's also somewhat time-consuming to test. I've made a list of basic stuff I need to have ready before I can test that, and I'm just working down the list...there are a lot of parts to FlightPredictor and just sitting down with no plan can be overwhelming, which this helps with.

In the meantime I've been reading a lot about Silverlight and such, and I discovered this free ebook by Charles Petzold(!) about Windows Phone programming, which has been excellent so far. (ironically enough, it's been very handy to read on my TouchPad...) There's also the 31 Days of Mango covering features that are new (and less discussed elsewhere).

A few links:

- The Democrats are trying to extend the payroll tax cut, but the Republicans won't vote for it because they're paying for it by raising taxes on people that make more than one million dollars a year. People affected by the payroll tax: 160 million. People that make more than one million dollars a year: ~350K.

- News Corp (parent company of Fox News) made a gay marriage guide! I know corporations aren't people (ahem), but the cognitive dissonance level is still pretty high.

- Crazy story from the New York Times: apparently Ambien can help people in vegetative states become more responsive and, in some cases, recover to some extent. This makes the long-term prognosis for people in vegetative states even more uncertain...


More irritations with developing for Android
Mood: irritated
Posted on 2011-11-05 16:32:00
Tags: rant programming android
Words: 593

I'm nearing completion of my port of FlightPredictor for Android. While things aren't as bad as they seemed when I started learning Android development and I'm able to make progress at a reasonable rate, two things popped up over the last few days:

- One of the activities in the app is choosing an airport. There are more than 900 to choose from, so I put them in a big list with a TextView on top so you can type to filter. On the emulator, filtering is very very slow (it takes around 5 seconds from typing something until the results change), and while it won't be that slow on an actual device (please?), it will probably be noticeable. So I wanted to put up a loading indicator that goes away when it's done.

Calling the filter to do its filter-y thing is pretty simple, something like

where s is the string that's in the TextView. And you can pass another parameter to get a callback when a filtering operation is done.

But! If the user types in something like "bwi" in rapid succession, it will immediately try to filter on "b". (I tried to fix this problem for a while: the Filter class even has a setDelayer() method that does exactly what I want, but it's hidden...) While that's going on and the "w" is pressed, I call to filter "bw", and assuming the first request is still going on when "i" is pressed, I call to filter "bwi". When the "b" filtering is done, the filter (intelligently) then goes on to just filter "bwi". So this means I only get two callbacks that filtering has been done. So I have to essentially write a tiny state machine to keep track of, when I do a request whether there's one already pending, and if I've already done another one when there's one already pending.

I will agree that the code didn't take that long to write - most of the time was spent looking for some easier way because someone must have wanted to do this before, right? But no luck...

- FlightPredictor includes a bunch of maps of airport terminals. These are saved as image files, and there's around 20 MB worth of them. Unfortunately, as best I can tell, some (many? most?) Android phones ship with very limited space on the device itself, but come with a big SD card to store stuff on. (this is the case on David's Galaxy S phone, anyway) You can tell apps to install on the SD card, but this won't work in my case because I have a Service that runs in the background (to update flights), and you aren't supposed to let those kinds of apps install on an SD card because apparently any SD card apps get killed when the phone gets plugged in.

What I really want is to just keep the image files (stored in assets/) on the SD card, but as far as I can tell there's no way to do this. My options seem to be:
* Take the images out of the main app and make them a separate app that users would have to download. This is a terrible user experience.
* Take the images out of the main app and, the first time it's run, add logic for downloading the images and saving them to the SD card. This is what I'll have to do, I guess, but it's quite irritating to have to write and test all this extra code to work around limitations in the OS...


Software development tools I'm grateful for
Mood: grateful
Posted on 2011-09-06 16:42:00
Tags: essay programming
Words: 577

This morning I tracked down a bug with a data breakpoint. Turns out, it was a very stupid bug that I introduced myself last week, so it was a bit like cutting a piece of paper with a chainsaw. But it got me to thinking - I'm really glad we have a full shed of chainsaws when we need them! Here are the tools I use when developing software that I'm most grateful for, with brief descriptions:

- Breakpoints: OK, this is pretty basic, but many of the later things are built on this. Being able to say "stop on this line of code and let me see the values of stuff", as well as single-stepping through code, is good enough to track down most of the easy bugs I come across.

- Conditional breakpoints: Like breakpoints, but you can set a condition that gets evaluated each time and only stops if it's true. Very useful if you have to set a breakpoint in high-traffic areas of code, but you only care about a particular VI that's being processed.

- Data breakpoints: Like breakpoints but more awesomer! Seriously: you give it an address and the program will stop immediately when that value is changed. Very useful for tracking down memory corruptions, or just "what stupid code is stomping on my variable?" type situations. Slows down execution a bit, but it's irreplaceable when you need it.

- Tracepoints: Something I only learned about last month, but in Visual Studio you can configure a breakpoint to output something to the debug window instead of actually stopping. This is invaluable in tracking down weird race conditions and tricky timing problems where actually breaking into the debugger will stop it from happening.

- Source control: Pretty fundamental, but being able to see when a line of code was last changed is occasionally very helpful. (plus, of course, being able to undo whole checkins at once, etc., etc.)

- Remote debugging: This still seems magical to me: you can copy over a small program to any machine, then from my computer I can attach to it and debug it as if it was running on my machine! Great for when a bug requires a bunch of other stuff installed to happen. Made even better by:

- Symbol server/pdb files: We archive all of our official build symbols. So, that means, as long as a remote computer is using an official version of LabVIEW, I can debug it and get symbols to see what function we're broken into, etc. Not always perfect (it can be annoying if your source isn't the same as when the executable was built), but makes my life much much easier.

- VMWare images: a great alternative to needing a physical machine to reproduce a problem on, in case you need a bunch of extra stuff installed. Even better in that it makes possible:

- Replay debugging: This is seriously amazeballs. On a VMWare image, you can try to reproduce a problem. When you succeed, you've basically recorded everything that's happened up to that point, and so you can essentially step back in time to see what went wrong. Think Prince of Persia, but without needing that pesky sand. (also: apparently it makes things kinda slow) I have not had the opportunity to use replay debugging, but I hope I do some day because: wow!

Being grateful for things makes you happy. I'm pretty happy now. What are you grateful for? (or, what did I miss in my list?)


sort of links, without many actual links
Mood: busy
Posted on 2011-09-02 15:07:00
Tags: projects programming links
Words: 477

I watched two excellent movies this week: The Hunt for Red October, and Clue. Both included Tim Curry. Make of that what you will. (I forgot he was in Red October!)

I also went to a podiatrist and got some orthotics which I've probably needed for the last 10 years or so. (thanks Dr. Newman!) Apparently my feet are as flat as a pancake. It amuses me when I stand barefoot in front of podiatrists and they flinch...

An iOS Developer Takes on Android - I haven't developed in iOS, but I am also new to Android development and sympathize sooo hard! Especially:

"You’re going to just hate Eclipse. You’re going to hate it with the heat of a thousand suns. It’s going to feel slow and bloated and it won’t taste like real food."
It takes the Android Emulator ~2 minutes to boot up on my perfectly-modern machine. But what really hurts is the edit/debug cycle. Every time I change a bit of Java and need to rerun the app, it takes about 30 seconds to redeploy and start up in the Emulator. Compare that to 5 seconds on the iOS Simulator. It may not sound like much but remember you’ll be doing this hundreds of times throughout your day.

Fortunately, it turns out to be much quicker to deploy and boot up your app on a physical device over USB.
But the reality is, it’s not HTML and CSS and so it’s another thick layer of stuff that you have to learn and understand and fight with when things don’t work like you expect.
So here’s the catch with the wonderful flexible layout system in Android: You must be very careful. If you animate certain kinds of properties, you can easily force the CPU to do all that fancy, expensive layout on each animation frame. And the CPU is very busy right now parsing some JSON from a web API or something, OK?
Anyway, I'm considering getting a real version of Eclipse and just installing the standard Android extensions. And maybe buying a Nexus S to get a better feel for the phone (and make debugging easier), but I haven't decided whether I'm that dedicated to finishing the FlightPredictor port yet.

Samsung basically says they're not going to use webOS (although they left the door a tiny bit open, I guess) - after HTC said a similar thing, this is very very bad news. Maybe someone will license webOS but the future is looking pretty grim. More likely (it seems to me) is that someone will buy the patents off of HP and call it a day. And then I will be sad.

Cool stop-motion ad, which made me eat at the place that made the ad. (stating obliquely so as not to "spoil" it...as much as you can spoil an ad, anyway. Nevermind.)

1 comment

Developing for webOS versus Android
Mood: determined
Posted on 2011-08-29 13:20:00
Tags: essay palmpre projects programming android
Words: 544

Over the weekend, I spent some time working on porting LJ for WebOS to Enyo, and after I got stuck there, I worked on porting FlightPredictor to Android. Since I was developing in both environments, I thought I'd give some more thoughts on Android (here's part 1):

I'm definitely learning more as I go, but there are still some areas (well, most areas) that developing for webOS is way easier than Android. I admit I'm biased in the matter since I've done a ton of webOS development, but sheesh:
- The Eclipse I'm using is decent in some ways and really frustrating in others. (I'm using MOTODEV Studio, because...well, I had an email from them and I just picked something) Keep in mind I'm used to developing in a text editor so the bar's pretty low here. It took me a while to realize that when you start debugging something you then have to switch to the "Debug" perspective to actually see what's going on. And 90% of the time when the app crashes, the only info I can see is that a RuntimeException was thrown, somewhere. (even when it's something easy, like a NullPointerException in my code!) Worse is when none of my code is on the call stack and I don't have the Android source code, so it gives me very little idea of where to start looking.

Honestly, the Intellisense is nice, but it just doesn't feel quite right, and often gets in my way. Hopefully as I get used to it I'll stop fighting with it so much. And the emulator is slow - it takes 15-20 seconds to get from pressing "Debug" to the app actually starting, and given the limited information I get back from the debugger it's almost not worth doing. (although if I know where the problem is, I can set breakpoints, etc., which is nice)

- To include a big list of structured data (i.e. a list of airlines), for webOS I just have to include a simple .js file assigning a big JSON array to a variable. For Android I have to generate an XML file (or a JSON file, I guess? Didn't try that...), and write initialization code to parse the XML file and store it.

- In webOS, doing an asynchronous web request is drop-dead simple - just use an XmlHttpRequest! Yay! In Android you have to spin up a new thread and post messages back and forth and such. Just reading that section of the book I'm borrowing depressed me, and I'm sure when I get to that it will suck.

- I wanted to have a Spinner (a list selector) that displayed a string and would return the associated value if it was selected. You know, like EVERY LIST SELECTOR EVER. Except, no - you can easily display a Spinner with a list of strings, but if you want to do something crazy like associate a value with it, you have to implement some interface and things generally get more complicated. What's so hard about a freaking templatized value type?? This makes me angry.

Anyway, nothing so terrible that I'm giving up, but a lot of annoyances and I really have to be in a tolerant frame of mind or I will start throwing things and cursing.


401's, why have you forsaken me? (adventures in #webOS)
Mood: frustrated
Posted on 2011-08-28 12:33:00
Tags: essay palmpre programming
Words: 671

TL;DR - 401 errors are important! Handle them correctly!

I've started work on porting LJ for WebOS (LiveJournal client) to Enyo and the TouchPad...and it is not going well.

A little background: An XmlHttpRequest is a neat Javascript feature that let's you fetch web pages or other URLs in the background without requiring a page reload. (it's the "X" in "AJAX") Key to the whole app working is being able to access protected posts, and to do this, I have to use an XmlHttpRequest to fetch a post's webpage while adding "auth=digest" to it, saying I want to see the page as a logged-in user sees it. The dance continues with LJ returning a 401 Unauthorized HTTP error, but this has the necessary information to do another request with the proper authentication. (see the Digest authentication section of RFC 2617)

This is all a bit clumsy, but worked great in the existing LJ for WebOS. Last night I got to the point where I was actually trying to get posts in the new version. First up was developing in Chrome - one of the nicest part of the new webOS framework is that it's easy to test most things in Chrome instead of having to fire up the emulator and install it, etc. I noticed that when I tried this code for the first time, I got a bunch of popups in the browser asking for a username and password for various LJ sites. If I hit Cancel on all of them, things seemed to work - the code then saw the 401 error and proceeded to authenticate normally.

This seemed odd, to say the least, so I did a quick search which led me to this StackOverflow page (yay StackOverflow!), which says that's it's a known issue in Chrome and the only way to work around it doesn't work in my case (because I need to know what the headers on the 401 error are).

Well, that's pretty annoying, and seems clearly wrong to me - the user didn't go to this page, so why is she being asked for a username/password? I could see that you might want the option to do this in some cases, but the default should be off.

So I was already a bit irritated and, after a little bit more work, decided to try it in the webOS TouchPad emulator. I watched the logs scroll by as it got to the point that it did the first request for the post pages and then...nothing. Only when I quit the app by throwing the card away did I get some not-really-sensical error indications in the log.

I rebooted the emulator (as it seems to have a problem once you do too many HTTP requests or something), and the same thing happened. I was a bit at a loss - since I wasn't getting any of my log messages I couldn't see at all what was going on. Finally I fired up Wireshark to see the HTTP requests that the emulator was making to see if there was a clue there.

Much to my surprise, I saw it requesting the same pages over and over again! It would do a request, get back a 401 Unauthorized response, and then do a request again, seemingly trying to authenticate with an empty username and who knows what password.

I looked through my code and tried a few things to make sure that I wasn't causing this, but after some more searching I found a private thread confirming this behavior and that it was a bug.

*sigh* I can't make any more progress while this bug exists, and who knows when (or if?) it will be fixed in the OS. So, if anyone asks, this is why LJ for WebOS isn't on the TouchPad, and I guess I'll move on to other projects for now...

(this probably also mean that running the phone-sized version on the TouchPad won't work either, so...sorry about that. I don't have the heart to try it out right now.)


trying to learn Android and missing webOS
Mood: angry
Posted on 2011-08-22 21:53:00
Tags: rant essay programming android
Words: 596

So, here's my plan: try to port FlightPredictor to Android and Windows Phone 7. Hopefully this will lead me toward which platform I'd rather own. Note: I am definitely not abandoning webOS, but I need a new phone and since the Pre3 is only being released in Europe and the European Pre3's will work on AT&T or T-Mobile but won't get 3G service, that's not gonna work.

So! First up is Android, mostly because I can develop for it on Linux. But it's not off to a good start. I'm reading through the mounds of documentation and not getting much of anywhere.

The very first screen in FlightPredictor shows a list of flights with their status and such. So I need to define a list in the UI and each list item I want to display certain properties of the Flight object.

In Enyo (the newest webOS framework), this is pretty easy. My list item code looks like:

{kind: 'HFlexBox', name: 'downTarget', components: [
{kind: 'VFlexBox', components: [
{kind: 'HFlexBox', className: 'flightsList info', components: [
{name: 'flightAirlineNameAndNum', allowHtml: true, content: ''},
{name: 'flightSummary', allowHtml: true, className: 'summary', content: ''},
{kind: 'HFlexBox', name: 'cities', className: 'flightsList cities', components: [
{name: 'flightOriginCity', allowHtml: true, content: ''},
{allowHtml: true, content: ' to '},
{name: 'flightDestinationCity', allowHtml: true, content: ''},
{kind: 'HFlexBox', name: 'times', className: 'flightsList cities', components: [
{name: 'departureTime', allowHtml: true, content: ''},
{allowHtml: true, content: ' - '},
{name: 'arrivalTime', allowHtml: true, content: ''},
and so on, and the code to display it looks something like:
    this.$.flightAirlineNameAndNum.setContent(flight.airline.name + ' ' + flight.num + ' ');
if (flight.category == FlightPredictor.FlightCategories.Current) {
this.$.flightSummary.setClassName('summary ' + flight.summaryClass);
if (this.showCities && flight.category == FlightPredictor.FlightCategories.Current) {
} else {
and so forth.

So, I thought a good first step for tonight would be getting something like this to work in Android.

First of all, the tutorial for ListView seems to assume that you want the list to take up the whole screen. (I want buttons at the bottom for adding a new flight, etc.) Digging around a little more, I found a page that says how to make the list coexist with other controls, and it says to do so I just make a ListView with id "@android:id/list". This is already a bad sign: what if I (gasp!) want two lists in a screen?

But, OK, I'll use your magic id. Some ids look like "@android:id/list" and the rest look like "@+id/flightsHeader" when I'm defining them and "@id/flightsHeader" when I'm referring to them. (why? I have no idea)

So then I have to define my Flight class, with annoying getters and setters that aren't necessary in JavaScript. After another hour of searching and getting frustrated, do I use the ArrayAdapter or a SimpleAdapter? (I'm just trying to define my flights in code - storing them in a database or whatever is for another day) ArrayAdapter lets me pass in an array of them, which I need, but it doesn't let me map the object fields to id's in the list template like SimpleAdapter does. But SimpleAdapter seems to only work on XML objects, or something?

Lists are important. I'm frustrated that the ListView tutorial is so drop-dead simple that it only touches the very simplest use case for a list (yay, a list of strings!) It looks like what I want is something like this list example except
- that's an awful lot of code
- it still doesn't show me how to set values into things in a resource file, which is supposed to be how all non-trivial views are done! Arrrrrrrrgh.

Anyway. Not a productive night. I want to stick it out and accomplish something before giving up, because that seems like the right thing to do, right?


byte manipulation in Javascript = not fun
Mood: irritated
Posted on 2011-02-24 09:22:00
Tags: palmpre projects work programming
Words: 191

Google's two-factor authentication is a really good idea, so I'm trying to port the Google Authenticator to webOS. This involves calculating hash values, etc. in Javascript, which is a giant pain because Javascript knows nothing but signed 32-bit integers and 64-bit floating point datatypes. So every time I look at the sample Java code and it uses a byte[], that means I have to manually convert to the -128..127 range. And the sample code does some things like converting 4 bytes to an int, which is easy in Java but a pain in Javascript.

Anyway, finally I had the bright idea to actually compile the Java code so I could compare the results, because I generated at least twenty different codes trying various permutations of things, none of which were right.

Another thing I learned! If you return an error string from a constructor, this does essentially nothing. You still get a constructed object. So, if your library is doing this, there's a 99% chance that no one's going to notice without a lot of pain in suffering.

Adding to my troubles: I am involved in some heavy yak-shaving at work.


bridge update (or: programming is hard)
Mood: worried
Posted on 2010-10-27 14:00:00
Tags: bridge projects programming
Words: 477

After vacation and traveling and such, I've been working on the bridge program again. I'm at the point in the webOS client where I can now play a complete hand, bidding and all. It even looks relatively nice with cards sliding around, things fading into view, etc. Ask me to show it off if you're interested sometime!

So the next task is adding the AI for playing cards. I've decided to go with a rule-based system, where there are a bunch of rules that can decide if they apply. So I have "second hand low" and "third hand high" rules that go near the end (since they're fairly generic).

Last night I tried to sit down and really hammer out the rule that applies if you can determine exactly what will be played. If you're the last player on the trick, this is relatively easy - see if your partner is winning, if not win as cheaply as you can, if so (or if you can't win) throw some trash.

Even this is a little vague - what card should I throw away? Well, if there's only one suit I can play it's easy, but if not then it's hard. So I put this off with a TODO.

The tricky part is if you're the third player on the trick and the fourth player is the dummy. Then it's something like: look at all dummy's cards and the declarer's card that was played (if it's currently winning), then see if I can beat all these. If so, play the cheapest. If not....well, we are in the third hand and we'd like to force dummy's best card if we can. So find dummy's second best card and see if we can at least beat that. If not...well, at least we should try to beat the current high card played by declarer. If we can't do that, then just toss something.


This took a while to code up, and I need to test it thoroughly because I'm very much not confident in it. And it still leaves a lot of details out - what if dummy has AK6 and I have the 57 - under the algorithm I'd play the 5 which is pretty clearly wrong. It seems like to handle this case I should have been keeping track of which cards are winners in their suit, which is another layer of complexity!

And all this is for a (all things considered) pretty simple case where I can see all the cards. I'd also like to keep track of how many cards of each suit each person has (based on the bidding), which would help but would also add enormous complexity.

Maybe the AI should cheat and see all the cards? Is there some better thing to keep track of?

It could be a long time before the game seems to play intelligently...


minnesota, civ 5, bridge
Mood: busy
Posted on 2010-09-20 12:07:00
Tags: pictures bridge projects programming
Words: 208

I went to Minnesota last week on a recruiting trip and took a few pictures:

Civilization 5 is releasing tomorrow! The full manual is available online (warning: 233 page .pdf) and it look spretty good. I do miss having a paper manual, though.

I've been working on the bridge program for webOS. It's coming along nicely - the framework is all in place to bid and play, and I've done some of the graphical stuff so you can almost actually play a hand on the webOS emulator. The next big thing to do is the AI for bidding and playing, which will probably be big tasks. We're going to be fairly busy over the next month or two so I'm not expecting much progress for a while. (also, Civ 5 coming out certainly doesn't help :-) )

Happily, since webOS is Javascript-based it was pretty easy to make a webpage version of the game, which will make testing out AI algorithms, etc. much easier.

Eric Fischer took census data on race and mapped where people live in major cities. (here's Houston, Austin, and San Antonio) Some patterns definitely jump out at you, but if you zoom in you can see there's nowhere that's exclusively one race, which is a good thing.

1 comment

bridge it is!
Mood: thoughtful
Posted on 2010-09-07 15:10:00
Tags: palm projects programming
Words: 107

After some thought and some more research about what webOS app to work on next, it looks like the mystery option is going to be somewhere between difficult and impossible, and it's going to be a while before I can tell which.

So, bridge it is! It's going to be pretty tough, but a few people have volunteered to help, and it's better to aim high, right? So far it can deal out hands and count points...

I can't decide whether the hardest part will be the AI for bidding, playing, or just making the graphics look nice (with pictures of cards, etc). I guess we'll see!


new external HD, project, life
Mood: excited
Posted on 2010-04-05 13:51:00
Tags: palmpre projects programming
Words: 433

My backup strategy as of last week was to copy the really important stuff onto my 16GB USB thumbdrive relatively frequently, and more infrequently copy everything onto an external hard drive. Feeling rather clever at the time, I bought an enclosure like this, so I could buy a cheaper SATA hard drive and stick it in there and plug it into my computer via USB. Then when I wanted to upgrade drives I could just buy another SATA drive.

This sounded good in theory, and indeed kinda worked, but it was very slow for some reason - copying 200GB over to the drive and then untarring the 200GB file took around 3 days. Because of this I backed up way less often than I should have.

As luck would have it, the SATA drive seemed to die last week, and so after doing a little research I bought a Western Digital Elements USB Hard Drive with 1TB of space for $90. It was easy to get it to mount on Linux, and took less than 24 hours to do the same copying/untarring procedure. And it doesn't look totally cobbled together! So I'd recommended it if you're looking for an easy backup solution. You can also get a version with fancy software and an e-ink label on the outside that was actually kind of tempting.

I've been working for a little while now on my next WebOS app, which is called FlightPredictor. (unless I decide to change the name :-) ) It's an interface to FlightCaster.com, which can predict airline delays ahead of time and generally has lots of interesting data. (see a sample report) They already have iPhone and BlackBerry apps, so I'm happy to bring a third platform into the mix. (and before Android, even!) I'm just putting the finishing touches on it and incorporating some API changes, but right now the best part is that it does WebOS notifications to let you know when the information has changed so you can always keep on top of things.

The last few weeks have been quite busy, and I'm starting to wear down a bit. Between squeezing any time I can to work on FlightPredictor, plus a lot of yardwork we've been doing (planting a few new trees, making a rock garden), plus the usual weekend errands, plus volunteering at the tax center...it's getting to be a bit much. Hopefully things may quiet down soon but it's conceivable we/I'll start flaking on things to stay sane...

Baseball season has started! Here's a handy map to what parts of the country root for which teams. Go Astros!


Palm and LJ for WebOS
Mood: thoughtful
Posted on 2010-03-04 13:14:00
Tags: lj for webos essay palmpre projects programming
Words: 747

I worry about Palm sometimes. They recently lowered their guidance for this quarter, analysts don't seem too upbeat, and their stock price for the last year looked promising when they released the Pre in July, but has dropped dramatically since then.

More concerning is the fact that, 9 months after releasing the first WebOS phone, Gartner estimates that 0.7% of smartphones are running WebOS. Hopefully this will improve now that they're on Verizon (and rumor is they'll be on AT&T sometime this year) and once they launch in more countries.

The good news is that the mobile phone market isn't quite like, say, the social networking website market, which has a very strong network effect. If all your friends are leaving Friendster for Facebook, then Friendster is less valuable to you, and you'll probably switch to Facebook. But I can still use the web just fine from my Palm Pre even if the rest of the world switches to iPhones and Droids and Nexus Ones. There is somewhat of a problem that if fewer people use WebOS, fewer people will write apps for it, but this is more of a slow process. Also, at least in the US, most people are under contract for their phones and so they only have an opportunity to switch cheaply every one or two years. I'm really hoping Palm can keep turning things around - just yesterday they released an update to the Facebook app that makes it much nicer.

Speaking of apps...

LJ for WebOS has been doing pretty well since my last update - as of this very moment I've sold 72 copies. It seems fairly random how many copies are sold a day - thanks to the new app My WebOS Apps I have a nifty graph on my phone with these totals for the last week: 5, 1, 1, 2, 4, 2, 0. So...who knows?

One of the frustrating parts has been seeing bad reviews indicating that it just isn't working for a few people. Most of these reviews came early and I'm pretty sure I've fixed the bugs since then, but most people don't go back and edit their review when problems get fixed, and I have no way of contacting them to ask them if it's working for them and to try to diagnose their problem if not. I've been trying to make it more and more obvious how to contact me to the point that if you can't load the posts a dialog comes up with a button to email me the problem. We'll see if this helps at all. Encouragingly, more of the recent reviews have been good than bad, bringing the average back up to 3.5/5 stars.

I spent a lot of last week working on a new feature that I really wanted to add: the ability to browse other people's journals. I even wrote the parsing code before I got stuck on a problem that I've gotten stuck on before - the inability to properly authenticate so that the client can load friends-only posts. The API way to do this is to call the sessiongenerate method - unfortunately LiveJournal's cookie scheme has changed because of some security holes, and no one's gone back and updated or added a new API.

Every time I run into this problem, I spent some time trying to "fake it" by POSTing the right thing to the login page, essentially pretending I'm a regular user signing in from a browser. As in the past, I can't get this to work and I'm not sure why, and it's really frustrating to try to debug because it's all guesswork.

So I was a little down about that, and the last release (made it to the App Catalog yesterday) only had a few small features like deleting posts. This week I took a stab at some random not-quite-featurey things that have been on my list for a while, and everything just kinda worked. The next release will use a lot less bandwidth on the initial request (66% less in my test case!), and it fixes a bug with comments not posting by showing the CAPTCHA dialog that LiveJournal was returning. I was amazed that both of these basically worked the first time, so that was a nice pick me up :-) I'll probably submit it to the App Catalog tonight after a bit more testing.

I'm running out of features to work on that are actually possible to do, so I'm very open to suggestions!


LJ for WebOS mentioned!
Mood: happy
Posted on 2010-02-04 23:31:00
Tags: lj for webos projects programming
Words: 26

Yay! LJ for WebOS was mentioned in the latest news post. If anyone's interested, here's the homepage, and I'm happy to answer any questions about it.


LJ for WebOS released!
Mood: accomplished
Posted on 2010-02-01 10:50:00
Tags: lj for webos projects programming
Words: 57

I'm happy to announce that my LiveJournal client for WebOS devices has just been released on the Palm App Catalog! It includes support for:
- viewing your friends post and own posts
- making posts with custom security, userpics, and tags
- uploading photos to imgur.com and including them in posts
- viewing cached posts offline

Here's the official home page.


my first WebOS app published!
Mood: excited
Posted on 2009-12-16 16:49:00
Tags: palmpre programming
Words: 235

PasswordHash is now officially available on the Palm App Catalog! After some initial hiccups I was able to install it to my Pre and it works just fine. Hopefully my other app will be approved soon...

Speaking of my Pre, I had been having some problems with it lately - it thought that headphones were plugged in to the headphone jack all the time, and so I couldn't listen to music or use the phone except on speakerphone, which got annoying pretty quickly. I tried some internet-suggested remedies that worked for a little while, but this weekend even those stopped working, so I took it in to Sprint to see what they could do.

I dropped my phone off at the nearest Sprint service center, got lunch and returned to have them tell me I'd be shipped a new phone and it would probably arrive the next day. And they let me keep my phone in the meantime (and ship it back when the new phone arrived). Lo and behold, yesterday it arrived, today I took it in to be activated, and it seems to work like a charm. After being careful not to nuke my existing backup (not actually sure if this is a problem anymore, but better safe than sorry!), it transferred over my contacts, apps I had installed, and even bookmarks! So +1 for Palm and Sprint for taking care of the problem.


WebOSJournal reviewed!
Mood: happy
Posted on 2009-12-11 13:26:00
Tags: palmpre programming
Words: 39

So WebOSJournal (my LiveJournal client for the Palm Pre/Pixi) got a nice review. I've submitted it to the Palm App Catalog, and hopefully it'll be up there soonish!

As a bonus link, The Year In Ideas is pretty interesting.


links for what should be friday
Mood: hyper
Posted on 2009-12-03 15:10:00
Tags: palmpre programming links
Words: 137

Seriously...long week anyone?

I put up WebOSJournal on the PreCentral homebrew gallery. Still plenty of work to be done, of course...

What every programmer should know about memory - really long article (and it's only Part 1!) but I learned a lot. (and remembered a lot from college classes :-) )

Apropos of net neutrality, How Robber Barons hijacked the "Victorian Internet" (i.e. the telegraph system). The short version is that there was no government regulation and Western Union/AP fixed an election and conspired to keep non-AP newspapers out of business. Less government regulation doesn't always equal more open markets!

Drug-Makers Paying Off Competitors To Keep Cheap Generics Off Market - pretty much what the headline says. It sucks, but apparently it's legal.

No one wants America to be the sole global superpower, but no one wants to share the load.


rethinking the plan
Mood: tired
Posted on 2009-11-24 10:30:00
Tags: palmpre projects programming
Words: 121

I signed up for the Houston Turkey Trot 5K with my family. Unfortunately, it starts at 8 AM (which is very early for me these days) and I'm not in great shape and it's going to be cold, which doesn't do so great on my lungs. Hopefully I survive!

Been working on WebOSJournal (the LiveJournal client for the Palm Pre/Pixi) - it's coming along decently but I'm running into some frustration trying to allow replying to posts/comments. The authorization scheme is tricky and of course you don't get useful feedback. I actually downloaded the LiveJournal source code to try to figure out what I'm doing wrong, but it's hard to find my way around...

Obama kicks off massive science education effort - yay!


Test post
Posted on 2009-11-17 22:18:00
Tags: palmpre programming
Words: 5

First post from Palm Pre?


life update
Mood: busy
Posted on 2009-10-21 13:24:00
Tags: palmpre house programming
Words: 244

We finally got our roof replaced (and got a gutter on the back of our house to boot), and are almost done with getting the insurance settlement and paying the roofers and whatnot. I'm looking forward to putting this behind us. How often do roofs get damaged like this, anyway? The damage happened in March and we're still not quite done...

I've been getting back into origami lately - found a really cheap source of paper and ordered a bunch that should arrive today. My sorta-goal is to make all five Platonic solids - so far I've made a (bad) tetrahedron, a decent octahedron and a pretty cool cube. Anything with equilateral triangles = hard.

We cooked something out of the Joy of Cooking last night! Some kinda chicken dish with garlic and lime and potatoes. It turned out pretty well.

Planning on volunteering with the Community Tax Center in the new year. It's been a while (too long!) since I've done any volunteering, so it'll be good to get back into it.

As I mentioned earlier, I've been working on a LiveJournal client for the Palm Pre/Pixi. It works decently for me, but when I gave it to a few people to test it freezes. (luckily, you can just kill it so it doesn't mess up the whole phone) I'm at a bit of a loss as to why this is happening and it's pretty disheartening, so I haven't worked on it in a few weeks.


developing for the Palm Pre
Mood: contemplative
Posted on 2009-10-09 10:18:00
Tags: palmpre programming
Words: 179

This week Palm announced how you'll be able to publish apps for the Pre, Pixi, etc. There seem to be three options that I'd consider:
- Join the Palm Developer Program (PDP) for $99/year and pay $50/app to get it in the App Catalog on the phone (paid or free)
- Join PDP for $99/year and distribute it on the web (paid or free)
- Release it open source and distribute it on the web (no need to join PDP in this case)

Basically, if I wanted to sell apps, I'd have to choose options 1 or 2, which are a bit more expensive than I'd like for a hobbyist project. I've been working on a LiveJournal client under the assumption that I'd be selling it for $2.99 or so - I haven't seen any other clients and I put a fair bit of work into parsing posts and comments, etc.

But I think I'll probably just release it on the web open source and put up a Donate button or something. Costs me nothing and it will let more people use it.

1 comment

Rice pictures
Mood: determined
Posted on 2009-10-01 10:20:00
Tags: pictures programming
Words: 388

I visited Rice for some recruiting this week, and tried to document all the things that have changed since I graduated. The result is here:

This has been, for various reasons, a crazy few weeks. I'm hoping things have settled down a bit now.

One of the things I've been doing is competing in the Google Code Jam, an online programming contest. The structure is interesting - each problem has a small set and a large set of data to solve. The small set you can usually get by doing a straightforward program/algorithm (although not always!), and you have 4 minutes to download and submit your solution. If it's incorrect you get notified right away and you can try again on a different data set. The large set is much, well, larger, and you usually have to do something pretty clever to solve it. You get 8 minutes to download and submit it, but you only get one shot and you don't know whether you succeeded or not until the contest is over.

The first round was the qualification round (you had to get one small set and one large set to advance) and after that was Round 1A. There were also rounds 1B and 1C, you could compete in any/all of them, and the top 1000 from each round advanced to round 2. I did well enough to advance, so I started doing more practice rounds from previous years with varying levels of success. This culminated in trying round 2 from last year the day before the contest and getting 0 points. I was kinda frustrated at this but it sorta worked out well because I was resigned to probably not advancing. In the actual round 2 I got 2 small sets and 1 large set, which I considered respectable - it put me in 1078/3000 people, but only the top 500 advance (and get a T-shirt :-) ). So I'll probably do it again next year but without a lot of practice or something I'm not sure I'd ever see the ways to do most of these; Google publishes a contest analysis of each round after the fact, and those round 2 problems are doozies.

The upside is that it made me want to jump back in to my other programming projects, so hopefully I'll get to do that soon!


still liking my Pre
Mood: confused
Music: Michael Giacchino - "Star Trek" soundtrack
Posted on 2009-08-20 14:17:00
Tags: palmpre programming politics links
Words: 131

Palm is now accepting submissions for the Pre App Catalog, so I went ahead and submitted PasswordHash. That one will be free but I'm thinking of charging $2 for my next one...

LabVIEW has a lot of handy keyboard shortcuts.

codepad.org is a neat place to quickly try out/share code in a bunch of different languages.

The latest in health care: maybe the Democrats are going to give up trying to compromise since the Republicans don't really seem interested, although Chuck Grassley may or may not be. Honestly, I've kinda given up trying to figure out what the hell is going on - I hope something good passes but hearing how laws that affect us actually get passed makes me sad.

The Longest Poem in the World made out of rhyming tweets.


link friday!
Mood: groggy
Posted on 2009-07-10 09:58:00
Tags: movies programming links
Words: 380

And since it's 8(!) days until the wedding, presumably my last one as a swinging bachelor.

This fascinating NY Times graphic shows the various business cycles and the recession we're in now. And maybe some hope that things will be better in 6 months.

A 60 foot Gundam robot has risen in Tokyo. I really really hope this is real.

Short article on the 40th anniversary of Stonewall. Found this surprising:

In 1966, three years before Stonewall, Time, then the voice of middlebrow, middle-class respectability, published a long essay on “The Homosexual in America.” The magazine, while acknowledging that “homosexuals are present in every walk of life,” concluded that homosexuality
is a pathetic little second-rate substitute for reality, a pitiable flight from life. As such it deserves fairness, compassion, understanding and, when possible, treatment. But it deserves no encouragement, no glamorization, no rationalization, no fake status as minority martyrdom, no sophistry about simple differences in taste—and, above all, no pretense that it is anything but a pernicious sickness.

Sacha Baron Cohen's (Borat, Ali G, etc.) new movie Bruno comes out today- this character is a gay fashion person or something. He went out of character on Letterman describing how he interviewed a terrorist. Salon thinks it's bad for the gays, Slate thinks it's good for the gays, and Ebert surprised me by liking the movie.

Ricky Gervais's The Invention of Lying looks like a good movie, and it has "every other Hollywood actor and comic in the world" (Patrick Stewart! Tina Fey! Jason Bateman and Jeffrey Tambor! John Hodgman!)

Cool jQuery plugin to dynamically make nice-looking graphs - uses the new canvas feature, like my marriage map does. Wish I had thought of this!

California v. Texas - take that, California! Although
Despite all this, it still seems too early to cede America’s future to the Lone Star state. To begin with, that lean Texan model has its own problems. It has not invested enough in education, and many experts rightly worry about a “lost generation” of mostly Hispanic Texans with insufficient skills for the demands of the knowledge economy. Now immigration is likely to reconvert Texas from Republican red to Democratic blue; Latinos may justly demand a bigger, more “Californian” state to educate them and provide them with decent health care.


Optimizing iTunesAnalysis: faster database access
Mood: tired
Posted on 2009-07-09 09:44:00
Tags: optimization essay projects programming
Words: 280

The second in an occasional series
Idea: Now I'm focusing on improving the time to insert the iTunes data into database. Where we left off last time, our script took 71 seconds to run, ~50 seconds of which was database operations. The idea I had to speed this up was to batch a bunch of queries together and thus make fewer calls to the database. It turns out this actually slowed things down.

So I did a little research and it turns out if you insert data with the same query structure over and over again (but with different bind variables), the database doesn't have to reparse the query which speeds things up a lot. I tried doing this with pyPgSql but couldn't find any documentation how it was supposed to work, so I switched to using psycopg2 and changed the query for inserting the playlist data. Just switching to a psycopg2 sped things up a lot, it seems. I tried switching to a similar sort of query for inserting track data, but that actually slowed things down.

Anyway, the new script runs in 25 seconds, and it looks like only around 9 seconds for database operations. This is a 400% speedup in the database time! Overall, this step improved performance by ~180%, and since we started at 114 seconds we've improved ~350%.

Conclusion: Another big success, and I'm not sure how much more I can squeeze out of the iTunesInfo.py script. Next time I'll focus on the analyzedb.py script, which does the analysis from the database - right now it's taking between 5 and 12 minutes to run on my library of 6400 tracks.

Source files:
- old script
- new script


Optimizing iTunesAnalysis through smarter parsing
Mood: geeky
Posted on 2009-07-08 10:31:00
Tags: optimization essay projects programming
Words: 631

The first in an occasional series
Intro: A while back I wrote a script to analyze an iTunes library and find your favorite artists, albums, etc. It works pretty well and I regularly use it to update my own analysis. Unfortunately, it generally takes a long time to run, which is sort of OK for me (because I just start it running and go do something else) but less good for people who are running the analysis through the web site.

So I'd like to make it run faster, and I have a number of ideas to do so.

Idea: There are two main parts to the system - parsing the iTunes Music Library.xml file into a database, and running the analysis on the database. First I'm focusing on the parsing part.

The first version of the parsing script uses Python's xml.dom.minidom package to completely parse the library file.

After profiling the first version by running python -m cProfile -o profiledata.oldway iTunesInfo.py "iTunes Music Library.xml", I see that the whole parsing process takes 114 seconds. The major parts of this are 60 seconds for the xml.dom.minidom.parse method and 46 seconds for the database operations. Note that this only leaves ~8 seconds for figuring out the track information - clearly this is not the bottleneck!

So I'd like to improve parsing speed. There are two basic kinds of XML parsers - what we're using now is a DOM or Document Object Model-style parser, which means that the parser reads the entire file in and returns a parsed structure containing all the data. (I remember writing a simple XML parser that did this as a project in COMP 314. Ah, memories...) The advantage to this method is that after the parsing is done, it's easy to traverse the DOM tree and find the data that you're interested in. The downside is that, well, it's slow. Also, the entire document has to be read into memory which means that your memory usage is proportional to the size of the file you're processing, which adds to the slowness and can lead to out of memory problems on huge files (although we weren't seeing that here).

The other basic kind of XML parser is known as SAX, or Simple API for XML. You provide callback functions that are called whenever the parser runs across the start of a tag, end of a tag, character data, and...that's it. Whatever processing you want to do you have to do in those callback functions. So if you're just, say, counting the number of <key> tags in a document this works really well. It's also much faster than the DOM-style parser, since it doesn't have to generate a giant tree structure. But doing the sorts of processing we're doing on the library file seems a bit more tricky.

Anyway, I take a stab at it, and after a bit end up with version 2 of the script. Notice that the logic in the Handler class is a bit twisted - we have to keep track of where we are in the document (so if things get out of order we'll have problems) and use a state-based system which is a bit brittle and unclear.

But how does it perform? The old version of the script ran in 114 seconds, and this version runs in 71 seconds for a ~60% increase in speed. But really, it's better than that, because the database operations still take around 50 seconds - if we subtract that from both we get 64 seconds versus 21 seconds which is a ~200% increase in the speed of the parsing.

Conclusion: This was a big success! Most of the time is now in the database layer, which I have some ideas for speeding up next time.

Source files:
- old script
- new script


Games database
Mood: cheerful
Posted on 2009-06-27 20:20:00
Tags: projects programming
Words: 58

Just got done working on a database of the games we own. Not everything we own is in there, but most of it is, and I'll try to get the rest added soon. You can search based on number of players and the length of the game. Give it a shot and let me know what you think!


faster pretty pictures
Mood: cheerful
Posted on 2009-04-22 11:05:00
Tags: projects programming
Words: 56

Just finished rewriting the backend to Pretty Pictures from Genetic Algorithms in C++ (it used to be in Python). It's easy to forget how much faster C++ can be than Python - I even turned on an option by default to make all images color, which made things unacceptably slow in Python. It's fun to play with!


links for everyone!
Mood: busy
Music: Official Lost Podcast
Posted on 2009-03-02 12:11:00
Tags: projects programming politics links
Words: 212

So I won the code bounty! I've really been enjoying writing Firefox extensions - it's easy to get started and fast to see results, and I'm starting to understand XUL better. Anyway, Kate (who offered the bounty) is going to polish it up and release it, at which point I might consider using it - it's convenient and is a better solution to passwords than a tiered password scheme.

I also got the $100 Amazon gift card, which I'm not sure what to do with. Is this good because I'm less materialistic or bad because I'm not excited about anything in particular at the moment?


- Happy Texas Independence Day!

- If you like crazy Obama theories, you'll like this Daily Show segment! Sometimes I wonder where they find these people...

- The government is bailing out AIG some more, but the real story is that AIG lost $62 billion in 3 months!

- In a recent study, only 29% of people supported gay marriage, but this number went up to 43% if assurances were made that no church would be required to perform gay marriages. Which is kind of weird, because no church is required to perform particular kinds of marriages today.

- Obama frames things as people versus corporations and industries rather than Republican versus Democrat.


more whereslunch.org progress
Mood: happy
Posted on 2009-01-26 12:39:00
Tags: whereslunch programming
Words: 122

Over the weekend I
- added geolocation to the front page of whereslunch.org - it should automatically forward you to your closest city now!
- made the map use more horizontal space (and got rid of the ugly gap between the map and the sidebar)
- made the page statically load the list of restaurants, which should improve load time a little and should help the ads be more relevant, hopefully.
- upgraded to the latest jQuery which makes things look a bit different, especially the sidebar. Not sure whether to keep this new color scheme (especially the filtering sliders), but I'll leave it for a few days and see what I think then. This should also improve performance a bit.
- a few minor random performance improvements


Try whereslunch.org!
Mood: excited
Posted on 2009-01-16 10:11:00
Tags: whereslunch programming
Words: 122

I've been working on whereslunch.org for a while now, and I've finished everything I thought I'd need to release it. But I'd really appreciate any feedback y'all have about how to make it better before I do that :-) Please give it a shot!

Some things that might have changed since you last used it:
- Fully supports IE7 and IE6
- You can edit tags on restaurants you create
- There's a small FAQ
- less buggy!

Also under the category of "things I wrote", I'd like to make my ThumbnailCopy extension for Firefox public (so anyone can download it, rather than having to be logged in), but it needs more reviews before they'll do that. So if that's your kind of thing, I'd appreciate it!


ThumbnailCopy firefox extension
Mood: happy
Posted on 2009-01-03 12:19:00
Tags: pictures projects programming
Words: 38

So I wrote this ThumbnailCopy extension (see Kurt's post) and in order for it to go public people have to register for an account there and review it. So...anyone mind doing that? :-)

Also, Texas Bowl pictures are up.


"I wish I had done this!"
Mood: geeky
Posted on 2008-11-26 10:16:00
Tags: essay math programming links
Words: 387

"I wish I had done this!" is my highest praise for a website. The last time I can remember using it was for wowjutsu, which tracks raid progression in WoW by looking up what gear people are wearing in the Armory and matching that with where that gear came from. Simple idea, useful, interesting, but the technology behind it is something I totally could have done.

My newest "I wish I had done this!" is StateStats. You enter a search term, it finds which states in the US search for that term more per capita, then gives you a nice heat map of the US. But then it correlates that with a host of other state rankings: obesity, income, high school graduation rates, voted for bush, percent youth, etc., etc., etc. So you can see that searches for "prius" are correlated with income and negatively correlated with energy consumption. Or searches for "gay" are correlated with density (i.e. more urban states) and negatively correlated with voted for bush. Or searches for "lsu" are highly correlated with, well, being Louisiana. Or searches for "coke" are highly correlated with obesity, while searches for "soda" are highly negatively correlated with obesity. (huh?) Or searches for "tea" are correlated with income and negatively correlated with voted for bush.

Anyway, it's a ton of fun to play with, and the example queries ("garth brooks" is highly correlated with voted for bush!) are interesting, but it's even more fun to think of a common search term and see what pops up.

The correlation metric it's using is just based on rank and not intensity - i.e. it's just the order of the 50 states that matter, not how much the first place one is bigger than the second place one. This probably leads to some false positives when the numbers are very close together. Also, I'd imagine there's a natural inaccuracy determining which state a particular query is coming from, and since you're looking at things only on a state by state level (as opposed to county by county or something) it's not as precise as it theoretically could be.

And don't forget correlation is not causation - searching for "hockey" does not make it colder outside, or make you richer.

I award StateStats the official "I wish I had done this!" seal of approval.


the usual
Mood: busy
Music: Peter Gabriel - "Down to Earth" (from Wall-E soundtrack)
Posted on 2008-08-12 14:31:00
Tags: pictures programming links
Words: 90

Since we're leaving tomorrow, I put up the pictures I had built up. Here are a few more ASMC pictures (check out my awesome chart!), here are a few recent random ones, and the most recent puzzle we did.

Strong Bad's Cool Game for Attractive People, Episode 1: Homestar Ruiner is out!

Barack Roll - memes upon memes...

I downloaded the iPhone SDK and may play around with it some, although I don't have a great idea for an app.

My throat gets tired talking for hour-long chunks at a time!


not the ideal situation for starting 3 solid weeks of rehearsals
Mood: tired
Music: Nas - "Black President"
Posted on 2008-07-09 13:33:00
Tags: asmc worldofwarcraft programming politics links sleep
Words: 260

I've been unable to get to sleep the last two nights. Last night was particularly bad so eventually I got up and programmed some and played a little WoW. (won a close AB battleground on my 46 pally) Even then after returning to bed it took me a while to sleep. I had no sodas yesterday, which is the most common cause of insomnia for me. Hopefully it just kinda disappears. The alternative is that I get so tired that falling asleep is no problem, but I'd kinda like to avoid that :-)

White House, 2005: Setting the Record Straight: Democrats On An Artificial Timetable In Iraq

The President Explains Why Timetables For Withdrawal Are Bad Strategy. PRES. BUSH: "Setting an artificial deadline to withdraw would send a message across the world that America is a weak and an unreliable ally. Setting an artificial deadline to withdraw would send a signal to our enemies - that if they wait long enough, America will cut and run and abandon its friends. And setting an artificial deadline to withdraw would vindicate the terrorists' tactics of beheadings and suicide bombings and mass murder - and invite new attacks on America." (President Bush, Remarks On The War On Terror, Annapolis, MD, 11/30/05)

Iraq, yesterday: Iraq Wants Withdrawal Timetable In U.S. Pact
Iraq's national security adviser said Tuesday that his government would not sign an agreement governing the future role of U.S. troops in Iraq unless it includes a timetable for their withdrawal.

Unrelatedly: you can't protect your HTML source and it's a waste of time to try.


a short list of things that anger me
Mood: enraged
Posted on 2008-07-07 14:44:00
Tags: work programming
Words: 27

- Visual Basic 6
- bugs that don't happen when you set a breakpoint but do otherwise
- bugs that make you think you're losing your mind
- Visual Basic 6


ah, Heisenbugs
Mood: annoyed
Posted on 2008-07-02 09:40:00
Tags: work programming
Words: 68

Yesterday I was going to say that, when debugging a problem, I had caused another problem by popping up debugging MessageBoxes. (tip: VB6 does not like coercing non-string-like objects to strings implicitly) But after I fixed that, it turns out that popping up MessageBoxes actually seems to fix the real problem. Arrrrgh.

Happy 4th! Here are 21 hilariously hyperbolic pro-America songs. I recommend "America, Fuck Yeah" (number 21).


rattling around in my brain
Mood: pleased
Music: Run Lola Run Soundtrack - "Running One"
Posted on 2008-04-15 13:54:00
Tags: projects programming
Words: 54

I like being around people. I like having people over/visiting people. I'm sorry if I don't give out that impression.

watchdog.net launched today and it looks pretty neat - I volunteered my services. One of the founders wrote this essay on how to be more productive.

jQuery is now my default choice of JavaScript library.


a day of unnecessary stress
Mood: busy
Posted on 2008-02-19 10:01:00
Tags: django work programming
Words: 282

First I was stressed about a recruiting meeting, which ended fine and without my being asked any questions. Then I was stressed about playing bridge with onefishclappin and the gang - it went fine, I had a lot of fun and I think I played at least well enough not to embarrass myself (or onefishclappin). Duplicate is a really nice format, and the bidding boxes are very elegant. 24 hands in 3 hours is a lot, though - my brain was fried by the end!

I was also stressed by a horrendously tricky bug at work, but that's worth stressing over, at least during work hours.

Carpet costs more than we thought it would (anyone surprised?) but it'll be fine. I just want to get the damn thing installed, really.

Now I'm stressed about taking my MacBook in today to see if they can fix my "Control" and "v" keys. That would make me a happy person.

Django question: (sorry yerfdogyrag, meant to ask you last night but see earlier bit about brain-frying) On my main page, I want a drop-down box that selects which person to see gifts for, and then some way to do that. Two options I've come up with:

- Use a <form> with a <select>, and GET the results to a different page. The problem with this is that the URLs get kinda ugly, which I'm not thrilled with. Should I do a redirect to the nicer-looking URL?

- Do some javascript stuff in the <select> element that changes the target of a link to the nicer-looking URL. But, I'm currently using the newforms library to generate the form - is there a good way to insert javascript in the onChange handler?


happy v-day!
Mood: creative
Posted on 2008-02-14 10:13:00
Tags: haskell programming politics links
Words: 272

Appropriately, the ban on selling sex toys in Texas was overturned (although it doesn't take effect until March). I swear, the laws we pass in this state. I wonder if there's a count somewhere of which state has had the most laws overturned by a federal court?

I think my stomachache yesterday was caused by the large mocha (four shots of espresso!) I had in the morning to try to curb my headache. Then I wasn't hungry for lunch, then I was very hungry for dinner. Anyway, that's all sorted out, and my head is doing significantly better. I slept much better than I thought I would. Hooray!

From this article about Clinton campaigning in Texas:

Delia Guajardo, 63, a retired administrative secretary, was among several in the crowd who said they don't foresee Obama swaying them: "Just the Obama name scares me. It's not a common name."
Seriously? Seriously?? Maybe that's why Kucinich never wins the primary?

Apparently the right way to read binary files in Haskell is to use this BitSyntax library. bitSyn is totally awesome but I wish I knew how it worked - the source is available but it uses some crazy template stuff. (here's an example of parsing a JPEG file with it) I think what I need to do is work on two projects - one in Haskell or O'CaML or whatever crazy language strikes my fancy and another in Python or C++ or something I know better to keep from losing my mind. Maybe I'll give Django another shot!

The writer's strike ended! Clinton and Obama will debate at UT next Thursday! (although good luck finding tickets...)


a collection of links that suddenly accumulated in my firefox tabs
Mood: hopeful
Music: Radiohead - "15 Step"
Posted on 2008-01-29 13:04:00
Tags: activism programming politics links
Words: 245

5 attributes of highly effective programmers - the title is cheesy, but it's a very good article, and it quotes The Screwtape Letters which was a nice surprise. Also on that site is The Effective Software Developer's Book List which is an impressive collection.

For Gay Democrats, a Primary Where Rights Are Not an Issue, This Time - the headline says most of it, except for this gem:

In an address last week honoring the Rev. Dr. Martin Luther King Jr. at a black church in Atlanta, Senator Obama made waves by lecturing the audience about homophobia. “We have scorned our gay brothers and sisters instead of embracing them,” he said during the speech at Ebenezer Baptist Church, where Dr. King served as co-pastor with his father.

Joe Solmonese, the president of the Human Rights Campaign, a gay lobbying group, said he thought Mr. Obama’s speech was the first time a presidential candidate had brought up gay issues in front of a nongay audience without being prompted to do so. “This is dramatically refreshing,” he said. “It’s a great day when we can look at a field of candidates and determine that we are comfortable with all of them on gay rights and move on to other issues.”
Awesome! (via dailykos)

I saw this press release from the National Organization of Women (NY chapter) about how Ted Kennedy betrayed women by supporting Obama instead of Clinton. It made me angry, until this rebuttal made me feel better.


weird phone thing
Mood: confused
Posted on 2008-01-25 16:04:00
Tags: phone programming
Words: 124

I was beginning to get a little worried about the new phone, since I charged it night before last and it ran out of power before bedtime. So I charged it last night and now it's still showing full power. Hopefully it's just a weird battery quirk, or maybe I had one of the side buttons pressed in my pocket? Either way, weird.

Also, I hate PHP. Any language where the expression "id" + 1 evaluates to 1 instead of (best case) "id1" or (not as good but still acceptable case) erroring should die. It doesn't help that I have no way of debugging it, just running the Greasemonkey script and inferring from the fact that the database is empty that something went wrong. Sheesh.


random WTF
Mood: amused
Posted on 2007-12-05 17:50:00
Tags: programming links
Words: 51

I don't usually post stuff like this, but this is possibly the best programming WTF ever. Add() reallocates memory each time (although I guess maybe the C# Array class does the doubling thing for you?), and Remove()...ow, my eyes! O(n^2) FTW.

The explanation at the end, of course, seals the deal.


my bad
Mood: tired
Posted on 2007-07-18 08:12:00
Tags: programming
Words: 69

After doing dinner (at the food court!) and Harry Potter last night (verdict: pretty good), I found a few horrendous bugs in Pretty Pictures. The upshot is that I think the scripts won't hang anymore, and breeding (which was horribly horribly wrong) seems to work better now. If you tried it yesterday and ran into problems, try it again (hitting Shift-Reload to make sure you get the new scripts).


pretty pictures is done
Mood: cheerful
Posted on 2007-07-17 09:15:00
Tags: programming
Words: 101

Play around with it here. It can be a little slow, so be patient. There's also some problem with the script getting hung up, but I couldn't track that down last night. Also, I forgot to put my favorite genotype up - try

(abs (sub-wrap (neg (ccrgb (ru (ru (num 0.6614554081190059))) (log-clip (abs (div-clip (abs (cchsl (ccrgb (num 0.5266760532380341) (num 0.2037882282789114) y) x x)) (mul-wrap (sub-wrap (atan-wrap (num 0.2710625018544712)) (num 0.46994645301411775)) (abs x))))) (rd (cchsl (num 0.811607978527779) y x)))) (abs y)))

Leave your favorites in the comments!
Other things I was planning on adding but didn't:
- going back a generation
- Perlin noise


link explosion
Mood: calm
Posted on 2007-06-18 09:56:00
Tags: facebook programming links
Words: 318

I'm going do to something called the "compliment sandwich", where I talk about something good, then something that could use improvement, then end with something good.

You look like snoopy, and it makes me smile.

I tried to work on my new facebook app, and made the same little amount of progress that I had on the previous one, but I just felt confused. I think the ideas I have are all pretty heavily friends based which seems hard to do with the API. I'm probably gonna let this idea die unless I get a huge burst of inspiration, which seems unlikely. Supposedly this is why "fun" projects are better than work - if they don't end up working out, you don't have to force it. I still feel bad about it though, like I'm failing and not good at "this stuff".

You really wowed that rep from the Cincinnati office! (thanks for the real quote, which was a little hard to find)

I had a long weird dream Friday night, the culmination of which was that searching for minnesota guacamole was very important. The results, as you can probably imagine, are not interesting.

Links I was going to share on my now-defunct neato Facebook thing:

Places I visit every day:Places I go when I'm bored:


heroic mechanar, and facebook
Mood: hopeful
Music: Avenue Q - "It Sucks To Be Me"
Posted on 2007-06-14 11:20:00
Tags: facebook worldofwarcraft programming
Words: 455

The day off yesterday was really nice. We watched some TV - BSG season 1 finale (shocker!), Lost season finale (also shocker although hard to figure out what's going on!), and I worked on getting a Facebook app working. I partially succeeded, and started to try to hack around the problems with the Friend of a Friend application before deciding that, even if I could get it to work it wouldn't be very reliable. So, my new idea is to have a Daily/Cool Links app which will be kinda like Digg/Reddit except you only see links/votes from people that you are friends with. I don't know how useful or interesting it will be, but at least it's technically possible. I'll let y'all know when it's done :-)

Another thing I did yesterday was run my first Heroic instance! Details behind the cut for you non-WoW people...

After an abortive attempt with djedi earlier (we didn't even do one full pull before the tank said he had to leave :-/), I got in with a different group, and after some trouble (some people don't realize you have to be keyed...) we got going. It was four people from Daggerfall - two paladins (one tanking, one healing), a warlock, hunter, and me (mage).

One thing I had always heard about heroic instances is that the trash hits hard. This is incontrovertibly true. The normal guys were OK, but the big guys (guardians, maybe?) consistently one-shotted me if I pulled aggro. One time we were running away to the elevator and he killed me without even stopping. We wiped I think five times just pulling one of those guardians (to be fair, the healer was having lag issues, but yikes).

The first "interesting" boss was the guy that in normal mode, turns green for spell reflect and red for melee reflect. In the heroic mode he does something totally different! Every so often he will cause each person to be either positively or negatively "charged". If two people with opposite polarity stand near each other, every five seconds they both take 2000 damage. If two people with the same polarity stand near each other, both of their damage is increased by 100%. So we had the rule that the healer would stand still, and depending on your charge you'd run up to him or away from him. Not too difficult but kinda neat :-)

We killed three bosses but couldn't kill Pantheon the Calculator - got him down to 20% a few times before giving up. I collected my three Badges of Justice and went home happy. It would be nice to run an easier heroic first, but I still need to rep grind with Thrallmar, Cenarion Expedition, Lower City, and Keepers of Time.


camping, facebook
Mood: geeky
Posted on 2007-06-11 12:27:00
Tags: facebook programming
Words: 232

This weekend blamantin and Alex and djedi went camping. It was, in a word, fun! We hiked on a trail in Rocky Gap State Park (in western MD) that was 4.5 miles - quite a good workout! After swimming in the lake, We drove to some campgrounds and played games, roasted marshmallows (mmmmmmmm), etc. We had been a little worried about the weather, as there was a chance of rain, but the weather for the hike was pretty darn good, felt around 80 degrees or so. It ended up being pretty cold at night - luckily djedi and I had brought sweaters so we wore them to bed and huddled together to conserve heat, which worked decently well :-) I took a bunch of pictures that I'll put up soon.

I'm planning on writing a Facebook application as my next project. My thought was to make a "friend of a friend" sort of thing, like LiveJournal has (but you have to be paid or maybe plus to see it - sorry!) which shows people that lots of your friends have friended but you haven't. It seems like a good way to discover people on Facebook that you didn't realize were there, or people that you might be interested in meeting :-) Anyway, for various technical reasons step 1 is to upgrade my Apache web server from 1.3 to 2.0, so hopefully I can make that work tonight.


I love data
Mood: giddy
Posted on 2007-04-25 16:05:00
Tags: programming ngrams
Words: 80

Thanks to Peter Norvig's (he wrote my AI textbook! and is director of research at Google!!) article about writing a spelling corrector, I was reminded that Google released a giiiiant list of n-grams found on the web. Unfortunately, it's only for noncommercial use unless you join and pay thousands of dollars (noncommercial is OK) and costs $180(!) to buy and ship. On the other hand, it's 6 DVDs of compressed data (24 GBs of gzipped files). This is soooo tempting.


new project: congress ratings
Mood: excited
Posted on 2007-03-19 18:48:00
Tags: haskell projects programming congressvotes
Words: 421

So my next project is fun and neat. I can't remember how exactly I got here, but I noticed that all roll-call votes in congress are accessable in a handy XML format (Here's an example vote - note that the page you see is that XML file after being processed with an XSLT stylesheet, so you'll have to hit View Source to see the raw data). My idea (a little vague at this point) is to take all of the votes, and then figure out which issues I care about and which way I would have voted, and then "rate" representatives as to how closely their votes align with mine. This is a little grand in scope.

Anyway, it took me a while to get HaXml (a Haskell XML parser) installed, because I kept not being able to compile it from source for various stupid reasons. Anyway, I finally figured out that it was in fact in Debian in the libghc6-haxml-dev package, which made my life about 5 times easier.

So I saved a sample vote and have it parsing and I'm extracting simple data from it, which is exciting! I have a few questions, though: does anyone know the answer to these?

- I have these two functions:

nothing :: Maybe a -> Bool
nothing Nothing = True
nothing (Just _) = False

findElementContent :: String -> Content -> Maybe Element
findElementContent target (CElem el) = findElement target el
findElementContent target _ = Nothing

findElementContents :: String -> [Content] -> Maybe Element
findElementContents _ [] = Nothing
findElementContents target (c:cs) = if (nothing (findElementContent target c))
then findElementContents target cs
else findElementContent target c

findElementContent takes in a target tag and some data (Content), and returns the element that has that tag name if it exists, and Nothing otherwise. (findElementContents is just a helper function to do the same thing with a list of Content) But findElementContents looks pretty ugly to me - what I want it to do is return findElementContent target c if that isn't Nothing, and otherwise recur on the rest of the list. The code is correct, but is it inefficient since I'm calling findElementContent target c twice? My limited understanding says no, since findElementContent is referentially transparent since it doesn't use monads (i.e. if you call it again with the same inputs it will return the same thing, always), but I'm not entirely clear on this.

- As I mentioned, findElementContents seems a little inelegant - is there a better way to do this? Is there some builtin nothing that I couldn't find?

Resources I've been using:
- HaXml reference
- standard library reference, including the Prelude


What does Thursday mean to you?
Mood: cheerful
Posted on 2007-03-15 07:55:00
Tags: haskell worldofwarcraft tivo programming
Words: 361

It hit 70 degrees here yesterday! It was absolutely beautiful. Of course, it's going to be cold again tomorrow, but it was nice to have a touch of spring.

I finished and posted my latest WoW guild data project. It was fun, but nothing too tricky about it.

So I read about the Amazon Unbox downloads on the TiVo, so I signed up. (free $15 if you link the accounts by the end of April!) We rented A Scanner Darkly to try it out (you have to watch it within 30 days, and once you watch it it expires in 24 hours), and it downloaded in like an hour. Very very cool stuff. It's also neat because if you buy something, Amazon saves it and you can redownload it at any time. (except if the stupid copyright holder doesn't let you, grr) Also, they have a small number of TV series including 24, Firefly, and Arrested Development.

My attempt at comprehending Haskell continues. I read through A Gentle Introduction to Haskell which, despite the title, is not really gentle at all. I'm also looking through the Haskell Wikibook which is a little too slow-paced for me, but it's a nice change of pace after the other one! I'm starting to get some of the ideas, and remembering why functional languages are so cool. As a crazy example, you can do something like

y = x * 2
x = 2

and it will still work, since variables are only assigned to once, so I guess it can figure out the order things should run in. (I should note that I can't find the reference for this anymore, so it may not be true. Also, this seems like a crazy thing to do in a functional language anyway.) Maybe my next Haskell project will be to write a bot for WoW! (*)

I've gotten in to work at 7:45 every day this week, surprisingly enough. I've also been to the gym every day.

I got two nice "Congratulations!" cards in the mail from my relatives for my engagement. Yay!

(*) - Just kidding. My next project won't be WoW-related. Also, this is borderline crazy to try to do.


Mood: pensive
Posted on 2007-03-14 08:09:00
Tags: haskell programming
Words: 289

So it's been a little while since I've learned a new language, and I keep hearing rumblings about Haskell (programming.reddit.com really loves it). So I started to read A Gentle Introduction to Haskell, and...wow. I haven't done deep functional stuff since college, and I made it about halfway through before my brain up and revolted.

It has a lot of neat features that you would expect in a functional language, and it has the neat Prologish way of defining functions by pattern matching. For example, here's my first Haskell program!

leng [] = 0
leng (_:xs) = 1 + leng xs

main = print (leng ['a','b','c'])

This is defining a function leng ("length" is built in to the standard library) that returns the length of a list. Note that you don't say what to do to calculate the length, exactly, you specify that the length of an empty list is 0, and the length of any other list is 1 plus the length of the rest of the list.

It took me a while to figure out the exact syntax to define this, but I finally did and compiled it with the Glasgow Haskell Compiler (ghc). It took 6 seconds, which seems like a really long time for something so simple. Maybe there's a fixed amount of overhead or something. And the resulting executable was 370KB!! I'm guessing this is because it compiles it to a binary executable, which I wasn't expecting, so you don't get the benefit of an interpreter. It runs instantly, as you would expect.

Anyway, I'll keep reading and thinking what would be a good project to do in it. I/O looked weird, but that was about when my brain shut down yesterday so maybe I'll start there today...


Look what we can do!
Mood: accomplished
Posted on 2007-03-09 09:07:00
Tags: pictures worldofwarcraft programming
Words: 242

Last night, I accomplished the following things:

- grocery shopping
- mole check
- wrote a script to download data from the World of Warcraft Armory, and set it up to download every guild member's stats nightly. At some point I'll do something neat with the data. (to do this I used the new Python XML library elementtree, which is better than the builtin stuff, but not nearly as nice as e4x) I had to solve a bit of a mystery - when I would fetch the URL http://armory.worldofwarcraft.com/character-sheet.xml?r=Maiev&n=Tsouzer in Firefox and the HTML page would show up, but I could view source and see the raw XML data that I wanted. When I fetched it with wget or curl, I would always get the HTML page, not the XML data. The key is to fake out the user-agent: I guess if it thinks you support the fancy XML processing it does (with XSLT, I think?) it will give you the data, otherwise it gives you standard HTML any browser can handle. Neat idea, I guess.
- fixed my automatic del.icio.us bookmark backup script (the API had changed a little).
- seemingly fixed the problems with sending email from my computer, although it still complains that it can't write to the log file.
- put up a (very) few pictures.
- updated my backup script to back up some more databases

A lot of this stuff had been hanging over my head for a while, so I feel great today!


little bit o' everything
Mood: cheerful
Posted on 2007-02-27 10:25:00
Tags: microresolution worldofwarcraft programming
Words: 286

We went to an Oscar party this weekend. It was fun! Ellen did a good job hosting.

We've been playing a lot of World of Warcraft. I posted to the guild site about an analysis I did (using my auctioneer data lookup) about what potions you can craft to make money. I'm gonna try taking my own advice and see if it works! I also have been working on an Adobe Flex version of that interface which is almost done. The SDK is free and works on Linux, and since Flash Player 9 was just (a month or so ago) released for Linux I thought it would be a good time to try it out. Working in a data-driven language takes some getting used to but the resulting code is a lot simpler than the AJAX version. And it looks nice!

Via Fire Joe Morgan there's this delightful piece which compares sports bloggers to homeless people or...something. It's really hard to tell what the point of the column is, although from what I can tell the point was to be stupid.

The cablecards in our TiVo seem to be going on the fritz. Luckily Heroes recorded last night, although we haven't watched it yet. I reeeeealy don't want to have to call Comcast and get them to come out and fix them, especially since they work some of the time so you just know they're gonna work when the people come out.

I really really need to address and send those darn letters. I think it's the final step that makes everything "real" and I don't want to deal with. But sheesh, get over it. Microresolution: address the letters by Thursday night, send them by Monday.


take that, damn spammers!
Mood: triumphant
Posted on 2006-12-03 22:33:00
Tags: programming sql
Words: 316

So I've noticed off and on for a while that my gallery has been getting some weird hits - mostly google searches for "hentay" (I'm guessing they meant hentai, which is most definitely a NSFW search term) and such. At first I laughed it off because they were obviously mistaken, but I couldn't figure out why it was happening. Apparently I have a knack for missing the blindingly obvious, so I was a bit shocked today when I looked on the front page of the gallery, again trying to figure out what was going on, and saw comment spam. Lots and lots of comment spam. (people posting links to their web sites to gain PageRank) Unfortunately, this meant I had to disable anonymous commenting, so if you want to comment just open an account and I'll approve it :-)

More unfortunately, there were still over 2000 comments (all of them spam) on the front page! I looked around and it turns out there's no way to bulk delete comments as of yet. And I sure wasn't going to sit there clicking more than 4000 times (one to delete, one to confirm). So I dug into the SQL and beat those bastards!

Anyway, in case anyone else runs into this problem, here's what I did: open up mysql and the first thing I found was that the root album had a g_id of 7. So I did these commands to first isolate the ID's of all the comments, and then delete them. There are probably better ways to do this but I'm a novice at this so I was aiming for safety...

CREATE TEMPORARY TABLE tmptable SELECT g2_Entity.g_id FROM g2_ChildEntity, g2_Entity WHERE g2_ChildEntity.g_id=g2_Entity.g_id AND g2_ChildEntity.g_parentId=7 AND g2_Entity.g_entityType='GalleryComment';

DELETE FROM g2_Comment, g2_Entity, g2_ChildEntity USING g2_Comment, g2_Entity, g2_ChildEntity, tmptable WHERE g2_Comment.g_id=tmptable.g_id AND g2_Entity.g_id=tmptable.g_id AND g2_ChildEntity.g_id=tmptable.g_id;

Hopefully this will be helpful to someone else, or at least a future version of myself :-)


does the program exhibit the buddha way?
Mood: hopeful
Posted on 2006-10-04 07:37:00
Tags: netflixprize programming
Words: 402

I was trying to track down a particularly odd bug at work yesterday and I suddenly realized I was effectively playing a game of Zendo with the program. I had my theory about what sequence of events made the bug show up in the program (i.e. what koan of events had the buddha way), and I pretty early on got a repeatable koan that always exhibited the buddha way. After some time trying to look at the code to figure out what was going on, I had my theory about what made a sequence of events exhibit the buddha way and I started trying to make it happen some other way. Just before I left I discovered something completely unexpected that seems to determine whether a sequence of events exhibits the buddha way based on something entirely unexpected. That's the thing about debugging versus Zendo - the rules are a lot looser! Anyway, I miss playing Zendo, but I guess I can have my own little game at work :-)

I spent some time working on Netflix Prize stuff last night - I tried to parse all the data but it ran out of memory, so I closed Firefox, tightened up my class that was representing each rating (12 bytes to 8 bytes makes a big difference...), and got it to successfully parse all 100 million+ entries. The only slight concern is that that doesn't leave a whole lot of memory to do any sort of analysis (I only have 1G, 800M or so of which will be used up by the ratings), but I think I can do just O(n) stuff in space and get by. Or, I could buy more memory (woot!), but the last time I tried to put more than 1G of RAM in my computer, bad things happened. Maybe I'm just being superstitious.

The other problem is that it takes around 40 minutes just to read in all the data. I had it calculate the average rating over all ratings last night and it gave me something reasonable, so I was thinking of writing some code to do the ratings on the test set and just rate everything the average and submit that. This would give me a good baseline - although they give you that baseline already, I think, it would be nice to have an official entry submitted, even if it's crap. So maybe I'll get to that tonight...


netflix prize away!
Mood: excited
Posted on 2006-10-03 09:09:00
Tags: netflixprize programming
Words: 162

Sunday I was feeling a little down, mostly because I didn't have a project to be working on and none of the ones I had on my long-term radar sounded very interesting (or possible). The very next day Netflix announced the Netflix prize, which is basically a competition to improve their suggestions engine (i.e. if you liked movie X and Y you'll probably like movie Z but not movie W). And they're releasing 700M worth of data to train on. I think I'll work on that next!

Since the amount of data is so massive, I decided to work in C++ instead of Ruby or Python - I'm a little worried about keeping all the data structures in memory, but we'll see how that goes. I got some basic parsing of the data done and hope to submit a very basic entry soon! (you can submit an entry once a week and it will score it to let you know how you're doing)

1 comment

Mood: mellow
Music: The Killers - "Mr. Brightside"
Posted on 2006-06-15 10:25:00
Tags: java programming
Words: 689

I was feeling a little blah yesterday, so I did what usually clears that up for me - start a new project! (this probably isn't a great long-term solution, but it works for now) I decided to write a program that would turn my iTunes playlists (which are stored in the iTunes XML file) into plain ol' .m3us so I could listen to them served up from my computer. Not too tricky, since I already have an analysis of my songs in iTunes that parses the iTunes XML file.

To make it more exciting, I started writing it in Java, which I used extensively at Rice, but hardly at all since then. I was going to parse the XML file and then use XPath to get the stuff I needed, but the file is so big (200000+ lines!) that Java runs out of memory getting the nodes. Sigh. So I went back to plain ol' parsing it line by line and just picking out the parts I need, which turned out to be pretty easy.

It's not done yet, but I've already extracted the playlists and which songs they refer to (by referencing the database that is created when I do the analysis of my songs), so I just need to write the code to output them as .m3us which is pretty straightforward. But the point of this post is to point out some new stuff in JDK 5 (which is really 1.5) that are very nice compared with what we had back at Rice :-) The link describes them, but I'll point out the highlights:

So, anyway those made programming in Java a bit less painful, but it's still awkward going from a dynamically-typed language (Ruby, Python) to a statically-typed one (Java). Just takes some getting used to, is all.

On a random note, I rearranged my home page a bit, hopefully for better clarity and for putting the most important/interesting stuff towards the top.


ruby scoping
Mood: confused
Posted on 2006-04-18 09:41:00
Tags: ruby weight programming
Words: 218

I weighed in this morning, and I'm at my lowest weight since starting to track it (oddly enough, despite eating many tasty cookies and candy on Sunday). Hooray!

On a completely separate note, I'm learning Ruby to use in my next project. It seems like a nice language, but I have a complaint already. Consider the following code:

[1,2,3].each {|e| print e, " " }

This prints out "1 2 3 " - the each operator iterates over the elements of the array, and the block that follows it ({|e| print e, " " }) operates on each one of them. So that's fine, and e is undefined after the block executes, as I would expect.

What I don't like is that the following code:
e = 10
[1,2,3].each {|e| print e, " " }
print e

prints out "1 2 3 3" - the e argument to the block ends up overwriting the e variable declared outside the block! This seems like a big violation of scoping - shouldn't the e inside the block be unrelated to the e outside the block? Can anyone (omega697, perhaps) explain this to me?

I mean, I can see why it might be useful (if you want to save the last value that the block executed on), but the penalty of possibly overwriting a variable outside the block's scope seems a bit much to handle...


This backup was done by LJBackup.